Project 4. Linking Assessment and Measurement to Performance in PHEP Drills and Exercises (Biddinger (MGH)) will start at the fourth step of the Cycle by examining how best to employ drills and exercises to measure public health response capabilities, with a major focus on validity and reliability of emerging metrics and scales. While all 50 states are using exercises that simulate emergencies for PHEP training and planning purposes, work is still needed to address a major gap in using such exercises to systematically evaluate PHEP and initiate true systems-level improvement. This project assesses whether current metrics intended for use in exercises are valid and reliable measures of PHEP, identifies critical elements in the design, conduct and structure of exercises to allow for measurement of PHEP capabilities, and defines how exercises can most effectively be integrated into ongoing public health practice.
There are three specific aims of the "PHEP Drills and Exercises" project, namely:
1. The first specific aim of the PHEP Drills and Exercises project is to assess whether metrics intended for use during exercises are valid and reliable measures of PHEP. Building upon our HSPH CPHP work, as well as the critical efforts of programs within the U.S. Department of Health and Human Services (DHHS) and Department of Homeland Security (DHS), we have cataloged a preliminary set of measures to assess PHEP capabilities. We will extend this preliminary work to evaluate the validity and reliability of these performance measures through a more structured series of exercises designed specifically for measurement purposes. Further, we will identify gaps where such measures are lacking.
2. The second aim is to identify critical elements in the design, conduct, and structure of exercises that maximize their utility for measuring PHEP. While we hypothesize that a well-designed exercise can serve as a useful approximation of a real emergency, we recognize that each exercise has its own set of artificialities and assumptions. Moreover, the intrinsic characteristics of an exercise (e.g., how it is designed, facilitated, and structured) affect its utility in measuring performance. We intend to employ modeling and other techniques to validate and refine these artificialities and assumptions and subsequently establish standards to guide the design and implementation of high fidelity exercises.
3.
The final aim is to define how
exercises can most effectively be integrated into ongoing public health
practice to produce measurable improvements to systems-level preparedness. By working closely with local public health
agencies and their partners within the PHEP system, we plan to identify the
practical factors enabling local public health systems to implement exercises
utilizing our metrics to measure preparedness, allowing for continued and
self-sustaining assessment of PHEP.
Principal Investigator for Project 4:
Paul Biddinger, MD serves as Lead Investigator for
Project 4. Dr. Biddinger is
Associate Director of the HSPH CPHP, Director of the Scientific Core of the
HSPH CPHP and Assistant Professor of Health Policy and Management at HSPH as
well as Assistant Professor of Surgery at HMS. An Emergency Physician and Director
of Disaster Medicine at the Massachusetts General Hospital in Boston, MA, he is
one of the leading national figures in the design, conduct, evaluation and
utility of drills and exercises. He has authored several of the critical
peer-reviewed publications on the use of exercises to evaluate PHEP. He has authored
several of the critical peer-reviewed publications on the use of exercises to
evaluate PHEP. Additionally, Dr. Biddinger has been awarded several separate
research grants from the Agency for Healthcare Research and Quality and others
on the study of disaster preparedness in the United States. Dr. Biddinger
chaired a national collaboration group jointly convened by the CDC and the Association
of Schools of Public Health (ASPH) to investigate PHEP measurements using
exercises and drills.
Co-Investigator:
Michael A. Stoto, PhD serves as the Co-Investigator for Project 4. One of the leading PHEP performance
measurement experts in the country, Dr. Stoto is currently an Adjunct Professor
of Biostatistics at HSPH, Director of the Evaluation Core of the HSPH CPHP, and
Professor of Health Systems Administration and Population Health at Georgetown
University. He currently leads the
evaluation team for the DC Healthcare Coalition Emergency Management
Partnership, funded by the DHHS Office of the Assistant Secretary for
Preparedness and Response (ASPR), and recently completed a multi-site project
investigating regional approaches to PHEP funded by the Robert Wood Johnson
Foundations Public Health Systems Research initiative. Previously, he led an evaluation of NACCHO's Project Public Health Ready to recognize
local health departments prepared for PHEP, as well as a case study of the
public health response to outbreaks of West Nile Virus, SARS, monkeypox, and
hepatitis A in the U.S. He has also worked as an evaluator of
syndromic surveillance systems. With
extensive experience in the application of quantitative and qualitative
research methods to guide public health policy and practice, including the
development of Healthy People 2000, Dr. Stoto has conducted research on PHEP
measurement and improvement, as well as on public health interventions, and
ethical issues in research and practice. He has published extensively, is a co-editor of an IOM report that
provided public health data for the Washington, DC area, and authored the
chapter on evaluation in Public Health
Administration, now in its second edition.
New Research Investigator:
Elena Savoia, MD, MPH serves as New Research Investigator for Project 4. Dr. Savoia is
a research scientist in the Division of Public Health Practice at HSPH.
She joined the HSPH-CPHP in 2004 and has worked on the development of
evaluation methods to assess the reliability and validity of capacity,
and performance measures designed to assess the public health system's
response to large scale emergencies. In particular, she has implemented
over 100 surveys in the specific field of emergency preparedness and
collected and analyzed data from over 5,000 respondents. In year 1, as
a New Research Investigator, Dr. Savoia will lead the Core Pilot
project entitled "Linking Assessment & Measurement to Performance in National Surveys"
and will be responsible for supervising the monitoring and evaluation
of the four program projects. In years 2-5, Dr. Savoia will serve as a
Co-Investigator in the Core.
Contact Information:
Jessica Preston
Scientific Core Assistant
Center for Public Health Preparedness
Division of Public Health Practice
Harvard School of Public Health
677 Huntington Avenue
Landmark Center, 3rd Floor East
Boston, MA 02115
Email: jpreston@hsph.harvard.edu
Phone: (617) 496-1087
Fax: (617) 495-8543