Measuring the Facilitators and Barriers in the Implementation of Performance Measures in Emergency Medical Services for Children Open Access
Downloadable ContentDownload PDF
Background: The Federal Emergency Medical Services for Children (EMSC) Program provides funding to States, the District of Columbia, and the Territories (herein referred to as States) to improve their emergency care infrastructure. The Federal Program has developed specific performance measures (PMs) to measure the ability of state grantees to make health system change. The Program has not evaluated whether States have been successful with PM implementation and achievement and what factors affect such success. Specific Aims: This dissertation used a process and outcome evaluation to determine micro, meso, and macro level factors that impact State's ability to implement and achieve PMs. Results of the evaluation were used to develop policy recommendations. In Specific Aim 1 a process evaluation was conducted to describe how States implemented PMs and explore why States varied in their ability to implement the measures. In Specific Aim 2 an outcome evaluation was used to describe the degree to which each State met 2007 PM targets and explore the facilitators and barriers to meeting the PMs. In Specific Aim 3 policy recommendations were developed to help EMSC and other Programs maximize health system improvements through PMs.Methods: This was an exploratory study using a cross-sectional study design. A conceptual model was developed by evaluating existing literature on models of health system change. Data from the Health Resources and Services Administration (HRSA) and the National EMSC Data Analysis Resource Center (NEDARC) were used for the dependent variables on the implementation and achievement of PMs. Independent variables on the facilitators and barriers to PM implementation were collected through a newly developed 56-item survey of 54 State EMSC Programs. Bivariate and multivariate analyses were conducted to assess relationships between dependent and independent variables.Results: Forty-seven States (87%) responded to the dissertation survey. The results indicated that 54% of States met data collection standards and 38% were able to verify the accuracy of data they submitted to HRSA. The most significant factors in helping states meet data collection standards included the percent time spent by the EMSC manager on PM-related activities and the presence of statutes, rules and regulations. On average, States met 61% of the PM achievement targets for the year 2007. Micro-level factors affecting PM achievement included the effort of the EMSC manager, importance managers placed on PMs, and job satisfaction of the EMSC manager. Job satisfaction alone explained 40% of the variance in PM achievement. Meso level factors that affected PM achievement included the degree of change that PMs represented from prior grant priorities as well as several advisory committee related factors including: presence of an advisory committee, political will of the committee to move PMs forward, variables related to stakeholder management, advisory committee reporting hierarchy, and commitment of advisory board members. Macro level factors that affected PM achievement included the presence of statutes, rules, and regulations (mandates), involvement of the EMS medical director, and identification of a champion (change facilitator). The presence of a change facilitator and use of statutes, rules, and regulations accounted for 39% of the variation in PM achievement. Conclusions: Major factors that affect health system change in EMSC Program include individual manager effort, buy-in, understanding, and job satisfaction. Other significant factors include the presence of statutes, rules, and regulations and the presence of a champion. PM data alone may not reliably measure Program performance given the complex, varied, and nuanced nature of the Federal Programs, such as EMSC. New approaches to performance measurement and evaluation need to be considered.