Evaluation and Technical Support Capacity
All partners in this consortium are experienced in the use of course pilot testing with focus groups representing the target audience, pre- and post-tests and overall course evaluations as a process of continuous quality improvement in training. These processes are routinely used in the preceding HRSA-funded Bioterrorism Curriculum Development Program project at Columbia University, all courses developed by the two CPHPs, and the New York-New Jersey Public Health Training Center (NY-NJ PHTC), with which this effort is closely aligned. These routine steps will be continued for all courses developed or offered through the NYCEPCE. Experience to date indicates that faculty and course developers are responsive to using this type of evaluative approach. Audience members informed about course changes over time have also expressed appreciation for the changes that are responsive to the needs of their sites and situations.
Comprehensive evaluation
While course-to-course assessments are helpful, it can be difficult for project staff deeply immersed in the daily activities of a program to maintain the perspective essential for a critical evaluation necessary for overall continued improvement. Accordingly, the NYCEPCE will use an external evaluation team experienced in assessment of emergency preparedness and emergency preparedness training to examine and indicate which project objectives are being met. The Kansas Public Health Workforce and Leadership Development (WALD) Center (www.waldcenter.org) will serve as this evaluator. The selection of this center was based on the congruence of philosophy with the goals of the NYCEPCE and the familiarity of the WALD staff with the challenges and realities of emergency preparedness education for a widely dispersed audience.
The WALD Center uses a combination of quantitative and qualitative designs in its training venues and related evaluation activities. These evaluation efforts include such methods as: cross-sectional surveys, focus groups, cohort follow-up, nested groups, and train-the-trainer methods. When possible, cohort follow-up for training evaluations is used. Emphasis is placed on competency-based training within the ecological model using epidemiologic evaluation procedures and measures. The use of a particular evaluative approach depends on the nature of the project. For example, evaluation of the HRSA “Can it Happen in Kansas?” project has used a nested cohort design within a pre-test/post-test evaluation. The nested cohort is followed by mailed surveys at three months and ten months following training as well as by focus groups. In contrast, mental health preparedness training workshops use a cohort follow-up design within a pre-test/post-test format for each training session.
The specific evaluation of the programs developed within the cooperative agreement will use a mixed qualitative/quantitative model. In addition to the process evaluation and pre and post-tests implemented in the preceding HRSA-funded Bioterrorism Curriculum Development Program project at Columbia University and used by all three of the consortium members in the past, this proposal outlines cohort follow-up measures of 6 and 12 months post-training, focus groups, and assessment of the ICS simulation exercise.
Formative Evaluation
A number of measures will be taken to address formative and process measures. Feedback loops will be implemented as needed and determined collaboratively with the Project Advisory Committee and the project staff at the three institutions to improve curricula and process.
Summative Evaluation
Quantitative
- Extend assessment of impact of existing modules to 6 and 12 months post training through sequential analysis to assess knowledge decay and training effectiveness; expand to reflect the larger population of respondents to assess whether modules are effective across population segments through analysis of differences across stratified sample.
- Web-based course(s) (electronic format) to assess: 1) accessibility by busy professionals, 2) reaching the larger, intended audience, 3) agencies sharing of curricula. Comparison of target population and current-user population. Training schedule will be analyzed for differences. Cost of web-based compared to face-to-face will be compared for differences.
- Assessment for simulation exercise - An ICS template for a tabletop exercise, using an organizational chart to move beyond the “awareness level” of participants’ competencies to measure the value of utilizing competencies.
- Power analyses will be conducted throughout.
- Design pathways through an interactive tabletop learning for adult learners (correct/ incorrect responses)
- Participants would respond to “big picture” ICS questions (how would you respond to this crisis, chain of command)
- Identify best practices – what works, could be repeated (i.e. physicians notifying attending staff), what decision-making resources would you utilize in your staff and in the organization if an event occurred. It is useful to learn from others about what works well, which can lead to the sharing of most effective and useful practices, eliminating duplication of effort.
- Economic evaluation so activities can be as cost-effective as possible
- Other evaluation components as needed and determined collaboratively with the Project Advisory Committee and the project staff at the three institutions.
Qualitative
- Focus groups and/or structured interviews with random sample of simulation participants. Will be audio recorded, transcribed, and analyzed for concurrent and divergent findings.
- Focus groups and/or structured interviews with random sample of participants using existing assessment modules. Will be audio recorded, transcribed, and analyzed for concurrent and divergent findings.
- Other qualitative evaluation components as needed and determined collaboratively with the Project Advisory Committee and the project staff at the three institutions.
Technology Resources and Electronic Distance Learning Methodologies and Capabilities
The University at Albany School of Public Health (UA SPH) has extensive experience and resources for distance learning tailored to the needs of working healthcare and public health professionals. UA’s continuing education infrastructure supports satellite broadcasts, web-based training, live regional workshops, and “blended learning.” The impetus for distance learning is based on the School’s need to serve professionals across New York State and the nation. An electronic subscription and registration service includes a database of over 8,000 healthcare and public health professionals who subscribe to receive updates on professional development opportunities. This electronic system handles registrations, distribution of handouts, and collection of evaluations in all 50 states.
UA SPH has three on-going broadcast series: Third Thursday Breakfast Broadcasts target local health departments with monthly updates on public health topics. Women’s Health Grand Rounds target healthcare professionals in hospitals and health centers, and the Center for Public Health Preparedness Grand Rounds reach emergency preparedness personnel in public health, healthcare, and public safety. Professional video producer Amy Hart is on staff. UA SPH is fully equipped with digital cameras and editing capability and has access to the professional broadcast studios and uplinks of NYNetwork of the State University of New York:
Satellite broadcasts have proven an effective way to distribute “awareness level” training and updates to practitioners with limited access to continuing education. UA SPH broadcasts routinely reach rural county health departments and hospitals as well as urban organizations serving the medically underserved. All 57 county health departments in New York State now have videoconferencing capability thanks to BT preparedness funding. The NYS DOH publicizes and re-broadcasts all UA SPH programs via this internal network. Likewise, the Adirondack Area Network routinely distributes UA SPH programs to its members in the rural “North Country” and upper Hudson including small hospitals in the Iroquois Alliance, the Hudson Headwaters primary care network of providers, and BOCES (centralized K-12 services).
For web-based training, UA SPH partners with U Albany’s Professional Development Program Instructional Technology Unit (PDP ITU) to produce on-line learning. ITU has a creative team of instructional designers and programmers who work with subject matter experts to produce highly interactive non-instructor-led courses. Courses use scenario-based learning to engage participants and encourage transfer of learning to the work setting. The development involves multiple iterations of usability and content. The courses include exercises to complete and the option for a certificate of completion or continuing education credit (CEU, CME, Nursing, or CHES). The structure of the courses enables the collection of in-depth evaluation data on demographics, usage and completion patterns.
Current on-line courses include: Orientation to Public Health and Exploring Cross-Cultural Communication for the NYNJ Public Health Training Center and for the Center for Public Health Preparedness: Terrorism, Preparedness, and Public Health: An Introduction and Introduction to Basic Microbiology. Three courses are in production for Fall 2005: Pandemic Flu; Applied Case Studies in Cross-Cultural Communication; and Evidence-based Public Health.
The online Introduction to Basic Microbiology is a successful example of “blended” learning. The module serves as a prerequisite for the Basic Environment Health Course, a classroom-based training program offered by UA SPH and the NYS Department of Health. The online prerequisite frees the class-room instructor from the necessity of reviewing microbiology before covering new material. Additionally, the online module helps train new public health practitioners, including Bioterrorism Coordinators, who have little or no preparation in microbiology.
Despite the success of distance learning, UA SPH also recognizes the importance of face-to-face training for more in-depth learning. Its staff supports delivery of live regional workshops and on-campus institutes for over 1,500 participants annually.
Impact: The impact of these distance learning programs lives long past the initial development and launch. Satellite broadcasts are available on videotape, DVD, and via archived webstreaming. NYSACHO and local health departments and hospitals maintain video libraries for in-house training. For both satellite broadcasts and web-based courses, practitioners have on-going access to CME, nursing contact hours, and CHES by completing a post-test and evaluation on the web.
An astounding 15,000 people have registered for UA SPH web-based training between the launch of Orientation to Public Health in February 2003 and May 2005. Successful completion rates vary from 56% to 26% - well above industry standards for non-required Internet training. Participant evaluations are consistently high. Since its launch in June 2004 until May 2005, over 6,000 individuals from 50 states and international locations have registered for the six module, interactive, on-line course: Terrorism, Preparedness, and Public Health: An Introduction.
The impact of these programs extends beyond their original objectives. Because the programs are free, accessible 24/7, and of high quality, a number of agencies and college professors have adopted them as parts of larger training and education programs. For example, Orientation to Public Health is required as a homework assignment in Community Health Nursing and Health Policy courses at various universities nationwide. The Terrorism Preparedness, Orientation, and Cross-Cultural Communication courses have been adopted by a number of local and state health departments for employee training and orientation.