Program Evaluation as a Key Tool in Health and Human Services
Program Evaluation as a Key Tool in Health and Human Services Maria Delos Angeles Mora HCA460 Research Methods in Health and Human Services Professor TyKeysha Boone April 22, 2013 Program Evaluation as a Key Tool in Health and Human Services In this competitive health care environment, consumers want and expect better health care services and hospital systems are concerned about maintaining their overall image. There is also attention to ways in which patient satisfaction measurement can be integrated into an overall measure of clinical quality. As lots of information is available to be used in a hypothetical evaluation.
The American Red Cross is my selection due to that I worked with them for several years as a voluntary and telephonic representative to answer incoming calls that needed to be checked for different parts of the United States and commonwealth territories. The fundamental Principles of the Global Red Cross Network are based on humanity- the Red Cross, born of a desire to bring assistance without discrimination to the wounded on the battlefield, endeavors-in its international and national capacity-to prevent and alleviate human suffering wherever it may be found.
Its purpose is to protect life and health and to ensure respect for the human being. It promotes mutual understanding, friendship, and cooperation lasting peace amongst all peoples, impartiality-it makes no discrimination as to nationality, race, religious beliefs, class or political opinions. It endeavors to relieve the suffering of individuals, being guided solely by their needs, and to give priority to the most urgent cases of distress, neutrality- In order to continue to enjoy the confidence of all, the Red Cross may not take sides in hostilities or engage at any time in controversies of a political, racial, religious or deological nature, independence-since the Red Cross is considered is independent. The national societies, while auxiliaries in the humanitarian services of their governments and subject to the laws of their respective countries, must always maintain their autonomy so that they may be able at all times to act in accordance with Red Cross principles, voluntary service-is a voluntary relief movement not prompted in any manner by desire for gain, unity-is there is a Red Cross society in any one country no one can be turned out as it may be open to all.
It must carry on its humanitarian work throughout its territory, and universality-as the Red Cross is a worldwide institution in which all societies have equal status and share equal responsibilities and duties in helping each other. In the continuing effort to improve human service programs, funders, policymakers, and service providers are increasingly recognizing the importance of rigorous program evaluations. They want to know what the programs accomplish, what they cost, and how they should be operated to achieve maximum cost-effectiveness.
They want to know which programs work for which groups, and they want conclusions based on evidence, rather than testimonials and impassioned pleas. This paper lays out, for the non-technician, the basic principles of program evaluation design. It signals common pitfalls, identifies constraints that need to be considered, and presents ideas for solving potential problems. These principles are general and can be applied to a wide range of human service programs.
We illustrate these principles here with examples from programs for vulnerable children and youth. Evaluation of these programs is particularly challenging because they address a wide diversity of problems and possible solutions, often include multiple agencies and clients, and change over time to meet shifting service needs. It is very important to follow the steps in selecting the Appropriate Evaluation Design. The first step in the process of selecting an evaluation design is to clarify the questions that need to be answered.
The next step is to develop a logic model that lays out the expected causal linkages between the program (and program components) and the program goals. Without tracing these anticipated links it is impossible to interpret the evaluation evidence that is collected. The third step is to review the program to assess its readiness for evaluation. These three steps can be done at the same time or in overlapping stages. Clarifying the Evaluation Questions is a design of any evaluation begins by defining the audience for the evaluation findings, what they need to know, and when.
The questions used are determine which of the following four major types of evaluation should be chosen such as: The Impact evaluations focus on questions of causality. Did the program have its intended effects? If so, who was helped and what activities or characteristics of the program created the impact? Did the program have any unintended consequences, positive or negative? How performance monitoring does provides information on key aspects of how a system or program is operating and the extent to which specified program objectives are being attained (e. g. numbers of youth served compared to target goals, reductions in school dropouts compared to target goals). Results are used by service providers, funders, and policymakers to assess the program’s performance and accomplishments. Process evaluations answer questions about how the program operates and document the procedures and activities undertaken in service delivery. Such evaluations help identify problems faced in delivering services and strategies for overcoming these problems. They are useful to practitioners and service providers in replicating or adapting program strategies.
Cost evaluations address how much the program or program components cost, preferably in relation to alternative uses of the same resources and to the benefits being produced by the program. In the current fiscal environment, programs must expect to defend their costs against alternative uses. As the comprehensive evaluation will include all these activities. Sometimes, however, the questions raised, the target audience for findings, or the available resources limit the evaluation focus to one or two of these activities.
Whether to provide preliminary evaluations to staff for use in improving program operations and developing additional services is an issue that needs to be faced. Preliminary results can be effectively used to identify operational problems and develop the capacity of program staff to conduct their own ongoing evaluation and monitoring activities (Connell, J. P. , Kubisch, A. C. , Schorr, L. B. , and Weiss, C. H. (1995). But this use of evaluation findings, called formative evaluations, presents a challenge to evaluators who are faced with the much more ifficult task of estimating the impact of an evolving intervention. When the program itself is continuing to change, measuring impact requires ongoing measurement of the types and level of service provided. The danger in formative evaluations is that the line between program operations and assessment will be blurred. The extra effort and resources required for impact analysis in formative evaluations has to be measured against the potential gains to the program from ongoing improvements and the greater usefulness of the final evaluation findings.
Performance monitoring involves identification and collection of specific data on program outputs, outcomes, and accomplishments. Although they may measure subjective factors such as client satisfaction, the data are numeric, consisting of frequency counts, statistical averages, ratios, or percentages. Output measures reflect internal activities: the amount of work done within the program or organization. Outcome measures (immediate and longer term) reflect progress towards program goals. Often the same measurements (e. g. number/percent of youth who stopped or reduced substance abuse) may be used for performance monitoring and impact evaluation. However, unlike impact evaluation, performance monitoring does not make any rigorous effort to determine whether these were caused by program efforts or by other external events. The way that we are looking at Design Variations is when programs are operating in a number of communities, the sites are likely to vary in mission, structure, the nature and extent of project implementation, primary clients/targets, and timeliness.
They may offer somewhat different sets of services, or have identified somewhat different goals. In such situations, it is advisable to construct a “core” set of performance measures to be used by all, and to supplement these with “local” performance indicators that reflect differences. For example, some youth programs will collect detailed data on youth school performance, including grades, attendance, and disciplinary actions, while others will simply have data on promotion to the next grade or whether the youth is still enrolled or has dropped out.
A multi-school performance monitoring system might require data on promotion and enrollment for all schools, and specify more detailed or specialized indicators on attendance or disciplinary actions for one or a subset of schools to use in their own performance monitoring. Another look is at the Considerations/Limitations when selecting performance indicators, evaluators and service providers need it is important to consider: The relevance of potential measures to the mission/objective of the local program or national initiative. The comprehensiveness of the set of measures. The program’s control over the factor being measured.
The validity of the measure and the reliability and accuracy of the measure, feasibility of collecting the data. How much effort and money is required to generate each measure? Practical Issues. The set of performance indicators should be simple, limited to a few key indicators of priority outcomes. Too many indicators burden the data collection and analysis and make it less likely that managers will understand and use reported information. Regular measurement, ideally quarterly, is important so that the system provides the information in time to make shifts in program operations and to capture changes over time.
However, pressures for timely reporting should not be allowed to sacrifice data quality. For the performance monitoring to take place in a reliable and timely way, the evaluation should include adequate support and plans for training and technical assistance for data collection. Routine quality control procedures should be established to check on data entry accuracy and missing information. At the point of analysis, procedures for verifying trends should be in place, particularly if the results are unexpected. The costs of performance monitoring are modest relative to impact evaluations, but still vary widely depending on the data used.
Most performance indicator data come from records maintained by service providers. The added expense involves regularly collecting and analyzing these records, as well as preparing and disseminating reports to those concerned. This is typically a part-time work assignment for a supervisor within the agency. The expense will be greater if client satisfaction surveys are used to measure outcomes. An outside survey organization may be required for a large-scale survey of past clients; alternatively, a self-administered exit questionnaire can be given to clients at the end of services.
In either case, the assistance of professional researchers is needed in preparing data sets, analyses, and reports. Process Analysis key element in process analysis is a systematic, focused plan for collecting data to: (1) determine whatever the program model is being implemented as specified and, if not, how operations differ from those initially planned; (2) identify unintended consequences and unanticipated outcomes; and (3) understand the program from the perspectives of staff, participants, and the community.
The design variation is the systemic procedure used to collect data for process evaluation often include case studies, focus groups, and ethnography. As strong pressures demonstrates program impacts dictates making evaluation activities a required and intrinsic part of program activities from the start. At the very least, evaluation activities should include performance monitoring.
The collection and analysis of data on program progress and process builds the capacity for self-evaluation and contributes to good program management and efforts to obtain support for program continuation-for example, when the funding is serving as “seed” money for a program that is intended, if successful, to continue under local sponsorship. Performance monitoring can be extended to non-experimental evaluation with additional analysis of program records and/or client surveys. These evaluation activities may be conducted either by program staff with research training or by an independent evaluator.
In either case, training and technical assistance to support program evaluation efforts will be needed to maintain data quality and assist in appropriate analysis and use of the findings. There are several strong arguments for evaluation designs that go further in documenting program impact. Only experimental or quasi-experimental designs provide convincing evidence that program funds are well invested, and that the program is making a real difference to the well-being of the population served. These evaluations need to be conducted by experienced researchers and supported by adequate budgets.
A good strategy may be implementing small-scale programs to test alternative models of service delivery in settings that will allow a stronger impact evaluation design than is possible in a large scale, national program. Often program evaluation should proceed in stages. The first year of program operations can be devoted to process studies and performance monitoring, the information from which can serve as a basis for more extensive evaluation efforts once operations are running smoothly. Finally, planning to obtain support for the evaluation at every level-community, program staff, agency leadership and funder-should be extensive.
Each of these has a stake in the results. Each should have a voice in planning. And each should perceive clear benefits from the results. Only in this way will the results be acknowledged as valid and actually used for program improvement. Reference Connell, J. P. , Kubisch, A. C. , Schorr, L. B. , and Weiss, C. H. (1995) New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: The Aspen Institute. Ellickson, P. L. , Bell, R. M. , and McGuigan, K. (1993) “Preventing Adolescent Drug Use: Long- Term Results of a Junior High School Program. American Journal of Public Health 83(6): 856-861. Engle, R-F and Granger, CW. J. (1987) “Cointegration and Error Correction: Representation, Estimation and Testing. ” Retrieved from: Econometrica 55: 25 1-276. Evaluation Strategies for Human Service Programs. Retrieved from http://www. ojp. usdoj. gov/BJA/evaluation/guide/documents/evaluation_strategies. html#p 6. Heckman, J. J. (1979) “Sample Selection Bias as a Specification Error. ” Econometrica 47:153-162. IRB Forum. Retrieved from www. irbforum. rg. Joreskog, K. G. (1977) “Structural Equation Models in the Social Sciences. ” In P. R. Krishnaiah (ed. ), Applications of Statistics, 265-287. Amsterdam: North-Holland; Bryk, A. S. and Raudenbush, S. W. (1992) Hierarchical Linear Models: Applications and Meta- Analysis Techniques. Newbury Park, CA: Sage Kalbfleish, J. D. , and Prentice, K. L. (1980) the Statistical Analysis of Failure Time Data. New York: Wiley. Kumpfer, K. L, Shur, G. H. , Ross, J. H. , Bunnell, K. K. , Librett, J. J. and Milward, A. R. 1993) Measurements in Prevention: A Manual on Selecting and Using Instruments to Evaluate Prevention Programs. Retrieved from: Public Health Service, U. S. Department of Health and Human Services, (SMA) 93-2041. Monette, Duane R. , Thomas J. Sullivan, Cornell R. DeJong. Applied Social Research: A Tool for the Human Services, 8th Edition. Wadsworth, 2014-03-11. . MREL Appendix A. Retrieved from: http://www. ecs. org/html/educationIssues/Research/primer/appendixA. asp. Program Evaluation 101: A Workshop. Retrieved from: http://aetcnec. ucsf. edu/evaluation/pacific_evaluation%5B1%5D. ppt.
