EVALUATION AND REPORTING Presenter Iris E. Pettigrew, RN, MS, ScD, CPHQ Enterprise Services and Technologies, Inc. (EST) 912 Thayer Avenue, Suite 202 Silver Spring, MD 20910 TodayÕs Presentation ¥ Why we are having this presentation ¥ Review of the Logic Model (Evaluation Model) ¥ Sample model format received ¥ Reporting ¥ Summary of Changes ¥ Questions Evaluation as a Tool Evaluation ¥ Provides a common language ¥ Helps us differentiate between Òwhat we doÓand ÒresultsÓ ---outcomes ¥ Increases understanding about program ¥ Guides and helps focus work ¥ Leads to improved planning andmanagement ¥ Increases intentionality and purpose ¥ Provides coherence across complex tasks, diverse environments Evaluation (cont.) ¥ Enhances team work ¥ Guides prioritization and allocation of resources ¥ Motivates staff ¥ Helps to identify important variables to measure; use evaluation resources wisely ¥ Increases resources, opportunities, recognition ¥ Supports replication ¥ Often is required! So, why bother? WhatÕs in this for you? ÒThis seems like a lot of work.Ó ÒWhere in the world would I get all the information to put in a logic model? ÒIÕm a right brain type of person Ð this isnÕt for me.Ó ÒEven if we created one, what would we do with it?Ó Benefits for Your Program 3/4 Short term 3/4 Medium 3/4 Long term Benefits to ADD 3/4To have valid comparable results between programs 3/4To fully examine and describe the effective parts of a program for duplication elsewhere. 3/4To verify or show data that the program accomplished its goals ADD-YIC Evaluation Expectations Process Evaluation Outcome Evaluation that is: 3/4 quantitative -focus on numbers, percentages of target reached based on goal 3/4 qualitative -focus on how individuals and groups view and understand the program and gain meaningfrom their experiences. (based on participantobservation, interview, etc) Both related to the goals stated in the evaluation model The Logic Model An Evaluation Tool Where are you going? How will you get there? What will tell you that youÕve arrived? A logic model is your program ROAD MAP What Is A Logic Model ¥ Picture of your program or intervention ¥ Graphic representation of the Òtheory of actionÓ Ð what is invested, what is done, and what are the results ¥ Core of planning and evaluation Provides a common framework for your work Definition LOGIC ¥ the principles of reasoning ¥ reasonable ¥ the relationship of elements to each other and awhole MODEL ¥ small object, representing another, often largerobject (represents reality, isnÕt reality) ¥ preliminary pattern serving as a plan ¥ tentative description of a system or theory thataccounts for all of its known properties The American Heritage Dictionary, 2nd Ed The Accountability Era ¥ What gets measured gets done ¥ If you donÕt measure results, you canÕt tell success from failure ¥ If you canÕt see success, you canÕt reward it ¥ If you canÕt reward success, youÕre probably rewarding failure ¥ If you canÕt see success, you canÕt learn from it ¥ If you canÕt recognize failure, you canÕt correct it. ¥ If you can demonstrate results, you can win public support. Re-inventing government, Osborne and Gaebler, 1992 Generic logic model Strategy Results A diagram of the theory of how a program is supposed to work A graphic depiction of relationships between activities and results PLANNING INPUTS Program Investments OUTPUTS OUTCOMES (IMPACT OBJECTIVES) Activities Participation Short Medium Long-Term EVALUATION Understanding Components OUTCOMES (IMPACT OBJECTIVES) Short Medium Long-Term What the short term results are What the medium term results are What the ultimate impact is Learning Action Conditions Awareness Behavior Social Knowledge Practice Economic Attitudes Decisions Civic Skills Policies Environmental Opinions Social action Aspirations Motivations Measurement OUTPUTS OUTCOMES Activities Participation Short Medium Long-Term INPUTS Program Investments EVALUATION What does a logic model look like? ¥ Ð Relationships, linkages ¥ Any shape possible Ð Circular, dynamic Ð Cultural adaptations; storyboards ¥ Level of detail ¥ simple ¥ complex ¥ Multiple models EXAMPLES Example Every day logic model Ð Family Vacation Example: Financial management program Situation: Individuals with limited knowledge and skills in basic financial management are unable to meet their financial goals and manage money to meet their needs. Program We conduct a variety Participants gain invests time and of educational activities knowledge, change practices and have resources targeted to individuals improved financial who participate well-being WHAT WE DO (How Many WHAT WE INVEST Groups or Persons) WHAT RESULTS Example: One component of a comprehensive parent education and support initiative Situation: During a county needs assessment, majority of parents reported that they were having difficulty parenting and felt stressed as a result INPUTS OUTCOMES Example: Logic model training session Situation: ADD requires grantees to include a logic model in conducting their program evaluation; grantees have limited understanding of logic models and are unable to fulfill the reporting requirement INPUTS OUTPUTS OUTCOMES Increase knowledge of logic Trainer ¥2.5 hour models Budget training Use logic models Improved ¥TA sessions in planning and planning ÐEquipment Increase ability to evaluation Ð in programs ¥Practice achieve positive Training create a results curriculum ¥Q and A meaningful logic model of program Model quality logic model practice Improved evaluation more credibleIncrease and useful dataconfidence in using logic models Short-term Medium Long-term Component Logic Model Youth: Youth Advocating for Policy Change Outcomes -Impact Inputs Activities Reach Short Medium Long Establish baseline for members Coalition policy change in community with help from youth Time Educate youth and adults on policy change options and how to achieve them Funding Identify partners, including youth serving Partners organizations and schools, for engaging ¥Local youth in policy change ¥ Regional ¥ State Develop strategy for and promote engagement of youth in Effective practice policy change strategies Assist with development of youth advocacy skills Local media outlets Promote community support for youth involvement in community affairs/ policy change Multiple logic models Multiple models may be needed to describe and explaincomplex initiatives or systems. 1. Multi-level programs: A series of linked models that depict varying levels such as national-state-countylevels OR, institution-division-unit levels 2. Multi-component programs: A series of models to depict various components (goals, sites, target populations) within a comprehensive initiative REPORTING USING THE MODEL TO WRITE YOUR PROGRESS REPORT ¥ Submitting the updated model with your semi-annual reports ¥ Using the model to enhance what is reported ¥ Incorporating the model results into the current reporting format EXAMPLE EVALUATION: What do you (and others) want to know about this program? What amount Were all Did all parents To what extent To what To what of $ and time sessions attend that we did knowledge extent did extent are were delivered? intended? Who and skills behaviors relations invested? How did/not not?Did they increase? For change? For improved? effectively? attend all sessions? whom? Why? whom? Why? Does this What else What else result in happened? happened? stronger families? REPORT FORMAT Goals/Objectives ¥ Major Activities and Accomplishments ¥ Problems ¥ Significant Findings and Events ¥ Dissemination activities ¥ Other Activities ¥ Activities Planned for Next Period: as before SUMMARY What do you want to know? How will you know it? Limitations Logic ModelÉ ¥ Represents reality, is not reality ¥ Focuses on expected outcomes ¥ Challenge of causal attribution 9Many factors influence process and outcomes ¥ DoesnÕt address: Are we doing the right thing? Logic Model and Evaluation Process evaluation: How is program implemented? Fidelity of implementation? Are activities delivered as intended? Are participants being reached as intended? What are participant reactions? Outcome evaluation: To what extent are desired changes occurring? For whom? Is the program making a difference? What seems to work? Not work? What are unintended outcomes? Semi-annual Report Goals/Objectives ¥ Major Activities and Accomplishments ¥ Problems ¥ Significant Findings and Events ¥ Dissemination activities ¥ Other Activities ¥ Activities Planned for Next Period: as before QUESTIONS