At WISE’s 2017 Summit on Education, CNN’s Fareed Zakaria was asked the ONE thing that children need to learn. His answer? “Intellectual discipline.” He defined it as the ability to wade through noise, discern the facts, analyze findings and form a point of view. We echo this sentiment for the social sector. You need to know what to pay attention to and what not to pay attention to. You need to be a good consumer of market research (side note: today’s market research is getting sloppier and can be biased, so pay close attention to the source and the methodology before using it.) You also need to avoid letting perfect be the enemy of the good. One of the best ways to ensure “intellectual discipline” in your impact management process is to develop an evaluation plan.
You don’t have to be a data scientist or evaluator to be able to build an evaluation plan; anyone can do it. The most challenging part of impact management is developing a theory of change and logic model, and deciding what to measure. Once this is done, you just need a plan to collect, analyze and communicate your results. We encourage you to “right-size” your evaluation planning to the needs of your organization, using sharp discernment to collect only what you need. An evaluation plan is a written document that describes how you will monitor and evaluate your organization/program, as well as how you intend to use evaluation results for program improvement and decision-making.
To construct an evaluation plan:
Define the purpose of the evaluation with key questions, such as:
- Is there existing research on your issue/program area? (Think back to your program design)
- Is the program new or existing?
- Who is your audience (e.g., funders, donors, board, staff)?
- What budget and resources do you have to implement it?
- How will results be utilized (e.g., annual report, performance evaluation)?
- What are your goals (e.g., descriptive, predictive, prescriptive)?
- What type of evaluation(s) are you planning (e.g., impact, performance, formative, summative, developmental)?
Based on your answers, determine your needs and collect the best performance indicators to meet these needs.
Clarify organization/program goals and objectives and develop evaluation questions around the following areas:
- Planning & Implementation
For example, who participated and how did they find out about us?
- Attainment of Objectives
For example, how many people participated and what was the frequency/intensity (e.g. number of classes)?
- Participant Impact
For example, what has changed and are participants satisfied?
- Community Impact
For example, what community goal did we contribute to and what was the return on investment?
Once you clarify your evaluation questions, then determine the best approach for collecting the results, including demographic databases, participant surveys, participant focus group/interviews, community impact surveys and cost/benefit analysis.
Plan How and When?
Develop a project plan matrix with the following categories:
- Evaluation Question
What result is being measured?
- Indicator/Performance Measure
How can it be measured?
How is the indicator being tracked?
How often will the indicator be tracked?
Who is responsible for tracking it?
Which stakeholders need the results?
Once you have collected the data based on your evaluation plan, analyze the results regularly and conduct quarterly reviews to monitor and adjust as needed. Once the results are finalized, report them to the appropriate stakeholders.
The most important step of evaluation planning comes at the end – after the data is collected, how do you turn it into something meaningful? The focus on Big Data by the media – and even the coining of the phrase – miss the point. Data is just an input. The real gem is the insight gleaned from the data, which ultimately leads to program and organizational improvements. By measuring what matters and right sizing it for your organization, you will be able to focus on insights that produce positive results for your clients and best tell your organization’s story. We welcome your feedback on your impact management process and look forward to hearing how you cut through the noise to measure what matters.