At a recent forum, CNN’s Fareed Zakaria was asked the ONE thing that children need to learn. His answer? “Intellectual discipline.” He defined it as the ability to wade through noise, discern the facts, analyze findings and form a point of view. We echo this sentiment for the social sector. You need to know what to pay attention to and what not to pay attention to. You need to be a good consumer of market research (side note: today’s market research is getting sloppier and can be biased, so pay close attention to the source and the methodology before using it.) You also need to avoid letting perfect be the enemy of the good. One of the best ways to ensure “intellectual discipline” in your impact management process is to develop an evaluation plan.
You don’t have to be a data scientist or evaluator to be able to build an evaluation plan; anyone can do it. The most challenging part of impact measurement (some are calling it impact management now) is developing a theory of change and logic model, and deciding which measurements matter. Once this is done, you just need a plan to collect, analyze and communicate your results. We encourage you to “right-size” your evaluation planning to the needs of your organization, using sharp discernment to collect only what you really need now and can use in the future for continuous improvement. An evaluation plan is a written document that describes how you will monitor and evaluate your organization/program as well as how you intend to use evaluation results for program improvement and decision-making.
To construct an evaluation plan, follow these 3 easy steps:
Determine Why
Define the purpose of the evaluation with key questions, such as:
-
- Think back to program design – is there existing research on your issue/program area?
- Is the program new or existing?
- Who is your audience (e.g., funders, donors, board, staff)?
- What budget and resources do you have to implement the evaluation plan?
- How will results be utilized (e.g., annual report, performance evaluation)?
- What are your goals (e.g., descriptive, predictive, prescriptive)?
- What type of evaluation(s) are you planning (e.g., impact, performance, formative, summative, developmental)?
Based on your answers, determine your needs and collect the best performance indicators to meet these needs.
Determine What to Evaluate
Clarify organization/program goals and objectives and develop evaluation questions around the following areas:
-
- Planning & Implementation – how well did we plan?
For example, who participated and how did they find out about us? - Attainment of Objectives – how well did we meet objectives?
For example, how many people participated and what was the frequency/intensity (e.g., number of classes)? - Participant Impact – how much and what kind of change occurred?
For example, what has changed and are participants satisfied? Think about positive results as well as unintended consequences. - Community Impact – how much and what kind of difference has been made?
For example, what community goal did we contribute to and what was the return on investment?
- Planning & Implementation – how well did we plan?
Once you clarify your evaluation questions, then determine the best approach for collecting the results, including demographic databases, participant surveys (including satisfaction and change in knowledge, attitude and behavior), participant focus groups/interviews, community impact surveys and cost/benefit analysis.
Determine How and When
Develop a project plan matrix with the following categories:
-
- Evaluation Question
What result/outcome are you interested in? - Indicator/Performance Measure
How can it be measured? - Method/Data Sources
How can this indicator be tracked? What are the possible data sources? - Frequency
How often will the indicator be tracked? - Responsibility
Who is responsible for tracking it? - Reporting
Which stakeholders need the results?
- Evaluation Question
Once you have collected the data based on your evaluation plan, analyze the results regularly and conduct quarterly reviews using dashboards to monitor and adjust as needed. Once the results are finalized, report them to the appropriate stakeholders.
The most important step of evaluation planning comes after the data is collected – how do you turn it into something meaningful? The focus on “big data” by the media (and even the coining of the phrase) misses the point. Data is just an input. The real gem is the insight gleaned from the data, which, ultimately, leads to program and organizational improvements. By “right-sizing” evaluation planning for your organization and measuring what matters, you can focus on insights that produce positive results for your clients and help tell your organization’s story. We welcome your feedback on your impact management process and look forward to hearing how you cut through the noise to measure what matters.