Monday, June 21, 2010

Evaluation - Evaluation Questions: What's So versus So What?

Having recently read a multitude of evaluation reports as part of a literature review I was conducting, I realized that too many evaluations do not ask evaluative questions. That is, they fail to get at the heart of the actual evaluation.

What do I mean by this?

Namely, that many evaluation questions that are asked, while critically addressing important process or summative issues, do not ask actual evaluative questions. This may be best explained as the evaluation questions addressing "What's so?" while failing to address "So what?"

"What's so" questions generally ask about certain states - that is, they are descriptive or causal in nature. Take for example the following questions:

• What are drivers of and barriers to the successful implementation of this project?
• Who has enrolled in this program and to what degree do enrollees represent the target population?
• To what degree did participants' skills increase as result of their participation?
• How have participants' views changed as a result of attending this conference?

These are good questions, don't get me wrong. However, if these are the types of questions driving your evaluation you've only asked "What's So?" and failed to ask "So What?". "So what" questions are questions about the value of findings, changes, outcomes, etc. They are at the heart of the definition of evaluation (the process of determining the merit, worth, and value of things).

So what are examples of "So what" questions?

As noted, "So what" questions ask about the value, merit, and worth of such things as outcomes. So a truly evaluative question might be:

• How well does this program meet participants' and other stakeholders' needs?
• To what degree are these outcomes critical to changing persons' behaviors?
• How effective / cost-effective is this program compared to ones like it?
• Is this project more effective with some participants over others?

For example, I worked with a friend to identify some evaluation questions for a project that she hopes to evaluate that supports teaching basic math and budgeting to low-income families. Below are six questions I identified for this evaluation. Questions 1 - 5 are "What's so" questions; 6 is a "So What" question:

1. How many persons have enrolled and to what degree do they represent the surrounding community / target audience?
2. How do participants perceive all aspects of this program, including: working with volunteers and attending educational courses?
3. What could be improved?
4. To what degree are participants gaining more knowledge about basic math skills and budgeting?
5. To what degree have participants' budgeting processes been developed or changed as a result of this program?
6. Is this program worth extending to other low-income families?

Hopefully my explanations and examples will help you to think more evaluatively about your evaluations and the questions you ask. As the questions should guide your entire evaluation they merit great consideration.