Evaluation: What is the value added of utilizing an (external) evaluator?

This question (what is the value added of utilizing an (external) evaluator?) was recently explored as part of the thought leaders forum on AEA’s website. One person responded, “I think the unique benefit an evaluator can bring is an explicit evaluative framework (criteria, standards, synthesis) - not just saying "I know what is good - and this is/is not" (content expert connoisseur approach to evaluation) or "Most people said it was good/not good" (social researcher approach to evaluation).” I agree with this statement and think it’s very well articulated (and apologize that I cannot remember who said it).

I’ve been thinking about this comment for a while and wanted to expand upon it with some of my own thoughts. Here’s how I would respond if asked this question.

Objectivity

Perhaps the first one that should be noted is that an external evaluator brings objectivity to the evaluation. The external evaluator’s investment should be in the quality of the evaluation they conduct as opposed to the findings of the evaluation or the program/project they are investigating. Keeping objective needs to be and remain in the forefront of the external evaluator’s mind.

Knowledge of evaluation methodologies or theories of evaluation

An external evaluator should have knowledge of evaluation methodology or theories of evaluation and be able to ascertain which methodologies / theories will most likely produce the most reliable and valid data for the questions being asked.

Knowledge of and adherence to evaluation standards

Perhaps because evaluation has been described as a “political activity” all external evaluators should know and adhere to evaluation standards, whether AEA’s Guiding Principle or the Evaluation Standards developed by the Joint Committee on Educational Evaluation Standards. These should guide all aspects of the evaluation, from start to finish, with equal emphasis placed on all standards.

Knowledge of multiple types of evaluation (e.g., formative, summative, cost-benefit analysis, etc.)

External evaluators should be able to conduct multiple types of evaluation and understand which types of evaluations are needed to best evaluate the program under study. While not all external evaluators need to be experts at all types of evaluation, they should be able to ascertain what types of evaluation will produce the most reliable and valid answers to the questions being asked.

Broader view of impacts to include unintended outcomes

Sometimes programs have unintended outcomes, and whether positive or negative, evaluators should know to be on the lookout for such outcomes. Whereas an internal evaluator may not be aware to look for unintended outcomes, or may be too focused on intended programmatic outcomes, external evaluators should always be assessing what changes have occurred due to actual programming.

Surveying - Assessing Survey Reliability

Just a quick note that the PowerPoint I presented as part of the American Evaluation Association / Center for Disease Control (AEA/CDC) Summer Evaluation Institute about improving survey reliability can be found at the following link: Improving Survey Reliability

In it I include screenshots to help viewers understand the steps one must take to assess the uni-dimensionality of a survey scale and then how to calculate a scale's reliability using Cronbach's alpha.

If you have any questions, don't hesitate to contact me at agermuth@evalworks.com!

Evaluation - Evaluation Questions: What's So versus So What?

Having recently read a multitude of evaluation reports as part of a literature review I was conducting, I realized that too many evaluations do not ask evaluative questions. That is, they fail to get at the heart of the actual evaluation.

What do I mean by this?

Namely, that many evaluation questions that are asked, while critically addressing important process or summative issues, do not ask actual evaluative questions. This may be best explained as the evaluation questions addressing "What's so?" while failing to address "So what?"

"What's so" questions generally ask about certain states - that is, they are descriptive or causal in nature. Take for example the following questions:

• What are drivers of and barriers to the successful implementation of this project?
• Who has enrolled in this program and to what degree do enrollees represent the target population?
• To what degree did participants' skills increase as result of their participation?
• How have participants' views changed as a result of attending this conference?

These are good questions, don't get me wrong. However, if these are the types of questions driving your evaluation you've only asked "What's So?" and failed to ask "So What?". "So what" questions are questions about the value of findings, changes, outcomes, etc. They are at the heart of the definition of evaluation (the process of determining the merit, worth, and value of things).

So what are examples of "So what" questions?

As noted, "So what" questions ask about the value, merit, and worth of such things as outcomes. So a truly evaluative question might be:

• How well does this program meet participants' and other stakeholders' needs?
• To what degree are these outcomes critical to changing persons' behaviors?
• How effective / cost-effective is this program compared to ones like it?
• Is this project more effective with some participants over others?

For example, I worked with a friend to identify some evaluation questions for a project that she hopes to evaluate that supports teaching basic math and budgeting to low-income families. Below are six questions I identified for this evaluation. Questions 1 - 5 are "What's so" questions; 6 is a "So What" question:

1. How many persons have enrolled and to what degree do they represent the surrounding community / target audience?
2. How do participants perceive all aspects of this program, including: working with volunteers and attending educational courses?
3. What could be improved?
4. To what degree are participants gaining more knowledge about basic math skills and budgeting?
5. To what degree have participants' budgeting processes been developed or changed as a result of this program?
6. Is this program worth extending to other low-income families?

Hopefully my explanations and examples will help you to think more evaluatively about your evaluations and the questions you ask. As the questions should guide your entire evaluation they merit great consideration.

Consulting: Test Yourself

Seth Kravitz recently was a guest writer for the Harvard Business Review website. In his article he identified twenty reality-check statements one might want to test himself or herself against when deciding whether to become an entrepreneur (think consultant) Warning: As this was in response to another blog about consulting by someone wearing rosy-colored glasses, these are relatively negative statements. However, they point to the other side of consulting - it isn't always easy to be a consultant. As one friend noted, "It's the best 80 hours a week you'll ever work", and those 80 hours may be tedious, frustrating, non-paying, and yet necessary. For me, consulting has been very rewarding personally and professionally and I hope to continue as a consultant for a long time. But consulting is also scary at times and I appreciate someone willing to bring conversations about consulting / being an entrepreneur back to reality.

Here are Seth's statements. To read more, please see his article: 20 (More) Reality-Checking Questions for Would-Be Entrepreneurs

1. I am willing to lose everything.
2. I embrace failure.
3. I am always willing to do tedious work.
4. I can handle watching my dreams fall apart.
5. Even if I am puking my guts out with the flu and my mother passed away last week, there is nothing that will keep me from being ready to work.
6. My relationship/marriage is so strong, nothing work-related could ever damage it.
7. My family doesn't need an income.
8. This is a connected world and I don't need alone time. I want to be reachable 24/7 by my employees, customers, and business partners.
9. I like instability and I live for uncertainty.
10. I don't need a vacation for years at a time.
11. I accept that not everyone likes my ideas and that it's quite likely that many of my ideas are garbage.
12. If I go into business with friends or family, I am okay with losing that relationship forever if things end badly.
13. I don't have existing anxiety issues and I handle stress with ease.
14. I am willing to fire or lay off anyone no matter what — how good of a friend they are, if they are my own sibling, if they just had a baby, if they have worked with me for 20 years, if their spouse also just lost their job, if I know they might end up homeless, if they have cancer but no outside medical insurance, or any other horrible scenario millions of bosses and HR people have faced countless times.
15. I am okay with being socially cut–off and walking away from my friends when work beckons.
16. I love naysayers and I won't explode or give up when a family member, friend, customer, business associate, partner, or anyone for that matter tells me my idea, product, or service is a terrible idea, a waste of time, will never work, or that I must be a moron.
17. I accept the fact that I can do everything right, can work 70 hours a week for years, can hire all the right people, can arrange amazing business deals, and still lose everything in a flash because of something out of my control.
18. I accept that I may hire people that are much better at my job than I am and I will get out of their way.
19. I realize and accept that I am wrong ten times more than I am right.
20. I am willing to walk away if it doesn't work out.

Evaluation - Esther Duflo and Experimental Design

Great article by Ian Parker in the May 17 New Yorker about Esther Duflo, economist, MacArthur genius, and most recently recipient of the John Bates Clark Medal. The article discusses her work with the Jameel Poverty Action Lab (J-PAL), a research network specializing in randomized evaluations of social programs. Very thought provoking as it does make me wonder how we can use more randomized evaluations in education - as well as whether we should.

See part of the article here (the rest is for New Yorker subscribers only): The Poverty Lab

Evaluation - Again, Saving Pie for Dessert

Although not a new concept (Edward Tufte addresses this same topic in many of his bools), Stephen Few of Perceptual Edge addresses the overuse of pie and other circular charts and more importantly their failure to convey meaning to data in his latest blog post (see Our Irresistible Fascination with All Things Circular). I find such posts as these very helpful for rethinking ways to visualize data meaningfully as opposed to just artistically, which seems to be all for which some people strive. I also appreciate that Few finds one of the circular graphs well enough designed to keep, but am still left to wonder if the pie chart is useful at any time as a graphic when compared to a data table. If anyone can show me such an instance then I'll happily link to it from my website as I've not seen one yet.

As a note, I track Stephen Few's blog regularly and find him as interesting and exciting as Tufte in terms of pushing data visualization. Another person very influential in that area and in my reporting is Garr Reynolds. Between these two, Tufte, and what Windows 7 offers in terms of graphing I feel like I've moved my reporting to a whole new level. Just remember - simple can be elegant.

Evaluation: Seven Critical Books

I was recently asked by a friend identify books I think every evaluator should read. It proved to be an interesting discussion as we are both evaluators and have different training backgrounds (psychology versus education) and interests (qualitative versus quantitative). I thought it might be interesting to share my list; here it is in no particular order:

Dillman, D., Smyth, J., and Christian, L. (2008). Internet, mail, and mixed-mode surveys: The tailored design method (3rd. Edition). New York, NY: Wiley & Sons.

Campbell, D. & Stanly, J. (1963). Experimental and quasi-experimental designs for research. New York, NY: Wadsworth Publishing.

Davidson, E.J. (2004). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage Publications, Inc.

Scriven, M. (1991) Evaluation thesaurus (4th. Edition). Thousand Oaks, CA: Sage Publications, Inc.

Alkin, M. (2004) Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage Publications, Inc.

Patton, M.Q.(1990). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage Publications, Inc.

Yin, R. (2002). Case study research: Design and methods, (3rd. Edition), Applied Social Research Methods Series, Vol. 5. Thousand Oaks, CA: Sage Publications, Inc.

Interestingly, she agreed with my choices of the Dillman book, Campbell and Stanly ("a classic"), and likes Patton but would have chosen his book on Utilization-focused Evaluation as opposed to the one I chose.

What ones would you identify?