Evaluation: Evaluation Post-Mortems

My partner is a self-proclaimed computer nerd who writes software. As part of how she develops software, she and her business partners analyze, design, code, test, re-factor and document specific features of the software they are designing. At the end of this procedure (usually a 30-day period) they engage in a formal post-mortem where they analyze the success and failures of the project to-date, adding their findings to their general knowledge base for future use.

Conducting post-mortems is a common practice in business, yet I have never heard anyone talk abut doing such an analysis as a systematic process within evaluation. There is very little written about evaluations that don’t work, including why they don’t work. When we have that information (often form our own unfortunate experiences) we often keep that information to ourselves, as reporting it would be reporting our failures.

What if we made a post-mortem part of the evaluation process? Isn’t evaluation an iterative and cyclical process for those of us who design and conduct them? What would a formal post-mortem process at interim periods in an evaluation look like and would it be beneficial (and if so, why and how?)?

When used to assess business projects, some persons engage in a two-step process. The first step is to provide the persons involved a list of questions about the project that they think about and respond to on their own. The second step involves bringing all persons and their responses together to share what they thought and discuss lessons learned.

Michael Greer (http://www.michaelgreer.com/postmortem.htm) has developed a lost of general questions to guide post-mortems. They include such questions as:

1. Are you proud of our finished deliverables (project work products)? If yes, what's so good about them? If no, what's wrong with them?

2. What was the single most frustrating part of our project?

3. How would you do things differently next time to avoid this frustration?

4. What was the most gratifying or professionally satisfying part of the project?

5. Which methods or processes worked particularly well?

6. Which methods or processes were difficult or frustrating to use?

7. If you could wave a magic wand and change anything about the project, what would you change?

8. Did our stakeholders, senior managers, customers, and sponsor(s) participate effectively? If not, how could we improve their participation?

For evaluations, specifically, we might ask such questions as:

- How accurate were our original estimates of the time, cost, and other resources required of the evaluation? What did we over- or under-estimate?

- Knowing what we know now, would we have chosen the same type of evaluation design as the one we used? If not, what could have pointed us to a design that would have been better suited for such a project?

- Were our evaluation questions the best ones, or were there other questions we did not fully explore with stakeholders, through or evaluation, etc. that needed addressing?

- How would we rate the quality of the data we gathered and what could we have done to have collected more convincing data for formative and summative purposes?

- Did our presentation of results highlight the data so that stakeholders could make their own interpretations or understand the ones we made?

- What did we do to help stakeholders understand and use the evaluation findings?

While the list is endless, questions could be further identified by management areas (identifying the evaluand, evaluation method, data collection, data analysis, reporting, etc.).

I view a post-mortem as separate from just following the program evaluation standards (as some of these questions do get at) as it has a very formal outcome, the identification of lessons learned. And I see it as clearly separate from a meta-evaluation as meta-evaluations are themselves evaluations and not designed to identify lessons learned, as much as to identify the value in the evaluation that was conducted. Also, for meta-evaluation to be viewed as unbiased, they do need to be conducted by someone outside of the original evaluation, whereas post-mortems are specifically designed to engage the original evaluators.

I’d be interested in what others think and what other practice. Is there any group who is doing this in a formal and systematic manner where questions are identified and discussed and lessons learned are developed? What are good questions we should be considering if we want to make this part of our formal evaluation practice? Is anyone doing this in collaboration with the former evaluand?