Dealing with Small Sample Sizes

The most recent ATE newsletter developed by Western Michigan University for NSF Advanced Technology Education projects and centers has a nice article about how to handle small sample sizes (along with some other great articles).

Small sample sizes are something all evaluators face at some point in their professional lives.  And while evaluators may want ignore small sample sizes and treat such data as they might had they had larger sample sizes, evaluators might want to consider small sample sizes as opportunities! Specifically an opportunity to collect qualitative data that might substantiate results and potentially provide more powerful stories than numbers alone. 

Other recommendations ( these from Eboni Zamani-Gallaher) include:

·         Try to gather data on everyone by using a census, rather than a sample. Just remember to limit data analysis to descriptive statistics, rather than inferential.

·         Be upfront about the limitations and document your sampling strategies, decisions, and criteria.

·         See it as an opportunity to keep evaluation costs low recognizing that a large study without sufficient resources can under-power results.

Additionally:

·         Hesitate to report percentages, or don't at all - report fraction instead, as percents can be misleading and may overstate results.

·         Do not conduct quantitative tests of statistical inference where data requirements are at best ignored and at worst violated.

·         If small sampling sizes are the result of missing data, then there may be other possibilities for dealing with this issue.  One way is imputing missing data , of which there are multiple methods, including mean substitution, regression, and more intricate imputations such as multiple imputation using virtual datasets.

Perfect versus Interesting: Where should evaluation sit?

 I read a lot of different blogs as part of my personal and professional learning and last year began reading Seth Godin's blog. An entrepreneur, marketer, and author, he wrote the following in April 2011:

"There are two jobs available to most of us:

You can be the person or the organization that's perfect. The one that always ships on time, without typos, that delivers flawlessly and dots every i. You can be the hosting company or the doctor that might be boring, but is always right.

Or you can be the person or the organization that's interesting. The thing about being interesting, making a ruckus, creating remarkable products and being magnetic is that you only have to be that way once in a while. No one is expected to be interesting all the time.

When an interesting person is momentarily not-interesting, I wait patiently. When a perfect organization, the boring one that's constantly using its policies to dumb things down, is imperfect, I get annoyed. Because perfect has to be perfect all the time."

This post made me wonder where evaluation and evaluators fit?  Should evaluation/evaluators  be perfect or should it/we be interesting?

I would argue that good evaluation is perfect whereas better evaluation is interesting - and thus most likely imperfect some of the time. And that such imperfection is just the cost that comes when taking risks.

Let me try to explain: One example of a huge hit has been evaluators' forays into data visualization and reporting. Attendees at the American Evaluators Association meeting in 2011 could attend multiple sessions on using data visualization that were intended to improve client's understanding of,  interest in, and use of data and evaluation. The evaluators presenting these materials had all taken risks with their evaluations and clients to use new techniques to increase greater stakeholder buy-in. Examples included attempting different reporting formats, using data dashboards, and creating new data visualizations.

Of course, there are is always the flip-sides to the same coin. I recently took a risk when sharing some data with clients by putting together a Prezi presentation, thinking "why not try something different from PowerPoint"?  While they loved the presentation, they were much more interested in what they could do with Prezi than the actual data I presented. It took a lot of time and effort to steer the conversation back to the data!

But my imperfection had a few benefits.  It reminded me of the value of my data  message and the need to ensure that nothing obscures that message when sharing it with clients and /or stakeholders.  It also allowed my clients to see that I was taking risks to better reach them, which they appreciated, and knew I would do when helping them with their own outreach and messaging. So that old saying, 'nothing ventured, nothing gained' was true in this case.

So the moral to the story, for me at least, is to try to be more interesting, even at the risk of not being perfect. I think risk is necessary to growing evaluation and meta-evaluation and truly making evaluation a trans-discipline, to use a term from Michael Scriven.

 

 

What I'm Reading

I'm always interested in what evaluators are reading, so thought I would share below some of the books and blogs I’ve been reading lately. I would love to know what you are reading and how is it improving your evaluation and consulting practices.  Please leave me a post!

Books

I’m constantly referring to this book for ideas – definitely a classic: Campbell, D. & Stanly, J. (1963). Experimental and quasi-experimental designs for research. New York, NY: Wadsworth Publishing.

Okay – not a straight read – but another go-to book: Scriven, M. (1991).  Evaluation thesaurus (4^th Edition). Thousand Oaks, CA: Sage Publications, Inc.

Very useful for helping me understand the history of evaluation: Alkin, M. (2004).  Evaluation roots: Tracing theorists' views and influences. Thousand Oaks, CA: Sage Publications, Inc.

Because almost all evaluators are consultants as well: Block, P. (2011).  Flawless consulting: A guide to getting your expertise used (3^rd Edition).  Hoboken, NJ: John Wiley & Sons.

Blogs

Stephen Few on data visualization (perceptualedge.com ) and Nancy Duarte on presenting blog.duarte.com/), because both are central to my evaluation work. 

For Pleasure

A beautiful and haunting “evaluation” of the Mann Gulch Fire.  This is probably the 5^th time I’ve read it: Maclean, N. (1993). Young men and fire. Chicago, IL: University of Chicago Press

What I Hope to Read Next

Michael Quinn Patton - Essentials of Utilization-Focused Evaluation. Just arrived!

Analyzing and Reporting Likert and Numerical Data

Like most evaluators, I gather lots of Likert and numerical data.  Over the years I have changed the way I report these data so that my clients can better grasp what the results mean.

Likert Data

Almost all of my Likert data are numerically anchored.  So, for example, Strongly Disagree is anchored at a 1 and Strongly Agree would be anchored as a 5, with all other qualitative options anchored numerically in-between.  In these occasions I tend to report the sample size (n), minimum value, maximum value, mean, and standard deviation, as well as the percent of persons who Agree or Strongly Agree (i.e., those who selected a 4 or 5).  I can report the mean and standard deviation because I have numerically anchored m Likert scale. This allows my clients to see right away how persons responded, on average. By reporting the standard deviation I am able to show them how “spread out” the data are, another point of information for my clients. Last, by reporting the percent of persons selecting a 4 or 5, I provide my clients an additional data point for them to consider.

Numerical Data

Like most evaluators reporting numerical data, I report the sample size (n), minimum value, maximum value, as well as the mean and standard deviation. Lately I’ve also begun reviewing the median value in a dataset and reporting that value as well. Oftentimes this data point is very revealing when compared to the mean, especially if they are very different in value. Explore what this means for your dataset and share with your client.

ALL DATA

Even better than presenting these data in table format, consider providing them in graphical form.  Excel allows you to do most graphs you might need.  Using text boxes and shapes such as arrows I can usually point out the mean and median values and the interval that is bounded by the sd.

Exemplary Evaluation and Exemplary Evaluation Firms

I eagerly awaited the newest issue of AJE as my colleague Melanie Hwalek and her company SPEC Associates were to be featured.  After returning from a week of site visits in CENLA, there it was. (By the way CENLA is Central Louisiana, not Central Los Angeles - much less shopping - but LA doesn't have crawfish, does it?).

The article, "Aspects of Successful Evaluation Practice at an Established Private Evaluation Firm" by Paul Brandon, Nick Smith, and Melanie did not disappoint. Like many of my conversations with Melanie, I came away with multiple ideas about how to make my evaluation work more meaningful to my clients and myself, my firm more successful, and my clients more knowledgeable about evaluation.

While I highly suggest all evaluators read this article, it has interesting insights for those of us who consider ourselves evaluation consultants. Particularly interesting to me were the conditions Melanie identified as necessary for exemplary evaluation practice to occur. These include:

1. Clients who seek to learn, change, and improve;
2. Internal "evaluation champions" within client organizations who will support the conduct of evaluation studies and use of results;
3. Effective feedback mechanisms from evaluators to clients so evaluation information is understood and used; and
4. Trust between evaluators and client organizations.

Are there other conditions that need to be in place for exemplary evaluation to occur - whether on the evaluator side or client side?  Certainly evaluators must come with lots of skills - hard evaluation skills and soft evaluation skills (as Melanie notes, good evaluators listen more and talk less), but what other conditions should be in place?  

And, as a corollary, what conditions are necessary for a firm to be an exemplary evaluation firm?

The authors suggest that  organizational flexibility within the evaluation firm to take on more and/or different work (often by hiring subject experts) was critical to building this exemplary  evaluation practice. And there is definitely a culture of learning within Melanie's firm.  But what else is necessary?

Like a great red wine, SPEC Associates seems to have only gotten better with time, and I look forward to learning more from Melanie and her experiences.

    

Cheers my friend and a toast to many jobs done quite well!