Surveying: How Big Should Our Sample Be? Part 2

Well, I was hit with the same question again a week ago: “How big should our sample be?” In this case I have five treatment schools and need to identify comparison schools so I can truly ascertain whether any changes are related to the program I am evaluating.

A first note – yes, I must make comparisons at the school level. Even though I will be using test scores and I would love to aggregate them at the classroom/teacher or keep them at the student level, I am restricted to the school level for two reasons – 1) the intervention is a school-wide intervention, it’s not limited to certain teachers or students, and 2) the treatment schools were purposefully selected based on school-level characteristics.

What else do I know? Mainly that I will need a large number of schools in my comparison group as I am hoping to detect an intervention that may have a small effect size (.2 – maybe a little less), according to Cohen. Remember, Effect size = (Treatment group mean – Control group mean) / Pooled standard deviation. Thus I will need to design my study so that it will account for a potentially small effect size.

Luckily, I can again use a power analysis to help me out. For this analysis, I will set alpha to .05 (this limits my Type I error to 5%, in other words, limits the chances that my test accepts an effect that does not exist to 5%) and I will set beta, my Type II error rate, to 20% (thus limiting the chance that my test rejects an effect that actually exists to 20%).

Using these limits, I find that with 5 treatment schools, depending upon the number of comparison schools I include, the effect size my test will detect increases as the number of comparison schools increase. For example, with 5 treatment schools, the effect sizes I can detect depending upon the number of comparisons schools I include are as follows:

Comparison schools Effect Size detected
5 .297
10 .239
20 .189
40 .164

Right now I would like to use comparison schools in the same district so will probably try to identify the 20 elementary schools that match the closes on key school indicators (size, %FRL, % teachers highly qualified, etc.). That will allow me to identify an effect size of .189 or greater, if one exists.

(With special thanks to John Keltz at UW-Madison and part of the Center for Educator Compensation reform who gave a great talk on Effect Size and Power)