The most recent ATE newsletter developed by Western Michigan University for NSF Advanced Technology Education projects and centers has a nice article about how to handle small sample sizes (along with some other great articles).
Small sample sizes are something all evaluators face at some point in their professional lives. And while evaluators may want ignore small sample sizes and treat such data as they might had they had larger sample sizes, evaluators might want to consider small sample sizes as opportunities! Specifically an opportunity to collect qualitative data that might substantiate results and potentially provide more powerful stories than numbers alone.
Other recommendations ( these from Eboni Zamani-Gallaher) include:
· Try to gather data on everyone by using a census, rather than a sample. Just remember to limit data analysis to descriptive statistics, rather than inferential.
· Be upfront about the limitations and document your sampling strategies, decisions, and criteria.
· See it as an opportunity to keep evaluation costs low recognizing that a large study without sufficient resources can under-power results.
Additionally:
· Hesitate to report percentages, or don't at all - report fraction instead, as percents can be misleading and may overstate results.
· Do not conduct quantitative tests of statistical inference where data requirements are at best ignored and at worst violated.
· If small sampling sizes are the result of missing data, then there may be other possibilities for dealing with this issue. One way is imputing missing data , of which there are multiple methods, including mean substitution, regression, and more intricate imputations such as multiple imputation using virtual datasets.