Monday, February 22, 2010

Evaluation: Quasi-Experimental Designs - Advantages and Disadvantages of Each

Because after all my talk about Causality and Rubin it's very hard to do experimental designs in education. So here's an overview of quasi-experimental designs - some which are worthy of trying! But not all!

One-Shot Pre Test Only Design:
O X
Ha-Ha - I made this one up! Who would do this, although I guess it could occur if you ran out of funding (or everyone dropped out of the treatment pool). I have seen stranger things proposed.....

One-shot Post Test Only Design:

X O
Sigh. How do you know there was even a change? Enough said.

One-shot Pre-Post Test Design:

O X O
Double sigh. Okay - you may be able to detect a change but how can you attribute it to the treatment? You need another group. However, I see this proposed WAY TOO OFTEN.

Post-test Only Intact group Design:

X O
- O
Another sigh (but only one). Good that you now have two groups. However, you don't know whether your two groups started at the same place. For example, if your treatment group scores high on a test (a good thing) and your control group does not, you don't know whether your treatment group would have scored higher than the control group to begin with (and whether their final score is really one of NO CHANGE).

Pre-Test Post-test Intact Group Design:

O X O
O - O
Now we're cooking! Two groups, hopefully equivalent. If persons were randomly assigned to a group this would be an experimental design (and we would be cooking with gas!). Without random assignment it remains a quasi-experimental design. However, if we used propensity score matching to match groups it would be a HIGH QUALITY quasi-experimental design. Or we could use regression discontinuity and use a cut score to determine the two groups. Again, that would result in a HIGH QUALITY quasi-experimental design.

Now - go forth . And design better evaluations!

3 comments:

Anonymous said...

I agree with everything you say here, but in my experience these levels of quality in design are directly tied to the amount of funding received, and I often don't have the resources to test two groups pre and post-

Anonymous said...
This comment has been removed by a blog administrator.
Free Online Survey said...

This is one of my favorite blog because whenever i visit this blog found something interested and different,you are doing very well job,keep it up