Surveying: The Fantastic Five Checklist for Improving Survey Reliability Writing Better Questions

Surveying is both an art and a science and developing a high quality survey question is not always easy to do. Even factual information is a challenge to measure, as reliability and validity can easily be affected by question wording

The “Fantastic Five” checklist includes five question that I have gathered from various sources, to ask about any survey question you have written. An answer of no for any single question below suggests that the survey question you have written may not be one that respondents can reliably answer and thus may need rewriting.

The five questions are:

1. Can the question be consistently understood?
Example: How many times have you been hospitalized in your life?
What counts as hospitalization? A 23-hour admit, Your birth? A day-op surgery?
Clearly define any events that may be viewed inconsistently.

2. Does the question communicate what constitutes a good answer?
Example: When did you first purchase a car?
A year ago, after college, in 1979, etc.
Indicate the answer you are looking for: In what MONTH did you first purchase a car?

3. Do all respondents have access to the information needed to answer the question?
Example: What was the annual premium for your health insurance last year?
Most persons would need their insurance records or check register to accurately answer this question. If respondents need such materials, make sure they are aware of that upfront.

4. Is the question one which all respondents will be willing to answer?
Example: Have you been tested for HIV in the last year?
Many people will respond no or not respond because they fear an answer of yes suggests they are involved in what they consider deviant activities. Instead, if an alternative, less threatening question can get at the same answer, use it. For example, one could ask: Have you donated blood in the last year?

5. Can the question be consistently communicated to respondents?
Example: What was your annual income for 2008?
This may be better written as , "Including all forms of income (e.g., wages, gratuities, social security, rent paid to you, dividend earned, tips, annuities, and alimony) what was your annual income for 2008?" However, note that this is an easier question to ask on a written survey than a survey conducted by an interviewer. Questions that may be hard to consistently administer to respondents might be better off asked as a series of questions.

Good resources for improving your survey question (and where these questions came from) are the following:

DeVellis, R.F. (2003). Scale development: Theory and applications, 2nd edition. Thousand Oaks, CA: Sage

Dillman, D. (1999). Mail and Internet surveys: The tailored design method, 2nd Edition. New York: John Wiley Company.

Fowler, F. J. Jr. (1995). Improving survey questions: Design and evaluation. London: Sage.

Methdology: Rasch Analyses - what they might add

The past two blogs I have written have been about factor analysis - a data reduction technique I have used frequently. For example, I have often used exploratory factor analyses to determine which items to keep as part of survey constructs. However, I was recently shown some Rasch analyses which showed me such characteristics as item fit and construct reliability, as well as how well items discriminate among persons (e.g. are they easy to agree too or hard to agree to) and how persons viewed response options.

For example, while I was able to use factor analyses to determine that a specific item loaded onto my construct at a weight of .765, I could have found the same thing ("good infit") using Rasch analyses. However, the use of Rasch analyses also showed that this was one of the easier items to agree to (meaning it had low discrimination among persons, not what I wanted). Rasch analyses also told me something about my response options: the Rasch analyses showed that persons did not follow a trajectory from Highly disagree to Somewhat disagree, to Neither agree or disagree, to Somewhat agree, to Highly agree. Rather, a person would go through this sequence but without using the middle response option (Neither agree or disagree). This suggests that instead of a 5-point scale with a middle "neutral" response option, respondents in actuality responded as if the response options represented a 4-point scale, treating the middle option as "Not applicable".

That's a lot of information from one analysis! I plan to try conducting a Rasch analyses next time I would have used factor analyses. One popular program used to conduct such analyses is Winsteps (www.Winsteps.com). It has a primer / how-to book, but I would also suggest reading Bond and Fox's book, "Applying the Rasch Model". It shows some examples using Winsteps as well as the text one could use to reproduce their findings.

*With special thanks to Andrew Swanlund at Learning Point Associates for making me a believer!

Methodology: Factor Analysis - Best Practices Part 2

Following factor analysis, one usually computes factor scores  for use in subsequent analyses.  These scores can be computed in multiple ways, each with different features.  Below I explore such ways so that the reader can make a better choice for him or herself as to which way to compute them.

Below are 5 "non-refined" or simple methods for computing factor scores

1.  Sum or average raw scores (including negative scores, if that's what was produced) corresponding to all items that load on a factor

2.  Sum or average raw scores corresponding to all items that load above a certain cutoff point (e.g., .4 or above)

4.  Calculate a weighted sum or average of raw scores corresponding to all items that load on a factor or only those that load above a certain cutoff point, using the factor loading as the weight.

5.  Calculate a sum, average, or weighted sum or average using the standardized scores  - either for all items or those above a certain cutoff point.  This involves standardizing raw scores to the same mean and standard deviation.  This is most often used when standard deviations vary greatly across the items.

The above five scores are frequently used because of simplicity's sake - the choice as to which to use is guided more by the researcher's decisions than hard science.  One additional method that is worth mentioning, however, is the use of regression scores.  This is considered a "refined" method, meaning linear combination is used to calculate factor scores versus simple addition.  Scores are calculated by weighting the raw scores by their regression coefficients and then standardizing the scores  to a mean of 0.  These scores can easily be obtained using SPSS or SAS and are often used because these popular programs can calculate them without much additional work needed from the researcher.  Researchers often believe that this method produces scores that have greater validity than those produced by "unrefined" methods.