Surveying is both an art and a science and developing a high quality survey question is not always easy to do. Even factual information is a challenge to measure, as reliability and validity can easily be affected by question wording
The “Fantastic Five” checklist includes five question that I have gathered from various sources, to ask about any survey question you have written. An answer of no for any single question below suggests that the survey question you have written may not be one that respondents can reliably answer and thus may need rewriting.
The five questions are:
1. Can the question be consistently understood?
Example: How many times have you been hospitalized in your life?
What counts as hospitalization? A 23-hour admit, Your birth? A day-op surgery?
Clearly define any events that may be viewed inconsistently.
2. Does the question communicate what constitutes a good answer?
Example: When did you first purchase a car?
A year ago, after college, in 1979, etc.
Indicate the answer you are looking for: In what MONTH did you first purchase a car?
3. Do all respondents have access to the information needed to answer the question?
Example: What was the annual premium for your health insurance last year?
Most persons would need their insurance records or check register to accurately answer this question. If respondents need such materials, make sure they are aware of that upfront.
4. Is the question one which all respondents will be willing to answer?
Example: Have you been tested for HIV in the last year?
Many people will respond no or not respond because they fear an answer of yes suggests they are involved in what they consider deviant activities. Instead, if an alternative, less threatening question can get at the same answer, use it. For example, one could ask: Have you donated blood in the last year?
5. Can the question be consistently communicated to respondents?
Example: What was your annual income for 2008?
This may be better written as , "Including all forms of income (e.g., wages, gratuities, social security, rent paid to you, dividend earned, tips, annuities, and alimony) what was your annual income for 2008?" However, note that this is an easier question to ask on a written survey than a survey conducted by an interviewer. Questions that may be hard to consistently administer to respondents might be better off asked as a series of questions.
Good resources for improving your survey question (and where these questions came from) are the following:
DeVellis, R.F. (2003). Scale development: Theory and applications, 2nd edition. Thousand Oaks, CA: Sage
Dillman, D. (1999). Mail and Internet surveys: The tailored design method, 2nd Edition. New York: John Wiley Company.
Fowler, F. J. Jr. (1995). Improving survey questions: Design and evaluation. London: Sage.