4

Somewhat similar to this question, but looking for resources (books/articles, etc) for undergrad students who are not very familiar with designing effective surveys, (for example, one student is creating a survey in SurveyMonkey to evaluate nurses' attitudes toward their work). I'm looking for a resource to help my students develop survey questions that will return meaningful, non-biases responses.

Note: this is in the social sciences, but would welcome resources that will be helpful to undergrads in any field.

  • By "develop," do you mean "build and deploy" (such as SurveyMonkey will do)? Or do you mean "develop good questions," i.e., questions that don't inadvertently introduce bias? – J.R. Oct 18 '13 at 1:36
  • @J.R., I meant 'develop good questions' which do not inadvertently introduce bias. 'Building and deploying the survey' would be another question... – J. Zimmerman Oct 18 '13 at 1:40
  • just a tool, qualtrics is popular – user-2147482637 Aug 10 '14 at 12:35
5

What they need is proper training in survey design and analysis. Let's say, around 80 hours of teaching and then a lot of practical work, assuming they've already got a basic stats grounding.

It's a serious technical skill, and you won't do them any favours if you don't treat it as seriously as any other tool they might use.

One option might be to work with a colleague who does teach survey design and analysis. They may well have students who are looking for some material on which to practice their new skills. Perhaps your colleague can set for them, as homework, the task of working with one of your students on the survey design. That way, your students get to introduce their students to their subject (learning by teaching), and they get to see survey science done reasonably well (assuming your colleague has taught them well).

Just because survey design looks easy and online tools enable any fool to do it, doesn't mean that any fool should do it.

A complaint about "giving them a well when they only need a drink", doesn't hold water. How could any teacher encourage their students to do bad science, or cultivate a contempt for other experts' fields?
(are they physicists?)
enter image description here

  • That would be giving them the well when they only need a drink!...unfortunately neither they nor I have the time and resources for 80 hrs of teaching w/ lots of practical work. So its a good answer, except that I am looking for something I can point them toward and let them learn on their own; pithy advice that's accessible even to a neophyte! – J. Zimmerman Oct 18 '13 at 21:41
  • I want to give my students (undergrads at a junior college) exposure to survey design & analysis, without overwhelming them--hence the comment about the well versus a drink. Ideally, the intro I can give will inspire them to learn more, and more deeply, later in their career. Survey design isn't officially part of these courses, so my ultimate goal in including it is to inspire students to explore it on their own after my introduction. I am NOT trying to "encourage [my] students to do bad science, or cultivate a contempt for other experts' fields"! – J. Zimmerman Oct 19 '13 at 23:30
  • @J.Zimmerman in that case, get them working with other students who have been taught this stuff: that gives them the exposure, without having them do bad science. – EnergyNumbers Oct 20 '13 at 8:46
  • @EnergyNumbers, you would be unpleasantly surprised to find out that the fraction of universities (even research level ones) that have a decent survey methodologist somewhere in the academic staff is probably single to low double digits. Here's an exhaustive list of programs granting degrees or certificates in survey methodology (which usually means there are 3+ people who can teach alternating courses): amstat.org/sections/srms/college.html. If the OP is a local of any of the cities, trying to connect to these folks, even if they are in a different school, might work. – StasK Nov 5 '13 at 23:40
  • +1 for finding areas of mutual benefits through collaboration for students – user-2147482637 Aug 10 '14 at 12:37
2

Zimmerman I think I can help you out. I have a resource where students can create and deploy surveys. It also has a tutorial that helps you create a survey from beginning to end including tips on creating neutral non biased questions.

Check out SurveySidekick.com

The site should be especially useful for beginner survey designers. It was created by Teachers College Columbia University and meant for any higher-ed students so I think this is appropriate for your students.

Hope this helps!

  • Sounds good, but I don't like that it requires an access code for access to the site. +1 for the tutorial--that's the kind of resource I was looking for. – J. Zimmerman Nov 5 '13 at 22:59
2

Everybody can design a good question, right? Just like everybody can build a car engine. Or everybody can cook Boeuf Bourguignonne a la Julia Child. Or everybody can write a speech for a candidate in a state governor election. Instrument design is a professional work that requires understanding how people respond to questions, which in turn requires some psychology on the respondent's end, some statistics on the data user's end, some computer graphics on the GUI end, etc. As a professional survey statistician, my professional duty is to discourage your creating a false sense of "doability" here. Rectifying the user-written instruments is an unpleasant part of the job that a team of survey methodologists in my company has to perform more often than we would have liked to.

Having said that, I would encourage you to lookup something like "questionnaire design class syllabus". The JPSM/UNC class looks good, and refers to right books. The reading list of the UIC course is very comprehensive, if not intimidating. If you don't have the time to read any books, the minimum self-check list is available through the RTI's Question Appraisal System.

  • +1. Good resources, @StasK. Since I am looking to pique my students' curiosity and appreciation for survey design, I appreciate having a way to introduce rigorous methods without giving these students a huge reading list. – J. Zimmerman Nov 6 '13 at 0:43
1

I have found several resources that are helpful for introducing motivated undergrads to the concepts of survey design and analysis. My own favorites are

These books provide a good overview, and are accessible to beginners.

1

I know an answer has already been accepted but I will add this for future searchers.

Coursera has a module called Questionnaire Design for Social Surveys. Since it's free and you can pick and choose which video lectures to watch, your students might find it quite helpful.

One example of the lectures included is "Measurement Error: Bias and Variance" but there are also plenty of others to not just measure bias but also reduce it.

0

Some advice, but not a resource.

  • As something that you want to happen quickly, EnergyNumbers' advice to find a colleague who does this or teaches a class on it is appropriate. Ask this individual if they would be willing to come to your class for one or two sessions to give the students an overview of the process, including examples of good and bad survey questions. A whole lot of success can be achieved by mimicking the successful behavior of others, even if you do not have time to get into the underlying theory (cf. memorizing basic arithmetic facts instead of teaching children to construct them a priori).

  • As an exercise, have each student write two or three survey questions independently and test them on the rest of the class. Depending on the number of students in the class, extreme biases should emerge under review by the class.

  • Strive for a neutral tone in the questions. If you are asking an opinion question, do not ask how strongly they agree/disagree with one opinion on the issue. Give them a range of opinions and let them pick the one they agree with most. Let the respondents provide the bias (otherwise, what are you looking for). Let's go with the nurse example proposed in the question. Let's look at two questions about shift length:

To what degree do you agree or disagree to the following statement: "My shifts are usually longer than I would prefer."

  1. Strongly agree.
  2. Somewhat agree.
  3. Neither agree nor disagree.
  4. Somewhat disagree.
  5. Strongly disagree.

Please select the response that most closely matches your opinion regarding the lengths of your shifts.

  1. Most of my shifts are too short. I could work longer shifts if it was needed.
  2. I sometimes have shifts that atr shorter than I would like, but most of my shifts are of an acceptable length.
  3. I like the lengths of my shifts. They are neither too long nor too short.
  4. I sometimes have shifts that are longer than I would like, but most of my shifts are of an acceptable length.
  5. Most of my shifts are too long. I would prefer to work shorter shifts if possible.
  6. Other (provide a spot for written comments).

Both questions are after the same info - how nurses feel about the length of their shifts. The first question is biased - it is asking nurses whether they agree with just one (negative) opinion about shift length. You have given them just one opinion to agree with about an issue instead of a range of opinions to agree with. The second one is not as biased. It goes after the same information, but in a different way - by providing a list of five opinions about shift length running the gamut from too short to too long and asking each respondent to pick which one they like most.

  • Provide an other option. Notice that my second question has this option. That way you can capture the few of the more unusual opinions without railroading the respondents into just the choices you provided.

  • Use simple language. Do not use flowery language or more complex wording than necessary. Notice that my second question did not read:

Please meditate on the durations of your shifts and select the response that most closely matches the harmonious resonances of your soul.

  • Avoid technical jargon unless that technical jargon is understood by all of your respondents (and then think twice about it) or if the survey is about technical aspects of the respondents' work. Jargon related to the nursing field would be appropriate, but be careful. Jargon used by a geriatric nurse might not be understandable to a nurse anesthetist. Since you student probably does not know much nursing jargon, such jargon should be avoided.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.