What is the best way to design a paper questionnaire to support scanning and converting to raw data?

Question

When surveying my students for various reasons (which have nothing to do with my institution which does not care about, nor fund, any of my "research"), I find that online surveys get a very low response rate. Sometimes, if I'm lucky, I can get up to 50% of students to respond (sometimes only 10%). However, if I hand questionnaires out during class time I can usually get 80-90% completion rates.

The problem comes that I end up with hundreds of paper surveys and keying them into a computer takes hours. There must be a better way.

I am actually looking for two pieces:

1. How can I design the paper survey to best support scanning and having a computer convert the scan into raw data which I can then explore
2. What kind of software can do the scanning and conversion (I realize this might be a question for softwarerecs.SE)

For point 2, I usually use multiple choice or Likert-type scales so full OCR is not required.

Are there standards for questionnaire design which support this process?

Answer 1

I can offer some help with scanning paper questionnaires as this is something I've done in a previous job. But, first, I would suggest it's quite involved so you may want to at least revisit improving your 'digital' response rate first.

Audience response devices

One option may be to allow the students to use 'clickers' to respond to questions. I would recommend having no more than 10 questions that can be answered on a multiple choice or Likert scale, which fits your requirements. Our department use something like this: http://en.wikipedia.org/wiki/Audience_response We've found that about 10 questions can be answered in about 10 minutes.

Re-consider your online/digital surveys

If clickers aren't suitable - for example because the students are not physically situated together - you could consider reviewing your digital survey, purely because it's significantly easier to obtain data from responses. To improve response rates, I tried to follow these rules of thumb:

• No matrices, ever.
• No more than three questions per page.
• No more than six pages.

Obviously this is going to vary depending on how much detail you need, but these reminded me to keep the questions light.

Scanning paper surveys

So, with all that in mind if you still need the paper copy route, here goes. To make it worthwhile you need:

• a scanner designed for automatic throughput. That is, a single sheet personal/small office scanner is not going to speed things up for you if you have to manually change ~200 sheets. Something like this Fujitsu ScanSnap might do the trick.
• Software that recognises marks. I used Eyes and Hands, which is deprecated and has been superceded by ReadSoft as far as I can tell. ReadSoft can recommend a compatible scanner.
• There are some requirements about how you set your paper questionnaire up, and this needs to be precise. Therefore I recommend something like QuarkXPress or Adobe InDesign.
• You need four recognition marks for each page, one for each corner. I often found a character from the header or text ok for the left hand side, but on the right hand side if text wasn't justified I needed to manually add some marks.
• You need to lay our tick or text boxes quite clearly. For example, it's quite common to see tables without any padding used for response boxes on paper surveys but you need a gap between them for the scanning software to recognise them unambiguously.

With these basic requirements you can set up a survey to be automatically scanned and entered in to a spreadsheet or database.

Clearly this is quite involved so you may be able to find a company who can do this for your commercially. Some market research or survey companies might be able to do the heavy lifting for you.

Answer 2

In terms of the software solution, as one option I would suggest playing with the AutoMultipleChoice software package if you have access to a Linux computer. The software can certainly output a CSV file with each response recorded for you to do your own data analysis (i.e. you can completely ignore the "grading" part of the software and just use it to capture data).

A few caveats:

• As the software is designed for multiple choice exams, each page is barcoded. You will have to convince the students yourself of their anonymity (if that's necessary); this can be achieved by randomly distributing the copies.
• The software supports autoshuffling of the multiple choice responses, as well as the question ordering. This can be used to great effect (if you want to avoid biases due to ordering of things), but sometimes you have to pay attention (questions of the type "on the scale of 1 to 5, rank blah" really shouldn't have the answers ordered "4, 1, 3, 2, 5".)
• The software was written by French people; the English documentation reads fine but with some slightly quirky word choices and grammar constructions.
• Make sure the students use a dark black pen to mark the questionnaire, and make sure they fill in the boxes. Experience has told me that blue ink or pencil, or tick marks or "X"-marks are often missed if you scan them using the default settings. You have to play with the contrast and darkness settings on your scanner a bit to find the right settings.
• The scanning can be painful if you have a large stack and not one of those automatic-feed photocopiers.

Answer 3

This question covers quite a wide area to be comprehensively answered. The first point that comes to mind is "Why are you doing a survey and keying the results in?"

Is it because your institution requires it of you? Do they require the survey to be performed as part of a teaching quality process? Is the desire for the survey your own and on your own initiative? (You just said "for various reasons") I say this because when someone starts doing a task that is difficult I ask "Do you need to do that?"

The surveys could be for you to learn about your teaching in some way in order to make a teaching improvement, perhaps. The surveys could be caused by you performing research on your class students in order to collect data, perhaps. The surveys could be done because you are teaching the students something about the role of surveys in business or marketing, perhaps. Maybe you are doing the survey as a substitute for other forms of summative or formative assessment, perhaps. (... and even more reasons I could speculate on).

I'll address these points in more detail. If you are surveying the class for your own personal teaching quality improvement, then I suggest that typing the data is not necessary. You can collect two kinds of data (qualitative and quantitative). The numerical and statistical parts are probably less important if the only consumer of the results is yourself. What is important are the detailed comments from the students. These involve reading all the responses irrespective of whether the survey was done on paper or online. I have used all three systems over the years (online survey, scanned paper survey, just paper survey) and for qualitative personal feedback it takes the same time to read them all.

If you are surveying the class for personal research data, then research ethics approval comes into play. In most research ethical approval and participant informed consent is required and this usually rules out mass questionnaires of classes of undergraduates! If mass surveys are required then some form of funding for appropriate data collection tools should have been part of the research plan.

If the surveys are part of teaching business and marketing processes then perhaps the investigation of how to do them better should be part of your teaching preparation, because helping students learn how to solve these problems in a business context is what you might need to include in the course, perhaps.

Having challenged the need for the survey or the data entry, lets assume your premise that it needs to be done.

I would next look at the relationship between yourself and the institution. Does the institution expect an online survey of the class and are they providing the web facilities for this that they expect you to use, or do they expect you to organise some form of online form yourself and provide them with the data? If the institution is expecting you to collect the data by a method of your choice and enter the data into their systems then they are leaving themselves open to external criticism of their quality mechanisms. This means that there is no audit mechanism that the data collected about any course from any group of students is valid and meaningful. One should challenge it through the various management and committee structures that exist in an institution for that purpose. The goal would be to either get the need for data collection to be properly supported or abandoned.

If you are trying to improve the response rate to an institutional provided online student questionnaire by substituting a paper copy and inputting the data yourself also sounds suspect. If the institutional data collection permits someone other than the accredited student to give feedback, then again this leaves the institutional mechanisms open to criticisms if any external audit were done on the data. It also means that you could equally criticise the invalidity or validity of other courses or faculty data as untrustworthy.

(Does my analysis begin to hint that perhaps your question is a little weak in construction, because this feels like answering an undergraduate course assignment from a business school; you can grade me later..)

OK. Lets continue to assume that doing a paper survey to collect quantitative and qualitative data is sensible and valid. What software and facilities are available? There are quite a few vendors that offer surveying capabilities in bulk supported by combined online and paper surveys. These have been adopted by quite a few educational institutions, my own included. A quick google search for such things shows many many vendors. There are online survey makers (some free), there are OCR questionnaire tool makers and those that do a combined job. There are those that do the data analysis for you also, and some that operate at an institutional level. Just so many to choose from. Perhaps you were just asking us to sort through the many offerings - oh! if it were that easy to answer.

Its hard to get cheap and good together. Those tools that provide what you need are often priced in a way that only make it economic for adoption at an institutional level, which is why action for a solution at an institutional level is often the best route to a solution.

I have used (and written) software to handle the OCR and data extraction from paper sources for large populations (~100,000). It is not easy, and also depends on your technical skills and processing and customising your data handling.

The first technical task to consider is the paper handling. How do you plan to OCR the physical material? If you have to use a single sheet scanner and turn the papers by hand then you have a problem. It is not a sensible task for an academic to perform, it is time consuming, tedious and error prone. If you have so few sheets that it is not tedious, time consuming and error prone, then it would still be faster to process the data by hand by reading and calculating yourself! You need a bulk sheet scanner that does accurate paper placement and has high speed sheet feed. These are not cheap. The salaries of clerical staff to do the scanning and paper handling are also not cheap. These are further reasons why solutions are best sourced at an institutional level and not a personal one.

OK, lets continue to assume that we can get the paper OCR processed in a reasonable manner. We need software post-processing. What software to use? It either comes with the package you buy/adopt or you have to customise. How capable are you at the customisation. I wrote textual matching algorithms using regular expression pattern matching combined with structural parsing of the resultant text generated by the OCR. Handling user errors was not easy , but you can structure the questions in a manner that permits regular expression matching to find the necessary glyphs.

For example:

 1. This is a Question?                             X
 2. This does not have an answer

 3. This question has a textual free form answer:
    =====================================================

    Answer goes where we can find it

    =====================================================

Can be pattern matched to detect the glyph mark after a question or the absence of a mark. The problem with real paper is that dirt and coffee stains can also look like response glyphs. Human post processing is required. I had to write a data error checker before I could write a useful data capture phase. If you are not into heavy professional coding, then one must adopt a professional and probably pricey solution or service provider. (Sorry, institutional again).

I also assume you know there is a whole science and scientific discipline to making surveys and questionnaires?

(For example: The Duke Inititiative on Survey Methodology, Odum Institute an UNC)

Perhaps you should be involving some professional input to your problem, rather than us dilettantes? (But then I'm back to an institutional level again)

Answer 4

Have you looked at Akindi? https://akindi.com/ You or the students can print out the answer sheets which are similar to those used by Scantron (you know, the SAT's and such).

I will also add to the notion made elsewhere that with so many kinds bringing computers and smartphones to class that using something like SurveyMonkey.com (or similar) will probably be even easier than going the scanning software route.