Search RESINED


powered by FreeFind

Home

 

Beginning Research | Action Research | Case Study | Interviews | Observation Techniques | Education Research in the Postmodern

Evaluation Research in Education | NarrativePresentations | Qualitative Research | Quantitative Methods | Questionnaires | Writing up Research

 

Questionnaires in Education Research


Prepared by Professor Andrew Hannan

Now led by Dr. Julie Anderson

© A Hannan, Faculty of Education, University of Plymouth, 2007


CONTENTS

 


A.     INTRODUCTION

1) You will no doubt have had numerous experiences of having to fill in a questionnaire, everything from the Census itself to forms to get your motor insurance or library card.

2) Such questionnaires wouldn’t be so popular if they weren’t in some measure successful in getting the information required in a form it can be usefully analysed.

3) I want to begin by asking you the following questions.

Jot down your answers before reading on.  Use the lists you have created to check against the points made below.

4) Notice how I used a form of open-ended question to try to get you state your views, having decided that this was better than listing potential advantages and disadvantages and asking you to tick to indicate the ones with which you agreed. The issue of how best to pose questions is one to which we shall return.

5) Straightforward written questions requiring an answer by ticking the appropriate box are very efficient ways of collecting facts.

6) Questionnaires are also employed as devices to gather information about people’s opinions, often asking respondents to indicate how strongly they agree or disagree with a statement given, but sometimes merely posing a question and giving respondents space in which to formulate their own replies.

7) One of the obvious advantages of questionnaires is that they provide data amenable to quantification, either through the simple counting of boxes or through the content analysis of written responses.

8) Problems arise, however, when the facts themselves are difficult to establish, when the question posed contains ambiguity or bias or when the range of available questions or answers does not allow the respondent the opportunity to state what he or she wishes. The agenda is normally set by the researcher with the respondent being somewhat constrained so as to follow planned pathways; there is little room for the unexpected. The picture presented is often static, with facts and views given as more concrete and fixed than they may be in the dynamic flow of personal formation and social interaction.

9) Let us have a look at how questionnaires are put together.

Back to CONTENTS list


B.    QUESTIONNAIRE DESIGN - How to do it

1) Ask yourself, why should I use a questionnaire? It is worth being self reflective when beginning to construct your own questionnaire, by writing down your reasons for choosing such a research instrument rather than another (say interviews or observation), for inventing your own rather than using one already available in the literature, and for posing the sorts of questions you want to use. Such notes may be useful when you come to write the ‘methods’ chapter/section of your research report.

2) The fundamental question that must then be asked is, what are you trying to find out? Every questionnaire must have a purpose, ie it must draw from some underlying hypotheses about what are the important facts or opinions and even make some predictions about which facts may be relevant in explaining the opinions expressed.

3) Write your own rationale, in terms of statements like, ‘I need to know whether or not senior members of staff are more likely to support the moves to introduce appraisal and what reasons they have for the positions they express. I need to find out why junior staff seem opposed, ie are they misinformed about the nature of the reforms or are they protecting weaker colleagues from what is seen as scapegoating in an under-funded profession of whom too much is demanded with too little support?’

4) This can be developed so as to produce a justification for every question used, eg ‘I asked this so as to probe the extent to which those of various positions in the hierarchy valued staff consensus and the feeling of shared purpose, with the intention of seeing whether those who were strongly committed to such views were also more or less opposed to staff appraisal’. If you can’t come up with a good rationale, drop the question.

5) Many questions can be closed-ended, ie the respondent has simply to tick the appropriate box, although these are most suitable for the gathering of unproblematic facts. Such a device can be employed to ascertain the viewpoints of respondents but there are more problems involved in both posing the questions and offering a range of possible answers.

6) The best descriptions I have come across of the issues involved are those given in Munn & Drever (1999), which I strongly recommend you should read.

7) You need to decide how to pose your questions and the form of coding you might use, to ensure that your survey produces data that you can analyse. You need to avoid ambiguity and bias, and to refrain from leading your respondents.  For further guidance consult  the types of questions; decisions about question content; decisions about question wording; decisions about response format; and, question placement and sequence pages in the The Research Methods Knowledge Base

8) I would personally like to recommend the use of open-ended questions that allow respondents to state their opinions in ways not pre-selected by the researcher. These give the possibility of discovering things that were unsuspected and enable some respondents to challenge the sort of assumptions that may have been made. The disadvantage of such questions is that computation is very difficult and can only follow a process of categorisation, which in any case has to be undertaken by the researcher.

9) However, a combination of closed-ended and open-ended questions has its advantages in that it preserves the possibility of easy computation whilst providing respondents with the space to develop their own ideas, eg

‘To what extent are you satisfied with the current proposals for staff appraisal?

Very satisfied    Satisfied     Neutral    Not satisfied     Very dissatisfied

1                  2                3                  4                      5

Please circle as appropriate and explain your response in the space below: ....’

The Likert scale used here (1-5) also serves as a self-coding for any explanation given.

10) Let me identify some other important points: -

  1. Make sure you show a full draft to someone else, preferably a tutor, before trying it out.
  2. Make use of a pilot trial run first if at all possible - it is amazing just how flawed the product of hours of solitary effort can be once it is put into practice. Use a parallel smaller sample or a sub-sample of the target population concerned. If all else fails get 3 or 4 friends to pretend they fit the respondent categories. Ask those in your pilot sample to feed back to you their views about the questionnaire itself, eg how long it took to complete, which questions they found ambiguous or leading or biased, etc. Don’t forget to analyse the results to see if you can make sense of the data that you have collected. It’s a lot better to find out you’ve made a mistake at this stage than to do so when it’s too late!
  3. Bear in mind how you propose to use the data so collected - it is better to build in a coding device for closed-ended responses from the very beginning and to check in advance that you have sufficient information to undertake a statistical analysis (see Munn and Drever [1999] on ‘analysing the results’, although don’t necessarily take their advice on avoiding computers).
  4. Attempt to obtain as big a response as possible, the whole population would be best (!) but otherwise you will need to seek a random or structured sample (see Sampling in The Research Methods Knowledge Base), not forgetting that you will need a minimum of 30 respondents to do statistical analysis of anything more than a very low level kind. If, for example, you decided that you wanted to know what primary school teachers thought about the National Numeracy Strategy, you would be best to ask all of them. As this is probably beyond your scope, for reasons of the costs and time involved, you may feel it best to confine yourself to a sample, eg all primary school teachers in Devon. Even this is probably too many to cope with, so you may decide on all those in, say, East Devon, or all those in Exmouth, or, even, all those in one particular school. Alternatively, you might give every primary school in the country (or Devon) a number and choose a 10% random sample, using a table of random numbers, and survey all the teachers in those schools. You might try and put together a stratified sample of schools typical of the different varieties that you know to exist to include in your survey. You might deliberately seek unusual schools where you know teachers have taken a particular stance to the teaching of mathematics in order to find out more about the extremes or about ‘vanguard’ cases where ideas were being tried in ways which others were likely to follow. Whatever choice of sample you make you need to justify it, ie to make a case to the reader who examines your results that he/she has good grounds for taking your findings seriously in terms of their representativeness.
  5. Ensure that you pose your questions in a manner that makes them easy to answer and that your whole questionnaire is short enough to mean that most people will complete it. Beware of using survey methods that make it likely that a significant proportion of your target population won’t have the chance to respond, eg using email when not all have access to it or forms of written presentation beyond the literacy level of some of your respondents.
  6. You need to be careful about obtaining the highest possible response rate otherwise the answers you get may not be representative of the overall population or of the sample you chose. Presentation and politeness are important here, remember the respondents are doing you a favour and be sure to thank them! Postal surveys normally obtain very low response rates, even when pre-paid envelopes are supplied. It helps to make it as simple as possible for respondents to return their forms. You need to convince members of your target group that it is worth their while to complete and return your form - tell them how much it matters, how it will have real consequences, how they can find out the results. Don’t forget to reassure them about confidentiality. If you are asking them to identify themselves (not normally a good idea), you must explain to them how you will use and present the information you gather, in accordance with your ‘ethics protocol’ (which you may wish to incorporate in an abbreviated form in the introduction to the questionnaire itself). If you get a low response rate you need to ask yourself if those who replied are likely to differ substantially from those who didn’t, ie are the non-respondents likely to be from particular groups or to take different views of the issues involved? A response rate of over 60% is fairly respectable. One of less than 50% is a concern and you need some strong arguments to justify taking the results seriously.

Back to CONTENTS list


C.   QUESTIONNAIRE DESIGN - examples

1) The first example given is the text of a questionnaire I distributed to all Devon primary school headteachers in the summer of 1993.  Please click here and have a look at it with reference to the design recommendations made above and make a list of its strengths and weaknesses- you can use this as a self-corrective guide when designing your own questionnaire!

2) The second example is from the Families and Children Study conducted by the National Centre for Social Research.  Please click here to have a look at the self-completion questionnaire issued to a sample of school children aged 11-15 in 2006.  Note how the designers have done their best to produce a child-friendly form, complete with smiley faces!

Back to CONTENTS list


D.    QUESTIONNAIRE ANALYSIS - How to do it

In general

1) When you get the forms back you will need to number each completed form (the ID for each respondent).

2) You should then sort out the easy stuff first, ie the answers to closed-ended questions which you can simply count in order to give you frequencies. Question 1 in section A of the headteachers’ questionnaire given earlier is one of these, as are each of the Likert scale options which set out the various government proposals for questions 1-7 in section B. The responses to the closed-ended items should be entered onto a manual or computer generated spreadsheet, with the respondents’ numbers (ID) given for each row and the answers coded by number or letter for each column. The spreadsheet below has the first ten entries from the actual survey:

ID

A1

A2

A3

B1

B2

B3

B4

B5

B6

B7

01

6

220

9.05

5

5

3

2

3

2

3

02

5

105

4.7

5

3

2

1

4

2

2

03

5

186

-

5

4

4

4

5

2

2

04

6

350

12.8

5

-

4

2

4

2

2

05

5

47

2.5

5

4

4

2

5

2

2

06

5

35

2.2

5

5

5

5

5

4

4

07

5

420

16

5

2

3

2

5

2

2

08

5

65

3.3

5

3

3

1

3

1

1

09

5

40

2.2

5

5

5

2

5

3

3

10

5

60

2.7

5

2

1

1

5

2

-

Note how a hyphen has been entered where there is a missing or unclassifiable response. Items A2 (number of pupils on roll) and A3 (number of FTE teachers) would require further categorisation before analysis, eg enter ‘1’ for schools with less than 50 pupils, ‘2’ for those 50-99, ‘3’ 100-199, ‘4’ 200-299, ‘5’ 300-399, ‘6’ 400-499.

If you have designed your questionnaire correctly the respondents will have done the coding for you by ringing the appropriate number which you can then enter into the spreadsheet. It’s not difficult to find out how many respondents gave each sort of answer. These numbers can easily be turned into percentages (showing what percentage of the respondents gave each sort of response). These figures are already very useful as analytical tools.

3) Very often questionnaires have a ‘background characteristics’ section like the first part of the headteachers’ survey. These frequently give information on the age or sex of respondents. The next stage in statistical analysis is to take whichever of these ‘variables’ interests you and to look at how the patterns of response vary, eg comparing males and females in terms of the answers they gave to all the other questions, or headteachers of infant and junior schools in the same manner. This is known as crosstabulation and is easiest done by computer programme (especially SPSS). You can then ask such analytical questions of the data as, ‘Are there gender differences at work?’ or, in terms of the example given, ‘Are infant school heads more in favour of moving teacher training into schools and out of universities than junior school heads’?

4) Of course, far more sophisticated statistical analysis can also take place, with measures of significance and correlation, etc. But these are not covered here. If you are interested please consult the ‘Quantitative Methods in Education Research’ component.  You may also wish to consult the Analysis page from the The Research Methods Knowledge Base.

5) The headteachers’ questionnaire was part of a survey that also involved slightly different questionnaires issued to parents of pupils at seven Devon primary schools and teacher training students and tutors at what was then Rolle Faculty of Education. The results were first analysed in the sort of fashion described above (using Excel spreadsheets and SPSS), with comparisons being made between the different categories of respondent. See my paper The Initial Training of Primary School Teachers: Response to the DFE, an Interim Report. Note in particular the elements of the research report, viz the description of the research instrument, samples and response rates, and the manner in which the question asked is presented alongside the information collected in response.

6) The comments provided by the headteachers, students, tutors and parents in explanation of their Likert scale responses and in answer to the open-ended question posed in section C were entered on computer into a qualitative data software package called HyperQual. The Likert responses were then used to code that data, enabling me to gather together information to answer analytical questions such as, ‘What were the comments made by headteachers who strongly opposed the ‘Mums’ Army’ idea?’ and, ‘What were their general views of government policies?’.

7) To see how an analysis of this more qualitative data was undertaken see my article ‘The Case for School-led Primary Teacher Training’, Journal of Education for Teaching, 21, 1, 25 -35, 1995.

8) In this example, HyperQual was used to sort answers into analytical categories, but content analysis of the different points made was then carried out manually. In practice this meant that every statement was analysed for content and placed under an appropriate heading, along with any others which were sufficiently similar. These were then grouped under more general umbrella headings to produce the description of points made with reference to their nature, range and frequency.

 

Back to CONTENTS list


Using IT

1) It helps to make sure when you design your questionnaire that it is amenable to computer analysis.

2) There are software packages that facilitate content analysis of responses to open-ended questions, the most popular one being Survey Monkey which provides a limited degree of functonality for free.  Weft QDA is a free qualitative analysis software application - 'an easy-to-use tool to assist in the analysis of textual data', which can be downloaded from http://www.pressure.to/qda/NVivo is a very common analysis tool for qualitative data and is provided over the University of Plymouth server to all networked PCs or as a use at home package

 

3) For quantifiable data, MS Works and Excel spreadsheets will all produce a wide range of computations and forms of presentation such as graphs and charts as well as carrying out some statistical operations. 

 

4) However, to undertake sophisticated statistical analysis to produce tables of results as well as figures and charts there is nothing to compare with SPSS, which is provided over the University of Plymouth server to all networked PCs or as a use at home software package.

5) It is a fairly straightforward matter to enter the data on computer yourself. There are considerable advantages in doing so rather than following the advice of Munn and Drever (1999) and doing it all manually, in that once the data are entered the computer packages are wonderfully quick and flexible tools of analysis.

6) SPSS is capable of performing any number of ‘t’ tests, Chi- squares, etc but it is not beneath undertaking relatively simple tasks such as crosstabulation (producing tables separating out various categories of respondent for each set of answers) and producing row, column and cumulative percentages.

7) You will need to use the in-built 'tutorials' and the 'help' menu to teach yourself how to use SPSS.


Not Using Computers

1) Get a copy of Munn and Drever (1999). This book takes you through a pencil-and paper method ideal for computer-phobes.


E.    TASKS

(NB: only for those University of Plymouth students undertaking the ‘Research in Education’ module as part of the preparation for the submission of a MA dissertation proposal)

Tasks, once completed, should be sent to resined@plymouth.ac.uk, making clear:

It will then be passed on to the component leader (and copied to your supervisor). The component leader will get back to you with comments and advice which we hope will be educative and which will help you in preparing your dissertation proposal once you are ready. (Remember that these tasks are formative and that it is the proposal which forms the summative assessment for the MERS501 (resined) module.) This email address is checked daily so please use it for all correspondence about RESINED other than that directed to particular individuals for specific reasons.

 

TASK B (DATA COLLECTION)

NB For a QUESTIONNAIRE survey it is often preferable to incorporate the ethical provisions in the introduction to the questionnaire form itself, ie telling informants what the project is about (informed consent), giving them the choice not to respond to individual items or the form as a whole (right to withdraw), setting out how feedback may be obtained (debriefing), describing the provisions for confidentiality (particularly if the form is not anonymous), etc.  This section of the questionnaire is in effect the 'ethics protocol' and can be submitted for approval.

 

TASK C (DATA ANALYSIS)

 

Back to CONTENTS list

 


F.    FURTHER READING

 

CD-ROM

Barrett, Elizabeth; Lally, Vic; Purcell, S & Thresh, Robert (1999) Signposts for Educational Research CD-ROM: A Multimedia Resource for the Beginning Researcher.  Sage Publications, London.  (This CD-ROM includes a section called 'Travelogues' that gives advice on three commonly used methods of data collection - Interviews, Observation and Questionnaire surveys.)

 

WEBSITES

Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: http://www.socialresearchmethods.net/kb/ (version current as of July 01, 2008).   This is the excellent site referred [and linked] to several times in the sections presented above.

The Survey Question Bank has many examples of lists of questions used in structured interviews as well as what it calls 'self-completion questionnaires'.  It's a good place to look for examples, which you can normally use without worrying about copyright (although you will need to acknowledge the sources in the normal fashion).

 

BOOKS

Munn, Pamela & Drever, Eric (1999) Using Questionnaires in Small-Scale Research: A Teachers’ Guide Scottish Council for Research in Education, Edinburgh.       

Robson, C. (2002) Real world research : A resource for social scientists and practitioner-researchers,  Oxford, Blackwell [has an excellent chapter on surveys/questionnaires] 

For more on statistical analysis, from basic averages, standard deviation etc. to co-variant analysis, try...

Wright, D. & London, K. (2009) First and Second Steps in Statistics, (2nd edn.) London, SAGE.

                                                                          

 

 

Back to CONTENTS list

 

 

Beginning Research | Action Research | Case Study | Interviews | Observation Techniques | Education Research in the Postmodern

Evaluation Research in Education | NarrativePresentations | Qualitative Research | Quantitative Methods | Questionnaires | Writing up Research

 

Home

 

© A Hannan, Faculty of Education, University of Plymouth, 2007


 

http://u1.extreme-dm.com/i.gif