Beginning Research | Action Research | Case Study | Interviews | Observation Techniques | Education Research in the Postmodern

Evaluation Research in Education | Narrative|  Presentations | Qualitative Research | Quantitative Methods | Questionnaires | Writing up Research

Quantitative Methods in Education Research

Dr Ulrike Hohmann

Originally prepared by Professor John Berry
© J Berry, Centre for Teaching Mathematics, University of Plymouth, 2005

(links updated August 2006)

• A.    INTRODUCTION
• B.     QUANTITATIVE AND QUALITATIVE RESEARCH
• C.     INGREDIENTS OF QUANTITATIVE RESEARCH
• D.    STATISTICS
• E.     STATISTICAL CONCEPTS & QUANTITATIVE PROCEDURES
• F.    TASKS
• G.    FURTHER READING

• H.    WEBSITES

A.    INTRODUCTION

This component is unable to do more than help you to begin thinking about quantitative methods in educational research. Its aim is to give you an insight into the issues should you choose quantitative methods as part of your research methodology.

We will briefly address the following questions:-

• What are quantitative methods?
  
• What are the ingredients of quantitative methods?
  
• How do you go about research design?
  

Education research has moved away largely from the numbers approach in recent years, and the emphasis has been on qualitative methods. However, the use of numbers can be a very useful tool, either as part of a larger project that employs many different methods or as a basis for a complete piece of work. With the use of sophisticated software packages such as SPSS it is relatively easy to deal with the computation side of things and it is possible to come up with numerous tables and charts almost instantly once your data is installed. However, it is very important that the underlying principles of statistical analysis are understood if sense is to be made of the results spewed out by such a package in terms of your research.

This component consists of two sections; we begin with an overview of quantitative methods and finish with a brief introduction to some of the basic statistical concepts to be looking for when you read research papers that use quantitative methods of research.

Back to CONTENTS list

B.     QUANTITATIVE AND QUALITATIVE RESEARCH

In simple terms we can think of two approaches to investigations in educational research: qualitative and quantitative. In the former we use words to describe the outcomes and in the latter we use numbers.

Quantitative research methods were originally developed in the natural sciences to study natural phenomena. However examples of quantitative methods now well accepted in the social sciences and education include:

• surveys;
• laboratory experiments;
• formal methods such as econometrics:
• numerical methods such as mathematical modelling.

Qualitative research methods were developed in the social sciences to enable researchers to study social and cultural phenomenon. Examples of qualitative methods include:

• action research aims to contribute both to the practical concerns of people in an immediate problematic situation and to the goals of social science by joint collaboration within a mutually acceptable ethical framework;
• case study research - a case study is an empirical enquiry that investigates a contemporary phenomenon within its real-life context;
• ethnography- the ethnographer immerses her/himself in the life of people s/he studies and seeks to place the phenomena studied in its social and cultural context.

Other components of this module cover various qualitative research methods.

When setting out on educational research you will be (have been) encouraged by your supervisor to read appropriate publications and this is a good way of identifying the methods of research that seem most used in your research area. A typical structure for a research paper is summarised in the table below:

Table 1 Structure of typical research paper

Back to CONTENTS list

C.     INGREDIENTS OF QUANTITATIVE RESEARCH

As part of your research you will be looking at certain characteristics (variables) and endeavouring to show something interesting about how they are distributed within a certain population. The nature of your research will determine the variables in which you are interested. A variable needs to be measured for the purpose of quantitative analysis.

We may collect data concerning many variables, perhaps through a questionnaire, or choose to measure just two or several variables by observation or testing. The variables we are interested in may be dependent or independent. There will be other features present in the problem that may be constant or confounding.

Using the data that you have collected then you can:

• Describe variables in terms of distribution: frequency, central tendency and measures and form of dispersion. Descriptive statistics include averages, frequencies, cumulative distributions, percentages, variance and standard deviations, associations and correlations. Variables can be displayed graphically by tables, bar or pie charts for instance.
  This may be all the statistics you need and you can make deductions from your descriptions. In fact univariate (one variable) analysis can only be descriptive.
  But descriptive statistics can be used to describe a significant relationship between two variables (bivariate data) or more variables (multivariate).
  
• Infer significant generalisable relationships between variables. The tests employed are designed to find out whether or not your data is due to chance or because something interesting is going on.

See the section on Variablesin The Research Methods Knowledge Base.

Often it is not possible to undertake a true experiment as part of your research and a common research approach in educational research is called quasi-experimental design represented in the following diagram:

In this figure O1 and O3 represent initial testing of the two groups; X represents some intervention or experimentation strategy with one of the groups and O2 and O4 represent final testing of the two groups. We would use the test results to investigate whether the experimental teaching approach has led to an improvement in the feature being tested.

Back to CONTENTS list

Perhaps the best way to begin to appreciate the kind of statistics that you might employ in your own research is to have a look at what others have done.

Read the paper by John Berry and Pasi Sahlberg: Investigating Pupils' Ideas of Learning. Learning and Instruction. 6 No 1, pp 19-36, 1996 and attempt to identify the statistics that are used.

• The section ‘Findings’ on pages 28 – 32 contains some of the important measures that we use in quantitative research methods. The mean and standard deviation tell us about the average and spread of the data. The bar graphs on page 31 allow a visual comparison of the means between the two groups of samples.

• The symbol ‘p’ represents the probability of a significant difference between the two groups. This is probably the most difficult concept to grasp because in some senses it is counter intuitive. The probability of an event happening lies between 0 and 1. A large probability (i.e. p close to 1) implies a high likelihood of the event happening. For example if you are told that there is a 95% chance of winning a game (p = 0.95) then put your money on winning! On the other hand if there is a 5% chance of you winning (p = 0.05) it’s probably best not to bet on yourself!

• So if p is small (close to 0) the event is less likely to happen than if p is large (close to 1). Let’s see what this means in the context of educational research. As an example look at page 31 Berry and Sahlberg. If we compare the means of the scores of the UK and Finland pupils for the statement "I learn better by doing work by myself than by watching the teacher", then there is a probability of less than 0.001 (i.e. 0.1%) that the difference in the means will occur by chance. In ordinary language the probability of it happening by chance is so small that we say it is a significant result. Because it is unlikely to occur, the reason that it does is significant.

• In our analysis we look for small probabilities usually less than 5% or p = 0.05. Then we say that the result is significant at the 5% level of significance.

• There is also evidence from Figure 3 that there is some difference between the UK and Finland pupils. For Q2 the UK rating is positive and the Finland rating is negative.

• Very often a comparison of the means as in Figure 3 is a clue. For example look at Q7 in Figure 3. For the UK pupils the rating is positive whereas for the Finland pupils it is negative. The ‘p – value’ for this feature is less than 0.001 (0.1%) and so we conclude that there is a significant difference between the two groups of pupils for the statement "I like most teachers in my school".

• One of the first steps in the design of a piece of quantitative research is setting up what is called a hypothesis. For example, we might propose the hypothesis that there is no difference between the UK pupils and Finnish pupils views of their teachers (item 7 in Berry and Sahlberg). This is called the null hypothesis. We then need to use the pupil responses to try to disprove this hypothesis.

• To be able to compare quantities we need to define a statistic whose distribution is known. In the paper by Berry and Sahlberg the t-statistic is used as a measure of the difference between the means of the two groups of pupils. Having calculated the value of the t-statistic for the feature under investigation we then look up in tables the probability of the feature occurring. (At this stage don’t worry about how it is calculated!) You can see on page 31 the t-statistic and its associated p-value.

At the heart of quantitative research methods is some very sound statistical theory. If you are planning to carry out a research investigation using quantitative research methods you do not need a thorough grounding in this theory but you will need an understanding of the statistical methods. We use statistical software packages to do the arithmetical calculations so the important skill is not doing the mathematics but is interpreting the results.

In what follows we have gathered together some of the essential statistical ideas needed for quantitative research. It is a summary with some examples to provide a flavour of the ideas.

Back to CONTENTS list

1.    Variables

• numerical measurements: person’s age or weight; size of a family

• non-numerical measurements: position on a scale indicating a level of agreement e.g. Likert rating scale

• continuous data: measurement that can, in principle, take any value within a certain range e.g. time, age and weight

• categorical data: (or discrete data): measurements that can take only known discrete values e.g. the number of rooms in a house, the number of children in a family
  • nominal data: numerical values are assigned to categories as codes e.g. in coding a questionnaire for computer analysis, the response ‘male’ might be coded as ‘1’; and ‘female’ as ‘2’. No mathematical analysis is usually possible and no ordering is implied.
  • ordinal data: numerical values are assigned in accordance with a qualitative scale e.g. in coding a questionnaire for computer analysis, the responses ‘very good’, ‘good’, ‘poor’ and ‘very poor’ are coded ‘4’, ‘3’, ‘2’ and ‘1’ respectively.

See also the section on Variables in The Research Methods Knowledge Base.

2.    Basic Measures

• mean: is a measure of the central location or average of a set of numbers, e.g. the mean of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.75

• standard deviation: is the square root of the variance!!

• variance:   is a measure of dispersion (or spread) of a set of data calculated in the following way:   

• median: is the centre or middle number of a data set, e.g. the median of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.5

• quartiles: divide a distribution of values into four equal parts. The three corresponding values of the variable are denoted by Q₁, Q₂ (equal to the median) and Q₃

• range: is a measure of dispersion equal to the difference between the largest and smallest value.

A distribution is symmetrical if the difference between the mean and the median is zero. An appropriate pictorial representation of the data, (histogram, stem and leaf diagram etc.) would produce a mirror image about the centre:

A distribution is positively skewed (or skewed to the right) if the mean - median is greater than zero. Such data when represented by a histogram would have a right tail that is longer than the left tail:

A distribution is negatively skewed (or skewed to the left) if the mean - median is less than zero. Such data when represented by a histogram would have a left tail that is longer than the right tail:

If data are skewed then the best measure of location is the median and the best measure of dispersion is the interquartile range.

If data are symmetrical then the best measure of location is the mean and the best measure of dispersion is the standard deviation or variance.

This is an important concept in statistics and is an important part of our story.

It is defined in the following way: if an experiment has n equally likely outcomes and q of them are the event E, then the probability of the event E, P(E), occurring is

P(E) =

Some simple examples:

the probability of getting a head from the toss of a fair coin is

the probability of getting a six from the throw of a fair die is

the probability of getting the ace of spades from cutting of a pack of cards is

The smaller the probability the more unlikely the event is to happen. This is an important concept in quantitative methods in education research as we shall see.

There is an important link between probability and the frequency distribution. Consider again the hybrid apple example above. We saw that the proportion of apples weighing less than 150 grams was 0.168. If we pick up one of the apples ‘at random’ then it could weigh less than 150 g and be rejected or it could weigh more than 150 g and be accepted by the supermarket.

There are two basic concepts to grasp before starting out on testing an hypothesis.

• Firstly, the tests are designed to disprove hypotheses. We never set out to prove anything; our aim is to show that an idea is untenable as it leads to an unsatisfactorily small probability.

• Secondly, the hypothesis that we are trying to disprove is always chosen to be the one in which there is no change. For example there is no difference between the two population means.

This is referred to as the null hypothesis and is labelled H₀. The conclusions of a hypothesis test lead either to acceptance of the null hypothesis or its rejection in favour of the alternative hypothesis H₁.

Hypothesis testing: a hypothesis test or significance test is a rule that decides on the acceptance or rejection of the null hypothesis based on the results of a random sample of the population under consideration.

For more on this see the section on Hypotheses in The Research Methods Knowledge Base.

7.    Statistical tests

t tests

A two-tailed test is one where H₁ would be of the form m ¹ 3.

Click here for more information on t-tests.

is used to test the null hypothesis H₀: the population mean equals m. If the test statistic lies in the critical region whose critical values are found from the distribution of T_{n, a}, H₀ is rejected in favour of the alternative hypothesis H₁. n are the degrees of freedom and for a single sample test n = n-1, and a is the significance level of the test.

The test statistic T is distributed T_n, where n =(m-1)+(n-1) for two unpaired samples. If the test statistic lies in the critical region whose critical values are found from the distribution of T_{n, a}, H₀ is rejected in favour of the alternative hypothesis H₁.

Beginning Research | Action Research | Case Study | Interviews | Observation Techniques | Education Research in the Postmodern

Evaluation Research in Education | Narrative|  Presentations | Qualitative Research | Quantitative Methods | Questionnaires | Writing up Research