Professor Paul Smith first began working on censuses 25 years ago, at the Office of National Statistics (ONS). “Population censuses are an important part of a nation’s statistical system,” says Paul.
But there was a problem with the traditional way census data was being collected and analysed.
“A traditional census asks each household to complete a questionnaire. Not everyone responds. Certain population groups are under-represented, while others are over-represented.”
Southampton’s social statistics researchers, including Paul over the last decade, would change this. Their work has paved the way for 'virtual censuses' that are based largely on existing data and yet are more accurate.
Setting out to better estimate the size of populations
Southampton researchers worked on the idea of carrying out a follow-up survey once the census had been collected. This approach was inspired by ecologists using repeated samples to estimate populations, so-called 'dual system estimation'.
University social statistics researchers carried out more work to refine this ‘dual system’ approach for the UK censuses of 2011 and 2021. This involved improving the design of the coverage survey for the additional data collection that would allow adjustments to be made for undercounted populations.
This work paid off, Paul says. The ONS said that the statistical work improved the 2011 and 2021 censuses.
The problem of some groups not being fully represented in official statistics is faced around the world.
The main challenge in a traditional or administrative data census is to estimate population sizes using imperfect data.
How the team worked with new sources of population data
The team took the opportunity to improve censuses using administrative data around the world.
A population register gives the Dutch Government a more efficient way to carry out a census than a questionnaire. However, population groups can still be under-represented, as some people are unregistered. One of Paul's colleagues, Professor Peter van der Heijden, developed a way to correct this.
He analysed data for specific population groups such as on their usual residence, and police records. "Dual system estimation could then be used to estimate the size of these specific population groups," Paul says.
Another problem with the data from the register is that people do not always de-register if they move abroad, leading to over-coverage. Southampton researchers found a way to overcome this.
Professor Li-Chun Zhang proposed an innovative method to correct estimates based on administrative sources that are likely to over-represent some population groups, the 'trimmed dual system estimator'.
The team is also using dual system estimation accounting for different measurement methods to estimate the size of New Zealand's Maori and Pacific Islander populations. The work, with Statistics New Zealand, involves analysing linked census and administrative data.
The results of this research could have wide-reaching implications.
Inspiring nations to adopt 'virtual' censuses with less data collection
Statistics Netherlands used the methods Southampton researchers developed to estimate the country’s unregistered population. The population size affects the number of seats in the European Parliament and therefore the influence of the Netherlands in Europe.
New Zealand, the UK and Ireland are seeking to move away from periodic large-scale censuses. Their goal is to have a ‘virtual census’ based largely on administrative data sources. Southampton’s ‘trimmed dual system estimator’ approach has been investigated as part of this change in all 3 countries.
Paul comments: "These developments give us a much better knowledge of the size and characteristics of previously undercounted population groups. They also show how to make more frequent estimates while spreading costs more evenly."
