
Customer Success Manager
It’s no small task to process the huge amount of Census data into local insights. Continuing the story of our Census 2021 preparation, Glenn shares some behind the scenes tales of how .id approaches this complex job.
Keep up-to-date on all Census 2021 news.
The 2021 Census is expected to show some significant changes in the way we live in Australia, as well as things we’ve been talking about for a while, like a slowdown in population growth. The results should paint a unique picture of Australia at August 2021, living in a pandemic, with lockdowns across large parts of the nation. Before we can get that though, ABS has to process the data, and we have to do a lot of background work for our clients.
Getting Census data from the ABS
One important thing to remember is that .id doesn’t get the Census data earlier than anyone else. Although we work closely with the ABS, we are an external organisation and we can only get the data when it’s released to the public. The exact release date is not set as yet, but we’ve been told first release is in June 2022. It will likely be available form 11:30am on the day of release, when the ABS will lift the embargo on the raw dataset. We have our data order in (more on that in another blog) and should receive it on release day.
I’ve personally collected our order from the ABS office in Melbourne the last two Censuses. In 2016 the Dropbox we were supposed to use wasn’t working, so I ended up cycling over to the ABS office to pick up the entire Census results on CD-ROM!
Census 2021 data release timeline
The June release date represents about a 10-month processing time from Census day. This has been roughly the same for the past 4 Censuses, back to 2001. Prior to that the data could take 18 months to 2 years to be released. We haven’t seen any recent reductions in processing time, which you might have expected given that around 80% of households complete the Census online.
About 75% of Census data items are released in June. All your basic demographic info is in there: age/sex, birthplace, language, religion, education, the new health topic, incomes, citizenship etc. The second release is a bit later – expected around October this year. Second release includes topics such as employment, occupation and industry datasets, detailed field of qualification data, and population movement over 1 and 5 years. These topics can’t be automatically read by computer as many rely on written responses (eg describing main duties at work) so need some manual interpretation to code.
A third release will happen in early to mid-2023, covering topics requiring additional processing, such as distance to work, socio-economic indexes for areas (SEIFA).
Adding new Census data to .id information tools
We can’t get the data in on the day of release, but we’re working on our processes to have it in the community profile as soon as possible after release date.
Every five years the release of Census data sends our technical and data processing teams into something of a flurry. The task can be simply described: copy the data from spreadsheets provided by the ABS into our internal database tables so they can be processed into insights on our online tools. Actually doing it – and making sure it’s right – is far less simple.
One of our newer software engineers recently asked, “Won’t we just use the same process you used last time?” The thing is, we’ve done it quite differently every Census. 5 years is a long time, and there are always new processes and technologies available. Our all-consuming goal is to process the data as quickly as possible without causing data issues, so our clients can start acting on the insights provided.
In the lead up to the Christmas break, our software and database team spent time trialing out different methods for processing the data. Last Census we built our own data-consumption software. This Census we’ll be focusing on using Python code to speed up our data processing. We’re know writing the required code and running “fire drills” to make sure our technical processes are set up and ready for release day. Each recent Census we’ve significantly shrunk the amount of time between data being released and it appearing on our tools; we’re hoping to see a similar reduction this Census as well.
In profile.id (the community profile tool) there will be a topic-by-topic update, likely starting with the basic age structure and total population data. So there will be a few months in which profile.id will be a hybrid of the 2021 and 2016 Censuses (of course we retain past Census data too), until after the second release. We’ll clearly display which topics have been updated to the new data. Topics such as SEIFA will have to wait until the the final release in 2023.
Our social atlas (atlas.id) and housing monitor (housing.id) tools will be updated in parallel with the community profile over the same timeframe. The profile.id Communities of Interest optional modules are the most complex dataset and will follow from second release. Census data in the economic profile (economy.id) is primarily second release, so won’t be updated until after October. However, economy.id’s core data source is the annual modelled data on jobs, industry and economic value, independent of the Census. The next update to that dataset is due soon, well ahead of the first Census release.
In the meantime we’re working in the background to make sure this update is as smooth as possible for all our clients.
A few of the things we’re working on at the moment:
We’ll keep you updated on our processes over the next few months, ensuring our clients have an up-to-date demographic profile of their area as soon as the Census data are available.
Not sure if profile is available for your area? Check out our list here and let us know if you’d like to add it, if your LGA is not on the list!
Our specialists have deep expertise in demographics and spatial analysis, urban economics, housing research, social research and population forecasting
Head of Sales & Retention (Gov.)
Sales & Account Management (Industry)
Product Marketing Manager (Industry)
Demographic Consultant
Sales & Account Management (WA)
Customer Success Manager
Head of Product (Government)
Sales & Account Management (VIC/SA/TAS)
Views Delivery
Forecaster
Customer Success Manager
Customer Success Manager