We’re helping to deliver mission three of the 2022 to 2025 roadmap for digital and data by leading the build of the cross government Integrated Data Service (IDS). The service will make it easier for analysts across government - and eventually academics and others - to access a catalogue of quality indexed data sources which will inform policy decisions and improve public services.
Better data and analysis
Continual focus on meeting user needs with better data and analysis is critical to IDS - the data and tools we share with other government organisations will help them to investigate problems facing citizens, identify root causes, and deliver solutions through new legislation, services, and support. Just as critical is our protection of personal information – we go to great lengths to de-identify the data, making sure safe insights are generated.
One of the biggest data challenges for government are the boundaries that exist between departments which slow down the sharing of government data - complex legal frameworks and isolated working arrangements delay, or in some cases prohibit, important and innovative research.
The IDS provides a platform that allows us to share timely and relevant data securely across government, academia and the wider research community to support effective analyses and decision-making, while maintaining high standards of data security and governance.
The technology
IDS uses a modern multi-cloud platform to build our services. We learnt a lot from delivering the Census 2021 about cloud scaling, security, and implementation. We’ve also been through a tech refresh. Moving to the Google Cloud Platform earlier this year allowed us to use a set of products and tools that gave us both flexibility and assurance.
The data is stored in Google's BigQuery which gives us fine grained control of information for each piece of analysis. With this we can keep to the five safes model which are: safe people, safe projects, safe settings, safe outputs and safe data.
We have built analysis tools like Jupyter notebooks with supporting services from GitHub and Artifactory, and our infrastructure is built with role based access controls at the centre of the design, using Terraform and Kubernetes.
Innovative solutions
IDS data is accessed through a web browser. There are brilliant web-based tools to run data science and statistical methods in web browsers. To ensure that data is protected as much as possible but still allows analysts to use the IDS platform, we worked with Cloudflare to further develop their browser isolation tools. For example, stopping copy and paste activities between the browser and the operating system but still allowing copy and paste within the tab and workspace.
Collaborating across government
We’re not working alone on IDS. We're making sure colleagues across government are aware of our work and collaborate with them to help make sure the platform is meeting their needs.
We’re partnering with Central Digital and Data Office on the Government Data Exchange (GDX) so that IDS will be able to take advantage of the standards and data that are being developed and we’re encouraging others to contribute and share their data with IDS. Ultimately, we want to allow users to bring their data and analysis to the platform and combine and share it with others.
Meeting needs through different sharing models
We know that not everyone will give us data in the same way, so we developed 3 models that a provider can use.
- Donate model, where we ingest and link data on behalf of the data provider, or the commercial provider if we are purchasing data.
- Deposit model, where we provide a staging area inside the IDS platform and the data provider manages it - essentially, providing the cloud space for the provider to do the data architecture and linking.
- Enable model, where we use cloud virtualisation which works across all major cloud providers, to bring together data but keep within the boundaries of the donating organisations cloud environment.
Launch
We have learnt a lot so far are planning a public beta launch in March. Building at pace challenges us to keep the focus on usability, to provide enough support to new users, and to incorporate feedback to enhance the platform.
It is indeed exciting times. We’re using the technology to deliver an innovative platform that will provide the ability to answer questions that we don’t yet know we need to answer.
Leave a comment