Northland Regional Council: Transforming Environmental Data Management for a Sustainable Future
Every year at the ALGIM conference, local councils compete for the ICT Project of the Year, an award that recognises outstanding project management success. The award always has exceptional entries and Psoda proudly maintains our tradition of sponsoring this prestigious award.
Every year we have the honour of introducing the three finalists, the second of which for 2023 is Northland Regional Council.
Background
How do we streamline environmental data processing to meet the growing future needs head-on? Through innovation and automation.
The Resource Management Act of 1991 entrusted regional councils with collecting, processing and storing environmental data, particularly for water resource management. As New Zealand Aotearoa grapples with climate change and the importance of preserving its limited water resources, the demand for accurate and timely data has surged.
However, the manual processing of real-time data from hundreds of sensors led to substantial delays, resulting in up to a 15-month backlog of unprocessed data. This issue is exacerbated as data collection technology advances, contributing to exponential data volume growth.
Recognising the need for a faster data processing solution, Northland Regional Council decided to look for a digital solution for the current processing challenges while preparing for a data-rich future.
The project
Prior to kick off, this Council surveyed other Councils to understand what solutions were currently available, as well as identifying any gaps.
Northland Regional Council engaged in proactive collaboration, partnering with various stakeholders from other regional councils, Orbica and Auckland University. They synchronised their efforts, identified automation opportunities and explored mutual benefits. The insights gained from this cooperative effort were invaluable in mapping the project’s direction.
As a result, a full set of requirements were developed. The Council needed the tool to be able to:
- Automate the quality assurance and control process. It needed to be able to quality code to the National Environmental Monitoring Standards, prepared on authority from the Regional Chief Executive Officers and the MfE.
- Capture the inherent expertise and knowledge of senior analysts.
- Develop an algorithm to find anomalies that require attention instead of just quality data.
- Detects and alerts users to anomalies while providing options to deal with different issues (such as spikes, flatlines or gaps).
- Provide an audit trail of all decisions taken while creating model parameters to detect and edit data.
- Provide an XML file output that can be imported to the timeseries database. This means that the tool is independent of any system, enabling seamless integration by another agency.
- Be maintained and hosted externally in the cloud.
- Use an open-source programming language to provide flexibility and scalability for changes to be made easily to scripts and to add new datasets, extending the tools’ capabilities.
Council selected Orbica to develop the tool. It is built upon the Orbica Geospatial Platform, utilising Geo AI services and functions, with a simple and intuitive front-end application built on PostgreSQL and AWS S3.
The solution leverages AI to enhance business efficiency, democratise data and foster innovation. Initially focusing on automating water level data processing (5.5 million data points collected annually across 80 sites), Northland Regional Council’s project evolved to be capable of automating the handling and cleaning of all continuous environmental data (e.g., water quality, groundwater, etc.) using standardised methodologies. The project adheres to the National Environmental Monitoring Standards (NEMS) for consistency across the sector.
Outcome
The DatPro project has achieved remarkable results:
- Automating the quality assurance process, significantly reducing data processing time.
- Ensuring consistency across individual and governmental agencies.
- Capturing invaluable knowledge and experience and integrating it into an efficient tool.
- Processing up to 100,000 data points in just two hours, a monumental time-saving accomplishment.
- Adhering to open-source programming for flexibility and scalability.
The DatPro project signifies a significant leap toward efficient environmental data processing. It has bridged the gap in data consistency and paved the way for future collaboration and integration within the regional council sector. Its potential to significantly reduce processing time and enhance data quality underscores its significance in the era of climate change and data-driven decision-making.
Northland Regional Council’s DatPro project exemplifies the power of innovation and collaboration, setting a high standard for environmental data management in New Zealand and beyond.
The awards
Northland Regional Council is one of three finalists in this year’s ALGIM ICT Project of the Year award sponsored by Psoda. The winner will be announced during the ALGIM 2023 conference in Wellington.
Other finalists
