Predicting water pipeline ruptures
Artesia Consulting, a data science consultancy for water utilities, approached Rocketmakers with a challenge in early 2019. They wanted to deploy their Machine Learning algorithm for predicting ruptures in pipes in a database which would provide real time alerts to their clients.
After our initial Discovery Workshop it became really clear that the ability to provide reliable early warning signals (based on pressure sensor information from many thousands of sensors around the UK) would be a game changer in time/money for water utility companies when dealing with major leaks in the water network
The algorithm developed by Artesia’s data scientists worked extremely well - that was not the issue. To be truly transformative, the algorithm needed to be running continuously, calculating real time alerts across a vast network of sensor sites. Artesia was keen for this to happen, but were aware that the running costs of always-on cloud processing could be huge. Since this technology had yet to be tested in the market, keeping costs low was a priority.
Rocketmakers software engineer Rob Butler helped design a cloud-based database and processors for Artesia with support from CTO Keith Walker. For Rob, this was the perfect challenge.
“I love working with lots of data,” he explained. “I like the feeling that things are working in the background. You can’t see it, but it’s always happening, with lots of data coming in and out.”
Rob was selected to head up the system design because of his previous experience working with Lambdas from Amazon Web Services (AWS). Unlike Amazon’s standard cloud-based server options, Lambdas are “event-driven,” which means application processes can run on-demand to respond to new events and new information. When there is no information, the virtual server is effectively turned off, which is a highly efficient way of keeping costs down without sacrificing any processing power. Lambda running costs can also be reduced further by limiting the frequency of server calls. All in all, AWS Lambdas were the perfect choice!
The Rocketmakers team also needed to construct an API to receive sensor data directly from water companies. This data was analysed by the Artesia algorithm, generating a twice daily email (morning and evening) with an overall view of a water network. This included lists of every sensor showing which are “okay”, which are in a “warning” state, and which have reached the “alarm” stage. Of course, emails are also generated whenever a sensor reaches the alarm stage.
Rocketmakers front-end developer and designer Patrick Benjamin then developed a dashboard which provided Artesia with a visual display of the entire system at any given moment. It also included controls to allow them to adjust server use (and therefore running costs) up or down.
The system was initially launched in April 2019 with data from just a few sensors. After this test proved the concept, the number of sensors was increased to hundreds, before ramping up to many thousands of sensors by April 2020.
“I was really excited when we transitioned from tens to hundreds of sensors and the system still worked,” Rob explained, “but I have to admit, the amount of data we were receiving was pushing the limits of what was possible with the database management system we were using.
“To minimise costs, we were using an open-source system called PostGreSQL. It is excellent software, but it didn’t quite have the capacity we needed in the current version. Keith and I knew that a new update (version 12) would be released soon and we were confident it would solve things.”
“I kept making tweaks to keep the current database alive, and then - finally - PostGreSQL 12 was available! I tested it, and it had more than enough capacity for what we needed. Finally we had some room to breathe!”
Building a cloud-based server database for Artesia Consulting was in many ways an ideal project for Rocketmakers. The challenge of providing a cost-effective way to run and to help tweak Artesia’s cutting edge ML algorithms was a genuine opportunity for innovation. The project also required the development of well-built, scalable architecture. And, by helping water providers better maintain their vast network of pipes for leaks and ruptures, Rocketmakers’ work would ultimately have a positive environmental impact.
The project was also a perfect opportunity for Rob: “I loved what this did and how it used technology to predict problems before they happen. And I loved having the chance to help use data to make the world a better place. The team at Artesia are extremely passionate about what they do, and were genuinely nice to work with.”
At Rocketmakers we leverage our experience to build innovative software for companies of all sizes. If you have a vision for a project and you need a technical partner to help you design, develop and deploy it, get in touch: [email protected]