Don’t miss the AC21 session, Improving your Utility’s Cybersecurity, Moving Past the Oldsmar Hack on Thursday, June 10. Register before April 23 to receive a $130 discount.
There are a few important lessons we can take away from the Oldsmar: we have a recent, tangible, and highly publicized example demonstrating that Water utility control systems are actively targeted for cyberattacks.
AWWA issued the Risk and Responsibility Guide for the Water Sector in 2019. There are several good examples of incidents that had severe consequences for the municipalities involved. However, those didn’t play out in real-time for many of us. Heck, even my Mom called me and asked what happened in Oldsmar and why they couldn’t just airgap the control system! (I’m paraphrasing for brevity)
There were a few initial reactions to reporting of the incident. One of the initial reactions was, ‘Why are they talking about this?’ There has been a long history of utilities keeping these types of incidents to themselves. Of course, we hear scuttlebutt of other utilities having similar experiences. To some extent, keeping these incidents quiet makes sense due to the legal and regulatory requirements utilities must adhere to, but it doesn’t help the bigger picture. One of the drivers for that silence is that we have a tendency to blame the cyberattack victim. Certainly, Oldsmar could have been doing many things better. Numerous articles, press releases, and notifications from DHS have articulated some of those things. However, most of us are in that boat. We need to stop blaming the victim, empathize, and through that – learn and get better.
Finally, and most importantly, the incident highlights the importance of our water system operators. They are the trusted humans in the process. Operators are critical to our collective mission of providing safe drinking water.
I have spent a great deal of time supporting utilities with all-hazards emergency response planning. One of the most important things that I have found is that when a utility empowers their staff to speak up and escalate concerns, they can improve their emergency preparedness, including relative to cyber incidents. So, my first suggestion is to speak up if you see something that is amiss. More than likely it isn’t anything malicious. Kudos to the Oldsmar operator for noticing and stopping the attack, and reporting it.
While that may seem like an obvious reaction, training on how control system cyberattacks can manifest, the consequences they can result in, and how to effectively respond and recovery are not widely practiced yet.
There are many initiatives ongoing to support cybersecurity education and more broadly to support building the workforce of the future. This is ongoing at the federal, state, and local levels with many stakeholders like utilities, professional organizations, and universities. All utilities should engage in this at some level to ensure they understand the resources available to them and are tapped into the talent pipeline.
One effort currently underway that I would like to highlight is Baywork’s Digital Worker Initiative. Baywork started as a consortium of water/wastewater utilities in the San Francisco Bay region working together to improve workforce reliability. Since its founding, it has grown geographically beyond the San Francisco Bay region.
Baywork has been working with numerous utilities and industry subject matter experts to redefine what skills the water and wastewater operators of the future should have. One of the primary skill sets is cybersecurity.
When their report is published, I suggest both frontline staff and leadership take these recommendations seriously. You can learn more about Baywork, and access the white paper The Digital Worker: Using Digital Tools to Deliver Water Services (publication is expected in late April 2021) at https://baywork.org/.
Build cybersecurity into any training, planning, design, and engineering process that your utility takes on. This may seem like a stretch. However, the Oldsmar hack highlights the need for better cybersecurity and more recent developments like Idaho National Laboratory’s (INL’s) Consequence-Drive Cyber-Informed Engineering (CCE; www.inl.gov/cce) provide us all with a process to engineer out cyber-physical risks associated with control system cyber-attacks.
While chronic underinvestment in cybersecurity is still a problem through most of our industry, I am generally hopeful. Four or five years ago, I would not have said that. I have changed my perspective for a few reasons. First, awareness of the importance of cybersecurity coupled with improved practices within the sector are pointing in a good direction.
AWWA has led the charge for many years on this. Everyone should check out their resources at www.awwa.org/cyber. In addition, there are several trainings that are available to utility staff through AWWA and funded by USDA and RCAP. Building on the work American Water Works Association (AWWA) has done within the sector, a number of other entities are now providing increasing support and resources for utilities. In addition to the organizations already mentioned, these include INL, EPA, and DHS CISA.
Given that there are over 50,000 community water systems in the United States we need the resources of these organizations and the competition of ideas to move the sector forward.
One of the great ideas that is moving forward with increasing momentum is INL’s CCE methodology. The methodology is focused on engineering out cyber-physical risk. We are seeing more and more utilities think along these lines in an effort to improve their engineering practices and operations capabilities. In the May 2021 issue of the Journal AWWA, my colleagues and I will have an article summarizing this thought process with some excellent case studies. Building on this article, we are currently in the process of writing the book and developing training in cooperation with INL and AWWA which we hope will change the way we, engineers and operators, engineer our water and wastewater systems. The book will include numerous case studies of water and wastewater utilities implementing CCE to improve their cyber-risk management. The anticipated release date will be in late 2021or early 2022.
Critical infrastructure operators are now paying more and more attention to cyber-physical resilience. CCE provides us all with a hopeful message that we as engineers can implement and improve the cyber-resilience of our systems.
We have some great speakers. David Ubert from CDM Smith, Gary Finco from INL, and Joe Oregón. David will discuss how a utility can develop a cybersecurity strategy. Gary and Joe will discuss some of the tools and trainings available to utility staff to improve their cybersecurity practices.
We are looking forward to all of the questions our attendees will have. I find that most utilities like to learn from other utilities’ experiences. While each presentation during our session is relatively brief, I suggest asking our presenters questions on the following themes:
I would like to touch on some trends that we are seeing that will change how we as a sector do business. First is the increasing adoption of SCADA in the cloud services. This trend has been building for a while and I bet a few people reading this are treating it as old news.
The cost-effectiveness and quality of service is quite good in many ways now. However, it is worth mentioning for two reasons. First, there is variability in the contract terms that SCADA in the cloud services provide. Be sure to read the terms of the contract and make sure that your utility has properly addressed any redundancy and back up requirements. Second, the cybersecurity practices of SCADA in the cloud service providers are highly variable. The awareness of third-party risk utilities accept when outsourcing SCADA is increasing, but like many sectors we do not yet manage it well.
Next, those cybersecurity practitioners that have been around a while, are familiar with the ‘assume breach’ philosophy. We and others in the field have started to take this a step further and not refer to it as ‘guaranteed breach.’
While some might find the idea that your networks are always vulnerable to be farfetched, there are good reasons to start with this. I’ll highlight two here.
First, I had the opportunity to conduct a cyber risk and resilience assessment for a client in the Washington D.C. National Capital Region. A DHS staff member attended a threat characterization workshop and he made the statement that every single network (some of these being owned by water/wastewater utilities) that DHS had assessed had some form of malware in it. That means that given all of the variability of the organizational and cybersecurity practices, no matter what the bad actors were in.
Second, say you assume “guaranteed breach,” what does that mean? Well a lot of things to be certain. But to my colleagues and I it means that we need to start with the engineering of the physical system and the people responsible for operating it. Luckily, INL has been working on this for years. Which leads me into the final trend.
As mentioned above, INL has developed the CCE methodology. I strongly urge all utilities to explore this and the possible outcomes. The most detailed account of the CCE methodology is included in Andy Bochman and Sarah Freeman’s book, Countering Cyber Sabotage; Introducing Consequence Driven, Cyber-Informed Engineering, which was published in January. INL has compiled some excellent and regularly updated resources on their website (www.inl.gov/cce). As noted above, West Yost is currently writing the book on applying CCE in the water sector. We are excited to be partnered with INL to bring CCE to the water sector.
Please contact Andrew Ohrt at [email protected] or 952.393.9905 with any questions.
Thank you to my colleagues, Dan Grovez, Joel Cox, and Michael Gruenbaum from Qest Yost, and Cheryl Davis from CKD Consulting for their input on this article.
My Profile
