The recent event at the City of Oldsmar, Florida is only the latest in a series of crises representing how exposed some of the nation’s most critical infrastructure is to outside, internet-based compromise. An attentive operator at the treatment facility was able to avert disaster when he noticed his operator console being used for unauthorized changes. Initially the console activity appeared to be routine, but the bad actor clicked through control systems and increased the concentration of sodium hydroxide to potentially toxic levels. The unauthorized change was quickly reversed and no harm was done to the residents of Oldsmar. This event should be considered a ‘wake up call’ for other cities to check and update systems security, as reported here.
Questions remain as to the identity and motivation of the threat actors, as well as what technology or process controls existed to help prevent this type of event from occurring. The application Teamviewer is largely accepted as a legitimate remote access tool, but improper management and configuration can expose an organization to compromise. Through our ongoing work in automating and protecting critical infrastructure, the team at Red Trident believes more needs to be done to improve critical infrastructure organizations’ capabilities in preventing unauthorized and malicious activity in an industrial or Operational Technology (OT) environment.
A post is forthcoming regarding technical analysis and specific cybersecurity recommendations, but Red Trident does have best practices to share at this point in time:
Check parameters at every process point
Most industrial systems in a water treatment plant can define an acceptable range and validate operator inputs to ensure they are realistic to prevent dangerous values from being processed from a Control System.
Require authentication to change critical variables, preferably with multi-factor authentication
We suggest requiring multi-factor authentication for all remote access vectors, tools, or software, to prevent accidentally leaked credentials from exposing access to critical infrastructure.
Log configuration changes
Typically, industrial automation devices log changes, though modern SCADA platforms generally provide a centralized logging database that could be easily accessed for audit purposes.
Configure and use realtime directed alarms
Email alerts are very common in modern industrial automation equipment, with advanced platforms providing text and automated telephonic alerts.
Recently in Texas, the Brazoria County Water Authority discovered a brain eating amoeba in the water supply, and the city of San Angelo is currently under a “do not use” water order. These incidents may be the result of a lack of automation and visibility. Events like these put municipalities at risk for remote exposure, regulatory, and compliance failures, not to mention fines and the lost confidence from their communities. The Texas Commission on Environmental Quality issued the highest percentage of judgements to date against Water Supply and Irrigation Systems.
Red Trident is focused on protecting OT, ICS, SCADA, DCS and other embedded systems. We support local, state and federal agencies, as well as enterprises that require our expertise. In 2020, more OT-related vulnerabilities were reported than any prior year. If you would like an assessment of your security posture or a partner in implementing automation or cybersecurity improvements, please contact us at your earliest convenience.