BETTER TOGETHER: IT/OT COLLABORATION

8.28.23 > Guest Blog Feature >Thomas Tomalla, Jr.

History and Conflicting Priorities

I’ve worked in situations where Information Technology (IT) teams and Operational Technology (OT) teams are different and don’t get along. The goal of this article is to demystify OT systems for IT teams, and help find common ground – Technology and Risk Management.

IT is the use of computers to handle data, rather than handling it with paper or manual processes. This should be centered around the needs of the business and therefore, the needs of the users.

OT is the use of computers to interface with things in the real world by sensing inputs and controlling outputs. This is typically centered around the needs of the business as well, but in this case, the business usually produces something physical like products, energy, or food. Users interact with these systems, but the main purpose of OT systems is to control or monitor something physical.

Early on, these were often separate groups for a few reasons. IT focused on data processing, efficiency, and business processes. It evolved from mainframes to PCs to the cloud, and now distributed architecture.

OT focused on control, safety, and uptime. It evolved from electro-mechanical controls like relays and timers, to Programmable Logic Controllers (PLCs). As time progressed, more IT-like systems were introduced like data historians and computer-based user interfaces. Now we see cloud and Machine Learning as part of OT environments similar to IT environments.

Yet these two teams often remain separate, and sometimes at odds with each other. The three pillars of Information Security are Confidentiality, Integrity, and Availability. IT would often put Confidentiality at the top of their list, while OT would put Availability at the top of their list. This is where conflicts begin. For example, IT wants to patch systems as soon as possible to minimize vulnerabilities, lock sessions after 15 minutes, and keep passwords long – thereby maximizing Confidentiality. OT wants to hold off on patches, and make sure operators can always access the system – maximizing Availability. The reality is that we can’t prioritize one pillar over another. They are ALL important to both IT and OT. The best way to make balanced decisions is to use risk as a deciding factor.

We are starting to see a combination of IT and OT in Industry 4.0. It is the fourth industrial revolution where we use information to make decisions, integrate that information from the top to the bottom of the organization, and also between its suppliers. Many industries have been collecting data for decades, but the barrier of entry for data analysis tools and skillsets has been high. These barriers have been coming down quickly as Machine Learning becomes commonplace. To accomplish Industry 4.0 ideals, IT/OT collaboration and teamwork is necessary. An air gap as a security measure is likely no longer an option, aside from a few very specific industries. Things will be connected, and both IT and OT need to understand each other for an optimal outcome.

Overview of Operational Technology

OT can take many forms. The majority of OT systems can be classified into two main categories: Building Automation Systems (BAS) and Industrial Control Systems (ICS). Then the lines get blurry with Internet of Things (IoT), and Industrial Internet of Things (IIoT). There are also specialty systems such as laboratory, medical device, and life safety systems that are beyond the scope of this conversation.

Building Automation Systems help maintain occupant comfort and safety. Heating, Ventilating and Air Conditioning (HVAC) controls, lighting controls, card access, and security systems would fall into this category. These systems are often optimized for a specific use, which drives down cost and makes scalability easier. BAS systems require proprietary software and tools to program and install them, and in larger systems there is often a user interface that maintenance and security staff can use to manage the system. In most cases, the manufacturer that provides the hardware also provides the user interface. When these systems need to communicate with each other, there is usually a common protocol such as BACnet that allows systems to send and receive data in real time.

Industrial Control Systems are used to control physical processes or systems that you would see in industries like manufacturing, energy, and transportation. Reliability is paramount, so these systems have many more components to make up a working system. The Purdue Model, also referred to as the Purdue Enterprise Reference Architecture, provides a good outline that will introduce many of the terms that might be encountered with industrial controls. It’s worth noting that not all ICS systems adhere to the Purdue model, and the efficacy of it in modern architectures is under debate. The Purdue model is comprised of five distinct levels.  Level 5 references the Internet, and Level 4 represents the organization’s Business Systems. Levels 4 and 5 are typically the responsibility of the IT department. Level 3 would include things like the Human Machine Interface (HMI), databases, and historians (trend data or time series data). Level 2 may include the Supervisory Control and Data Acquisition (SCADA) Systems, Distributed Control Systems (DCS), or a local HMI. Level 1 is where field controllers are typically found such as Programmable Logic Controllers (PLC), Remote Terminal Units (RTUs), and Safety Controllers. Level 0 is where the real world connects – sensors, motor controllers, actuators, lights, etc. This reference is not exhaustive, nor accurate for every industry.

With the basics out of the way, let’s compare the BAS and ICS.

The use cases for an ICS controller are non-specific. Many are freely programmable, so the same model of the controller can serve many purposes. It may need to make decisions in milliseconds to maintain the process. The controllers may reside outdoors or in harsh environments, and may be deployed in highly available architectures. Additionally, some enable the system engineer to change the program on the fly, without interrupting the process. Generally, the lifetime on these systems is very long.

The use cases for a BAS controller are usually specific. An HVAC controller generally controls HVAC equipment, and a lighting controller generally controls lights. The programming is sometimes locked to those specific cases although it varies by the manufacturer. While many controllers are capable of making decisions in milliseconds, the processes often don’t require that level of control. The controllers are often in conditioned spaces such as mechanical rooms or data closets – but not always. Highly available architecture is not typically a requirement, therefore, programming changes often require a controller restart (although that trend seems to be changing). These systems are expected to last a long time but controller failure is less of an issue and systems tend to get replaced as a building is remodeled or as energy efficiency requirements increase.

Now let’s tackle two other categories that may be classified as OT – IoT and IIoT. The reality is that the lines are often blurred, and what a manufacturer calls their device is irrelevant. Every organization needs to understand what is on their network and what it is communicating with.

IoT is the Internet of Things. These are devices that were born in the Internet era – and with Internet connectivity often being a requirement for them to be functional. They also tend to be paired with an app or service. Security and privacy may be an afterthought or loosely defined. Smart cameras, smart watches, smart doorbells, smart toasters, smart speakers – the list goes on and on. We run into complications when these cross out of the consumer environment and into a workplace network. Let’s explore a smart security camera. It’s cheap, easy to set up, and has an app that can provide remote viewing. For many small businesses without an IT department this is a natural choice to help improve physical security. It’s even tempting for medium businesses when a full-blown Network Video Recorder (NVR) and network camera system could easily cost ten times more. But what happens when the employee that set up the camera leaves? What happens when the manufacturer decides not to support the device anymore or the service goes defunct? I’m not saying an IoT device serves no place in business or that everyone must purchase business or industrial grade equipment. The business should assess the risks of IoT devices and their associated services and make an active risk decision, rather than a passive decision to accept that risk.

IIoT is the Industrial Internet of Things. These are also devices born in the Internet era, but usually focus on a business segment rather than the consumer market. In this category you might find things like vending machines, remote sensors, and cellular connectivity, as well as the associated platforms to manage the services they provide. Security and privacy are usually thought of early on and may be contractual. These devices also tend to have longevity in mind, whereas the consumer IoT market does not. Many vendors will provide security architecture details – ask for them, seek to understand the architecture, and ask questions. The business is accepting the risk, so make it an informed risk decision.

The Common Ground

It’s important to find common ground to set the stage for improved IT/OT collaboration.

First, Confidentiality, Integrity, and Availability are important to both IT and OT. We both want the right person to have access to the right data at the right time in order to make the best decisions. Second, both systems are Internet Protocol (IP) based. We’re often dealing with the same networking stack. Third, both systems are usually dealing with the same software stack (Microsoft Windows) at some level. Fourth, while the embedded microcontrollers used in OT systems are not general purpose computers, they are still computers. In some cases we are seeing general purpose single board computers (SBCs) make their way into OT systems. Most Importantly, both IT and OT make risk decisions on behalf of the organization, whether this is formalized or not.

Benefits of this Collaboration

Let’s point out some benefits of Information Technology and Operational Technology having a good relationship.

• Many of the skill sets required to operate modern OT systems are similar to those needed for IT systems. When things are down or your preferred vendor isn’t available it’s good to have another team to help work through problems.
• If IT is going to be a hindrance OT may be resourceful and find their own way (Shadow IT). That isn’t always good from an overall risk perspective. Having IT and OT on the same page will result in better overall risk decisions.
• If you are serious about Digital Transformation and Industry 4.0 you need consensus at all levels of the organization. IT and OT being on the same page is one piece of that puzzle.
• With technology accelerating faster than ever we need all the technology players working together.
• When you are evaluating new OT vendors, you must talk about security early on. Vendors won’t always include the most secure end result unless you ask for it. I think this is where there is the most opportunity for IT to assist. Once IT knows OT they can help with these evaluations.
• In general, the more teams know each other, the better they can understand why things are done a certain way.

Where do we go from here?

IT and OT should have a good working relationship. IT and OT are more alike than they are different. Each absolutely has their own dialect and are specialized in their own ways. Here are some ideas to open the conversation. If you are an IT person who dabbles in electronics, home automation, programming, or just likes learning new things – talk to an OT person and find some common ground. If you are an IT person and have no idea where to start – grab your personal protective equipment (PPE) and head to the field with an OT person. Just get a tour, ask them what is new or where there are pain points. If you are an OT person who knows just enough about IP addressing and network information to get by – befriend the IT person and make that connection for when you need an extra set of eyes. If you are an OT person who really doesn’t understand why an IT policy is the way it is – ask.

The collaboration between IT and OT is essential for modern industries. While IT and OT may operate within distinct realms, a close partnership is essential to address the challenges of a rapidly evolving technological landscape. By recognizing common ground and understanding the unique strengths each brings to the table, IT and OT professionals, practitioners, engineers, and technicians can bridge the gaps that have historically separated them. This alone won’t prepare an organization for digital transformation, but it is a necessary step. The traditional boundaries of technology will continue to blur even beyond IT and OT, and a unified approach to technology becomes more crucial than ever.

Tom Thomalla, Jr., CISSP, GICSP, GCIH

Thomas is the Director of Information Systems for Ever-Green Energy. He has over 20 years of experience in IT, building automation, industrial automation, and information security. He holds a Bachelor of Applied Science in Information Technology Infrastructure from the University of Minnesota.

You can Connect with Thomas on LinkedIn at https://www.linkedin.com/in/tthomallajr/