By Mark Bristow, Branch Chief, Cyber Defense Coordination (CDC) at Cybersecurity and Infrastructure Security Agency (CISA), (CS)²AI Fellow
August, 2021
We hosted a (CS)²AI Online™ seminar on August 11, 2021 that focused on Stop Tomorrow's Crisis Today - Developing and Leading a Top ICS Incident Response Team.
Here is a bit about the event:
Incident response can be one of the most challenging times a process may face. The key to success is pre-coordination, preparation and training. (CS)²AI founding fellow Mark Bristow will take you through strategies in setting up and training your ICS incident response capability to make sure you are ready for this challenging day. With the right staffing model, incident response plan, pre-arranged internal and external partnerships, pre-built mitigation strategies and the right frame of mind, responding to an OT cyber incident can be effectively managed. Mark has worked on hundreds of incident response efforts impacting or threating process control environments in his long career with CISA’s Threat Hunting teams (formerly ICS-CERT).
Speaker:
Mark Bristow is Branch Chief, Cyber Defense Coordination (CDC) at Cybersecurity and Infrastructure Security Agency (CISA). He previously served as Director of the US Department of Homeland Security's (DHS) National Cybersecurity and Communications Integration Center (NCCIC), responsible for Incident Response efforts of the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) and the United States Computer Readiness Team (US-CERT).
As always, we encourage our audience to participate throughout the event by contributing feedback and questions for our speakers. We weren't able to answer all of your questions, so we have asked some of our speakers to answer a few of them. Below, are some answers to a few questions posed during the event.
Do you want to have access to more content like this? Register for our next (CS)²AI Online™ seminar or symposium here, or consider becoming a member of (CS)²AI to access our entire library of recorded sessions.
*******************
On August 11th 2021 (CS)²AI hosted me in a conversation about developing and leading incident response teams. In the webinar we covered how incident response can be one of the most challenging times an organization may may face. We covered some keys to success and strategies in setting up and training your ICS incident response capability to make sure you are ready for this challenging day. You can see the presentation in full in the member library (https://www.cs2ai.org/member-resource-library) but we had way more questions than we had time to answer. We decided to take some of the best questions we were not able to get to and post them in blog format to continue this important conversation.
QUESTION:
Can you have Mark comment on his guidance on incident disclosure - what MUST be disclosed, and what might need to be disclosed, and what might be able to be kept internal? There is an intersection of ethics and business reputation that may need to be dealt with and also specific compliance requirements depending on the client - i.e. Fed client vs commercial client.
ANSWER:
This is a really complicated topic and there are no “one size fits all” solutions. Each organization needs to internally decide what if anything they are required to disclose, what they plan to disclose and to whom. Business and reputational risk must be carefully weighted and agreed on by management. Not having a pre-built communications plan for common incident scenarios is one of the biggest mistakes I see organizations make and trying to build such a plan during an incident leads to rushed and not fully contemplated decisions.
In my view, more disclosures are needed and details of incidents need to be shared with the broader community. Too often are organizations hit by similar activity because others have not shared their experiences. This is where disclosure to external organizations like the Cybersecurity and Infrastructure Security Agency (CISA) or an Information Sharing and Analysis Center (ISAC) can be really helpful in communicating the relevant tactics and observables in a way that can keep the identity of the organization confidential.
Ultimately, many incidents go undisclosed. I recently completed a survey of OT/ICS practitioners for SANSwhere this topic was explored more in depth. Most respondents indicated that they had an incident in the last 12 months with many organizations having multiple incidents with operational impacts. Outside of this and (CS)2AI’s own annual surveys and reports I’m not aware of many of these incidents being publicly acknowledged or disclosed. Until we get over the stigma of reporting, and view public incidents with empathy and understanding instead of snide remarks and scorn, incidents will continue to be under-reported and future victims will continue to not benefit from the hard-earned lessons from those already impacted.
QUESTION:
Do you have suggestions for practice lab environments?
ANSWER:
Building out an ICS lab is an important step to training and maturing your incident response team. The best plan for building out a lab is to use equipment representative of your current, as-built process so you can test how your response tactics will perform on the equipment you have in the field. I like to recommend taking virtual copies of your windows-based systems in your ICS and stand them up in a virtualized environment to aid in your lab setup. While the system will not have all of the expected IO for the process, this will provide a mostly realistic environment for how an adversary may view your process control network and allow you to test detections without the investment of setting up an entire mirror process.
If your environment isn’t yet ready to build out a fully replicated lab, you can work with external organizations such as the ICS Village to leverage labs already built in the community to accelerate your lab buildout.
QUESTION:
How can we build relationships with other security professionals in order to share intel?
ANSWER:
This is definitely a key challenge faced by many and can be solved in a few key ways. The first is to always ask questions. This may be tough for many but you’ll find that even when approaching ICS cyber “legends” almost all are willing to support you and help so don’t hesitate to reach out. Anyone not willing to help out or answer a reasonable question isn’t really worth YOUR time anyway. Joining communities like (CS)²AI is also a great way to meet others of like mind in the industry. Conferences and events can really help build your network of trusted partners as well even while most are currently virtual. The best advice is to not be afraid to ask questions of others and be receptive to others when you are asked.
QUESTION:
What changes would you recommend specifically to Corporate Cyber Incident Response plan to ensure it has correct inclusions for OT?
ANSWER:
If you have a corporate IR plan, you are off to a great start! The next step is to ensure that OT considerations and operations are reflected in the plan. I like to run a table top exercise (TTX) with the IT IR team and a few people from operations. It’s best to use whatever is in the news recently so ransomware is a perfect scenario. Try running the IT plan on an incident impacting OT and identify the gaps that emerge (there should be a few). Then take a look at any safety plans and emergency plans that the OT team already have on the shelf, and evaluate how the new plan can incorporate and compliment the existing safety plans. Finally add some OT specific scenarios to your overall IR plan to ensure that when you have an incident you are not starting from a blank page.
QUESTION:
What recommendations do you have for performing red team exercises?
ANSWER:
Red teaming is an important step in the maturity of an ICS IR team, however, to be effective it requires a level of maturity from the IR team and a red team who is proficient in emulating adversary behavior in an OT network. Make sure you have clearly defined exercise goals and rules of engagement and you’ve accomplished some pre-requisite actions such as comprehensive, continuous asset identification program, a developed collection framework to support your ICS security monitoring, correlation of security monitoring with process data, integration with IT monitoring, and have a robust risk management program. Without these emulating a high order adversary the red teaming will not be particularly effective and resources would be better invested in ensuring fundamentals and security postures are prepared for such an exercise. Finally, for an effective exercise it’s best to conduct it in an environment that replicates your production control system network, some tips covered above (Question 2) when addressing building a lab environment.
QUESTION:
Do you have (or can you point me to) a standard/document that can be used to design the turtle mode you described? Also do you have an estimate on what percentage of the industry is already using this technique?
ANSWER:
I’m not aware of anything out there that describes “turtle mode” but I know other ICS security experts have a similar concept in their repertoire. Developing a defensible cyber position is really something that needs to be tailored to the specific process, threat model, and management decisions on acceptable impacts from a defensible position. In the end, “turtle mode” is about having a plan to temporarily reduce your cyber risk surface area while minimizing process impacts. Some key things to consider are:
1) What connectivity can you temporarily disable safely?
2) Can you disconnect all pathways to the internet from the ICS?
3) Are there functional elements that might be able to temporarily disabled (perhaps with an impact to efficiency)?
4) Can you temporarily limit accounts that access the ICS from on or off site?
5) Can you temporarily move to a secure out of band communications framework?
6) Can you temporarily limit processes changes to ensure your baselines are validated?
QUESTION:
Where do you see plans fall apart the most? Is it right at the start or is there a common point along the plan execution you see it fall apart the most? Bit of an abstract question, but any insight would be great.
ANSWER:
There are two areas where I see organizations stumble, communications and experience. Most IR engagements begin to fall apart when it becomes necessary to communicate with partners, customers or the public and few plans have a communications plan as a key part of the IR. Every organization believes that all aspects of an incident can be handled either “in-house” or with existing resources, this is almost never the case for an effective response. Finally, often motivated by the above, organizations attempt to leverage only internal resources without the needed expertise and experience to handle an incident. This often leads not only to the response not being completed correctly but in the destruction of evidence or un-necessary impact to the process/business. It’s ok to admit you are in over your head, organizations who do this well know what they can handle in-house, and who they can call when they can’t, and the self-awareness to know the difference.
QUESTION:
When looking for partners, what are the key skill sets that you feel are hardest to find?
ANSWER:
Find partners who are strong where you are weak. If you have really great host analysis team, make sure your partners compliment you with strong network analysis capabilities. Perhaps you have a really mature team that doesn’t have the time to build and maintain an ICS testing lab. The community has really grown over the last few years and there are some really great organizations out there who can fill a lot of gaps. The hardest part is making an honest assessment about your capabilities and where you need to reach out for help.
Comments