A few weeks ago, NIST held another Cyber Security Framework Workshop, in Gaithersburg, Maryland. Post-Framework release, this workshop’s topics helped NIST better assess how the Framework is being used and what next steps are needed.
In one of the breakout sessions, the question of framework “Efficacy” was posed to participants in the context of helping to define near-term Science & Technology research needs. The question was important. Lack of efficacy testing and assessment of content approaches is why the term “Common Practices” is more apt than “Best Practices” when discussing the framework as it stands. Unfortunately, the breakout session did little to help, as it made a common mistake: The question asked generalized the problem at a very high level without actually shaping the discussion.
For instance, in order to determine effectiveness, goals and constraints must be defined. What is successful for one user may not have anything to do with what another user considers success. The Framework might help a large well-funded organization demonstrate partnership with the government but it is not so helpful at improving a small energy utility’s defensibility within its budget constraints.
In particular, by failing to ask more specific questions about what efficacy meant to users, the discussion helped perpetuate what is a deeply pervasive and problematic assumption made in the cyber security space: effective common practice implementation leads to effective incident management, which in turn leads to sustainable cyber risk reduction. These assumptions have not been shown to be true. In fact, with the present compromise rate (despite investment and organizational maturity), a case can be made that effective common practice implementation has little to do with sustained organizational risk reduction.
By failing to disambiguate these types of effectiveness from each other – or to identify constraints such as “required immediate and sustained investment needs” – it was difficult for real S&T research needs to be determined.
In the future, if any progress is to be made, any cyber security discussion where effectiveness is a goal (whether in NERC CIP, NIST Framework, or other contexts), a risk frame and threat models should be used to constrain the dialogue. At least five questions must be asked to start building these risk frames and threat models:
- What business consequences are being enabled or avoided and for whom?
- How would those consequences be made to occur and by whom?
- How much investment would make it worth controlling for the threat model that exists at the intersection of these questions?
- How many of these different risk frames are applicable to stakeholders?
- Which of these risk frames does the practice or regulatory body in question support, and how effectively?
Jack Whitsitt is a Senior Security Strategist for EnergySec. In addition to providing analyst expertise for the Tactical Analysis Center, Jack follows various federal activities closely. He is an expert on the NIST Cyber Security Framework, the ES-C2M2, and activities related to the National Infrastructure Protection Plan.