SIRAcon 2021 Talk | Baby Steps: Easing your company into a quantitative cyber risk program

I’m pleased to announce that my talk, “Baby Steps: Easing your company into a quantitative cyber risk program”, has been accepted to SIRAcon ‘21. SIRAcon is the annual conference for the Society of Information Risk Analysts (SIRA), a non-profit organization dedicated to advancing quantitative risk analysis. The talk is scheduled for Wednesday, August 4th, 2021 at 11:15 am Eastern US time.

In the talk, I’ll be sharing tips, techniques, successes, failures, and war stories from my career in designing, implementing, and running quantitative risk programs.

Here’s the talk abstract:

Baby Steps: Easing your company into a quantitative cyber risk program

Risk managers tasked with integrating quantitative methods into their risk programs - or even those just curious about it - may be wondering, Where do I start? Where do I get the mountain of data I need? What if my key stakeholders want to see risk communicated in colors?

Attendees will learn about common myths and misconceptions, learn how to get a program started, and receive tips on integrating analysis rigor into risk culture. When it comes to quant risk, ripping the Band-Aid off is a recipe for failure. Focusing on small wins in the beginning, building support from within, and a positive bedside manner is the key to long-term success.

Update: here’s the link to the recording

Additional Resources from the Talk

It was a challenge to cram all the information I wanted to cover into 30 minutes. Sit me down with a few beers in a bar and I could talk about risk all night. This blog post is a companion piece to the talk, linking to the resources I covered and providing extra details. This post matches the flow of the talk so you can follow along.

The One Takeaway

The one takeaway from the talk is: Just be better than you were yesterday.

If you are considering or are in the process of implementing quantitative risk modeling into your risk program and you need to pause or stop for any reason, like lack of internal support, competing priorities, your executive sponsor leaves, that’s ok. There are no quant risk police to come yell at you for using a heat map.

We - the royal we - need to move off the risk matrix. The risk matrix has been studied extensively by those who study those sorts of things: data and decision scientists, engineers, statisticians, and many more. It’s not a credible and defensible decision-making tool. Having said that, the use of the risk matrix is an institutional problem. Fixing the deep issues of perverse incentives and “finger in the wind” methodologies canonized in the information security field doesn’t fall on your shoulders. Just do the best you can with what you have. Add rigor to your program where you can and never stop learning.

The Four Steps to a Quant Risk Program

I have four general steps or phases to help build a quant risk program:

1. Pre-quant: What to expect when you’re expecting a quant risk program - you are considering quantitative risk and this is how you prepare for it.
2. Infancy: You’ve picked a model and methodology and you’re ready for your first few steps.
3. Adolescence: You have several quantitative risk assessments done and you’re fully ready to rage against the qualitative machine. Not so fast – don’t forget to bring everyone along!
4. Grown-up: Your program is mature and you’re making tweaks, making it better, and adding rigor.

(I’ve never made it past grown-up.)

You can follow these phases in your own program, of course modifying as you see fit until your program is all quant-based. Or, use as much of this or as little as you want, maturing your program as appropriate to your organization.

Step 1: What to Expect When You’re Expecting a Quant Risk Program

In this phase, you’re setting the foundational groundwork, going to training or self-study, and increasing the rigor of your existing qualitative program.

Training - Self-Study
Reading, of course, plenty of reading.

First, some books.

• Measuring and Managing Information Risk: A FAIR Approach by Jack Freund and Jack Jones
• The Failure of Risk Management: Why It’s Broken and How to Fix It by Douglas Hubbard
• How to Measure Anything in Cybersecurity Risk by Douglas Hubbard and Richard Seiersen
• Risk Analysis: A Quantitative Guide by David Vose

Even if you don’t plan on adopting Factor Analysis of Information Risk (FAIR), I think it’s worth reading some of the documentation to help you get started. Many aspects of risk measurement covered in FAIR port well to any risk model you end up adopting. Check out the Open Group’s whitepapers on OpenFAIR, webinars, and blogs from the FAIR Institute and RiskLens.

Blogs are also a great way to stay up-to-date on risk topics, often directly from practitioners. Here are my favorites:

• Adam Shostack’s blog
• Cyentia blog
• ERM Insights
• The Evolver blog
• Exploring Possibility Space - Russell Thomas’ blog
• FAIR Institute
• Governance, Risk Management and Compliance
• Graeme Keith’s blog
• Hubbard Decision Research
• Information Security Management - Daniel Blander’s blog
• ISACA Now
• Less Wrong - Extensive site dedicated to improving reasoning and decision-making skills
• The Risk Doctor - Jack Freund’s blog
• Norman Marks’s blog
• Probability Management - Sam Savage’s blog
• Refractor - Rob Terrin’s blog
• Risk Academy
• Risk & Cybersecurity - Phil Venable’s blog
• Starting Up Security
• Tony Martin-Vegue's blog - this blog
• Vose Software

Webinars

• SIRA’s webinar series
• FAIR Institute’s webinars
• ISACA webinars (surprising amount of quant content)

Structured Training & Classes

• FAIR / FAIR Institute
• FAIR / HealthGuard Security
• Hubbard Decision Research classes
• Probability Management
• Good Judgement Project - calibration training
• Coursera - classes on prediction, probability, forecasting – generally not IT Risk classes

Add rigor to the existing program
Risk scenario building is part of every formal risk management/risk assessment methodology. Some people skip this portion or do it informally in their qualitative risk programs. You can’t take this shortcut with qualitative risk; this is the first place the risk analyst scopes the assessment and starts to identify where and how to take risk measurements.

If not already, integrate a formal scenario-building process into your qualitative risk program. Document every step. This will make moving to quantitative risk much easier.

Some frameworks that have risk scoping components are:

• NIST 800-30
• OCTAVE Allegro / Forte
• ISO 27005
• FAIR (risk scoping portion)
• ISACA’s Risk IT Framework & Practitioner’s Guide, 2nd editions - you need to be an ISACA member, but this has an extensive list (60+) of pre-built scenarios. I recommend starting here.
• Open Information Security Risk Universe

Adopt a Model
What model are you going to use? Most people use FAIR, but there are others.

• FAIR
• There are several in How to Measure Anything in Cybersecurity Risk by Doug Hubbard
• Probability management (Excel plug-in's)
• There are many Value-at-Risk models you can use from business textbooks, Coursera classes, business forecasting, etc.
• If you have the chops, you can develop your own model

Collect Data Sources
Start collecting data sources in your qualitative program. If someone rates the likelihood and magnitude of a data breach as “high,” where could you go to get more information? Write these sources down, even if you’re not ready to start collecting data. Here are a few starting places:

• Lists of internal data sources: Audits, previous assessments, incident logs and reports, vuln scans, BCP reports
• External data: Your ISAC, VERIS / DBIR, Cyentia reports, SEC filings, news reports, fines & judgments from regulatory agencies
• Subject matter experts: Experts in each area you have a risk scenario for; people that inform the frequency of events and the magnitude (often not the same)

Step 2: Infancy

You’ve picked a model and methodology and you’re ready for your first few steps.

Perform a risk analysis on a management decision
Find someone that has a burning question and perform a risk assessment, outside of your normal risk management process and outside the risk register. The goal is to help this individual make a decision. Some examples:

Get stakeholders accustomed to numbers

• Words of Estimative Probability by Sherman Kent
• Probability Survey - try this at home
  
  

Step 3 – Adolescence

You have several quant risk assessments done and are fully ready to rage against the qualitative machine – but not so fast! Don’t forget to bring everyone along!

Perform more decision-based risk assessments
In this step, perform several more decision-based risk analyses. See the list in Step 2 for some ideas. At this point, you’ve probably realized that quantitative cyber risk analysis is not a sub-field of cybersecurity. It’s a sub-field of decision science.

Check out:

• Foundations of Decision Analysis by Ron Howard
• Using Risk Assessment to Support Decision Making

Build a Database of Forecasts

Record the frequency and magnitude forecasts for every single risk assessment you perform. You will find that over time, many assessments use the same data or, at least, can be used as a base rate. Building a library of forecasts will speed up assessments - the more you do, the faster they will be.

Some examples:

Watch Your Bedside Manner
This is the easiest tip and one that so few people do. It’s an unfortunate fact: the risk matrix is the de facto language of cyber / technology risk. It’s in the CISSP and CRISC, it’s an acceptable methodology to pass organizational audits like SOX, SOC2, and FFIEC and it’s what’s taught in university curriculums. When moving both organizations and people over to quantitative models, be kind and remember that this a long game.

Do this:

• Recognize the hard work people have put into existing qualitative risk programs
• Focus on improving the rigor and fidelity of analysis
• Talk about what I can do for you: help YOU make decisions

Don’t do this:

• Disparage previous work & effort on qualitative programs.
• Quote Tony Cox in the breakroom, even though he’s right. “[Risk matrices are] worse than useless.”
• Force all people to consume data one way – your way

Step 4: Grown-up

In this step, the quantitative risk program is mature and you’re making tweaks, making it better, adding rigor, and bringing people along for the risk journey. Work on converting the risk register (if you have one) and the formal risk program over to quantitative as your final big step.

Through your work, risk management and the risk register is not busywork; something you show auditors once a year. It’s used as an integral part of business forecasting, helping drive strategic decisions. It’s used by everyone, from the Board down to engineers and everyone in-between.

Here are some references to help with that transition:

• OpenFAIR - STIX Intergration
• Open FAIR – NIST Cybersecurity Framework Cookbook
• Open FAIR - ISO/IEC 27005 Cookbook
• COBIT 5 for Risk

Durable Quantitative Risk Programs

The last and final tip is how to make your program durable. You want it to last.

• Colors and adjectives are OK, but getting stakeholders thinking in numbers will make your program last.
• Reach far and wide to all areas of the business for SME inputs.
• Embed your program in all areas of business decision-making.

Final Thoughts

The greatest challenge I’ve found isn’t data (or lack of), which is the most common objection to cyber risk quantification. The greatest challenge is that everyone consumes data differently. Some people are perfectly comfortable with a 5-number summary, loss exceedance curve, and a histogram. Others will struggle with decision-making with colors or numbers. Bias, comfort with quantitative data, personal risk tolerance, and organizational risk culture all play a role.

I recommend the work of Edward Tufte to help risk analysts break through these barriers and present quantitative data in a way that is easily understood and facilitates decision-making.

***

I’m always interested in feedback. It helps me improve. Please let me know what you thought of this talk and/or blog post in the comments below. What would you like to see more of? Less of?