Guidance on Measurements
"You cannot manage what you cannot measure" - Peter Drucker
Ask five different people inside your security team about how the organization is doing in terms of security and prepare to receive five different answers. Having strong metrics that are well understood will go a long way to removing the ambiguity of the question.
Before we measure, we need to make sure that what we're doing aligns with the desires of the business.
Examine the diagram below. The work performed by the security team is driven by the business, the impact leads to results. Depending on the results, the business alters or continues the demand for the work performed.
Measurement Principles
- Know the difference between measuring effort, impact, and effectiveness
- Know the difference between a data point, and a metric
- Consider how much data is needed to make a metric useful
- Know how your metrics will be interpreted
- Consider who is interested in your metrics and why
- Metrics should drive change, or reinforce past behavior
Measuring effort is easy and important. However, measuring effort by and of itself is not valuable. We must understand the result of the work. Without context, input is just a data point.
Measuring impact is trickier. What happened as a result of the work our team did? This is impact. The risk that was reduced, the new alerts that were implemented, and the increased security awareness are all elements of impact.
Measuring effectiveness is what matters. Effectiveness get's us close to the return on investment for our input. If the effort and impact aren't matched, then we need to adjust our approach.
Consider your function in an organization. Does the business come to you for services? Or do you go to them with an edict that dictates that they must work with you?
Product-Market Fit
If you've never heard of this term, it simply means that you have a product, that matches the market. If you have found a product-market fit, then you probably know it. If you don't know if you have it, you probably don't.
Imagine vehicles do not have speedometers. Now imagine you're cruising down the road hoping to make it home in time for your strict curfew. You're not even the fastest person on the road, in fact, you feel like you're going at the speed of traffic.
You get pulled over and are issued a fine. The cop tells you, "We sent you a letter about the speed limit on this road last year".
You'd be frustrated right? Then they come out with speedometers and start adding signs every so often reminding you what the speed is. They even invent cruise control that lets you set your speed correctly.
You'd probably appreciate that and there would likely be people lining up at the door to get these speedometers installed in their vehicles.
Provide speedometers and road signs. Understand your market and provide a product that fits. Know your stakeholders and give them the value they need.
Another Page from the Business Handbook
Have you ever asked your stakeholders how you're doing? Customer Success teams have been doing this forever because it works.
Sending a survey after a team has interacted with you can indicate how your function and team are perceived.
Your IT team probably already does it. Your vendors probably do it. You should too!
The benefit of surveys is that not only can you get a feel for how well your people and processes are perceived, but you can also gauge mindset.
Red Team Specific Measurements
For each of the following metrics, consider the guiding principles listed above, as well as the effectiveness matrix.
Let's start with some easy-to-come-by metrics:
- Vulnerabilities discovered/remediated
- The success rate of phishing emails
- Amount of “operations” - categorized by operation type
Those data points are nice, but they don’t always address our primary stakeholders. Having a strong understanding of the blue team's weaknesses, and strengths as well as how the blue team measures their own performance will help the Red Team determine what metrics are important to gather.
Some of the basics are:
- Mean time to detect - Shows improvement in detection capabilities
- Mean time to respond - Shows improvement in response capabilities
- Mean time to initial access - Shows improvement in perimeter security/phishing controls
- Mean time to act on objectives from the beginning of op - Shows increase in defense posture holistically
- Mean time from reporting, to detection/control in place - Shows improvements in detection engineering capabilities
- Amount of detection’s trending activity - Shows improvements in detection and response capabilities based on Red Team activity
- Time to Deploy Infrastructure - Helps understand the effort that a Red Team operation takes.
- Time between Kill Chain periods - How long does it take to get from Initial Access to Act on Objectives?
- Amount of systems/accounts compromised - Helps paint a picture of how many hoops an adversary might need to jump through to Act on Objectives.
- Meta Data - Length of report, report creation time length, areas of the report the most time is spent on/least time, number of supporting documents, and time to compile artifacts.
Now that we have a good start for understanding what needs to be tracked at the macro level, we need solid tracking of the micro-events. One way to do this is to have a spreadsheet per operation that tracks all the relevant details of each action performed:
- TTP - Map to the Mitre ID, this allows a heat map to be created which identifies areas of strengths and weaknesses in detection capabilities.
- Kill Chain Step - Similar to the above, this lets us see detection and response capabilities strengths, and weaknesses
- Hostname
- Host type - This allows us to see which OS’s have various strengths and weaknesses
- Type of detection expected/lacking - This allows us to map which controls are performing well, or underperforming.
- Action timestamp - Make sure this is in a timestamp compatible with the SOC’s SIEM or standard.
- Detection timestamp - Make sure this is in a timestamp compatible with the SOC’s SIEM or standard.
- Response timestamp - Make sure this is in a timestamp compatible with the SOC’s SIEM or standard.
- Business Unit - This allows us to see which business units are lacking, and can identify business units that might need more collaboration with the security org.
- Whether or not detection was intentional - This allows us to include/exclude actions that may have intentionally alerted the defense team, it also allows us to see where the Red Team needs to improve preparation if we are generating alerts in places we don’t want.
For an example of a spreadsheet that tracks much of this data, check out Cedric Owens GitHub project: https://github.com/cedowens/Rolling_Op_Metrics
While there are many metrics and data points mentioned here, every organization will need to assess which ones make sense, and which areas need additional metrics. Keep in mind, that all metrics and data points should be compared against the Measurement Principles mentioned above.
References:
https://medium.com/starting-up-security/measuring-a-red-team-or-penetration-test-44ea373e5089
The Red Team specifics portion was adapted from my personal blog: https://jordanpotti.com/2020/11/23/measuring-your-red-team/