[KANALPARTNER] Many of you may have seen the recent results of MITRE ATT&CK’s emulation of APT29 [1] tradecraft and operational flows.
Emulations were run against a variety of EDR vendor solutions that were invited to participate in the evaluation. Interpreting the results might be a cumbersome exercise, so here is a little clarification to help understand them.
Testing methodology
The test itself consist of two scenarios that can be divided into stages shown in Figure 1.
Each of these stages is completed using selected Tactics, Techniques and Procedures (TTP) mapped in the MITRE ATT&CK framework. It is a curated knowledge base and model for cyber threat behaviour, reflecting the various phases of threat lifecycle and platforms they are known to target. In this testing round, two specific attacks flows were used consisting of 57 ATT&CK techniques with 134 test procedures in total.
Figure 3: Kibana visualisation of MITRE ATT&CK Round 2 Evaluation Results
Reading the results
The results of the tests might make for confusing reading if you have gotten used to at-a-glance Magic Quadrants. You can find the main results nicely illustrated in Kibana (like in Figure 3) here [2].
Figure 3: Kibana visualisation of MITRE ATT&CK Round 2 Evaluation Results
To help you to understand what those data visualisations at Kibana actually mean, have a look at Figure 4, created by our friends at SentinelOne [3]. Even though it is coming from the vendor, it is well aligned with MITRE’s idea of using TTPs to explain attack behaviour.
Figure 4: SentinelOne assessment of data and scoring in MITRE ATT&CK evaluation
In addition to the main categories, one or more modifiers may be optionally assigned to each result. The below table from F-Secure [4] neatly explains what each of the modifiers mean (as well as the already discussed main categories) with ‘Alert’ and ‘Correlated’ both helping reduce dwell time, and ‘Host Interrogation’ and ‘Residual Artifact’ useful in forensics.
Figure 5: Detection categories and modifiers explained (F-Secure)
In addition to the above categories, there is an MSSP category that indicates if the vendor used analysts on top of their product to detect or gather relevant data about an attack. This only applies to vendors who provide Managed Detection and Response (MDR) or outsourced SOC services. These used to be premium services but have become more affordable from some vendors.
What does it mean to my organisation?
Almost every participating vendor has published its own interpretation of the results and how it excelled in the tests. This is normal in a highly-competitive market space such as EDR, but is also aligned with the fact these products have slightly different philosophies behind how they work.
The good thing with MITRE ATT&CK tests is that they are open and transparent. It is still a lab environment and focusing on detection only, but the tests are based on real TTPs that are used in the wild. TTPs are becoming a common language between security solutions and many vendors in our portfolio support them or even contribute to MITRE ATT&CK.
What comes to the results, some quick conclusions are:
- The lower number of total misses is good.
- The higher number of TTPs helps you to map findings and detections in MITRE ATT&CK if that is what your organisation prefers to use.
- Pay attention to MSSP detections if you need an MDR service from the vendor.
- In some areas there are only small differences between vendors, indicating good performance from many of them. In that case your organisation should pay more attention to deployment times, how much time is taken maintaining the solution, how intuitive it is to use and so on.
If you want to understand more about MITRE’s test, check out the vendor blogs or reach out to your Exclusive Networks contact.
[1] A description of APT 29 group: https://attack.mitre.org/groups/G0016/
[2] Elastic Dashboard of MITRE ATT&CK Round 2 Evaluation Results:
https://demo.elastic.co/app/kibana#/dashboard/7a625e50-8418-11ea-930f-cd34a3ca7ca4
[3] MITRE 2020 ATT&CK Evaluation – What The Results Really Mean to CISO & SOC:
https://assets.sentinelone.com/c/mitre-2020-att-ck-ev?x=Oy5zHq
[4] F-Secure excels again in the MITRE ATT&CK Evaluation: https://www.f-secure.com/en/business/resources/f-secure-excels-again-in-the-mitre-attack-evaluation
LÄNK: https://www.exclusive-networks.com/se/resources/sakerhet/
