MITRE Engenuity ATT&CK Evaluations for Managed Services (menuPass + ALPHV BlackCat)

MITRE Engenuity™ has released the results from the latest round of ATT&CK® Evaluations for Managed Services, assessing the abilities of 11 vendors to detect, analyze, and accurately describe real-world adversary behavior.

This was the second round of ATT&CK Evaluations for Managed Services, initially launched in 2022, to help organizations better understand how offerings like Sophos MDR can help protect them against sophisticated, multi-stage attacks.

Watch this short video for an overview of the evaluation:

What was the scope of the ATT&CK Evaluations?

MITRE Engenuity ATT&CK Evaluations are designed to simulate a representative example of how organizations should expect a managed service provider to engage with them during a sophisticated attack.

The MITRE Engenuity team emulates the behaviors of known threat actors during the evaluation. A ‘black box’ approach was used in this round, whereby MITRE did not disclose the simulated threat actor(s) or the technique scope until the assessment was complete.

This evaluation emulated tactics and techniques used by two known threat groups – menuPass and ALPHV/BlackCat – and assessed each vendor’s abilities to detect and report specific adversary activities.

In total, the evaluation comprised 172 adversary activities (sub-steps) across 15 overall steps. Note, however, that only 43 of the sub-steps – those that MITRE Engenuity considered critical for attack sequence success – were included in the results.

The evaluation focused entirely on detection and reporting. The ability to block, respond to, or remediate threats was not assessed. It’s essential, therefore, to keep in mind that adversary behaviors emulated in this evaluation may have been blocked by protection technologies (e.g., next-gen endpoint tools), which vendors needed to deactivate during the evaluation.

Evaluation participants

Eleven managed security service providers participated in this evaluation round:

Sophos’ results

The results of MITRE ATT&CK Evaluations can be interpreted in multiple ways and MITRE Engenuity does not rank or declare any vendor a “winner” or a “leader”. Each vendor’s managed service reports information differently and each organization’s needs and preferences are just as important as the results themselves.

Sophos successfully “Reported” and accurately described 84% of the 43 adversary activities (sub-steps) selected by MITRE Engenuity – higher than the average among participating vendors. The majority (75%) of Sophos’ detections were also categorized as “Actionable”. “Reported” means the adversary activity was successfully identified, and sufficient context was provided. And, where the reported information also successfully addresses the “5 W’s” (Who, What, When, Where, and Why), the activity was further categorized as “Actionable”.

The results also include the number of alert emails sent by each vendor.

To ensure an effective, understandable, and actionable response, Sophos MDR focuses on providing high-value, human-written notifications containing the critical information and context that customers need to know.

During the 5-day MITRE ATT&CK Evaluation for Managed Services, Sophos MDR sent 24 emails. The average among other participants was over 120 emails, with some vendors sending more than 300 emails. Alert fatigue, caused by an overwhelming number of notifications from security solutions, is a major problem in cybersecurity. Sophos understands that your organization’s time is valuable, and when resources are limited, quality is typically better than quantity.

How to use results of MITRE Engenuity ATT&CK Evaluations

ATT&CK Evaluations are among the world’s most respected independent security tests, due in large part to the thoughtful construction and emulation of real-world attack scenarios, transparency of results, and richness of participant information.

When considering a Managed Detection and Response (MDR) service, be sure to review the results from MITRE Engenuity ATT&CK Evaluations alongside other reputable third-party proof points, including verified customer reviews, and analyst evaluations.

As you review the data available in MITRE Engenuity’s evaluation portal, look beyond the numbers and consider the following, keeping in mind that there are some questions about managed security services that the ATT&CK Evaluations cannot help you answer. For example:

• Does the service present information to you the way you want it, with high-value communications containing the critical information you need to know?

• Does the service assume you have an in-house security operations team, or can they provide a full ‘instant SOC’ with the ability to take action to eliminate threats on your behalf?

• Who will be engaging the managed service provider on a day-to-day basis? IT Administrators, experienced security analysts, or perhaps both?
• Can the service integrate with other technologies in your environment to detect and respond to multi-stage threats that extend beyond endpoints (e.g., firewall, email, cloud, identity, network, backup and recovery, etc.)?
• Does the service include full remote incident response, and are the included IR services limited to a fixed number of hours, or uncapped?

Why we participate

Sophos is committed to participating in MITRE Engenuity ATT&CK Evaluations alongside some of the best security vendors in the industry. As a community, we are united against a common enemy. These evaluations help make us better, individually and collectively, for the benefit of the organizations we defend.

Our participation in the latest evaluation further validates Sophos’ position as an industry-leading Managed Detection and Response (MDR) provider and trusted cybersecurity partner to over 22,000 customers.

Don’t take our word for it

Sophos Managed Detection and Response is the world’s most popular MDR solution. We secure more organizations than any other MDR provider and have extensive experience across all industries and sectors. Recent third-party proof points include:

• Sophos named a Leader in the 2024 IDC MarketScape for Worldwide Managed Detection and Response (MDR)
• Sophos named a Leader in Frost & Sullivan’s 2024 Frost Radar™ for Global Managed Detection and Response
• Sophos is the only vendor named a Gartner Customers’ Choice in Endpoint Protection Platforms, Managed Detection & Response Services, Network Firewalls, and Mobile Threat Defense
• Sophos was the only vendor named a Leader in EPP, EDR, MDR, XDR, and Firewall in the G2 Winter 2024 Reports
• Sophos named a Leader for the 14th consecutive time in the 2023 Gartner® Magic Quadrant™ for Endpoint Protection Platforms
• Sophos MDR declared the Winner of the ‘Best Managed Security Service’ award in the SC Awards Europe 2024

To learn more about Sophos MDR and how it can support you, visit our website or speak with a security expert today.