What does MITRE Evaluation mean for you?
MITRE ATT&CK Engenuity emulates well-known threat groups inspired by publicly available threat intelligence. Organizations should use them to determine which solutions best address their cybersecurity gaps, complement their existing protection implementation, and fit their organization’s needs and capabilities.
On September 20, MITRE Engenuity published their results for the 5th round. Today, we are proud to share our results in our first year of participation in this evaluation. This round focused on adversary behavior informed by Turla, a known Russia-based threat group. We invite you to explore the vendor cohort results in this technical brief to understand the evaluation and interpretation.
The evaluation comprises two tests: Detection and prevention evaluation.
We submitted WatchGuard EDR, designed to complement the existing protection solutions on endpoints. WatchGuard EDR does not have preventive technologies such as lookouts to the Collective Intelligence knowledge in the Cloud, contextual detection, anti-exploit, etc., that are part of WatchGuard EPDR. In consequence, the results of this evaluation do not reflect our protection capabilities.
Detection Evaluation
MITRE Engenuity emulates the attack flow, looking at the evaluated solutions' visibility with telemetry for later analysis or adding additional context through analytics. Telemetry or analytics require a security operations team to operationalize them to investigate and respond to the attack and improve the preventive capabilities in the future, looking for better and automated protection.
During the test, the attacker couldn't be blocked in his path. It is not a test where the automated prevention, detection, and response capabilities are evaluated. It evaluates only the visibility into the attacker's behavior with telemetry or MITRE ATT&CK technique identification (analytics).
Many people will only look at the summaries of detection results. However, it's essential to pay attention to other information that helps security teams make informed decisions:
- Type of visibility. Analytics coverage provides enriched detections that add critical context through ATT&CK technique mapping and alert descriptions. The telemetry coverage is helpful for hunters and incident responders who have to dive deeper into the threat actors’ behavior.
- Configuration changes. A “Config change” modifier means the vendor adjusted the sensor, the data processing, or the UX to show detection during the test. In other words, the product didn’t behave as it does in a real environment.
- Operational Efficiency. When designing an effective endpoint security solution, the critical principle is balancing the signal-to-noise ratio. Theoretically, creating a solution that achieves 100% detection is easy – simply detecting “everything,” being very noisy, and having many false positives to solve. Of course, such a solution would be next to useless.
We also recommend you evaluate if 100% sub-step coverage is what your organization needs depending on your security operation team's capability and the noise of false positives the solution generates, considering the following recommendations:
1. Organizations without a security operations team:
Their security strategy should be based on the following:
Organizations with a security operation team can be overwhelmed by inefficient solutions that detect everything. Noisy EDR solutions generate too many false positives to solve and excessive alerts to manage.
- Security layers from network to endpoint
- Automated prevention, detection, and response (EPP and EDR) capabilities
- Complemented with MDR (managed detection and response)
2- Organizations with a security operation team:
Their security strategy is based on a good balance between prevention, detection, and response managed by a security operation team with a rationalized workload.
Organizations with a security operation team can be overwhelmed by inefficient solutions that detect everything. Noisy EDR solutions generate too many false positives to solve and excessive alerts to manage.
3- MDR and mature security operations teams:
With hunters and their own security analytics based on detailed telemetry.
Learn more about our Endpoint Security Solutions, especially WatchGuard EPDR, our layer-based security approach, including the Zero- Trust Application Service and the Threat Hunting Service, which detect and block threats on top of the EDR technologies without a security operation team intervention. Additionally, you may be interested in knowing the differences between WatchGuard EDR and WatchGuard EPDR.
