Less Marketing Drivel, and More Quantifiable Results Please!

April 13th, 2016 by admin

LightCyber Announces New Attack Detection Metrics

Accuracy-Efficiency-MetricsToday, LightCyber announced the IT security industry’s first-ever Attack Detection Metrics, which objectively measure the quantity and quality of alerts produced by our Magna platform in live customer environments. Our Q1’16 metrics are: Efficiency – 1.1 alerts per 1000 endpoints per day; and Accuracy – 62% of all alerts dispositioned by analysts in a useful way. To our knowledge, this is the first time a security vendor has ever publicly released objective data about their customers’ actual user experience with their products, and we believe that’s because the data for conventional security products would be abysmal. We challenge other IT security vendors to produce the same objective data for their customer environments.

You gotta feel some serious sympathy for IT security buyers that endure endless vendor marketing claims that their solution will “solve world hunger” and ensure customers the “best security” available in the industry. IT security is one of the few product sectors that I know of that isn’t subjected to basic quantitative metrics and objective performance standards. Despite $28 Billion in annual product spending and growing fast, IT security vendors are essentially not expected to objectively demonstrate their cost or performance value proposition in quantifiable terms, and customers are expected to buy based upon…Faith? Claims? Purported independent references? We’re not only talking about 6- and 7-figure expenditures, but also risks that can make or break an IT security professional’s career. Not a good position for security buyers.

Well, that changes today! Today, LightCyber has announced objective attack detection standards and Efficiency and Accuracy metrics and has disclosed its own anonymized and aggregated reporting of customer experience to those metrics.

We’ve all seen the marketing drivel and unsubstantiated claims made by vendors. These vacuous claims do nothing to quantify a vendor’s ability to really solve critical security visibility and attack detection problems:

  • “No Compromise Security”
  • “Security That Thinks”
  • “Enterprise Immune System”

It’s shocking that this can be the status quo despite the fact that the IT security industry is in a state of complete meltdown due to the inability of vendor solutions to accurately and efficiently focus security operators on the critical few attacker events that are most important. Data breaches of major corporations and government organizations seem to happen weekly – even daily. Advanced persistent threats targeting single organizations are increasingly common. And every single survey confirms that IT security professionals are overwhelmed, if not completely paralyzed:

  • Ponemon Institute Cost of Malware 2015 report surveyed 700 enterprises and concluded that organizations received 16,937 alerts per week on average, were only able to research 4% of alerts, and that time spent researching false positive alerts cost $1.3M on average. (This is the proverbial firehose sucking contest!)
  • Mandiant’s M-Trends 2016 report crowed about the reduction in attack dwell time from 205 days to 146 days during 2015 – a mere 5 months when the average attacker operates with unfettered access to the network unbeknownst to the security team. (Yay! … ?)
  • Mandiant’s M-Trends 2016 also acknowledges that despite 5 months dwell time, greater than half (53%) of the attacks are not even detected by the internal IT security team! (Argh!!!)
  • Security Week just reported that another recent survey identified that 31% of all enterprise security organizations ignore more than half of all alerts due to lack of resources. (What!!!)

It’s hard to avoid the reality that these data demonstrate – security vendor solutions are not providing adequate alert accuracy and efficiency, resulting in security operations paralysis, blindness, and ineffectiveness. IT security professionals cannot dig themselves out of the ongoing breach crises without tools that accurately and efficiently focus their scarce time and resources on what’s important. As a vendor community, we must provide objective, quantifiable metrics to measure a product’s ability to focus security operators on what’s important – active attackers that have circumvented conventional threat prevention infrastructure.

I am proud to announce today that LightCyber is leading this charge by promoting and publishing our own Attack Detection Metrics that measure the volume of alerts received per day (aka, “Efficiency”) and the usefulness of those alerts as determined by the analyst’s disposition of the alerts (aka, “Accuracy”). The only way to close the gap on attackers is to measure the effectiveness of our products and disclose them for buyers to make choices. Vendors must either improve the above picture, or perish due to lack of product efficacy.

Please click to see the Attack Detection Metrics results.



LightCyber Delivers Alert Accuracy

All Alerts Accuracy:
% Actionable Alerts High % is Better

Click here to view larger image


LightCyber Alert Efficiency Metric

# Alerts/1.1k Hosts/Day:
Lower # is Better

Click here to view larger image


The game is on! We challenge our peers to publish their own metrics.


Leave a Reply

Your email address will not be published. Required fields are marked *