Behavioral Security Technologies: If it Quacks Like a Duck, It Probably Is a Duck

November 25th, 2015 by Jason Matlof

Quack Like a Duck imageThe shortcomings of legacy “artifact-based” prevention technologies have forced security vendors to embrace new detection techniques like sandboxing and endpoint detection and response (EDR), which are being billed as “behavioral detection” technologies. This is causing confusion among users regarding value and utility when compared to emerging “behavioral anomaly detection” techniques. To be clear, “behavioral detection” techniques such as sandboxing and EDR are only slight variations of the security techniques employed over the last 20 years, and they are nothing like behavioral anomaly detection.

Behavioral detection is all about identifying the malicious characteristics of malware. Behavioral anomaly detection is all about learning the normal pattern of human and machine activity for each network, and looking for attacker behaviors that stand out — nothing to do with malware at all.

Let me try to clarify by starting with the basics…

First, the old static artifact-based methods of hunting for malware are insufficient. This is well documented in growing numbers of industry tests demonstrating that leading threat prevention vendor technologies (NGFW, antivirus and sandboxing) are less than 100% effective in blocking known malware and not at all effective in blocking unknown. It is also evidenced by the almost daily announcements of major data breaches. Why is this so?

Legacy threat prevention systems are real-time or near real-time architectures that attempt to identify and block the attacker’s initial intrusion attempt – typically via malware. Antivirus represents the oldest, most mature method that utilizes static artifact-based file descriptors of malware in the form of either byte-string “fingerprints” or cryptographic hashes to identify bad stuff. It’s now widely understood that targeted attackers with sufficient incentive can always circumvent antivirus systems by creating new malware tools or variants with signatures not yet known by the antivirus vendors.

This cat-and-mouse game led to the development of a new malware identification technique: sandboxing. Instead of using statically-defined file descriptors, sandboxing techniques execute a copy of a file in an emulated environment to identify statically-defined behavior descriptors that are commonly associated with malicious code, e.g., suspicious system or network calls; kernel hooking; file installation or modification; etc. While sandboxing complements the weaknesses of file-based matching techniques by using behavior-based techniques, they still employ statically defined malware behavior definitions. As such, sophisticated attackers are able to construct methods to circumvent these behavioral identification techniques, since they are static and known. These are well documented.

Recently, yet another threat prevention approach has come to market: agent-based Endpoint Detection & Response (EDR). Like sandboxing techniques, EDR tools also employ statically-defined behavior descriptors. Instead of identifying malware behaviors in an emulation environment, EDR technologies look for those statically-defined behavior descriptors after the malware has been installed on the target device (endpoint). EDR products capture and record every single system behavior – what some are calling an endpoint “flight recorder” or DVR. EDR tools then continuously compare these events to a database of statically-defined behavior descriptors much like a sandbox. EDR looks for suspicious network calls; suspicious system calls; suspicious communications; file installation or modifications; etc. EDR vendors call these static behavior definitions Indicators of Compromise (IOCs). While certainly employing “behavioral detection”, EDR is just another form of statically-defined threat prevention – an evolution from sandboxing, which was an evolution from anti-virus. EDR is still subject to circumvention due to the static detection technique.

What is needed is a different approach to security that doesn’t employ “static” definitions to catch attackers, since attackers can so easily test their new approaches to circumvent such systems.

Behavioral anomaly detection is essential to find the operational activities of an active attacker, and it’s dramatically different than behavior-based detection found in either sandbox or EDR approaches. Behavioral anomaly detection presumes nothing and has no concept of static definitions of malicious activity. Instead, it builds a profile of “known good” behavior for network entities (users and hosts) that is specific to each customer network environment. From that baseline, behavioral anomaly detection identifies the anomalous behaviors that are indicative of an active attack on the network. Since the behavioral profile is specific to each environment, the attackers can’t reverse engineer or test an attack method against the system. Since the attacker is operating in a foreign environment, they can’t disguise the anomalous operations that they must perform to succeed in their attack (reconnaissance, lateral movement, etc). Conventional circumvention methods are not possible with behavior anomaly detection systems, at least in any traditional sense.

It should now be clear that behavioral anomaly detection is not subject to the shortcomings of all prior statically defined attack identification techniques. Let’s not be lulled into a false sense of safety by vendors touting the “behavioral” detection banner. Whether they are file-based descriptors (i.e., antivirus) or behavior-based descriptors (i.e., sandboxing or EDR), static definitions for any malware detection will ALWAYS be easily circumventable by a highly motivated, targeted attacker. And such technology will always be useless in identifying the significant attack activity that makes no use of malware at all.

Sandboxing and EDR technologies are relevant and valuable to those trying to build yet another layer of prevention technologies and incrementally reduce the attack surface. By comparison, however, they bear very little resemblance to behavioral anomaly detection, and are not useful in finding active attackers on your network, especially for the stages of the attack that don’t leverage malware.

Let’s not fool ourselves – sandboxing and EDR technologies are just another generation of statically-defined threat prevention systems. If it quacks like a duck – it’s probably a duck.

To find out more about behavior anomaly detection, read more here.


Leave a Reply

Your email address will not be published. Required fields are marked *