Index / Behavioural detection

Behavioural detection

Every program a device runs performs a sequence of actions: it opens files, writes to them, contacts network addresses, starts other programs, changes settings, reads memory belonging to other processes. Most of these actions are ordinary in isolation. What distinguishes malicious software is usually the combination, the order and the context — a document viewer that launches a scripting engine which then contacts an unfamiliar address and begins rewriting files in rapid succession is doing something no legitimate document viewer does.

Behavioural detection is the attempt to recognise those combinations as they occur. It is the direct answer to the limitation described in signature detection: a catalogue can only describe what has already been collected, whereas a description of hostile behaviour can apply to code written five minutes ago.

The techniques grouped under the term

Static heuristics

Examining a file before it runs for structural traits associated with malicious code — obfuscated sections, unusual packing, instructions that hide the program's real entry point. No execution is involved; the file is being read rather than watched.

Emulation and sandboxing

Running the suspect program in a confined, simulated environment and observing what it attempts. Anything it does affects only the sandbox. The limit is time and fidelity: malicious code can wait, or check whether it is being observed and behave innocently until it is not.

Runtime behavioural monitoring

Watching real processes on the real device and intervening when a sequence crosses a threshold. This is what most products mean by behavioural protection, and it is the only one of the three that is continuously active.

Rollback and journaling

Recording changes a suspicious process makes so that, if the process is later judged malicious, the changes can be reversed. The coverage is partial and bounded by whatever the product recorded.

Machine-learning classifiers

Models trained on large collections of benign and malicious programs that score new ones by similarity to patterns in the training data. They generalise beyond the exact samples they were trained on, and they cannot explain their verdicts in terms a reader can check.

Why mistakes are more common here

A signature match is a statement about identity and is either right or wrong. A behavioural verdict is a statement about intent inferred from actions, and intent is not directly observable. The same sequence — enumerate files, read each one, encrypt it, delete the original — describes both ransomware and a legitimate disk-encryption tool. The product has to decide from context that is frequently incomplete.

The result is that behavioural systems generate more false positives than signature systems, and they are concentrated in predictable places: backup software, disk utilities, installers, automation scripts, software development tools, remote-support tools and anything that is unsigned or rare. Readers who use such tools should expect occasional interruptions and should understand how to inspect a detection before overriding it, rather than developing a habit of dismissing warnings without reading them.

How an alert is usually structured

A behavioural alert tends to carry less specific information than a signature alert, because the product does not know what the thing is — only that its conduct crossed a line. Typical labels are generic family names, "suspicious behaviour" categories or heuristic identifiers. That is not evasiveness; it is an accurate reflection of what was determined.

When such an alert appears, the useful questions are factual ones: what was the program, where did it come from, was it started deliberately a moment ago, and does the reader recognise it. A program the reader has just downloaded and launched on purpose is a different situation from a program that started by itself at three in the morning. Products generally show the path and the parent process for exactly this reason.

Living off the land

A further difficulty, and the reason behavioural approaches are now central rather than supplementary, is that a growing share of hostile activity uses no distinctive program at all. Scripting engines, administrative utilities and remote-management tools already present on the system can accomplish most of what an attacker needs. There is no foreign file to catalogue, and the executables involved are legitimate, signed and in daily use.

Against this, identity-based detection has nothing to match. Only the pattern of use distinguishes an administrator from an intruder, which places the entire burden on behavioural analysis and on the context the product can gather about who or what initiated the activity.

What behavioural detection does not address

Behavioural systems watch programs. They are not watching the reader. When a person is persuaded to enter a password on a convincing imitation of a bank's site, nothing malicious runs on the device at all; no behaviour threshold is crossed because there is no hostile program. This is covered in phishing, and it is the single largest gap in what any scanner can do.

Equally, behavioural detection acts after a program has started. Prevention that keeps the program from arriving — restricting what can be installed, keeping software patched, filtering connections as described in firewalls — operates earlier and is not replaced by it.

Comparing the two detection styles in practice

The final row is the reason cloud backup is treated in this library as a security control rather than a convenience. Once a sufficiently fast encryption routine has run to completion, detection of any kind is a post-mortem.

Independent guidance

General advice on reducing the chance that unwanted software runs at all — including application control and patching, which sit upstream of any detection method — is published by the Australian Cyber Security Centre at cyber.gov.au.