Keeping a network safe requires a broad array of tools. Detection of anomalies plays the important role of spotting trouble early. Anomalies can happen in incoming, internal, and outgoing traffic. On the incoming side, recognition of attacks in progress provides an early warning. Within the network, malware that has successfully invaded one machine may launch attacks on others. Outgoing traffic could include attempts to contact command-and-control servers. When monitoring detects these events, security software or administrators can take action to stop the threat quickly.

How does anomaly detection work?

An anomaly is an abrupt and significant deviation from the normal behavior of a network. Recognizing one requires knowing what is normal. Light or heavy traffic isn't an anomaly by itself, and an anomaly doesn't always mean a huge number of bytes. Recognizing them requires a detailed characterization of acceptable network behavior.

Anomalies fall into three categories:

• Unusual but legitimate traffic. Sometimes an application has to engage in a burst of atypical network activity. This could be flagged once, but the monitoring system should adapt to it and not report it every time it happens.

• Errors and malfunctions. A buggy but not malicious application could generate an uncontrolled series of requests. A device that isn't responding might get repeated requests. These are problems that need fixing but aren't threats.

• Threats. Repeated probes and denial-of-service attempts are warnings of a concerted attempt to attack the network. Anomalies in internal or outgoing traffic could indicate a successful malware invasion or a hijacked account.

Monitoring software gathers vast amounts of data on network activity and looks for patterns. It can take a centralized approach or a distributed one. Centralized anomaly detection is better at assembling the big picture, while distributed detection may be able to catch particular events more quickly.

Anomaly detection vs. perimeter protection

The traditional approach to security has been to focus on the perimeter and endpoints. It relies on firewalls and anti-malware software at the network and device levels. This protection is still important, but it isn't enough by itself. Methods of attacking networks have become increasingly devious, and some can get past the front line of defense.

Users can make mistakes and run malicious software while reading an email attachment. Spam filtering will stop some but not all of those threats. Firewalls don't recognize all attacks, and some hostile traffic will get past them. Seemingly legitimate requests may exploit software bugs in Web applications.

Old-style firewalls and intrusion prevention and detection systems (IPS/IDS) rely on identification of specific features, or signatures, in network traffic. A certain pattern of bytes can indicate a particular malware program. This approach still has its uses, but new threats without known signatures appear at an ever increasing rate. A signature-based approach offers no protection against new and unknown threats.

Anomaly detection uses a behavior-based approach. Instead of looking for specific kinds of packets, it looks for anything which falls outside the limits of normal activity. This requires more sophisticated code than signature identification, but it can recognize problems which no one has previously categorized. As the rate at which threats appear has grown, behavior-based detection has become a critical part of network defense.

Types of anomalies

Anomaly detection relies on a variety of metrics. A major deviation in any one can indicate a problem, and simultaneous changes in multiple measurements are a stronger indicator.

• Volume of traffic. The amount of traffic on a network goes up and down, but a sudden increase can mean trouble. What counts as abnormal depends on factors such as the day and time.

• Use of IP space. Networks usually communicate with ranges of IP addresses in a consistent pattern. If there's a sudden burst of heavy traffic to addresses or domains that the network normally doesn't touch, that may indicate trouble. It could be a sign of a probing attempt, a denial-of-service attack, or malware running inside the network.

• User activity. A burst of abnormal activity from an account could mean it's been compromised. If packets go out from it to every computer in the network, the user's machine may be running malware, or an unauthorized person might be logged in.

• Use of protocols. Abnormal use of a protocol can point at certain types of attacks. Bursts of DNS requests, especially if they involve abnormally long domain names, are a sign of malware. Port scans are often a search for vulnerabilities. If they find a service which was never configured and never disabled, an attempt to exploit it will follow.

• Device status. A non-user device, such as a printer or router, will normally generate predictable types of traffic. If it starts issuing packets that have nothing to do with its function, it's likely to have been compromised.

Anomaly detection doesn't consider any of them in isolation but uses all the metrics to determine whether a deviation is significant enough to report.

A reported incident may be a false positive or indicate only a software or hardware defect. Other incidents can indicate several kinds of threats.

• Data theft. Malware or use of a compromised account can send confidential data to an outside server. The attacker will try to disguise the transfer and keep the rate of data acquisition low enough to avoid triggering alarms. Advanced detection methods can often catch these transfers in spite of attempts to hide them.

• Botnet activity. Botnet software starts by getting onto one machine in a network, typically through a phishing attack. Once it's running, it will try to connect to a command-and-control server. It will also try to replicate itself onto other machines on the network.

• Access to services. Malware running on a network device can access network services more easily than outside systems can. It will probe the network for databases, file servers, and other targets.

Machine learning

What falls within the limits of normal behavior is different for every network. The kind of work done on one will determine what is unusual but acceptable, and what is really a sign of trouble.

The process of defining anomalies is a statistical one. There isn't a hard line, but rather an increasing likelihood that going further out along a metric indicates a dangerous condition. Advanced statistical methods put numbers on the likelihood.

Machine learning techniques let a monitoring system calibrate the network's behavior. They gather information on the metrics of its behavior and apply statistical processes to identify ranges and clusters of typical values. They will change over time as network usage changes. When they change suddenly and fall outside the previously normal range, the software will issue an alert.

Administrative response to anomalies

Anomaly detection software has to achieve a balance between reporting non-problems and failing to report real issues. Too many false positives will push administrators toward ignoring alerts. The software will let the administrator set the threshold level, depending on how sensitive the network is and how many alerts are acceptable.

Alerts can come in several forms, including email, text messages, and automated phone calls. The type of alert will depend on the severity of the anomaly. The best software will provide a clear explanation of the issue, including the type of event and the risks it may pose. If an alert actually indicates legitimate activity, the administrator can mark it as not a problem, so that recurrences won't cause alerts. Admins need to be careful not to dismiss reported problems too hastily.

Defense in depth

Today's cyber threats keep changing, and they use techniques which are hard to detect. A successful defense against them needs to have multiple layers, so that when a threat gets past one defense, another is ready to catch and stop it. Anomaly detection is an essential part of this strategy. It can detect serious attacks which are in progress, as well as catching malware which has gotten a foothold in the network. A security strategy which works in depth will keep the network's uptime rate high and make the administrator's job easier.