Load Shedding in Network Monitoring Applications

What is CoMo?

CoMo is an open-source passive monitoring system that allows for fast implementation and deployment of network monitoring applications. CoMo follows a modular approach where users can easily define traffic queries as plug-in modules written in C, making use of a feature-rich API provided by the core platform.

Why is load shedding necessary?

A robust network monitoring system must cope inevitably with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic during its normal operation. Load shedding techniques reduce the load of a system when under severe stress, in the case of network monitoring in order to avoid uncontrolled packet loss. For example, this can be achieved by sampling the input traffic.

Our load shedding method

Without any initial knowledge of the plug-in modules, the load shedding scheme in CoMo infers the cost of each query from the relation between its actual resource usage and a pre-defined set of traffic features. A traffic feature is a counter that describes a specific property of the incoming traffic (e.g., number of packets, bytes, flows, unique IP destination addresses, etc.).

Figure 1: Prediction and load shedding subsystem

The prediction and load shedding subsystem (Figure 1) intercepts the packets from the filter before they are sent to the plug-in module implementing a traffic query. The system operates in four phases. First, it groups each 100ms of traffic in a “batch” of packets. Each batch is then processed to extract a large predefined set of traffic features. Then, the feature selection subsystem is in charge of selecting the most relevant ones according to the recent history of the query's CPU usage. This subset of relevant features is then given as input to the multiple linear regression subsystem to predict the CPU cycles required by the query to process the entire batch. If the prediction exceeds the system capacity, the load shedding subsystem pre-processes the batch to discard (via packet or flow sampling) a portion of the packets.

Selected talks

• Load Shedding in Network Monitoring Applications
  (Intel Research Berkeley, July 26th 2007)

• Load Shedding for Network Monitoring Systems
  (Barcelona, February 3rd 2006)

• Building CoMo
  (Intel Research Cambridge, September 10th 2004)

Source code

• Modified version of CoMo 0.5 with load shedding [source code]
• Last version of CoMo with load shedding [SourceForge]

People

• Pere Barlet Ros, UPC, Barcelona (Spain)
• Gianluca Iannaccone Intel Research, Berkeley (USA)
• Ignasi Paredes Oliva, UPC, Barcelona (Spain)
• Josep Sanjuàs Cuxart, UPC, Barcelona (Spain)
• Josep Solé Pareta, UPC Barcelona (Spain)

Former collaborators

• Diego Amores Lopez, UPC, Barcelona (Spain)