Related papers Selected talks Source Code Related links People

Load Shedding in Network Monitoring Applications

What is CoMo?

CoMo is an open-source passive monitoring system that allows for fast implementation and deployment of network monitoring applications. CoMo follows a modular approach where users can easily define traffic queries as plug-in modules written in C, making use of a feature-rich API provided by the core platform.

Why is load shedding necessary?

A robust network monitoring system must cope inevitably with the effects of overload situations due to the large volumes, high data rates and bursty nature of the network traffic during its normal operation. Load shedding techniques reduce the load of a system when under severe stress, in the case of network monitoring in order to avoid uncontrolled packet loss. For example, this can be achieved by sampling the input traffic.

Our load shedding method

Without any initial knowledge of the plug-in modules, the load shedding scheme in CoMo infers the cost of each query from the relation between its actual resource usage and a pre-defined set of traffic features. A traffic feature is a counter that describes a specific property of the incoming traffic (e.g., number of packets, bytes, flows, unique IP destination addresses, etc.).

Figure 1: Prediction and load shedding subsystem

The prediction and load shedding subsystem (Figure 1) intercepts the packets from the filter before they are sent to the plug-in module implementing a traffic query. The system operates in four phases. First, it groups each 100ms of traffic in a “batch” of packets. Each batch is then processed to extract a large predefined set of traffic features. Then, the feature selection subsystem is in charge of selecting the most relevant ones according to the recent history of the query's CPU usage. This subset of relevant features is then given as input to the multiple linear regression subsystem to predict the CPU cycles required by the query to process the entire batch. If the prediction exceeds the system capacity, the load shedding subsystem pre-processes the batch to discard (via packet or flow sampling) a portion of the packets.

Related papers

  • "Robust Resource Allocation for Online Network Monitoring", Pere Barlet-Ros, Josep Sanjuàs-Cuxart, Josep Solé-Pareta and Gianluca Iannaccone. Accepted for publication in the 4th International Workshop on QoS in Multiservice IP Networks (QoSIP), 2008.

  • "Load shedding in network monitoring applications", Pere Barlet-Ros, Gianluca Iannaccone, Josep Sanjuàs-Cuxart, Diego Amores-López and Josep Solé-Pareta in Proc. of USENIX Annual Technical Conference, 2007.
    [PDF] [Slides] [PDF Publisher] [html] [Bibtex]

  • "Resource Usage Modeling for Network Monitoring Applications", Josep Sanjuàs-Cuxart and Pere Barlet-Ros in First Workshop on Execution Environments for Distributed Computing, 2007.
    [PDF] [Slides] [PDF Publisher] [Bibtex]

  • "On-line Predictive Load Shedding for Network Monitoring", Pere Barlet-Ros, Diego Amores-López, Gianluca Iannaccone, Josep Sanjuàs-Cuxart and Josep Solé-Pareta in IFIP Networking, 2007.
    [PDF] [Slides] [PDF Publisher] [Bibtex]

  • "Predicting Resource Usage of Arbitrary Network Traffic Queries", Pere Barlet-Ros , Gianluca Iannaccone , Josep Sanjuàs-Cuxart, Diego Amores-López and Josep Solé-Pareta. Technical report, Technical University of Catalonia, 2006.
    [PDF] [Bibtex]

Selected talks

Source code

  • Modified version of CoMo 0.5 with load shedding [source code]
  • Last version of CoMo with load shedding [SourceForge]

Related links

Both CoMo live and Appmon show graphical displays from the Technical University of Catalonia (UPC) access link:

  • CoMoLive
  • Shows graphical displays from CoMo.

  • Appmon
  • Per-application network traffic characterization tool.

CoMo websites:


Former collaborators

  • Diego Amores Lopez, UPC, Barcelona (Spain)

Related papers Selected talks Source Code Related links People