Load Shedding in Network Monitoring Applications
What is CoMo?
is an open-source passive monitoring system that allows for fast
implementation and deployment of network monitoring applications. CoMo
follows a modular approach where users can easily define traffic queries
as plug-in modules written in C, making use of a feature-rich API provided
by the core platform.
Why is load shedding necessary?
A robust network monitoring system
must cope inevitably with the effects of overload situations due to the
large volumes, high data rates and bursty nature of the network traffic
during its normal operation.
Load shedding techniques reduce the load of a system when under severe
stress, in the case of network monitoring in order to avoid uncontrolled
packet loss. For example, this can be achieved by sampling the input traffic.
Our load shedding method
Without any initial knowledge of the plug-in modules, the load shedding scheme in
infers the cost of each query from the relation between its actual
resource usage and a pre-defined set of traffic features. A traffic feature
is a counter that describes a specific property of the incoming traffic
(e.g., number of packets, bytes, flows, unique IP destination addresses, etc.).
Figure 1: Prediction and load shedding subsystem
The prediction and load shedding subsystem (Figure 1) intercepts the packets
from the filter before they are sent to the plug-in module implementing a
traffic query. The system operates in four phases. First, it groups each 100ms
of traffic in a “batch” of packets. Each batch is then processed to extract a
large predefined set of traffic features. Then, the feature selection subsystem is
in charge of selecting the most relevant ones according to the recent
history of the query's CPU usage. This subset of relevant features is then
given as input to the multiple linear regression subsystem to predict the CPU
cycles required by the query to process the entire batch. If the prediction
exceeds the system capacity, the load shedding subsystem pre-processes the batch
to discard (via packet or flow sampling) a portion of the packets.
Robust Network Monitoring in the presence of Non-Cooperative Traffic Queries.
Pere Barlet-Ros, Gianluca Iannaccone, Josep Sanjuàs-Cuxart and Josep Solé-Pareta.
Computer Networks, Oct. 2008 (in press).
Robust Resource Allocation for Online Network Monitoring. Pere
Barlet-Ros, Josep Sanjuàs-Cuxart, Josep Solé-Pareta and Gianluca
Iannaccone. In Proc. of Intl. Workshop on QoS in Multiservice IP Networks (IT-NEWS/QoS-IP),
Venice, Italy, Feb. 2008.
Load Shedding in Network Monitoring Applications. Pere Barlet-Ros,
Gianluca Iannaccone, Josep Sanjuàs-Cuxart, Diego Amores-López and Josep Solé-Pareta.
In Proc. of USENIX Annual Technical Conference, Santa Clara, USA, June 2007.
Resource Usage Modeling for Network Monitoring Applications.
Josep Sanjuàs-Cuxart and Pere Barlet-Ros. In Proc. of Workshop on Execution
Environments for Distributed Computing (EEDC), Barcelona, Spain, June 2007.
On-line Predictive Load Shedding for Network Monitoring. Pere Barlet-Ros, Diego
Amores-López, Gianluca Iannaccone, Josep Sanjuàs-Cuxart and Josep Solé-Pareta.
In Proc. of IFIP-TC6 Networking, Atlanta, USA, May 2007.
Predicting Resource Usage of Arbitrary Network Traffic Queries, Pere Barlet-Ros , Gianluca
Iannaccone , Josep Sanjuàs-Cuxart, Diego Amores-López and Josep Solé-Pareta.
Technical report UPC-DAC-RR-2008-29, Technical University of Catalonia, Dec. 2006.
Modified version of CoMo 0.5 with load shedding [source code]
Last version of CoMo with load shedding [SourceForge]
Both CoMo live and Appmon show graphical displays from the Technical
University of Catalonia (UPC) access link:
Shows graphical displays from CoMo.
Per-application network traffic characterization tool.
Pere Barlet Ros, UPC, Barcelona (Spain)
Intel Research, Berkeley (USA)
Ignasi Paredes Oliva, UPC, Barcelona (Spain)
Valentín Carela Español, UPC, Barcelona (Spain)
Josep Sanjuàs Cuxart, UPC, Barcelona (Spain)
- Josep Solé Pareta, UPC, Barcelona (Spain)
Diego Amores López, UPC, Barcelona (Spain)