With the huge increase in the volume of network traffic, there is a need for network monitoring systems that capture network packets and provide packet features in near real time to protect from attacks. As a first step towards developing such a system using distributed computation, new system has been developed in Spark, a cluster computing system, which extracts packet features with less memory consumption and at a faster rate. Traffic analysis and extraction of packet features are carried out using streaming capability inherent in Spark. Analysing the network data features provide a means for detecting attacks. This paper describes a system for the analysis of network data using Spark streaming technology which focuses on real time stream processing, built on top of Spark.
K. V. Swetha, Shiju Sathyadevan, and Bilna, P., “Network Data Analysis Using Spark”, in Software Engineering in Intelligent Systems: Proceedings of the 4th Computer Science On-line Conference 2015 (CSOC2015), Vol 3: Software Engineering in Intelligent Systems, Cham, 2015, pp. 253–259.