Course

Unit 1

Big Data Foundations and Architectures

Scaling with Traditional Databases, Limitations of RDBMS, Need for NoSQL, First Principles of Distributed Systems, Desired Properties of Big Data Systems, Lambda Architecture Overview, Kappa Architecture, Data Lakes and Cloud Storage Concepts.

Unit 2

Distributed Storage Systems

Big Data Model and Properties, Fact-Based Modeling, Graph Schemas, Distributed File Systems, Partitioning and Replication Strategies, Hadoop Distributed File System (HDFS), Cloud Object Storage Concepts (Amazon S3 and Azure Data Lake), Cassandra Basics

Unit 3

Batch Processing Systems

Scalability Concepts, MapReduce Programming Model (Overview), Apache Spark Architecture, Spark Core and Spark SQL, Batch Layer Architecture, Design Overview and Workflow, Data Ingestion Techniques, Data Cleaning and Normalization.

Unit 4

Serving Layer and Query Systems

Performance Metrics in Distributed Systems, Serving Layer Requirements and Design, Query Processing in Distributed Systems, Indexing Strategies, NoSQL Query Models, Distributed Data Access Patterns

Unit 5

Real-Time and Stream Processing Systems

Real-Time Views, Kafka Architecture (Producers, Consumers, Topics), Stream Processing Concepts, Spark Structured Streaming, Windowing and Stateful Processing, Fault Tolerance and Scalability in Streaming Systems, Case Studies of Modern Data Pipelines.

Text Books / References

Text Books:

1. Nathan Marz, James Warren, “Big Data: Principles and best practices of scalable real-time data systems”, Manning Publications 2015.

References

1. Kleppmann, M. (2017). Designing data-intensive applications. O’Reilly Media.
2. Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: The definitive guide. O’Reilly Media.
3. Carpenter, J., & Hewitt, E. (2016). Cassandra: The definitive guide: Distributed data at web scale (2nd ed.). O’Reilly Media.
4. Akidau, T., Chernyak, S., & Lax, R. (2018). Streaming systems. O’Reilly Media.

This course introduces scalable storage and processing architectures for Big Data systems. It covers batch, serving, and speed layers under the Lambda Architecture paradigm, along with Hadoop, Cassandra, MapReduce, Apache Storm, and distributed storage principles.

Course Outcomes: After successful completion of the course, students will be able to

• CO1: Explain the need for scalable distributed architectures and Lambda architecture principles.
• CO2: Analyze and design batch processing systems using Hadoop and MapReduce.
• CO3: Implement distributed storage solutions and apply partitioning strategies.
• CO4: Evaluate serving layer and NoSQL databases such as Cassandra.
• CO5: Design real-time stream processing systems using Apache Storm.

CO-PO Mapping: