Course

Syllabus

Introduction to security for machine learning models – overview of machine learning and its applications, and common security challenges and vulnerabilities in ML models. Designing security architectures for ML – security design principles for ML models, implementing robust security frameworks for ML systems, and case studies of secure ML architectures.

Securing data and model training – techniques for securing training data and processes, data privacy and integrity in ML, and secure model training methodologies. Secure data pipelines for LLM fine-tuning and retrieval-augmented generation (RAG) systems, including data poisoning and provenance checks.

MITRE ATLAS framework – introduction to MITRE ATLAS, understanding the threat landscape for AI systems, and practical exercises using the ATLAS framework. Threat modeling for LLM and agentic AI applications using ATLAS and the OWASP Top 10 for Large Language Model Applications.

Advanced security techniques for ML – adversarial machine learning defenses, implementing differential privacy and secure multi-party computation, and monitoring and maintaining ML model security. LLM and generative AI security – prompt injection and jailbreak defenses, AI red teaming methodologies, and securing RAG pipelines against adversarial manipulation. AI model supply-chain integrity (model cards, provenance, and watermarking)

Text Books / References

• Machine Learning and Security: Protecting Systems with Data and Algorithms – Clarence Chio, David Freeman | O’Reilly
• Adversarial Machine Learning – Anthony D. Joseph, Blaine Nelson, Benjamin I. P. Rubinstein, J. D. Tygar | Cambridge University Press
• The Algorithmic Foundations of Differential Privacy – Cynthia Dwork, Aaron Roth | Foundations and Trends in Theoretical Computer Science
• Generative AI Security: Theories and Practices – Ken Huang et al. | Springer
• MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) – atlas.mitre.org
• OWASP Top 10 for Large Language Model Applications – owasp.org

This course provides a comprehensive foundation in designing, implementing, and securing machine learning systems against adversarial threats and vulnerabilities. It equips students with the knowledge and skills to identify security risks in ML pipelines, design robust security architectures, secure training data and model development processes, and apply structured threat-modeling frameworks such as MITRE ATLAS. The course also covers advanced defense techniques including adversarial robustness, differential privacy, and secure multi-party computation, while addressing emerging topics such as large language model (LLM) and generative AI security, AI red teaming and AI model supply-chain integrity

Course Objectives

• To understand the security challenges and vulnerabilities inherent to machine learning models and systems.
• To develop proficiency in designing and implementing secure architectures and robust security frameworks for ML systems.
• To apply techniques for securing training data, ensuring privacy and integrity throughout the model training process.
• To explore the MITRE ATLAS framework for understanding and mapping the AI/ML threat landscape.
• To implement advanced security techniques including adversarial defenses, differential privacy, and secure multi-party computation.
• To understand security considerations specific to large language models (LLMs) and generative AI systems, including prompt injection, jailbreaking and AI red teaming.

Course Outcomes

Prerequisites

• Machine Learning
• Network Security / Cryptography

CO-PO / PSO Mapping

Evaluation Pattern – 60:40

• Midterm Exam – 30%
• Viva – 10%
• Assignments & Case Study – 20%
• End Semester Exam – 40%