Course

Foundations of Generative AI and Secure AI Systems: Introduction to Generative AI, Foundation Models, Transformer Architecture, Self-Attention Mechanism, Embeddings and Vector Representations, Large Language Models (LLMs), Prompt Engineering, Fine-Tuning Techniques, Parameter-Efficient Fine-Tuning (PEFT), LoRA, Retrieval-Augmented Generation (RAG), Vector Databases, AI Agents and Autonomous Systems, Internet of Agents, Model Context Protocol (MCP), Secure AI Development Lifecycle, Model Hardening, Identity, Traceability and Secure Communication for AI Agents, Applications of Generative AI in Cyber Security.

AI Security and Trustworthy AI: AI Threat Landscape, AI Attack Surface, Security, Privacy and Trustworthiness in AI, AI Risk Management, Threat Modelling for AI Systems, Adversarial Machine Learning, Responsible AI, AI Governance and Ethics, Robust Machine Learning, Adversarial Training, Defensive Techniques, Secure Data Pipelines, Privacy-Preserving Machine Learning, Differential Privacy, Federated Learning Security, Secure Multi-Party Computation, Homomorphic Encryption, Explainable AI, Adversarial Attacks, Privacy Attacks, Model Extraction and Inference Attacks, MITRE ATLAS Framework, AI Supply Chain Security, Model Fingerprinting, Machine Unlearning, Model Provenance and Lineage, AI Bill of Materials (AI-BoM), Security of Deep Learning Models, AI Assurance and Case Studies in AI Security.

Security of Foundation Models and Large Language Models: Foundation Model Security, Large Language Model Security, Prompt Injection, Jailbreaking, Prompt Leakage, Hallucinations, Data Leakage, Model Theft, Security of Fine-Tuning, Security of Retrieval-Augmented Generation (RAG), Security of AI Agents and Autonomous Systems, Offensive AI for Security Evaluation, AI Hardening and Defensive Prompting, Input Validation, Output Guardrails, AI-assisted Security Operations, AI Security Frameworks and Standards, Secure AI Framework (SAIF), ISO/IEC 42001, NIST AI Risk Management Framework, EU AI Act, AI Compliance and Governance.

Prerequisites: 26CY731 – AI for Cyber Security Applications

Course Outcome

CO-PO Mapping

Text Books

1. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016.
2. Christopher M. Bishop and Hugh Bishop, Deep Learning: Foundations and Concepts, Springer, 2024.
3. Yevgeniy Vorobeychik and Murat Kantarcioglu, Adversarial Machine Learning, Morgan & Claypool Publishers, 2018.
4. Chip Huyen, AI Engineering: Building Applications with Foundation Models, O’Reilly Media, 2025.
5. Battista Biggio and Fabio Roli, Adversarial Machine Learning, Springer.

References

1. NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), National Institute of Standards and Technology, 2023.
2. OWASP Foundation, OWASP Top 10 for Large Language Model Applications.
3. Jay Alammar and Maarten Grootendorst, Hands-On Large Language Models, O’Reilly Media, 2024.
4. Nicolas Papernot et al., Practical Black-Box Attacks Against Machine Learning, ACM AsiaCCS.
5. OpenAI, Anthropic, Google DeepMind and Meta AI Technical Reports on Foundation Model and LLM Security.
6. MITRE, MITRE ATLAS – Adversarial Threat Landscape for AI Systems.
7. ISO/IEC 42001:2023, AI Management System.
8. Google, Secure AI Framework (SAIF).