Course

Syllabus

Low level assembly programming, identify common techniques and approaches for basic reverse engineering, disassemblers such as Ghidra/IDA and debugger aided debugging such as x32dbg and x64dbg, reverse engineering high level languages, identifying and defeating anti-disassembly techniques, anti-debugging techniques, code obfuscation. Windows PE file format overview, Windows API; COM overview, Malware persistence mechanisms (Registry by means of service, Trojans, DLL load order hijacking), user-mode rootkits, Privilege elevation mechanisms used by malwares, Malware execution(DLL injection, Process replacement, using Hooks and APC), Malware data encoding (common ciphers, custom encodings, Analyzing malwares. Behavioral Analysis of Advanced Malwares such as Ransomwares. Introduction to Symbolic Execution: Binary Analysis using Angr and Z3, AI-based approaches for Malware Detection: Supervised and Unsupervised Techniques, Deep Learning, Explainability

Text Books / References

1. Michael Sikorski and Andrew Honig, “PracticalMalware Analysis”, No Starch Press,2012
2. Bruce Dang, Alexandre Gazet, Elias Bachaalany and Sebastien Josse, Practical Reverse Engineering, First Edition, Wiley Publishers, 2014.
3. Eldad Eilam, Reversing: Secrets of Reverse Engineering, Wiley Publishers, 2005.

The course provides a comprehensive foundation in low-level programming, assembly language, executable file formats, debugging, and binary analysis techniques required for reverse engineering software applications. Students gain hands-on experience with industry-standard tools and frameworks for static and dynamic analysis of executable binaries and malware samples.

The course explores the design and behavior of contemporary malware, including trojans, ransomware, rootkits, and advanced persistent threats (APTs). It covers malware persistence mechanisms, privilege escalation techniques, process injection methods, anti-debugging and anti-disassembly strategies, code obfuscation, and packing techniques employed by adversaries to evade detection. Students learn to identify, analyze, and mitigate these threats through practical malware investigation and behavioral analysis exercises.

The course further introduces advanced topics such as symbolic execution, automated binary analysis, threat hunting using YARA rules, and Artificial Intelligence/Machine Learning-based approaches for malware detection and classification. Emphasis is placed on developing analytical and research skills necessary for cybersecurity professionals engaged in malware research, digital forensics, incident response, threat intelligence, and vulnerability analysis. Upon completion, students will be equipped to analyze complex software systems, investigate emerging cyber threats, and contribute to the development of advanced malware detection and defense solutions.

Course Objectives

• Understand the fundamentals of computer architecture, assembly language, and executable file formats relevant to reverse engineering.
• Learn static and dynamic analysis methodologies for malware investigation.
• Gain hands-on experience with industry-standard reverse engineering and debugging tools such as Ghidra, IDA Pro, x32dbg, x64dbg, and Angr.
• Analyze advanced malware techniques including persistence, privilege escalation, process injection, and anti-analysis mechanisms.
• Develop YARA rules and threat-hunting techniques for malware detection and classification.
• Explore AI/ML-based approaches for malware analysis, behavioral profiling, and automated threat detection.

Course Outcomes

• CO1: Understanding how to pick apart obfuscated systems systematically to understand their inner workings using reverse engineering techniques (PSO2)
• CO2: Learn how to detect malicious programs and classify them from benign programs and how malicious programs try to evade detection (PSO1, PSO2)
• CO3: Learn how to analyze and detect techniques used by malicious programs for activities such as persistence, data exfiltration etc. (PSO1, PSO3, PSO4)
• CO4: Understand how to analyze and defeat techniques used by programs such as anti-debugging and anti-disassembly to make their analysis (static/dynamic) harder (PSO4, PSO2, PSO1)
• CO5: Understand the application of AI for Malware Detection (PSO3, PSO4)

Prerequisites: Operating Systems, Programming, Data Structures and Algorithms, Computer Networking, Cryptography

CO-PO mapping:

Evaluation Pattern