Context Sensitive Anomaly Monitoring of Process Control Flow to Detect Mimicry ￡ Attacks a.pdf

Context Sensitive Anomaly Monitoring of Process Control Flow to Detect Mimicry Attacks and Impossible Paths Haizhi Xu, Wenliang Du, and Steve J. Chapin Systems Assurance Institute, Syracuse University, Syracuse NY 13244, USA
hxu02, wedu, chapin  @ Abstract. Many intrusions amplify rights or circumvent defenses by issuing sys- tem calls in ways that the original process did not. Defense against these attacks emphasizes preventing attacking code from being introduced to the system and detecting or preventing execution of the injected code. Another approach, where this paper fits in, is to assume that both injection and execution have occurred, and to detect and prevent the executing code from subverting the target system. We propose a method using waypoints: marks along the normal execution path that a process must follow to successfully access operating system services. Way- points actively log trustworthy context information as the program executes, al- lowing our anomaly monitor to both monitor control flow and restrict system call permissions to conform to the legitimate needs of application functions. We de- scribe our design and implementation of waypoints and present results showing that waypoint-based anomaly monitors can detect a subset of mimicry attacks and impossible paths. Keywords: anomaly detection, context sensitive, waypoint, control flow monitoring, mimicry attacks, impossible paths 1 Introduction Common remote attacks on computer systems have exploited implementation errors to inject code into running processes. Buffer overflow attacks are the best-known example of this type of attacks. For years, people have been working on preventing, detecting, and tolerating these attacks [1–13]. Despite these efforts, current systems are not secure. Attackers frequently find new vulnerabilities and quickly develop adaptive methods that circumvent security mechanisms. Host-based defense can take place at one of three stages: preventing code injection, preve