《一种基于双模式虚拟机的多态Shellcode检测方法.》.pdfVIP

计算机研 究与发展 7544／issn1OOO一1239．2014 JournalofComputerResearchandDevelopment 51(8)：1704—1714，2O14 一 种基于双模式虚拟机的多态 Shellcode检测方法 罗 杨 夏春和 李亚卓 魏 昭 梁晓艳 (北京航空航天大学网络技术北京市重点实验室 北京 100191) (veotax@sac．buaa．edu．en) A PolymorphicShellcodeDetectionM ethodBasedonDual-ModeVirtualM achine LuoYang，XiaChunhe，LiYazhuo，W eiZhao，andLiangXiaoyan (KeyLaboratoryofBeijingNetworkTechnology，BeihangUniversity，Beijing100191) Abstract M aliciouscode such asvirusand worm propagatesand attacksvia the Internetwhich compromisesnetwork securities seriously． By exploiting vulnerabilitiesof services running on a remotehost，themaliciouscodecaninjectshelleodeintothehostandgaincompletecontro1overit． Recently，shellcodeattackstendto evadenetwork detection by employingpolymorphictechniques． However，previousdetectionmethodslackthereal—timeperformance，resilienceagainstevasionsand the consideration ofdistinguishing polymorphic shellcode from normalshellprotected code． W e proposeanenhanced polymorphicshellcodedetection approach based on dual—modevirtualmachine． W efirstly filternetwork trafficvia an improvedGetPC locationmechanism and IA一32 instruction recognition method，subsequently implementa dual—mode virtualmachine to execute the target instructionflow bytransferring statesofthefiniteautomaton between control——flow modeand data—— flow modeaccordingtovariousdecisionconditions．A prototypesystem hasbeenimplementedforour approachand experimentalresultsindicate thatwhen detecting realnetwork data m ixed with the polymorphicshellcodesamples，theapproachweproposedsucceedsindifferentiatingthepolymorphic shellcodefrom shellprotected software without losing itsaccuracy and itstimecomplexity lies between the staticanalysis and dynamic emulation，which makes itan encouraging method for networkattack detectjon． Keywords polymorphieshellcode；OetPC location；ins