5th WCSET-2016 at Vietnam
Technical Session - 4
Title:
An Experimental Study on Identifying Obfuscation
Techniques in Packer
Authors:
Nguyen Minh Hai, Quan Thanh Tho
Abstract:
Malware is one of the most important problems in
computer security. There are two main approaches for
detecting malware, signature matching and virtual
emulation. Signature is a typical bit pattern, which
characterizes malwares. Most of industrial malware
detection methods depend on regular expression based
signature recognition. Virtual emulation prepares a
sandbox to explore behaviour of malwares, which requires
a deep encoding of system environments to emulate
windows APIs [1]. However, emulation requires finding a
suitable abstraction level which is very heavy task.
Moreover, these techniques are easily defeated by the
obfuscation techniques, e.g. indirect jump,
self-modifying code, Structured Exception Handling (SEH)
and many other techniques which are adopted in packer.
In fact, most of modern malware use packers for creating
a new variant which cheats the antivirus software,
According to a report of Semantic Lab [2], nearly 80% of
malware are packed by packer. This paper targets on the
problem of identifying the obfuscation techniques which
are adopted in some well-known packers. It proposes an
experimental study of obfuscation techniques which are
used in 7 popular packers which include UPX, FSG, NPACK,
ASPACK, PECOMPAT, PETITE, and YODA. We develop our
pushdown model generation of malware, BE-PUM as a
generic unpacker tool by implementing the
anti-anti-analysis techniques against the obfuscation
techniques in these packers. During the on-the-fly
disassembly, BE-PUM observes and measure the frequency
of obfuscation techniques adopted in packers. We have
performed the experiments in 8 packers using BE-PUM and
achieved very promising results.
Keywords: Concolic Testing, Pushdown
System, Malware Detection, Binary Code Analysis,
Self-Modifying Code, Packer Identification, Obfuscation
Technique
Pages:
201-205