x64Unpack: Hybrid Emulation Unpacker for 64-bit Windows Environments and Detailed Analysis Results on VMProtect 3.4

• Published in: IEEE access Vol. 8; pp. 127939 - 127953
• Main Authors: Choi, Seokwoo, Chang, Taejoo, Kim, Changhyun, Park, Yongsu
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Anti-forensics; Application programming interface; Binary codes; code obfuscation; Computer programming; computer security; Cryptography; Debugging; dynamic code analysis; Emulation; Encryption; Malware; Microsoft Windows; Performance analysis; Registers; Reverse engineering; Virtual memory systems
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2020.3008900

In spite of recent remarkable advances in binary code analysis, malware developers are still using complex anti-reversing techniques to make analysis difficult. To protect malware, they use packers, which are (commercial) tools that contain various anti-reverse engineering techniques such as code encryption, anti-debugging, and code virtualization. In this paper, we present x64Unpack: a hybrid emulation scheme that makes it easier to analyze packed executable files and automatically unpacks them in 64-bit Windows environments. The most distinguishable feature of x64Unpack compared to other dynamic analysis tools is that x64Unpack and the target program share virtual memory to support both instruction emulation and direct execution. Emulation runs slow but provides detailed information, whereas direct execution of the code chunk runs very fast and can handle complex cases regarding to operating systems or hardware devices. With x64Unpack, we can monitor major API (Application Programming Interface) function calls or conduct fine-grained analysis at the instruction-level. Furthermore, x64Unpack can detect anti-debugging code chunks, dump memory, and unpack the packed files. To verify the effectiveness of x64Unpack, experiments were conducted on the obfuscation tools: UPX 3.95, MPRESS 2.19, Themida 2.4.6, and VMProtect 3.4. Especially, VMProtect and Themida are considered as some of the most complex commercial packers in 64-bit Windows environments. Experimental results show that x64Unpack correctly emulates the packed executable files and successfully produces the unpacked version. Based on this, we provide the detailed analysis results on the obfuscated executable file that was generated by VMProtect 3.4.