Malware Analysis Advanced

Summary

Advanced malware analysis is the discipline of unpacking, reverse engineering, and understanding modern packed/obfuscated malware (UPX, VMProtect, Themida, Enigma, custom packers), identifying sandbox-evasion techniques (anti-VM, anti-debug, anti-analysis), analyzing rootkits (user-mode, kernel-mode, bootkits, UEFI), and authoring detection rules (YARA). This domain covers full unpacking workflows (static + dynamic), modern threat actor tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil), YARA rule authoring and optimization, and industry-standard analyst tooling (IDA Pro, Ghidra, Binary Ninja, radare2). Distinct from foundational binary-reverse — this skill focuses on dynamic unpacking, evasion triage, rootkit techniques, and analyst workflow automation.

Key Terms

• Packer — Tool that compresses + encrypts executable to evade AV (UPX, VMProtect, Themida)
• Unpacking — Recovering original executable from packed binary
• Anti-VM — Code that detects virtual machine (sandbox evasion)
• Anti-debug — Code that detects debugger presence (analyst evasion)
• Anti-analysis — Umbrella term for AV/VM/debug/sandbox evasion
• Rootkit — Tool that hides processes / files / network connections
• User-mode rootkit — Rootkit running in ring 3 (DLL injection, API hooking)
• Kernel-mode rootkit — Rootkit running in ring 0 (driver, syscall hooking)
• Bootkit — Rootkit that infects bootloader (MBR / VBR)
• UEFI rootkit — Rootkit that infects UEFI firmware (persistent across reinstall)
• YARA — Pattern-matching tool for malware identification
• PE — Portable Executable format (Windows binaries)
• ELF — Executable and Linkable Format (Linux binaries)
• Triage — Initial malware assessment (severity, family, capability)
• Sandbox — Isolated analysis environment (Cuckoo, JoeSandbox, Any.Run)

Scope

This skill covers advanced malware analysis:

• Unpacking modern packers (UPX, VMProtect, Themida, Enigma, custom)
• Sandbox-evasion detection (anti-VM, anti-debug, anti-analysis)
• Rootkit analysis (user-mode, kernel-mode, bootkits, UEFI)
• YARA rule authoring and optimization
• IDA Pro / Ghidra / Binary Ninja workflows
• Modern threat tooling (Emotet, TrickBot, Conti, LockBit, BlackCat/ALPHV, REvil)
• Automated triage pipelines

Out of scope: foundational RE (see binary-reverse), exploit development (see exploit-development), reverse engineering theory (see reverse-engineering-advanced).

Use Cases

• Unpacking packed malware: Recover original code from VMProtect/Themida binaries
• Sandbox evasion triage: Identify anti-VM/anti-debug techniques
• Rootkit detection: Find user-mode/kernel-mode rootkit techniques
• YARA rule authoring: Detect malware families by signature
• Threat actor tooling analysis: Reverse engineer Emotet, TrickBot, Conti, LockBit, BlackCat, REvil
• Automated triage pipeline: Build Cuckoo/JoeSandbox pipeline for mass triage
• Detection rule optimization: Tune YARA rules for performance
• Memory forensics: Use Volatility for in-memory malware analysis
• UEFI analysis: Identify UEFI rootkits (LoJax, MosaicRegressor)
• API hook detection: Find user-mode rootkit hooks

Core Tools

Tool	Purpose
IDA Pro	Industry-standard disassembler + decompiler
Ghidra	Open-source RE tool (NSA)
Binary Ninja	Modern disassembler + decompiler
radare2	Open-source disassembler
x64dbg	Windows dynamic debugger
WinDbg	Windows kernel debugger
gdb	GNU Debugger (Linux)
yara	Pattern-matching tool for malware detection
volatility3	Memory forensics framework
upx	UPX unpacker
vmprotect-devirt	VMProtect devirtualization
pe-sieve	PE artifact scanner (process memory)
hollows-hunter	Process hollowing detector
procdot	Visual malware analysis
process-hacker	Process explorer (Windows)
autoruns	Autostart entry scanner
pcap-ng-tools	Network capture analysis
zeek	Network behavior analyzer
suricata	IDS / IPS for malware traffic
pe-tree	Visual PE analysis
imhex	Modern hex editor

Methodology

Phase 1 — Static triage

# File hash
sha256sum malware.exe

# File type
file malware.exe

# PE analysis
pe-tree malware.exe

# Strings
strings -a malware.exe | grep -iE "http|dll|reg|cmd"

# Section entropy (packed indicator)
python3 -c "
import pefile
pe = pefile.PE('malware.exe')
for section in pe.sections:
    print(f'{section.Name.decode().strip():12s} entropy={section.get_entropy():.2f}')
"

Phase 2 — Unpacking

# UPX
upx -d malware.exe -o malware_unpacked.exe

# VMProtect / Themida / custom
# Use x64dbg + Scylla to dump from memory
# 1. Load in x64dbg
# 2. Set breakpoint on OEP (original entry point)
# 3. Run until OEP hit
# 4. Use Scylla to dump process memory
# 5. Fix IAT (Import Address Table)

Phase 3 — Dynamic analysis

# Run in sandbox (Cuckoo)
cuckoo submit malware.exe

# Manual analysis with x64dbg
# 1. Load binary
# 2. Set breakpoints on WinAPI calls (CreateFile, WriteFile, etc.)
# 3. Run, observe behavior
# 4. Capture network traffic (Wireshark)

# Volatility memory analysis
volatility -f memory.dmp windows.pslist
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind

Phase 4 — Sandbox evasion detection

# Static anti-VM strings
strings malware.exe | grep -iE "vmware|virtualbox|qemu|hyper-v|xen"
strings malware.exe | grep -iE "vbox|vmware tools|prl_"

# Anti-debug APIs
strings malware.exe | grep -iE "IsDebuggerPresent|CheckRemoteDebuggerPresent|NtQueryInformationProcess"

# Dynamic API tracing
# Use API monitor to capture all API calls

Phase 5 — Rootkit analysis

# User-mode rootkit detection
pe-sieve /pid 1234 /imp 3
hollows-hunter /pid 1234

# Kernel-mode rootkit detection
# Use WinDbg kernel mode
# List loaded drivers: lm t n
# Find hooked syscalls: !ssd

# Bootkit detection
# Check MBR / VBR / UEFI variables
bcdedit /enum firmware

Phase 6 — YARA rule authoring

rule Emotet_Loader_v4 {
    meta:
        author = "redteam"
        date = "2026-06-28"
        description = "Emotet v4 loader"
        reference = "https://attack.mitre.org/software/S0679/"
    strings:
        $s1 = "emotet" wide ascii nocase
        $s2 = { 6A 40 68 00 30 00 00 6A 14 8D 91 }
        $s3 = "%u%.4x" wide ascii
        $api1 = "CryptStringToBinaryA" wide
        $api2 = "InternetOpenA" wide
    condition:
        uint16(0) == 0x5A4D and
        3 of ($s*) and
        2 of ($api*)
}

Phase 7 — IDA Pro workflow

# IDA Python script: find anti-debug calls
import idautils, idc

for func_ea in idautils.Functions():
    name = idc.get_func_name(func_ea)
    if name in ["IsDebuggerPresent", "CheckRemoteDebuggerPresent"]:
        print(f"Anti-debug: {name} at {hex(func_ea)}")

# Decompile function
import ida_hexrays
cf = ida_hexrays.decompile(func_ea)
print(cf)

Phase 8 — Ghidra workflow

# Ghidra Python: find suspicious imports
from ghidra.program.model.symbol import SymbolType

sm = currentProgram.getSymbolTable()
for sym in sm.getAllSymbols(True):
    if sym.getSymbolType() == SymbolType.FUNCTION:
        name = sym.getName()
        if "VirtualProtect" in name or "WriteProcessMemory" in name:
            print(f"Inject: {name} at {sym.getAddress()}")

Phase 9 — Memory forensics (Volatility)

# Process listing
volatility -f memory.dmp windows.pslist

# Network connections
volatility -f memory.dmp windows.netscan

# Injected code detection
volatility -f memory.dmp windows.malfind

# DLL list per process
volatility -f memory.dmp windows.dlllist --pid 1234

# Kernel driver listing
volatility -f memory.dmp windows.modscan

Phase 10 — Reporting

Produce malware analysis report:

• Family + variant
• IOCs (hashes, domains, IPs, mutexes)
• TTP mapping (MITRE ATT&CK)
• YARA rules
• Detection recommendations

Practical Steps

Step 1 — Triage new sample

# Hash
sha256sum malware.exe > hash.txt

# VT lookup
curl -s "https://www.virustotal.com/api/v3/files/$(sha256sum malware.exe | cut -d' ' -f1)" \
  -H "x-apikey: $VT_KEY" | jq .

# PE analysis
pe-tree malware.exe

Step 2 — Unpack UPX sample

upx -d malware_packed.exe -o malware_unpacked.exe

# Verify
sha256sum malware_unpacked.exe
file malware_unpacked.exe
strings -a malware_unpacked.exe | grep -iE "http|dll"

Step 3 — Unpack VMProtect sample

# In x64dbg:
# 1. Load binary
# 2. Set memory breakpoint on .vmp section execution
# 3. Run until breakpoint
# 4. Step until OEP (look for typical MSVC entry point pattern)
# 5. Use Scylla plugin:
#    - Select process
#    - Click "IAT AutoSearch"
#    - Click "Get Imports"
#    - Click "Dump" → save unpacked.exe
#    - Click "Fix Dump" → fix IAT

Step 4 — Identify anti-VM

# Static
strings malware.exe | grep -iE "vmware|virtualbox|qemu"
strings malware.exe | grep -iE "vmware tools|vbox guest additions"

# Registry keys
strings malware.exe | grep -iE "SYSTEM\\\\CurrentControlSet\\\\Services\\\\VBoxGuest"

Step 5 — YARA rule authoring

rule BlackCat_ALPHV_Ransomware {
    meta:
        author = "redteam"
        description = "BlackCat/ALPHV Rust-based ransomware"
        reference = "https://attack.mitre.org/software/S1068/"
    strings:
        $rust = "rust_panic" wide ascii
        $s1 = "BlackCat" wide ascii nocase
        $s2 = "{ 52 75 73 74 }" // "Rust" in hex
        $api1 = "CryptEncrypt" wide
        $api2 = "BCryptEncrypt" wide
    condition:
        uint16(0) == 0x5A4D and
        $rust and
        any of ($s*) and
        any of ($api*)
}

Step 6 — Memory forensics

volatility -f memory.dmp windows.pslist | grep -v "Microsoft\|Windows"
volatility -f memory.dmp windows.netscan
volatility -f memory.dmp windows.malfind --pid 1234

Step 7 — Rootkit detection

# User-mode hook detection
pe-sieve /pid 1234 /imp 3

# Hollowed process detection
hollows-hunter /pid 1234

# Autoruns (autostart persistence)
autoruns -accepteula -a autostart.arn

Defense Perspective

Defenders must assume:

1. Packed malware evades AV signature — unpacking + behavioral detection required
2. Sandbox evasion defeats dynamic analysis — anti-VM must be bypassed
3. Rootkits hide in kernel — kernel-mode detection (PatchGuard, EDR) required
4. UEFI rootkits persist across reinstall — firmware scanning required
5. YARA rules need constant tuning — false positives / false negatives
6. Memory forensics catches fileless malware — Volatility essential
7. Threat actor tooling evolves rapidly — analyst workflow automation needed
8. Malware uses LOLBins — signed binaries (certutil, bitsadmin) bypass allowlist

Key defensive controls:

• Behavior-based detection (EDR / XDR)
• Memory scanning (pe-sieve, hollows-hunter)
• YARA scanning at egress + endpoint
• Volatility memory forensics for IR
• Application allowlisting (AppLocker, WDAC)
• Kernel-mode protection (PatchGuard)
• UEFI Secure Boot
• Behavioral baseline for processes

Packer Triage Cheat Sheet

Packer	Detection	Unpacking difficulty
UPX	Section ".UPX0/.UPX1"	Easy (upx -d)
ASPack	Section ".aspack"	Medium (manual)
Themida	Section ".Themida"	Hard (WinDbg)
VMProtect	Section ".vmp0/.vmp1"	Very Hard (devirt)
Enigma	Section ".enigma1/.enigma2"	Hard
Custom	High entropy + obfuscation	Very Hard

Sandbox Evasion Techniques

Technique	Detection	Bypass
CPUID VM bit	Static strings	Patch CPUID
Registry VM keys	Strings (VMware, VBox)	Registry scrub
MAC address OUI	Network adapter	Spoof MAC
Process count	Psapi enumeration	Inject extra processes
Sleep + check	Timing analysis	Hook sleep
Mouse movement	Cursor position	Virtual mouse
Disk size	<60GB = VM	Larger VMDK
Recent files	User profile age	Pre-populate

Rootkit Categories

Type	Ring	Persistence	Example
User-mode	3	Registry	Hacker Defender
Kernel-mode	0	Driver	Rustock
Bootkit	0	MBR/VBR	TDL4
UEFI	-1	Firmware	LoJax

Threat Actor Tooling

Family	Type	Packer	Notable Techniques
Emotet	Loader	Custom	Macro dropper, polymorphic
TrickBot	Banking trojan	Custom	Process hollowing, anti-VM
Conti	Ransomware	Custom	LockBit-shared code, Rclone exfil
LockBit 3.0	Ransomware	Custom	StealBit exfil, customizable
BlackCat/ALPHV	Ransomware	Rust	MEGA exfil, cross-platform
REvil	Ransomware	Custom	Affiliate program, onion leak

Engagement Workflow

1. Triage — hash, file type, PE analysis, VT lookup
2. Static analysis — strings, section entropy, import analysis
3. Unpacking — UPX/manual/VMProtect devirt
4. Dynamic analysis — sandbox + manual x64dbg
5. Evasion triage — anti-VM, anti-debug, anti-analysis
6. Rootkit detection — user/kernel/boot/UEFI
7. YARA authoring — detection rules
8. Reporting — IOCs, TTPs, detection recommendations

Lab Setup

# Cuckoo sandbox
git clone https://github.com/cuckoosandbox/cuckoo
cd cuckoo && python3 setup.py install

# Ghidra
wget https://github.com/NationalSecurityAgency/ghidra/releases/download/Ghidra_11.0_build/ghidra_11.0_PUBLIC_20231222.zip
unzip ghidra_11.0_PUBLIC_20231222.zip

# Volatility
pip install volatility3

# YARA
pip install yara-python

Quality Checklist

• File hash + VT lookup
• PE analysis complete
• Packer identified
• Sample unpacked
• Sandbox evasion identified
• Rootkit techniques analyzed
• YARA rule authored
• IOCs documented
• TTP mapping complete
• Final report delivered

Detection Methods

Static Analysis Detection

• AV signatures: YARA rules, ClamAV signatures match known malware families.
• Entropy analysis: PE sections with entropy > 7.0 (packed/encrypted).
• Import table anomalies: Missing imports (LoadLibrary, GetProcAddress); imports via hash resolution.
• PE structure anomalies: Imports via only LoadLibraryA/GetProcAddress; signature of dynamic resolution.

Dynamic Analysis Detection

• Sandbox detonation: Cuckoo, Joe Sandbox; behavioral signatures.
• API call sequences: Mimikatz signature (OpenProcess + ReadProcessMemory + WriteProcessMemory on lsass).
• Network anomalies: Connections to known C2 infrastructure (Cobalt Strike teamserver default ports).

SIEM Detection Rules

• Splunk SPL: index=malware sourcetype=yara | where rule matches "mimikatz/*"
• YARA rules: Continuous scanning of filesystem for known signatures.
• AMSI integration: Scan PowerShell, VBA, JavaScript content via AMSI.

Defense Evasion Techniques

Anti-Analysis

• Anti-debugging: IsDebuggerPresent, ntdll!KdUserExceptionDispatcher check, timing checks (rdtsc).
• Anti-VM: MAC address check (VMware 00:50:56), CPUID hypervisor bit, registry artifacts.
• Anti-sandbox: Mouse movement check (real users have jitter), recent documents check, uptime check.
• Anti-AV: Process enumeration looking for av processes; exit if found.

Code Obfuscation

• Packing: UPX, ASPack, Themida, VMProtect; detect via entropy.
• Polymorphic code: Decryptor changes; payload signature constant.
• Metamorphic code: Body rewritten each generation; no static signature.
• Control flow flattening: Switch dispatcher; defeats static analysis.
• Junk code insertion: No-op instructions between real instructions.
• String encryption: Encrypt sensitive strings; decrypt at runtime only.

Memory-Resident Evasion

• Reflective DLL injection: Load DLL from memory; no file artifacts.
• Process hollowing: Replace legitimate process memory; appears as legitimate process.
• Module stomping: Load legitimate DLL, overwrite; inherits module legitimacy.
• Phantom DLL hollowing: Hollow rarely-used DLL; less attention.

Modern AV/EDR Bypass

• AMSI bypass: Patch amsi.dll!AmsiScanBuffer in-memory.
• ETW bypass: Patch ntdll!EtwEventWrite in-memory.
• Direct syscalls: Bypass user-mode hooks (SysWhispers, HellsGate).
• BYOVD: Load vulnerable signed driver for kernel R/W.

References

• MITRE ATT&CK Defense Evasion — https://attack.mitre.org/tactics/TA0005/
• "Practical Malware Analysis" (Sikorski, Honig, 2012)
• "Malware Analyst's Cookbook" (Ligh, Adair, Hartstein, 2010)
• "Learning Malware Analysis" (Monnappa Ka, 2018)
• "The Art of Memory Forensics" (Hale Ligh, 2014)
• IDA Pro Book (Chris Eagle, 2nd Edition)
• Ghidra documentation — https://ghidra-sre.org/
• Volatility documentation — https://volatility3.readthedocs.io/
• YARA documentation — https://yara.readthedocs.io/
• CISA AA21-148A — DarkSide/Conti analysis
• Mandiant APT1 / APT41 reports
• CrowdStrike 2024 Global Threat Report
• MalwareBazaar — https://bazaar.abuse.ch/
• VirusTotal — https://www.virustotal.com/
• Any.Run sandbox — https://app.any.run/
• JoeSandbox — https://www.joesandbox.com/
• "VMProtect Devirtualization" (BlackHat 2023)
• "Rootkit Arsenal" (Blunden, 3rd Edition)
• "UEFI Rootkits" (ESET LoJax report 2018)