Malware Analysis Basics — How Defenders Study Dangerous Code Safely

What you'll learn

By the end of this lesson you will be able to:

• Explain the legal and ethical frame of malware analysis and the handling rules defenders follow
• Describe a safe analysis lab: isolated virtual machines, snapshots, a simulated network
• Follow the defender's workflow: intake → static triage → behavior observation → documentation
• Extract, defang and share indicators of compromise (IOCs), and judge which ones hurt an attacker most
• Write simple YARA-style detection rules and compare before/after snapshots of a system
• Map observed behavior to MITRE ATT&CK and structure a short analysis report

The biosafety lab 🧫

Scientists who study a dangerous virus do not work on a kitchen table. They use a sealed laboratory, protective equipment, strict procedures and decontamination. Their goal is never to spread the virus: it is to understand it well enough to build tests and vaccines.

Malware analysis is the same discipline for software. Defenders in security operations centers, incident-response teams, CERTs and security vendors study malicious programs to answer three questions: What does it do? How do we detect it? How do we clean up and protect others? The output is knowledge: indicators, detection rules and reports. Building or spreading malware is never part of the job.

About the samples in this lesson: every "sample" here is a harmless text, byte dump, log or file list written for teaching. No real malware appears anywhere, and you never need real malware to learn these skills.

The rules before the lab ⚖️

Malware work sits close to the line, so professionals follow explicit rules. They build on the rules of the road from the first lesson.

Work only in an authorized context. You analyze a sample because it is part of your job (an incident at your organization, a vendor's research process) or because a course gave you harmless practice material. Downloading live malware "to see what happens" is how people infect their own networks.

Never create, modify or spread malware. In most countries, writing or distributing malicious code without authorization and with the intent that it be used to commit offences is a crime. For example, Article 6 of the Council of Europe's Budapest Convention on Cybercrime asks its parties to criminalize producing or distributing programs designed to commit offences when this is done intentionally and without right, and it excludes authorized testing and protection of computer systems. Analysis is about understanding and defending, never about improving an attack.

Handle samples like hazardous material.

• Keep them in password-protected archives (by long convention the password is infected), clearly labeled, never on shared drives or in email bodies
• Rename files with a harmless extension (.bin) so nobody double-clicks them
• Transfer them only into the isolated lab, and never back out: what leaves the lab is text, such as hashes, notes and reports
• Do not upload a confidential file to a public scanning service: files submitted to services such as VirusTotal are shared with the security community. Search by hash instead
• Defang indicators when you share them (hxxp://example[.]com), so nobody opens them by accident

Never "hack back". Even when a sample reveals the attacker's server, connecting to it, disrupting it or taking data from it is unauthorized access. Report what you found to your incident lead, the hosting provider or the authorities. And as always, this is general education, not legal advice: rules differ by country and by contract.

A safe analysis lab

A typical small lab, built from free tools:

 ┌─────────────────────── your host computer ───────────────────────┐
 │                                                                  │
 │   ┌───────────────────────┐   isolated     ┌──────────────────┐  │
 │   │ analysis VM           │   host-only    │ "fake internet"  │  │
 │   │ (Windows + FLARE-VM   │◀──network ────▶│ VM: INetSim or   │  │
 │   │  or Linux + REMnux)   │   no route out │ FakeNet-NG       │  │
 │   │ clean snapshot taken  │                │                  │  │
 │   └───────────────────────┘                └──────────────────┘  │
 │      no shared folders · no shared clipboard · revert after use  │
 └──────────────────────────────────────────────────────────────────┘

• FLARE-VM (Windows) and REMnux (Linux) are free collections of analysis tools that turn a fresh VM into a workstation.
• INetSim and FakeNet-NG pretend to be the internet: they answer DNS, HTTP and other requests locally, so you can see what a program tries to contact without letting it reach anything real.
• Take a snapshot of the clean analysis VM before every session and revert afterwards.
• Keep the VM software up to date, keep the lab network away from your home or company network, and treat the whole lab as contaminated.
• Automated sandboxes such as CAPE run a sample in a disposable VM and produce a behavior report. They save time, but you still interpret the results yourself.

The workflow

1. Intake. Record where the file came from, compute its hashes (MD5, SHA-1, SHA-256), store it safely. Search the hash in your organization's records and public reputation services.
2. Static triage first. Everything from the earlier lessons, without running anything: file, strings, imports, sections and permissions, entropy (a hint of packing), embedded URLs or keys after decoding layers. Often this alone answers the main questions.
3. Behavior observation. Only when needed, in the lab: take a baseline, run the sample, and record processes started, files created or changed, registry changes on Windows, and network activity. Tools: Process Monitor and Sysmon on Windows, strace and auditd on Linux, Wireshark for traffic, Regshot-style before/after comparisons.
4. Document and share. Indicators, detection rules, an ATT&CK mapping and a short report that someone else can act on.

Static before dynamic is the golden rule: it is safer, cheaper, and tells you what to watch for when you do run the sample.

Indicators of compromise

An indicator of compromise (IOC) is an observable fact that suggests a system has met a particular threat: a file hash, an IP address, a domain or URL, a file path, a registry key, a mutex name, a distinctive string. Defenders search for IOCs in logs, block them at the edge, and share them so others can do the same.

Not all indicators are equally useful. David Bianco's Pyramid of Pain (2013) ranks them by how much it hurts the attacker when you detect them:

                 /\
                /  \        TTPs: tactics, techniques, procedures   (tough)
               /────\       tools                                   (challenging)
              /──────\      network and host artifacts              (annoying)
             /────────\     domain names                            (simple)
            /──────────\    IP addresses                            (easy)
           /────────────\   hash values                             (trivial)

Changing one byte changes a file's hash, and renting a new IP address is cheap, so those indicators are easy to evade. Detecting behavior (how a program persists, how it talks to its server) forces an attacker to change how they work, which is expensive. Collect the cheap indicators, but aim higher.

Defanging makes indicators safe to paste into tickets and chats: http becomes hxxp and dots become [.], so nothing turns into a clickable link. Refang before you feed them to a tool.

Safe example values. Documentation and training should never use real hosts. RFC 5737 reserves 192.0.2.0/24, 198.51.100.0/24 and 203.0.113.0/24 for documentation, and RFC 2606 reserves example.com, example.net, example.org and the top-level names .test, .example, .invalid and .localhost. Every indicator in this lesson uses them.

Sharing. Indicators are exchanged in standard formats such as STIX 2.1 (an OASIS standard), and labeled with the Traffic Light Protocol (TLP 2.0, by FIRST) to say who may receive them: TLP:RED, TLP:AMBER+STRICT, TLP:AMBER, TLP:GREEN or TLP:CLEAR.

Detection rules with YARA

Indicators describe one sample; a detection rule describes a family of files. The standard tool is YARA, an open-source pattern-matching engine created at VirusTotal and used across the security industry. A rule has a name, some strings (text or hex bytes) and a condition:

rule Notes_Updater_Config
{
    meta:
        description = "Teaching sample: notes-sync updater configuration"
    strings:
        $host  = "update.example.test"
        $agent = "TeachingAgent/1.0" nocase
        $key   = { DE AD BE EF }
    condition:
        2 of them
}

nocase ignores ASCII letter case, { ... } is a hex string, and the condition says how many strings must be present (any of them, all of them, 2 of them, or boolean logic such as $host and $key). Running yara -s rules.yar file prints every matching rule and, with -s, where each string was found.

Good rules are specific (strings that only this family has, never common words such as http or kernel32.dll), robust (several strings with N of them, so one change does not break the rule), and tested: run them against a collection of clean files to catch false positives before you deploy them. A packer's section names, such as UPX's UPX0 and UPX1, are a useful hint but not a verdict, because legitimate programs are packed too.

Watching behavior: snapshots and ATT&CK

When static triage leaves questions open, you observe the sample in the lab. The simplest powerful technique is a before/after comparison: record the state of the file system (and on Windows the registry), run the sample, record the state again, and compare. Tools such as Regshot automate this for the registry; on Linux you can hash every file before and after.

Most changes are noise (logs, caches). What deserves attention:

• Persistence locations, places that make a program start again after a reboot or login: on Windows the ...\CurrentVersion\Run registry keys and Startup folders; on Linux cron directories, systemd unit folders, XDG autostart entries (~/.config/autostart/) and shell start-up files such as ~/.bashrc
• New executables in temporary or user-writable folders
• Changes to security-relevant files, such as /etc/hosts
• Network activity: DNS lookups and connections, captured by the fake-internet VM

MITRE ATT&CK gives these behaviors shared names. It is a public knowledge base of adversary behavior organized as tactics (the goal, such as Persistence, TA0003, or Command and Control, TA0011) and techniques (the method). For example, T1547.001 is Registry Run Keys / Startup Folder, T1053.003 is Scheduled Task/Job: Cron, T1071.001 is Application Layer Protocol: Web Protocols and T1027 is Obfuscated Files or Information. Mapping your observations to these IDs lets other defenders search their own data for the same behavior, which is exactly the top of the Pyramid of Pain.

The point of knowing these locations is to check them: during an investigation, on your own systems, and in your detection rules.

Writing it down: the analysis report

Analysis that is not written down did not happen. A short report that a colleague can act on usually has these parts:

1. Summary        what it is, what it does, how confident you are   (3-5 sentences)
2. Sample         file names, size, type, MD5 / SHA-1 / SHA-256, where it came from
3. Static         format, notable strings, imports, packing, decoded configuration
4. Behavior       processes, files, registry, network, with timestamps from the lab run
5. Indicators     defanged, grouped by type, each with context
6. ATT&CK         technique IDs with the evidence for each
7. Detection      YARA rules, log searches, network signatures
8. Remediation    how to find affected machines and clean them
   Label          TLP marking, author, date, version

Write facts and assessments differently: "the sample creates notes-sync.desktop in ~/.config/autostart/" is a fact you observed; "this is probably meant to survive reboots" is an assessment, and it should say how confident you are. Never pad a report with indicators you did not verify, and remember that the report itself is shared material: defang it, label it with TLP and keep personal data out of it.

Summary and what to practice next

• Malware analysis is defensive work in an authorized context: never create or spread malware, never hack back, handle samples like hazardous material and share only text
• The lab is isolated: VMs with snapshots, no shared folders or clipboard, a fake internet such as INetSim or FakeNet-NG
• The workflow is intake → static triage → behavior observation → documentation, and static always comes first
• IOCs are hashes, IPs, domains, URLs, paths and keys; defang them, use documentation ranges in examples, share them with STIX and TLP, and aim up the Pyramid of Pain
• YARA rules describe families; before/after snapshots reveal persistence; ATT&CK gives behaviors shared names; a good report separates facts from assessments

Practice next: build a two-VM lab with REMnux or FLARE-VM and INetSim, then take and revert a snapshot until it is a habit. Run your snapshot tool on your own machine before and after installing a normal program, and see how much noise a legitimate installer makes. Write a YARA rule for the teaching sample of this lesson, test it against a folder of your own clean files, and fix every false positive. Then read a public analysis report from a CERT or security vendor and find each part of the report structure above. This is the last lesson of the track: take the rules of the law and ethics lesson into every analysis you do.