The Rootkit That Edits Files Only When You Read Them: A Linux Forensics Walkthrough

A Python service kept crashing on one server with an IndentationError. Reinstalling the package didn't help. Downgrading didn't help. The same package installed cleanly on any other machine. The file on disk was fine. The kernel was changing it every time a program read it.

This article walks through the investigation step by step, as a short course. The malware itself isn't the point. The point is the method: how to prove that a running system is lying to you, using only tools that system gives you, and why the result means you stop trusting it completely.

🧩 The symptom: one file, one server, always broken

The service failed at startup with a Python syntax error inside a small module called messages.py. The traceback pointed at a constructor call that had lost its arguments. A line like signature_id=... was simply missing, leaving an orphaned super().__init__( behind.

Three facts made this strange:

• The package checksum matched the official repository, byte for byte.
• The same package installed on a clean test machine worked perfectly.
• An engineer copied the file under a different name (xyz-test.py) and the copy was intact. Only files called messages.py came out damaged.

A second, unrelated problem showed up on the same host: a C library upgrade failed while replacing the dynamic loader, with rename failed - No data available. Keep that in mind; it comes back at the end.

🤔 Forming the hypothesis

"Content depends on the file name, not on the file bytes" rules out almost everything in user space. Packaging tools, editors and Python itself don't care what a file is called when they read it. Something below them does.

Comparing a broken copy with a good one showed a pattern: every missing line contained the lowercase word signature. Lines with Signature (capital S) survived.

That gives a testable hypothesis: the kernel's read() path drops any line containing signature from files with certain names. Why would anyone build that? Because messages and syslog are where Linux writes its system log, and a kernel module that loads without a valid signature leaves a line like module verification failed: signature and/or required key missing. A rootkit that wants to hide its own loading would filter exactly that word from exactly those files. The broken Python module is collateral damage: it happens to share a name with the system log and contain the forbidden word.

A hypothesis isn't proof. Here is how to turn it into proof.

🔗 Proof 1: same inode, two names, two answers

On Linux a hard link is a second name for the same data. Same inode, same blocks, same bytes. If two names for one inode return different content, the difference can't be on disk. It's being made at read time.

Work in a scratch directory you own (never inside application folders):

mkdir -p /root/readtest && cd /root/readtest
printf 'a\nsignature_id\nb\nSignature\nc\n' > messages.py
ln messages.py hl.txt            # hard link: same inode, second name

stat -c 'inode=%i links=%h size=%s name=%n' messages.py hl.txt
sha256sum messages.py hl.txt

On a healthy machine both hashes match. On the compromised one:

inode=1234567 links=2 size=29 name=messages.py
inode=1234567 links=2 size=29 name=hl.txt
d9c98e31...  messages.py
05cba207...  hl.txt

Same inode, same size, different hashes. You've already proved the kernel is involved. Now watch it at the syscall boundary with strace, so nobody can argue that some library in between did it:

strace -e trace=openat,read cat messages.py hl.txt 2>&1 | grep -A1 readtest

openat(AT_FDCWD, "/root/readtest/messages.py", O_RDONLY) = 3
read(3, "a\nb\nSignature\nc\n", 131072) = 16
openat(AT_FDCWD, "/root/readtest/hl.txt", O_RDONLY) = 3
read(3, "a\nsignature_id\nb\nSignature\nc\n", 131072) = 29

The kernel itself returned 16 bytes for one name and 29 for the other. The signature_id line is gone and Signature is kept: a case-sensitive, line-based filter. Rename the test file to syslog and you get the same result.

📏 A trap: why wc -c can't see it

An obvious first check is "compare the size stat reports with the number of bytes you actually read". The obvious tool for that is wc -c. It doesn't work.

GNU wc -c on a regular file takes a shortcut: it asks the kernel for st_size and never reads the content. So it reports the on-disk size and agrees with stat every time. Force a real read through a pipe instead:

stat -c %s messages.py           # size from metadata
wc -c < messages.py              # may still use st_size — not a real test
cat messages.py | wc -c          # real read() through a pipe

The lesson goes beyond this case. When you suspect the system is lying, know which syscall each of your tools actually makes. A check that never touches the filtered path proves nothing.

📦 Proof 2: a system binary nobody wrote to

Test files are convincing, but they're files you created. The second proof uses a file nobody on the server touched: the system logger binary. Package verification flagged it with a checksum mismatch, and the first instinct was "trojaned logger". That turned out to be wrong, and the way to disprove it is worth learning.

Download the official package for the exact same version on a separate, clean machine and extract the binary:

# on a clean workstation
rpm2cpio rsyslog-<version>.rpm | cpio -idm ./usr/sbin/rsyslogd
sha256sum usr/sbin/rsyslogd       # equals the hash in the server's package database
stat -c %s usr/sbin/rsyslogd      # equals st_size on the server

So the binary on disk is genuine: same size, same build. But on the server a real read returned 276 bytes fewer than stat reported. A compiled binary contains embedded strings, and one newline-delimited chunk of this one contained an error message with the word signature in it.

Now reproduce the filter offline on the clean copy and compare:

# remove every newline-delimited "line" containing "signature",
# trim the trailing newline grep adds, then hash
grep -av 'signature' usr/sbin/rsyslogd | head -c <bytes_read_on_server> | sha256sum

The hash matched what the server returned, byte for byte. The official binary, minus the one line containing signature, is exactly what the compromised kernel hands to anyone who reads it. That also tells you the name rule is broader than messages* and syslog: the logger binary is filtered too.

This is the strongest kind of proof in forensics: an independent reference (the vendor's package), a deterministic transformation, and an exact match.

📜 Proof 3: two logs that should agree

On systemd machines the same events usually reach two places: the binary journal and the text log file. The rootkit's filter acts on reads of files with certain names. The journal is stored in different files and read through a different tool. So compare them over the same time window:

# pick a window fully covered by the current /var/log/messages
journalctl --since 'YYYY-MM-DD 00:00:00' | grep -c signature
grep -c signature /var/log/messages

journal:          44
/var/log/messages: 0

Forty-four lines in the journal, zero in the text log. The missing lines were ordinary service messages, things like "loading signature database". The filter is blunt: it removes every matching line, malicious or not.

Bonus check: read the log file through a hard link with a neutral name, as in Proof 1. The "missing" lines appear. They were on disk the whole time.

🎭 Proof 4: the kernel hides its own taint

Linux keeps a "taint" bitmask. Loading an out-of-tree module sets flag O; loading an unsigned one sets E. Each module also shows its own taint. On an honest kernel the global value includes every module's flags.

cat /proc/sys/kernel/tainted
for m in /sys/module/*/taint; do t=$(cat "$m"); [ -n "$t" ] && echo "$m: $t"; done
journalctl -k | grep -m3 'Not tainted\|Tainted:'

On this server two legitimate out-of-tree modules (a livepatch module and a resource-limiting module) each reported taint O. The global value was 0. Kernel warning dumps printed "Not tainted". That's impossible on an unmodified kernel. Something is rewriting the kernel's own view of itself.

🔍 What we looked for and didn't find

A thorough sweep also covered the usual user-space persistence points: /etc/ld.so.preload, PAM modules, systemd units, cron, SSH keys, kernel modules under /lib/modules, ftrace and kprobe hooks, eBPF programs, and the process and socket lists compared against /proc. None of them showed an extra malicious item, and the implant itself was never found as a file on disk.

That isn't reassuring. It's expected. A component that can rewrite read() results and hide kernel taint can also hide its own module, its files and its hooks. "Not found" on a live compromised kernel means nothing.

Remember the failed C library upgrade? The package manager writes a new loader file and then renames it into place, which involves reading and checking the data. With a kernel that tampers with file reads, "No data available" on exactly that operation is no longer a mystery.

🧠 The logic, in one page

1. Notice the impossible. Content that depends on a file's name rather than its bytes can't come from user space.
2. Find the pattern. Diff broken vs good output until the rule is obvious (lowercase signature, line-based).
3. Remove the variables. Same inode under two names, so there's one set of bytes and two answers.
4. Go to the syscall. strace shows the kernel's own return value, with no library in between.
5. Know your tools. wc -c trusts st_size; a pipe forces a real read.
6. Use an outside reference. The vendor package, a reproduced transformation and an exact hash match.
7. Cross-check independent channels. Journal vs text log, module taint vs global taint.
8. Draw the right conclusion. Once the kernel is proven to lie, no check run on that system can prove it clean.

🛠️ What to do with a lying kernel

There's no in-place cleanup for this. Removing a file, reinstalling a package or rebooting into the same disk doesn't help, because you can't know what else the kernel is hiding or what it reinstalls at boot. The only trustworthy path:

• Plan the move: inventory the sites, databases and mail the server hosts.
• Build a fresh operating system installation from scratch on clean media. Don't restore a system-disk image, which brings the problem back.
• Migrate data only (site files, database dumps, mail), and scan it on the new host before it goes live. Don't copy binaries, system configs or startup scripts across.
• Treat every secret that ever existed on the old host as exposed: passwords, SSH keys, API tokens, TLS private keys.

The most useful takeaway: when a problem "only happens on this one machine" and every component checks out on its own, consider that the layer you're standing on might be the thing that's broken.

❓ FAQ

Can user-space malware filter file contents by name like this?

Not for every process at once. LD_PRELOAD tricks affect only dynamically linked programs that load the library, and strace would show the real read() result. When the syscall itself returns different data for two names of the same inode, the change is happening in the kernel.

Why didn't comparing stat size with wc -c catch it?

GNU wc -c uses the file's metadata size for regular files instead of reading the content. Pipe the file through cat so the bytes really pass through read().

Is a failed package checksum proof that a binary is trojaned?

No. Here the logger binary on disk was genuine. The checksum failed because the kernel altered what the verifier read. Compare against the vendor's package on a separate clean machine before calling a binary malicious.

Can the server be cleaned without reinstalling?

No reliable way exists. A kernel that rewrites reads and hides its own taint can hide anything else too. Rebuild from a fresh OS installation and migrate only data.