GCP crypto mining and startup-script backdoors: detection

TL;DR. Compute Engine abuse leaves clear Admin Activity traces. Persistence: setMetadata / setCommonInstanceMetadata entries whose delta adds startup-script, startup-script-url, ssh-keys, or removes enable-oslogin / block-project-ssh-keys. Mining: compute.instances.insert with guestAccelerators or a2-/a3-/g2- machine types, bursts of creations, VMs in regions you never use. The audit entry tells you which metadata keys changed; get the content from the VM or its disk.

Compute is where a compromised Google Cloud project most often turns into a bill. Google's Threat Horizons report of November 2021 found that 86% of a sample of 50 recently compromised Google Cloud instances were used for cryptocurrency mining, and that where timelines were available, 58% had mining software downloaded within 22 seconds of compromise. That is the "impact" side (T1496, Resource Hijacking). The persistence side, backdoors in VM metadata, is quieter and more dangerous.

Startup scripts: root on every boot

A startup script is a metadata value that the guest environment runs as root, at every boot on Linux (startup scripts on Linux VMs). Anyone with compute.instances.setMetadata on a VM, or compute.projects.setCommonInstanceMetadata on the project, can therefore run code as root on those VMs at the next reboot, and trigger that reboot with reset. Typical payloads: a reverse shell, a miner, or a script that reads the service account token from the metadata server.

The keys that execute code:

In the audit log, look for:

protoPayload.methodName=("v1.compute.instances.setMetadata" OR "v1.compute.projects.setCommonInstanceMetadata")

and read protoPayload.metadata.instanceMetadataDelta (or projectMetadataDelta), which lists addedMetadataKeys, modifiedMetadataKeys and deletedMetadataKeys. Compute operations are long-running, so one change can produce two entries (operation.first and operation.last); count them once.

Do not expect the script itself in the log. Retrieve it with gcloud compute instances describe VM --format=json (or project-info describe for project metadata), and snapshot the boot disk before anything else so you can examine what ran. Security Command Center has a comparable detection, Persistence: GCE Admin Added Startup Script. MITRE maps this to T1037 and, for running commands through the cloud control plane, T1651.

SSH keys and OS Login

Two ways to get a shell without touching IAM:

• Metadata SSH keys: adding a line to ssh-keys in instance or project metadata grants SSH to the VMs (add SSH keys to VMs). Project-wide keys apply to every VM that does not block them. MITRE T1098.004.
• Switching off OS Login: with enable-oslogin=TRUE, SSH access is controlled by IAM and metadata keys are ignored. Deleting or changing enable-oslogin (or block-project-ssh-keys) brings metadata keys back into play. It is the step that makes the SSH key useful, and it is rare in normal operations.

The combination "ssh-keys added and enable-oslogin deleted in the same setCommonInstanceMetadata call" is about as clear as cloud evidence gets. The fix is to re-enable OS Login and enforce the compute.requireOsLogin organization policy (OS Login).

Mining: GPU VMs, bursts and unused regions

The creation entries are v1.compute.instances.insert, bulkInsert and instanceTemplates.insert. The request shows what was asked for:

• guestAccelerators (attached GPUs), or accelerator-optimized machine types (a2-, a3-, a4-, g2- families) in machineType;
• a count: many instances by the same principal in a short time;
• a zone in a region where the project has no other activity. Attackers pick regions nobody watches (T1535, Unused/Unsupported Cloud Regions).

Quota errors are a clue too: an attacker hitting GPU quota limits leaves failed insert calls. And check billing: miners that run for hours show up there before anyone reads a log.

Firewalls opened to the Internet

v1.compute.firewalls.insert, patch or update with sourceRanges containing 0.0.0.0/0 (or ::/0) opens a port to everyone: SSH for the attacker, or a port for a web shell. Mapped to T1562.007.

Confirming with network evidence

Metadata changes say what was configured. VPC Flow Logs, if enabled on the subnet, say what happened: an inbound SSH session from the attacker IP after the reset, outbound connections from the VM to that IP or to mining pools, egress volume.

What the analyzer flags

The metadata keys are shown on each event as +startup-script, ~enable-oslogin, -block-project-ssh-keys. The remediation checklist asks to snapshot and inspect, then rebuild rather than clean, and to rotate the credentials the VM held.

Remediation

Snapshot disks of affected VMs for forensics, then rebuild them from a trusted image; remove unknown SSH keys and startup scripts from instance and project metadata; re-enable OS Login and enforce it; stop instances nobody approved; cap GPU quotas; restrict locations with gcp.resourceLocations. The VM's service account token must be considered stolen: review what that service account could do.

Frequently asked questions

How do I find a startup-script backdoor in Google Cloud?

Search Admin Activity logs for v1.compute.instances.setMetadata and v1.compute.projects.setCommonInstanceMetadata entries whose metadata delta adds or modifies startup-script, startup-script-url or the Windows equivalents. Then read the current value of the key on the VM or project and snapshot the disk.

How common is crypto mining after a Google Cloud compromise?

Google's November 2021 Threat Horizons report found that 86% of a sample of 50 recently compromised Google Cloud instances were used for cryptocurrency mining.

Further reading

• Compute Engine audit logging — Google Cloud documentation.
• About OS Login — Google Cloud documentation.
• Google Cloud incident response: a compromised project.