Leaked GCP service account key: how to investigate it

TL;DR. Every API call authenticated with a user-managed service account key carries protoPayload.authenticationInfo.serviceAccountKeyName, ending with /keys/<KEY_ID>. Filter on that key ID, list the caller IPs and user agents in time order, and look for the first public IP you cannot explain. From there, follow that IP: enumeration (bursts of PERMISSION_DENIED), SetIamPolicy, new keys, new service accounts. Disable the key first, investigate second.

Downloadable JSON keys are the most common way a Google Cloud project gets opened from the outside. They do not expire by default, they end up in Git repositories, CI variables, Docker images and laptops, and whoever holds the file is the service account. Google's key management best practices recommend avoiding them altogether; in the meantime, you need to know how to investigate one.

Step 1: identify the key

You usually start with one of these:

• the key file itself, whose private_key_id field is the key ID;
• a notification from Google that a key was exposed publicly;
• a finding on a service account doing things it never does.

List the keys of the service account with gcloud iam service-accounts keys list --iam-account=SA_EMAIL and note, for each user-managed key, the ID and the creation time. Then find the creation entry in the Admin Activity logs:

protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"

The creator is in authenticationInfo.principalEmail, the new key's name in protoPayload.response.name. A key created by a person from an office IP weeks before the incident tells a "leaked by mistake" story. A key created during the incident, by the attacker, tells a persistence story: MITRE ATT&CK T1098.001, additional cloud credentials. Also look for google.iam.admin.v1.UploadServiceAccountKey: an uploaded public key means someone outside holds the private half.

Step 2: find every use of the key

Google's example logs for service accounts show that calls made with a key include:

"serviceAccountKeyName": "//iam.googleapis.com/projects/PROJECT/serviceAccounts/SA_EMAIL/keys/KEY_ID"

So the query is simple:

protoPayload.authenticationInfo.serviceAccountKeyName:"KEY_ID"

Put the results in time order and build a small table: first seen, last seen, caller IP, user agent, count. What you are looking for:

Remember that read calls (GetIamPolicy, storage.objects.get, ListServiceAccounts) are Data Access logs. Without Data Access logging you will only see the key's writes. For read-only calls that fail with PERMISSION_DENIED, Google may redact the caller's email, but not when the caller is a service account (caller identities in audit logs), so a leaked key's failed reads stay attributable.

Step 3: separate CI from attacker

The hard part is not finding external IPs; it is that legitimate keys are also used from outside Google Cloud. A GitHub Actions runner, a partner's server, a developer laptop: all are "public IPs". Useful discriminators:

• User agent: Terraform and client libraries carry versioned strings; a gcloud CLI on Linux with interactive/True from a VPS is not your pipeline.
• Timing: CI runs on commits and schedules; an attacker session is a dense burst at an odd hour.
• Errors: pipelines rarely hit dozens of denied calls in a few minutes; enumeration does.
• Methods: pipelines repeat the same methods; attackers call GetIamPolicy on the organization, list secrets, billing and Security Command Center notification configs.

Step 4: follow the attacker IP, not just the key

Once you have the attacker IP, drop the key filter and search for the IP alone:

protoPayload.requestMetadata.callerIp="203.0.113.66"

Attackers switch identity as soon as they can: they grant Owner to an account they control (often a Gmail address) and continue as that account from the same machine. The IP ties the two phases together. See SetIamPolicy abuse for that second phase, and service account impersonation if the key was used to mint tokens for other service accounts.

Step 5: contain in the right order

Google's response guide for compromised credentials and the disable and enable keys page describe the mechanics. The order I use:

1. Disable the key (reversible, immediate). If the service account itself is compromised beyond the key, disable the service account.
2. Remove what the attacker added with it: IAM bindings, keys, service accounts, HMAC keys.
3. Deploy a replacement credential to the legitimate workload, preferably not a key: Workload Identity Federation for CI.
4. Delete the old key once the investigation has recorded everything about it.
5. Assume everything the service account could read is exposed: secrets, buckets, datasets. Rotate those secrets.

Then remove the root cause: purge the file from the repository history and CI logs, and look for other copies.

Prevention that actually works

• iam.disableServiceAccountKeyCreation organization policy: no new user-managed keys.
• iam.serviceAccountKeyExposureResponse set to DISABLE_KEY: Google detects keys exposed in public places and disables them, notifying owners and security contacts (Google Cloud blog).
• Key expiry for the keys you cannot avoid, and an inventory of the remaining ones.

What the analyzer flags

On an export dropped in the analyzer, the key story produces up to five findings:

The Entities tab lists each key with its creator and every IP that used it. One limit to keep in mind: "new IP" is relative to the export. If your export starts after the attacker's first use, the attacker IP becomes the baseline. Export a longer period when you can.

Frequently asked questions

How do I know if a leaked service account key was used?

Search the audit logs for protoPayload.authenticationInfo.serviceAccountKeyName ending with the key ID. Every call made with the key carries it. Calls from IP addresses you do not recognise, especially after the leak date, mean the key was used by someone else.

Does Google disable leaked service account keys automatically?

Google scans public repositories for exposed keys. With the iam.serviceAccountKeyExposureResponse organization policy set to DISABLE_KEY, detected keys are disabled automatically and project owners and security contacts are notified.

Should I delete the key immediately?

Disable it first: disabling is reversible and stops the attacker. Record the key ID and creation details, then delete it once the investigation no longer needs it and the legitimate workloads have a replacement.

Further reading

• Best practices for managing service account keys — Google Cloud documentation.
• MITRE ATT&CK T1078.004, Valid Accounts: Cloud Accounts.
• MITRE ATT&CK T1552.001, Unsecured Credentials: Credentials In Files — how key files leak.
• Google Cloud incident response: a compromised project.