Leaked Access Keys: The Silent Revolution in Cloud Security

Service accounts and automation are the lifeblood for efficiently operating cloud data centers and running business processes – and GCP is one vendor making an innovative move in this area. It comes in an ordinary, not-attention-seeking email about some policy change to its customers. A new setting allows users to configure how GCP reacts when a private service account key leaks. Is this the start of a larger shift in how vendors approach cloud security? Understanding the significance first requires some background.

Service Accounts Basics

Service accounts are a necessity for automation in every application architecture and data center. After proving their identity, they allow applications and scripts to access (cloud) resources. GCP offers various authentication options for service accounts, such as identities attached to resources (e.g., VMs), Workflow Identity Federation, or service account keys. Google actively warns against using the latter (Figure 1), though they are a necessity for some use cases, especially for access outside of GCP and/or the company's perimeter. So, GCP has such a feature (2).

Figure 1: Creating and managing access keys for service accounts in the Google Cloud Platform

Service account keys are as secure (or slightly more secure than) user/password combinations. However, over the recent years, we all learned that securing user accounts with only one factor is inadequate. Attackers too often succeed by tricking users into revealing their secrets – using phishing attacks, for example. With only one factor in place, attackers take over the user accounts and are in an organization's network. Thus, multi-factor authentication (MFA) – using Authenticator apps or (more legacy) text messages became state of the art. MFA prevents hackers from accessing accounts who stole the password with the second factor.

Service Account Security Risks

The challenge for service accounts is that MFA does not work, and network-level protection (IP filtering, VPN tunneling, etc.) is not consequently applied, primarily due to complexity and costs. Thus, service account key leaks often enable hackers to access company resources. While phishing is unusual in the context of service accounts, leakages are frequently the result of developers posting them (unintentionally) online, often in combination with code fragments that unveil the user to whom they apply. Discussion forums on the internet are one flavor; more virulent is uploading code and files with credentials to the public GitHub by mistake (Figure 2, 1). Then, hackers can simply scan the GitHub repositories and harvest cloud credentials (Figure 2, 2).

Figure 2: GitHub, Credentials, GCP – The Big Picture

Unintentional credential uploads to GitHub are obviously so prevalent that GitHub implemented a feature for scanning for secrets. It checks uploaded files and other information. Service providers (e.g., cloud or SaaS providers) can partner with GitHub to implement scans for their specific credentials’ formats. When GitHub finds such credentials, it informs the customers who uploaded the data and the service provider in which context the credentials are used (Figure 2, 3). In more detail:

The major cloud providers such as AWS, Azure, Google Cloud, and Alibaba Cloud collaborate with GitHub on security. Beyond the big public clouds, GitHub also scans for credentials related, e.g., to OpenAI, Slack, Tableau, or Dynatrace. The list of participating small and big tech companies is far from comprehensive. Still, the major cloud providers collaborate, which is a big step in protecting cloud data centers.

Handling Leaked Access Keys: Then and Now

The classic way to handle credential leaks is for GitHub to inform the customer (and service provider) about the leak. Then, the customer's operations and engineering teams fix it. The engineering team must get the information, react to it, and not deprioritize or postpone the task. The teams might even be reluctant to rotate the access key since it requires two simultaneous changes:

Both actions must occur in sync; otherwise, applications might be impacted. Thus, it might take a while for engineers to be comfortable rotating the key. In the meantime, hackers can break into the service account (Figure 3, upper part).

Figure 3: Fixing credential leakages - classic (top) and new approach (down)

Now, Google has changed the game with its recent policy change. If an access key appears in a public GitHub repository, GCP deactivates the key, no matter whether applications crash. Google's announcement marks a shift in the risk and priority tango. Gone are the days when patching vulnerabilities could take days or weeks. Welcome to the fast-paced cloud era. Zero-second attacks after credential leakages demand zero-second fixing. Preventing an external attack becomes more important than avoiding crashing customer applications – that is at least Google's opinion. While Google does not force its customers to use the feature, choosing a less strict setting is probably the less favorable choice for most customers.

The revolution in access key security fundamentally changes the dynamics between cloud providers and their customers in two areas:

What I am really curious about is whether Google's move will help reshape cloud security in an impactful way. Will Google roll out more of these proactive security features? How will other cloud vendors respond? Different cloud providers might depend more on security revenues and act differently. So, will the proactiveness of cloud vendors become the new norm, and will we see an end to seatbelt taxation? Stay tuned!