Breach Detection At Scale With PROJECT SPACECRAB

October 19th 2017 Dan Bourke in Security, Open Source

Every blog post needs a good truism, so here's ours:

Detection engineering is hard; detection engineering at scale is even harder.

Attackers have all the advantages and defenders have few asymmetries to exploit.

That's maybe two truisms, actually. Call it a bonus.

Today we’re releasing PROJECT SPACECRAB, an open source toolset that tips the balance in the right direction by enabling deployments of several thousand AWS honey tokens across your network and on every endpoint you own.

PROJECT SPACECRAB's spacecrab mascot


Good attackers are a combination of smart, dedicated and lazy. They've got some objectives they want to meet and they'll take any path they can to get there. One well-trodden path to those objectives is stolen credentials.

We know from experience running our cloud detection capability that the first thing most attackers do after they establish initial access, is search for passwords and other credentials. Then they try to use those credentials to access other services and escalate their privileges.

Passwords are extremely valuable to attackers (and you!), but in a world where much of the internet runs on Amazon’s infrastructure, AWS access keys have become the keys to the kingdom. Those keys give attackers the ability to call Amazon's API from the privacy of their own dodgy offshore VPS and manipulate whatever resources might be running in the attached account.

While AWS has extremely powerful policies and permissions language to prevent leaked keys from being abused in this way, a single misconfigured policy – or API that's not strictly forbidden – can lead to lateral movement, further information disclosure, or a sudden increase in your AWS bill as you start an enormous cryptocurrency mining farm for some reason.

Because AWS keys are very valuable and able to be secured pretty comprehensively, they're an excellent candidate for honey tokens. These tokens, which are credentials without the access to actually perform any actions, but set off alarms whenever they're used, are a great way to use attacker's methods against them. They function as tripwires or early warning alarms for breaches in your network or supply chain.

Tell me more about these "honey tokens"

As mentioned, AWS keys make great honey tokens because they're very attractive to attackers, as they could contain unlimited cosmic power. Finding an AWS key is like finding an unscratched instant lottery ticket. It might be nothing, or it might be the reason you can quit your job tomorrow and move to somewhere where there are beaches and no extradition treaties.

From a defender's point of view, AWS keys are cheap to generate, can be locked down to have minimal (zero) permissions, and are easy(ish) to audit and alert on. To continue our lottery ticket metaphor, it's easy to set them up so that when someone scratches the ticket, it has three of the same symbol but the prize is "jail time, 5 years" and it somehow calls the cops. It's ... listen, the fifth hard problem in computer science is "metaphors are difficult", let's move on.

AWS keys are also found "naturally" in lots of different locations, so attackers' delicate trap sensibilities won't be triggered by finding them in your chat logs, in environment variables in ephemeral EC2 instances, or in a file called "AWS secret access tokens for production account root user.txt" on your CISO's desktop.

So if you've got a few places you want to sprinkle some honey tokens to catch inquisitive attackers, it's easy for you to generate some tokens with locked down permissions and set up alerts in Cloudwatch that will tell you when someone tries to use them.

In fact, the team at Thinkst has made that trivial with their free canary token service. This approach is fine for small honey token deployments, but it is difficult and tedious to manage if you want to scale to, for instance, several thousand honey tokens deployed all across your network and on every endpoint you own.

That’s where PROJECT SPACECRAB (just SPACECRAB in informal contexts) comes in! Just log in to a brand new AWS account, run the bootstrap CLI, wait a while, and you’re ready to distribute thousands of honey tokens, automatically and securely, all over your network. Better yet, the detection logic is already set up and extensible to whatever alerting infrastructure you already use.

A screenshot of the CLI


SPACECRAB is a Cloudformation stack. That doesn't sound very impressive, but it's a Cloudformation stack that deploys a comprehensive set of tools, policies, datastores and alerting functions, and deploys flawlessly about 80% of the time.

(If your deployment is part of the 20% that doesn't work the first time, burn the stack down and deploy it again and it should be fine!)

SPACECRAB's three main components are:

  • A datastore for keeping records about honey tokens you've deployed (when they were produced, where they've been deployed, who owns them, etc.)
  • Some lambda functions for creating and managing honey tokens
  • An extensible alerting pipeline you can configure to set off alarms when one of your tokens is used

For instance:
If you're storing credentials in a third party service, and want to know if said third party has a breach, you can create a honeytoken with the annotations "stored in $THIRDPARTY, owned by $SECURITYTEAM". You then store it in your third party service with a juicy title like "AWS root user keys" and wait.

A while later, someone breaches their infrastructure and, eventually, pillages your account. They see "AWS root user keys" and, because it's the sensible thing to do, try to use those keys to list resources in your account.

They will get a series of permission denied messages, and, disappointed, go loot something else. But you will get a series of phonecalls from the nice PagerDuty robot telling you that someone has used the AWS keys stored in $THIRDPARTY and that you should initiate your incident response process.

Ideally you will also get phonecalls from $THIRDPARTY telling you to initiate your incident response process. But in the unlikely event that this does not happen in a timely fashion, you've given yourself a significant head start (the alert also contains the user-agent and source IP the attacker was using, which you can feed into your investigations, which might help).

Later, when you move to a new third party service, you can generate new tokens and sprinkle them there. In this way, you can set up a defensive perimeter in your supply chain.

Now, this example is something you can probably manage on your own because there's really only one location you're storing credentials. But if you want to store unique credentials on every EC2 instance you ever spin up – so you get notified not just what service was involved, but the specific server – you're going to need a more robust solution.

You're going to need SPACECRAB.

How does PROJECT SPACECRAB actually work?

⚡️ ☁️ AWS magic ☁️ ⚡️

Fine, yeah, that's not a good explanation. Here's a pretty picture: A diagram of how PROJECT SPACECRAB alerts work Basically, SPACECRAB generates new user tokens with a deny-all policy. The only action these tokens can perform is the 'sts get-caller-identity' action, which you literally cannot stop a token from doing.

SPACECRAB then stores metadata about these tokens, including the location you'll put them, in a database.

The account you're using to store and alert on tokens is configured with a comprehensive Cloudtrail logging policy. Any event that happens in that account is logged and sent to an S3 bucket, then a lambda function for analysis. Every logged event is checked to see if the principal is one of your honey tokens (against the records in the database). If it is one of your honey tokens, the event is sent to an AWS SNS topic.

That SNS topic triggers more lambdas, one for each of the alerting methods you've configured. Each lambda takes the event data from SNS, formats it appropriately, and triggers its configured alert action.

Currently available alerting actions include the PagerDuty Events API v2, and Amazon's SES (Simple Email Service), but you can write your own lambda functions to fire whatever alerts you need.


The extremely tl;dr version of how to use SPACECRAB is "clone the repo, run, answer the questions it asks you". There is a more detailed version in the repository itself, covering some steps you might want to do before and after, but that should cover it.

Let us know what you think!