In my previous post I talked about denv and how I switch between different Kubernetes clusters.

I also talked about the importance of being explicit about which environment you're currently working on, easily switching between and deactivating an environment so that we don't run accidental commands in the wrong context.

Continuing this series, in this post I will talk about how to effectively switch between different AWS accounts.

The problem

In my "day to day" I usually interact with multiple AWS accounts tied to various clients. I usually do this from the command line while using tools like terraform, packer, kops, AWS CLI etc.

These AWS accounts are usually set up with some best practices in mind:

• Mandatory MFA

This means that we require each user to sign into the AWS Console and set up a MFA device. This can be a virtual device like Google Authenticator or it can be a physical device like a Yubikey.

• No direct permissions are assigned to users
• Instead users have to assume a role depending on which permissions they need for the task at hand

At work we developed a terraform module that sets all of this up for us.

• Use of short lived STS session credentials

What this means is that we use "short term credentials" from AWS STS which we then use to request (and refresh) role based credentials if we're assuming a role. For certain endpoints AWS requires MFA when using STS so denv makes sure to enforce it. This means that the users aws_access_key_id and aws_secret_access_key are only used at the beginning to fetch the temporary STS credentials. The STS credentials are then cached and used to assume a role and fetch another set of credentials that have permission to actually do things on AWS (like create EC2 instances, RDS databases and similar). The user is prompted for the MFA token during the initial request and not again during the entire cache period.

Session credentials expire after 36 hours¹ and role credentials expire by default after 1 hour ², after which they need to be refreshed.³

When defining access credentials for AWS accounts you're going to be dealing with 2 files:

• ~/.aws/config

and

• ~/.aws/credentials

These files can be configured using the AWS CLI but since it's a straight forward ini format I usually edit them in my editor.

The non-solution

One could argue that you can just prefix each command with the AWS_PROFILE=myprofile environment variable and be done with it. This is true, and most tools do support this way of specifying which AWS profile to use. However, the trouble is it only works for the naive approach, just exporting the aws access key and secret key. It does not work if we're also using MFA and Role Assuming.

The solution

First things first.

If you have already setup your ~/.aws/config and ~/.aws/credentials files, make sure to delete (or rename) the [default] entry. We don't want to have a default profile. Instead, we want each profile to have its own unique name that forces us to specify the profile we're using.

NOTE: Make sure to set the correct permissions: chmod 600 ~/.aws/config && chmod 600 ~/.aws/credentials

Now let's look at 3 use-cases that you might encounter.

Case 1: MFA and assuming roles

In this case your configuration files will look like this:

#### ~/.aws/config ####

[profile project1]
region=us-east-1

[profile project1-prod-admin]
role_arn=arn:aws:iam::....:role/admin
mfa_serial=arn:aws:iam::....:mfa/deni
source_profile=project2


##### ~/.aws/credentials #####

[project1]
aws_access_key_id=.....
aws_secret_access_key=.....

Alright, let's break down what happens when we run the following command:

~ denv aws -p project1-prod-admin
Enter MFA code for device "arn:aws:iam::...:mfa/deni": xxxx

Once we enter our MFA code (using Google Authenticator or a similar app):

aws|project1-prod-admin  ~  export | grep AWS
AWS_ACCESS_KEY_ID=xxxx
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_SECRET_ACCESS_KEY=xxxx
AWS_SECURITY_TOKEN='xxxxx'
AWS_SESSION_TOKEN='xxxxxx'
_DENV_SET_VARS=AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN,AWS_SECURITY_TOKEN,AWS_DEFAULT_REGION,AWS_REGION,_OLD_DENV_PS1,_DENV_SET_VARS

aws|project1-prod-admin  ~  aws sts get-caller-identity
{
    "UserId": "...",
    "Account": "...",
    "Arn": "arn:aws:sts::...:assumed-role/admin/..."
}

We see that in the newly spawned shell we have all the required environment variables set that all the standard tools will look for. As with the kubernetes example in the previous post, the prompt is annotated with the account name so that we don't forget which account we're working on.

NOTE: aws sts get-caller-identity is just an example command. You might be more interested in running something like aws ec2 describe-instances or similar.

Once we're done, we can run denv deactivate to make sure that all the environment variables are unset from the current shell. This way we make sure that if we run any other AWS CLI command we're not accessing an account by mistake.

This form of using the denv aws subcommand is called the eval form. Since these credentials expire I recommend using the exec form that looks like this:

~ denv aws -p project1-prod-admin -- aws sts get-caller-identity

The exec form makes sure to execute your command in the required "context" with all the above variables set and once it's done, none of them get leaked to your currently running shell.

This form is preferable because it will always make sure to fetch fresh credentials if they're needed. This is not the case with a long running shell prompt because the credentials exposed in that shell might be stale.

Let's break down what denv aws -p project1-prod-admin does:

1. Parse and fetch the configuration and credentials from the standard files mentioned above
2. Check if we need to assume a role
3. If we do, check if we have valid cached credentials for that role
4. If we don't have valid cached credentials check if we have valid cached session credentials that we're using to fetch the above role credentials
5. If we don't have session credentials request new STS session credentials
6. Request new role credentials with the above session credentials
7. Export the role credentials into the current shell or run the supplied command while making the credentials available.

These series of steps are taken with each denv aws run. The cached credentials are stored in ~/.aws-env/.

Case 2: Assuming a role but without MFA

In this case your configuration would look like this:

#### ~/.aws/config ####

[profile project2]
region=us-west-2

[profile project2-prod-admin]
role_arn=arn:aws:iam::....:role/admin
source_profile=project2

##### ~/.aws/credentials #####

[project2]
aws_access_key_id=.....
aws_secret_access_key=.....

Activate this environment by running: denv -p project2-prod-admin. The main difference between this case and Case 1 is that we're not going to use intermediate session credentials from STS. Instead, we're going to use the raw credentials from ~/.aws/credentials to fetch our temporary role credentials. The reason we're skipping the session credentials is because AWS requires Multi Factor Authentication to access any AWS IAM APIs. ⁴

Case 3: Using raw credentials

#### ~/.aws/config ####

[profile project3]
region=us-west-2

##### ~/.aws/credentials #####

[project3]
aws_access_key_id=.....
aws_secret_access_key=.....

The last case is the "naive case" where denv is only exporting the aws_access_key_id and aws_secret_access_key defined in the above configuration file. It's not talking to the AWS API and fetching any short term or role credentials of any sort. I find denv is still useful in this case since it explicitly shows me which account I'm working on. Using the eval form for this case is safe and saves us a bunch of keystrokes (which I'm a big fan of).

How it works

We're using Haskell and some awesome libraries but the most interesting one is amazonka which we use to talk to the AWS APIs and fetch temporary session and role credentials.

The logic I outlined above is encoded like so:

env <- runMaybeT $
  MaybeT (runRIO awsEnv $ maybe nop (getFromRoleCache p) roleArn) <|>
  MaybeT (runRIO awsEnv $ maybe nop (maybe nop1 (getSTSWithRole awsenv region' lgr p sourceProfile) mfaSerial) roleArn) <|>
  MaybeT (runRIO awsEnv $ maybe nop (getFromSessionCache p sourceProfile) mfaSerial) <|>
  MaybeT (runRIO awsEnv $ maybe nop (getSTS awsenv region' lgr p sourceProfile) mfaSerial) <|>
  MaybeT (runRIO awsEnv $ maybe nop (useRawWithRole awsenv region' lgr p sourceProfile) roleArn) <|>
  MaybeT (runRIO awsEnv $ useRaw key' secret')

Fetching STS credentials is fairly straightforward:

mkStsSessionTokenRequest :: MfaSerial -> STS.GetSessionToken -> IO STS.GetSessionToken
mkStsSessionTokenRequest (MfaSerial mfaSerial) st = do
  req <- do
        tokenCode <-
          promptLine $
          "Enter MFA code for device " <> show mfaSerial <> ": "
        return $
          st & STS.gstTokenCode .~ (Just $ T.pack tokenCode) &
          STS.gstSerialNumber .~ (Just mfaSerial) &
          STS.gstDurationSeconds .~ (Just $ fromIntegral defaultSessionDurationSeconds)
  return req

And actually running the request:

execStsRequest :: AWS.AWSRequest a =>
     AWS.Env -> AWS.Logger -> AWS.Region ->  a -> IO (AWS.Rs a)
execStsRequest awsenv lgr region req = do
  ret <-
    runResourceT $
    AWS.runAWS (awsenv & AWS.envLogger .~ lgr) $
    AWS.within region $ AWS.send req
  return ret

Once we have the credentials it's just a matter of going to the entire process of exporting them into the current shell which I outlined in more detail in the previous post.

Summary

We've seen how denv aws helps us switch securely between multiple AWS accounts while using short term credentials. Using short lived credentials is encouraged as they are much easier to replace in case of a leak. They usually expire before it becomes a problem. It's also good that the raw credentials that the user gets from the AWS Console "travel the wire" much less in this scenario (even though it might be over a secure line).

Haskell helped me add this feature to denv fairly easily and while the initial version had even fewer lines than its bash counterpart⁵ it was much easier to maintain and reason about.

This feature is currently still in beta but I've been using it for a few months every day and have not found any bugs so far. I will likely remove the "beta" stamp once I allow for more configuration regarding cache expiry times.

Head over to the repo's releases page and try it out!

aws denv devops functional programming haskell haskell showroom open source