Chef + AWS KMS

This was a ton of fun to figure out. A challenge and a journey to discover how to optimize the integration of Chef and the new AWS KMS system. 

KMS is an interesting new product from AWS. It's server-side encryption, which means you're going to send it a payload of unencrypted bits, and they'll return to you the encrypted payload. This is not an encryption service in that it's not designed to encrypt large bundles of data like application packages or, in fact, anything over 4k.

Because of this we use KMS to encrypt our encryption keys that we then use to decrypt our payloads.
This can get pretty confusing, but in a nutshell here's the workflow:

1. Create 4k "password" that will be used with openssl to encrypt and decrypt large payloads.
2. Encrypt this payload with KMS and store the result in s3.
3. On the client side we pull the s3 payload down and use KMS to decrypt the payload, which will give us our decryption key.
4. Use that key to decrypt things like edb keys or validation pem files.  Possibly even larger payloads like application tarballs.

Let's kick this off by digging right into the code. This is my rake task for encrypting a payload with KMS:

namespace :encrypt do 
  task :payload, :filename, :service_name, :env_name do |t,args|
    cloud = AWSCloudHelper.new( args[:service_name], args[:env_name] )
    local_archive = args[:filename]

    Log.debug( "Getting key from s3." )
    s3 = cloud.get_s3()
    bucket_name = cloud.get_profile_name() ## logging-preproduction 
    enc_secret = s3.get_object({
      :key => "my secret key location",
      :bucket => bucket_name
    })
    Log.debug( "Key get complete." )

    Log.debug( "Decrypting key using KMS." )
    kms = cloud.get_kms().decrypt({ :ciphertext_blob => enc_secret.body.read })  #this is just a helper for getting a Aws::KMS::Client.new() object
    decrypted_secret = kms[:plaintext]
    puts decrypted_secret
    Log.debug( "Decryption complete." )

    Log.debug( "Encrypting payload." )
    cipher = OpenSSL::Cipher.new('super secret encryption method')
    cipher.encrypt
    cipher.key = decrypted_secret
    encrypted = cipher.update(File.read( local_archive ).chomp) + cipher.final

    f = File.open( "%s.enc" % local_archive, "w" )
    f.print( encrypted )
    f.close()

    Log.debug( "Encryption complete." )

    Log.debug( "Pushing to s3." )
    cmd_s3_push = "aws %s s3 cp %s.enc s3://%s/" % [cloud.get_aws_opts, local_archive, bucket_name]
    Log.debug( "CMD(s3_push): %s" % cmd_s3_push )
    system( cmd_s3_push )
    Log.debug( "Push complete." )
  end
end

So, if we're encrypting a EDB key we would crate the, store it into a file and do something like:

rake encrypt:payload["/my_edb.key", logging, preproduction"]

And we would end up with an encrypted payload stored in S3.

This is my lib function for getting kms-encrypted payloads in a chef recipe:

require "aws-sdk-core"

def get_kms_payload( payload )
  aws_access_key_id = ACCESS_KEY
  aws_secret_access_key = SECRET_KEY
  creds = Aws::Credentials.new( aws_access_key_id, aws_secret_access_key)

  s3 = Aws::S3::Client.new( :credentials => creds, :region => "us-east-1" )

  bucket_name = "%s-%s" % node.chef_environment.to_s.split( "-" )  ## logger-preproduction

  ## Get the encryption key used to encrypt everything.
  kms_payload = s3.get_object({
    :key => "this is where I keep my special secret payload",
    :bucket => bucket_name
  })  

  kms = Aws::KMS::Client.new( :credentials => creds, :region => "us-east-1" )
  res = kms.decrypt({ :ciphertext_blob => kms_payload.body.read })
  kms_encryption_key = res[:plaintext]

  secret_payload = s3.get_object({
    :key => payload,
    :bucket => bucket_name
  })  

  ## Now use the main decryption key with openssl to decrypt
  cipher = OpenSSL::Cipher.new('super secret encryption method')
  cipher.decrypt
  cipher.key = kms_encryption_key
  cipher.update( secret_payload.body.read ) + cipher.final
end

And this is my implementation:

(service_name, env_name) = node.chef_environment.to_s.split( "-" )
edb_secret = get_kms_payload( "%s.pem.enc" % env_name )
users = Chef::EncryptedDataBagItem.load( "logging", "users", edb_secret )
magic = Chef::EncryptedDataBagItem.load( "logging", "magic", edb_secret )

There are several neat things about this:

1. The EDB key is never actually stored on disk, so it's never persisted ( the security folks should enjoy this ).
2. KMS access is logged via CloudTrail, another +1 for the security folks.
3. IAM is used to control access to the s3 bucket, files, and of course KMS keys.
4. Eventually we can extend this to be more dynamic and do something crazy like roll out a new KMS key every time we build a new stack.

Great learnings in this little adventure.