Traildash on ECS

I’ve recently had some issues where I’ve had to investigate the AWS API usage on one of our accounts. Enabling Cloudtrail is a start but all it does is shove a load of gzipped json files into an S3 bucket which is no use if you actually want to make use of the data.

Enter Traildash a self contained ELK stack on a Docker image which will pull that bucket’s contents into Elasticsearch and display it usefully in a Kibana frontend.

“Docker?”, I thought to myself, “Doesn’t AWS have something that could do that?”. Of course it does. ECS, the EC2 Container Service, allows you to run your own docker cluster. So here’s how to set up Traildash on ECS

First of all you need to follow the instructions in the Traildash readme to setup traildash in AWS. This gets your Cloudtrail up and running and connected to SQS/SNS for pushing to Traildash. The one thing different is the IAM role which will be the ECS instance and service roles created as part of the cluster build below. If you already have running ECS instances, add the “SQS full access” and “S3 Read Only Access” managed policies to your ECS Instance and ECS Service IAM roles. If not, wait until your ECS cluster instances are built and add the policies after.

Next is creating a task and a cluster in ECS.

• If you’ve not used ECS before, follow the default set up wizard to get to the point where you have running cluster instances, otherwise you should be able to use your existing ECS configuration.
• Create a New Task Definition in your ECS console.
• Create a new volume (name doesn’t matter) and give it the source path /var/lib/elasticsearch/appliedtrust/traildash
• Then Add Container with the following settings:
  • Image: appliedtrust/traildash
  • Port mappings: Host: 7000 Container: 7000
  • Environment Variables:
    • AWS_SQS_URL
    • AWS_REGION
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
  • Mount point:
    • Source volume:
    • Container path: /home/traildash
• Create the Service with your task definition, a single task and an ELB if you want one. (You may need to edit the instance/ELB security group to allow port 7000 access. )
• Run the task and enjoy your Cloudtrail-y goodness on :7000/#/dashboard/file/default.json

It may take 10-15 minutes for your data to start to appear.

If you have Cloudtrail already working in your account and the data has been building up for a while, Traildash provides a backfill script to get it into your dashboard. In order to use the backfill script I changed it to use my aws credentials profile name:

#!/usr/bin/env python

import json
from os import environ

import boto3
boto3.setup_default_session( region_name='eu-west-1', profile_name='<your credentials profile name>')

AWS_S3_BUCKET= "<your bucket name>"
AWS_SQS_URL = "<your SQS url>"

bucket = boto3.resource('s3').Bucket(AWS_S3_BUCKET)
queue = boto3.resource('sqs').Queue(AWS_SQS_URL)


items_queued = 0
for item in bucket.objects.all():
    if not item.key.endswith('.json.gz'):
        continue

    queue.send_message(
        MessageBody=json.dumps({
            'Message': json.dumps({
                's3Bucket': AWS_S3_BUCKET,
                's3ObjectKey': [item.key]
            })
        })
    )
    items_queued += 1

print('Done! {} items were backfilled'.format(items_queued))

And you’re done. Enjoy your useful Cloudtrail data!