Traildash on ECS

I’ve recently had some issues where I’ve had to investigate the AWS API usage on one of our accounts. Enabling Cloudtrail is a start but all it does is shove a load of gzipped json files into an S3 bucket which is no use if you actually want to make use of the data.

Enter Traildash a self contained ELK stack on a Docker image which will pull that bucket’s contents into Elasticsearch and display it usefully in a Kibana frontend.

“Docker?”, I thought to myself, “Doesn’t AWS have something that could do that?”. Of course it does. ECS, the EC2 Container Service, allows you to run your own docker cluster. So here’s how to set up Traildash on ECS

First of all you need to follow the instructions in the Traildash readme to setup traildash in AWS. This gets your Cloudtrail up and running and connected to SQS/SNS for pushing to Traildash. The one thing different is the IAM role which will be the ECS instance and service roles created as part of the cluster build below. If you already have running ECS instances, add the “SQS full access” and “S3 Read Only Access” managed policies to your ECS Instance and ECS Service IAM roles. If not, wait until your ECS cluster instances are built and add the policies after.

Next is creating a task and a cluster in ECS.

It may take 10-15 minutes for your data to start to appear.

If you have Cloudtrail already working in your account and the data has been building up for a while, Traildash provides a backfill script to get it into your dashboard. In order to use the backfill script I changed it to use my aws credentials profile name:

#!/usr/bin/env python

import json
from os import environ

import boto3
boto3.setup_default_session( region_name='eu-west-1', profile_name='<your credentials profile name>')

AWS_S3_BUCKET= "<your bucket name>"
AWS_SQS_URL = "<your SQS url>"

bucket = boto3.resource('s3').Bucket(AWS_S3_BUCKET)
queue = boto3.resource('sqs').Queue(AWS_SQS_URL)

items_queued = 0
for item in bucket.objects.all():
    if not item.key.endswith('.json.gz'):

            'Message': json.dumps({
                's3Bucket': AWS_S3_BUCKET,
                's3ObjectKey': [item.key]
    items_queued += 1

print('Done! {} items were backfilled'.format(items_queued))

And you’re done. Enjoy your useful Cloudtrail data!