AWS Event Log Processing

Being provided with log events from a third party to an SQS queue we needed a way to make sense of the data for our web app to display.

Skills & Technologies

Message Processing

In order to process the incoming messages which were being recieved at a rate of about 1k/min. I created a Lamdba function in python, this was activated by AWS automatically when messages came in.

• Check the messages validity. Throw away if invalid.
• Transform the message from JSON to a delimited string.
• Push the message into a Kinesis Firehose.

To the frontend

The Kinsis Firehose was configured to stream the incoming data into a Redshift table. I then had the app grab summaries of the table on a scheduled basis and stored locally into its database system, which could then be pulled from by the frontend web app for display.