JASON NEUROHR

ECS Container Logs to Slack via AWS Lambda


Saturday, June 22, 2019

This was more of a discovery article to document the process for getting logs from ECS containers into a messaging service (Slack in this case). I used data from this website’s logs, specifically when the contact form receives spam data and when people go to suspicious paths that don’t exist (primarily automated scanners looking for the WordPress login page or phpMyAdmin…). The below topics are covered:

  • Configuring containers running in AWS ECS to send logs to AWS CloudWatch
  • Setting the AWS CloudWatch Log Group to Stream the data to an AWS Lambda function
  • The Lambda function code for sending the data as Slack messages

Configuring ECS Containers to Send Logs to CloudWatch

I’m assuming if your reading this you have familiarity with AWS ECS and so I’ll jump straight to the good bit.

From the ECS task definition, create a new revision. Select the relevant container definition for which you want the logs.

Scroll down to the Storage and Logging section and configure the Logging details. See the below example.

Setup Slack Incoming Webhook

To receive messages in Slack, a new app needs to be added to the workspace and an incoming webhook generated.

Navigate to https://api.slack.com/apps, select Create New App.

Enter an App Name and select a Development Slack Workspace. Click Create App.

From the basic information page, click Incoming Webhooks.

Activate the incoming webhook. Click Add New Webhook to Workspace.

Select the channel to add the webhook in. Click Install.

Copy the webhook URL for use later.

Create a Lambda Function to Send Messages to a Slack Incoming Webhook

In my case, I used the AWS Toolkit for Visual Studio Code to create the Lambda function. I won’t go into the details of how to use the extension here; however, at a high level, the extension is using Cloudformation to configure Lambda. The only significant change I made to the default template was to remove the API Gateway specifics (since I’m not using them here).

The app.py code I used to send messages to Slack is shown below.

At a high level:

  • event_decoder() is used to decode the incoming event data from CloudWatch.
  • send_notification() is used to send the actual slack message.
  • In main, event_decoder() is called, then using the return checked for some arbitrary message text, in this case, “Bad” which is part of the text when a bad contact form entry is made and “HTTP/1.1” 418" which occurs every time someone hits a suspicious URL like wp-login.php. See the references for more on 418.

Using the Visual Studio Extension, publish the Lambda function to the AWS account. This will require selecting a deployment S3 bucket, a region, and a function name.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

import json
import logging
from base64 import b64decode
from gzip import decompress
from io import StringIO
from os import getenv

import requests

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def event_decoder(event):
    logger.debug(f"Decoding event")

    event_data = event["awslogs"]["data"]
    cwl_compressed_payload = b64decode(event_data)
    cwl_uncompressed_payload = decompress(cwl_compressed_payload)
    cwl_payload = json.loads(cwl_uncompressed_payload)

    return cwl_payload


def send_notification(message, category):
    slack_uri = getenv("SLACK_WEBHOOK")

    # For reference - https://api.slack.com/docs/message-attachments
    attachment = dict(
        {
            "attachments": [
                {
                    "title": f"CloudWatch Log Alarm: {category.upper()}",
                    "text": message,
                    "color": "danger",
                }
            ]
        }
    )

    req_data = json.dumps(attachment)

    req_headers = {"Content-type": "application/json"}
    req = requests.post(url=slack_uri, headers=req_headers, data=req_data)


def lambda_handler(event, context):
    logger.info(f"lambda_handler entered")
    logger.debug(f"Log Stream Name: {context.log_stream_name}")
    logger.debug(f"Log Group Name: {context.log_group_name}")
    logger.debug(f"Log Events: {json.dumps(event)}")

    decoded_event = event_decoder(event)

    logger.debug(f"Decoded: {decoded_event}")

    for event_message in decoded_event["logEvents"]:
        message = event_message["message"]

        if str(message).find("Bad") >= 0:
            send_notification(message, "BAD CONTACT FORM")

        if str(message).find('HTTP/1.1" 418') >= 0:
            send_notification(message, "I'M A TEAPOT")

Be sure to check the GitHub repository for an updated version.

Once the Lambda function has successfully deployed, navigate to the settings and define an environment variable SLACK_WEBHOOK. Paste in the incoming webhook URL saved earlier as the value and save the changes.

Configure the AWS CloudWatch Log Group to Stream to AWS Lambda

Navigate to CloudWatch from the AWS Console, click Logs from the left-hand navigation.

Select the ECS log group, click Actions, click Stream to AWS Lambda.

Select the Lambda function created earlier. Click next.

Select the AWS Lambda log format, click next.

Click Start Streaming.

At this point logs coming from the ECS container/s matching the patterns from the Lambda function code will trigger a Slack message to be sent to the configured channel.

References