TrueWatch SIEM on AWS EKS with Teams Channel Notification

(Hands-on demo for TrueWatch users & partners)

1 Introduction

This workshop guides you to implement TrueWatch SIEM on top of AWS EKS logs, and deliver alerts to a Microsoft Teams channel. You will:

• Stream CloudWatch Logs → Kinesis Data Streams → Firehose (HTTP Endpoint) → TrueWatch.
• Verify ingestion in Logs → Source → aws_firehose.
• Create a Teams Notification Target and an Alert Strategy.
• Import and enable AWS EKS SIEM detection rules.
• Validate end‑to‑end with a simple kubectl exec test.

2 Prerequisites

Keep all resources in the same AWS Region to simplify IAM and reduce latency.

3 Stream CloudWatch Logs to TrueWatch

We will use the TrueWatch Firehose HTTP Endpoint integration: https://docs.truewatch.com/integrations/aws_firehose_http_endpoint/

Step 1 Create S3 Backup Bucket

Create an S3 bucket for Firehose backup (defaults are fine; enable versioning/KMS per policy).

Step 2 Create Kinesis Data Stream

Create a Kinesis Data Stream (start with 1–2 shards; scale as needed).

Step 3 Create Lambda for Record Transformation

Create a Lambda Function and paste code from /resources/lambda_function.py. Ensure the handler/runtime/timeouts meet expected throughput.

Update Lambda Timeout to 5 mins.

Step 4 Create Firehose Delivery Stream (HTTP Endpoint)

1. Source: Amazon Kinesis Data Streams
2. Destination: HTTP Endpoint
3. Provide the Kinesis Data Stream and the Lambda for record transformation.

Obtain your TrueWatch access token under Integrations → DataKit.

Configure the HTTP endpoint URL and paste the access token. Select the S3 backup bucket created earlier, then create the Firehose stream.

If you need lower alert latency, reduce Firehose buffer interval; higher buffers reduce cost at the expense of latency. Replace your HTTP endpoint URL based on your TrueWatch region, e.g., za1, id1, ap1, etc.

Step 5 Create CloudWatch Logs Subscription Filter

Navigate to CloudWatch → Log groups → /aws/eks/<your_cluster_name>/cluster. (Repeat for other application log groups as needed.)
Actions → Subscription filters → Create Kinesis subscription filter.

Select the Kinesis Data Stream, choose/attach a role with kinesis:PutRecord/PutRecords, keep defaults, and Start streaming.

If creation fails, it’s usually an IAM permission issue on the selected role.

4 Verify Ingestion in TrueWatch

Go to TrueWatch → Logs → Source → aws_firehose and confirm records are arriving from Firehose.

5 Create a Teams Notification Target

1. In Microsoft Teams, create an Incoming Webhook via Workflows and copy the Webhook URL:
   https://support.microsoft.com/en-us/office/create-incoming-webhooks-with-workflows-for-microsoft-teams-8ae491c7-0394-4861-ba59-055e33f75498
2. In TrueWatch → Monitoring → Notification Targets, select Teams.

Paste the Webhook details and click Confirm.

6 Create an Alert Strategy

In TrueWatch → Monitoring → Alert Strategies → Create, provide a name/description. Under Notification Rules, select the relevant criticality (e.g., All) and choose the Teams target created above.

7 Import & Enable AWS EKS SIEM Detection Rules

Navigate to TrueWatch → Security → SIEM → Detection Rules. Click Import and select:
/resources/AWS EKS Detection Rules.json

Use Batch to Enable rules and update alert configuration to use your Alert Strategy/Teams target.

8 Test & Validate

From AWS CloudShell (or your shell with kubectl configured):

# Get a DataKit pod name
kubectl get pods -n datakit
 
# Generate a simple log line
kubectl exec -n datakit <datakit_pod_name> -- echo "hello"

Confirm a TrueWatch Event and Alert exist, and a message appears in your Teams channel.

9 Troubleshooting

• No data in TrueWatch

  • Check CloudWatch → Log group → Subscription filters status.
  • Review Kinesis metrics (IncomingRecords) and Firehose metrics (DeliveryToHttpSuccess/Failure).
  • Check Lambda logs for transformation errors/timeouts.
  • Verify HTTP Endpoint URL/token in Firehose.

• Firehose 4xx/5xx to HTTP Endpoint

  • 401/403: wrong/missing access token.
  • 413/429: reduce record size in Lambda or adjust Firehose buffering.
  • 5xx: Firehose retries; ensure S3 backup for forensics.

• Subscription filter creation fails

  • Role must have kinesis:PutRecord/PutRecords permissions on the stream ARN.

• Teams message not delivered

  • Validate Webhook URL and tenant policies; some orgs restrict external webhooks.

10 References

• TrueWatch Firehose HTTP Endpoint: https://docs.truewatch.com/integrations/aws_firehose_http_endpoint/
• EKS reference setup: https://github.com/TrueWatchTech/idurar-demo-setup-guide

© 2025 TrueWatch Demo Workshop