Skip to main content
The AWS S3/SQS integration reads log files your AWS services already write to S3. Each new object is announced on an SQS queue dedicated to Cotool; Cotool assumes a role in your account, receives the notification, reads the object, and indexes its events into Cotool Logs. You can connect several named sources, for example one per environment or AWS organization, and pause, edit, or remove each independently.

​
Log types

Each source carries one log type. Supported formats are JSON, JSON arrays, newline-delimited JSON, and gzip-compressed versions of each. Objects in other formats (Parquet, ZIP, and so on) are reported with an actionable error on the source instead of being ingested.

​
Step 1: Create the queue and role

On the integration page, select Add log source and enter each bucket and prefix, and any customer-managed KMS keys that encrypt the objects. The setup section fills them, the Cotool principal, and your External ID into a Terraform configuration, an AWS CLI script, and a CloudFormation template. Each creates:
  • a dedicated SQS queue (cotool-log-ingest-<suffix>) with a dead-letter queue, and a queue policy that accepts notifications from your log buckets,
  • an IAM role (CotoolLogIngest-<suffix>) that trusts the Cotool principal only with your external ID, and can:
    • sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, and sqs:GetQueueAttributes on that queue,
    • s3:GetObject and s3:GetObjectVersion on each bucket prefix,
    • kms:Decrypt on the log keys, when objects are encrypted with customer-managed KMS keys.
<suffix> is a short random value generated for each setup, so several sources can live in the same account. Cotool only assumes roles whose name starts with CotoolLogIngest; keep that prefix if you rename the role.
Use a queue dedicated to Cotool. Cotool deletes messages once their objects are ingested, so sharing a queue with another SIEM would take messages away from it. If another tool already consumes the bucket’s notifications, have the bucket notify an SNS topic and subscribe both queues to it by adapting the generated configuration to use SNS.

​
Step 2: Deploy it

Deploy in the buckets’ account and region: S3 only notifies queues in its own region. S3 keeps a single notification configuration per bucket, so the generated notification script adds s3:ObjectCreated:* notifications for each prefix only to buckets that have none, and prints what to add to the others.
  1. Save the configuration in your infrastructure repository, and configure the AWS provider for the buckets’ account and region.
  2. Run terraform init, review terraform plan, then run terraform apply.
  3. Run the generated cotool-bucket-notifications.sh, which reads the queue from terraform output. The configuration only manages the notifications itself when you set manage_bucket_notifications = true, because aws_s3_bucket_notification replaces a bucket’s entire notification configuration. If Terraform already manages a bucket’s notifications, add the queue to that resource instead.
  4. terraform output prints queue_url and role_arn for the connection form.
To create the resources in the console instead, expand Setting it up in the console? for the principal and external ID the role must trust.

​
Step 3: Connect the source

  1. Connect: enter the queue URL (the region fills in from it), the role ARN, and each bucket and prefix Cotool may read. Objects outside these prefixes are skipped even if they reach the queue.
  2. Test: Cotool assumes the role, reads the queue attributes, samples up to ten messages, and reads one announced object. Sampled messages are returned to the queue, not deleted. If the queue is empty, paste a sample log.
  3. Format: name the source and choose AWS CloudTrail, Generic JSON, or Custom format, normalized to OCSF. Cotool previews the sampled events parsed with that format.
  4. Select Enable source, or Save paused to enable it later. Enabling needs a passing test.

​
Reliability

  • Messages are deleted only after the object’s events are archived and indexed. If Cotool restarts mid-run, the message becomes visible again and is reprocessed.
  • A temporary failure (throttling, a network error, or access denied while permissions are being fixed) keeps the message on the queue and retries it with backoff up to 15 minutes, for about a day with the template’s dead-letter settings.
  • Malformed lines are skipped individually and kept in the raw archive with the reason. Objects that can never be read (unsupported format, deleted, too large) are recorded on the source and their messages removed so they cannot block the queue.
  • Redelivered notifications do not create duplicates: CloudTrail events are keyed by eventID, and generic records by object version and position.

​
Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, the approximate queue depth, and recent errors with the object they concern.