> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cotool.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AWS S3/SQS Log Ingestion

> Ingest AWS CloudTrail and other JSON log files from S3 through a dedicated SQS queue

The AWS S3/SQS integration reads log files your AWS services already write to S3. Each new object is announced on an SQS queue dedicated to Cotool; Cotool assumes a role in your account, receives the notification, reads the object, and indexes its events into Cotool Logs.

```mermaid theme={null}
flowchart LR
    Producer["CloudTrail / log producer"] --> S3["S3 bucket"]
    S3 -- "ObjectCreated notification" --> SQS["Dedicated SQS queue"]
    SQS --> Cotool["Cotool (assumes your role)"]
    Cotool -- "s3:GetObject" --> S3
```

You can connect several named sources, for example one per environment or AWS organization, and pause, edit, or remove each independently.

## Log types

Each source carries one log type.

| Log type | Cotool Logs table | What Cotool does |
| - | - | - |
| AWS CloudTrail | `aws_cloudtrail` | Unwraps CloudTrail's `Records` envelope and maps actor, action, service, resource, source IP, outcome, account, and region into columns shared with `gcp_cloud_audit`. Digest files are skipped. Events are keyed by `eventID`. |
| Generic JSON | `aws_generic_logs` | Stores each JSON record with its original payload, top-level fields as a searchable `fields` map, and the object it came from. Records are not normalized and no detection coverage is implied. |

Supported formats are JSON, JSON arrays, newline-delimited JSON, and gzip-compressed versions of each. Objects in other formats (Parquet, ZIP, and so on) are reported with an actionable error on the source instead of being ingested.

## Step 1: Create the queue and role

On the integration page, select **Add log source** and enter each bucket and prefix, and any customer-managed KMS keys that encrypt the objects. The setup section fills them, the **Cotool principal**, and your **External ID** into a Terraform configuration, an AWS CLI script, and a CloudFormation template. Each creates:

* a dedicated SQS queue (`cotool-log-ingest-<suffix>`) with a dead-letter queue, and a queue policy that accepts notifications from your log buckets,
* an IAM role (`CotoolLogIngest-<suffix>`) that trusts the Cotool principal only with your external ID, and can:
  * `sqs:ReceiveMessage`, `sqs:DeleteMessage`, `sqs:ChangeMessageVisibility`, and `sqs:GetQueueAttributes` on that queue,
  * `s3:GetObject` and `s3:GetObjectVersion` on each bucket prefix,
  * `kms:Decrypt` on the log keys, when objects are encrypted with customer-managed KMS keys.

`<suffix>` is a short random value generated for each setup, so several sources can live in the same account. Cotool only assumes roles whose name starts with `CotoolLogIngest`; keep that prefix if you rename the role.

<Warning>
  Use a queue dedicated to Cotool. Cotool deletes messages once their objects
  are ingested, so sharing a queue with another SIEM would take messages away
  from it. If another tool already consumes the bucket's notifications, have
  the bucket notify an SNS topic and subscribe both queues to it by adapting
  the generated configuration to use SNS.
</Warning>

## Step 2: Deploy it

Deploy in the buckets' account and region: S3 only notifies queues in its own region. S3 keeps a single notification configuration per bucket, so the generated notification script adds `s3:ObjectCreated:*` notifications for each prefix only to buckets that have none, and prints what to add to the others.

<Tabs>
  <Tab title="Terraform">
    1. Save the configuration in your infrastructure repository, and configure the AWS provider for the buckets' account and region.
    2. Run `terraform init`, review `terraform plan`, then run `terraform apply`.
    3. Run the generated `cotool-bucket-notifications.sh`, which reads the queue from `terraform output`. The configuration only manages the notifications itself when you set `manage_bucket_notifications = true`, because `aws_s3_bucket_notification` replaces a bucket's entire notification configuration. If Terraform already manages a bucket's notifications, add the queue to that resource instead.
    4. `terraform output` prints `queue_url` and `role_arn` for the connection form.
  </Tab>

  <Tab title="AWS CLI">
    1. Save the script. It needs AWS CLI v2, signed in to the buckets' account with permission to create SQS queues and IAM roles and to change the buckets' notifications.
    2. Run `AWS_REGION=<bucket-region> bash cotool-log-ingest.sh`.
    3. It prints `queue_url` and `role_arn` for the connection form. If a bucket already has notifications, the script leaves them in place and prints the queue ARN to add to them.
  </Tab>

  <Tab title="CloudFormation">
    1. Save the template, and deploy it in the buckets' region with the generated `aws cloudformation deploy` command (stack `cotool-log-ingest-<suffix>`), or upload it under **Create stack** in the CloudFormation console.
    2. A stack can't change a bucket it didn't create, so add the notifications once the stack is up: run the generated `cotool-bucket-notifications.sh`, or in the S3 console open each bucket's **Properties > Event notifications** and send all object create events for the prefix to the `cotool-log-ingest-<suffix>` queue.
    3. The stack's `QueueUrl` and `RoleArn` outputs go into the connection form.
  </Tab>
</Tabs>

To create the resources in the console instead, expand **Setting it up in the console?** for the principal and external ID the role must trust.

## Step 3: Connect the source

1. **Connect**: enter the queue URL (the region fills in from it), the role ARN, and each bucket and prefix Cotool may read. Objects outside these prefixes are skipped even if they reach the queue.
2. **Test**: Cotool assumes the role, reads the queue attributes, samples up to ten messages, and reads one announced object. Sampled messages are returned to the queue, not deleted. If the queue is empty, paste a sample log.
3. **Format**: name the source and choose **AWS CloudTrail**, **Generic JSON**, or **Custom format, normalized to OCSF**. Cotool previews the sampled events parsed with that format.
4. Select **Enable source**, or **Save paused** to enable it later. Enabling needs a passing test.

## Reliability

* Messages are deleted only after the object's events are archived and indexed. If Cotool restarts mid-run, the message becomes visible again and is reprocessed.
* A temporary failure (throttling, a network error, or access denied while permissions are being fixed) keeps the message on the queue and retries it with backoff up to 15 minutes, for about a day with the template's dead-letter settings.
* Malformed lines are skipped individually and kept in the raw archive with the reason. Objects that can never be read (unsupported format, deleted, too large) are recorded on the source and their messages removed so they cannot block the queue.
* Redelivered notifications do not create duplicates: CloudTrail events are keyed by `eventID`, and generic records by object version and position.

## Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, the approximate queue depth, and recent errors with the object they concern.