> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cotool.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GCP Pub/Sub Log Ingestion

> Ingest GCP Cloud Audit Logs and other log entries from a dedicated Pub/Sub subscription

The GCP Pub/Sub integration reads log entries that a Cloud Logging sink publishes to a Pub/Sub topic in your project. Cotool pulls from a subscription dedicated to it and indexes the entries into Cotool Logs.

```mermaid theme={null}
flowchart LR
    Logging["Cloud Logging"] --> Sink["Filtered sink"]
    Sink --> Topic["Pub/Sub topic"]
    Topic --> Subscription["Dedicated subscription"]
    Subscription --> Cotool["Cotool (workload identity federation)"]
```

You can connect several named sources, for example one per project or organization-wide sink, and pause, edit, or remove each independently.

## Log types

Each source carries one log type.

| Log type | Cotool Logs table | What Cotool does |
| - | - | - |
| GCP Cloud Audit Logs | `gcp_cloud_audit` | Maps Admin Activity, Data Access, System Event, and Policy Denied entries into actor, action, service, resource, source IP, outcome, project, and region columns shared with `aws_cloudtrail`, plus impersonation chains and permissions. Other entries are skipped. |
| Generic JSON | `gcp_generic_logs` | Stores each entry with its original payload, top-level fields as a searchable `fields` map, and the Pub/Sub message attributes. Entries are not normalized and no detection coverage is implied. |

## Authentication

Cotool uses **workload identity federation**, so no service account key is created or stored. Cotool's runtime identity presents a Google-signed token whose audience is unique to your Cotool organization; your workload identity provider accepts only that subject and audience, so no other Cotool customer can use your setup. You can grant the subscription to the federated principal directly, or have Cotool impersonate a service account of yours.

<Note>
  If your organization restricts workload identity pool providers with the
  `iam.workloadIdentityPoolProviders` policy, allow
  `https://accounts.google.com`.
</Note>

## Step 1: Create the sink, subscription, and federation

On the integration page, select **Add log source** and enter your project ID. The setup section fills it, the logs to route, and Cotool's **Subject** and your **Audience** into a Terraform configuration and a gcloud script. Each creates:

* a topic and a subscription dedicated to Cotool (`cotool-logs-<suffix>`; 120-second ack deadline, 7-day retention, no expiry),
* a dead-letter topic and subscription (`cotool-logs-<suffix>-dead-letter`) that keep messages Pub/Sub could not deliver after 100 attempts, with the Pub/Sub service agent allowed to move them there,
* a log sink with the filter you build under **Logs to route**, with permission to publish to the topic,
* a workload identity pool (`cotool-<suffix>`) and OIDC provider that trust `https://accounts.google.com` only for Cotool's subject and your audience,
* `roles/pubsub.subscriber` on the subscription for the federated principal.

`<suffix>` is a short random value generated for each setup, so several sources can live in the same project.

Under **Logs to route**, choose one of:

* **Log types**: combine Cloud Audit Logs (the default), VPC flow logs, firewall rule logs, Cloud DNS queries, load balancer and Cloud Armor logs, GKE workloads, Cloud Run, and Compute Engine VMs.
* **All logs**: every entry in the project.
* **Custom filter**: any [Cloud Logging query](https://cloud.google.com/logging/docs/view/logging-query-language), for example `resource.type="k8s_container" AND severity>=WARNING`.

Cotool keeps the filter with the source, so the setup guide on its settings shows the same configuration later.

<Warning>
  Use a subscription dedicated to Cotool. Cotool acknowledges messages once
  they are ingested, so sharing a subscription with another SIEM would take
  messages away from it. Other tools can have their own subscription on the
  same topic.
</Warning>

Deploy it signed in as someone who can manage Pub/Sub, log sinks, IAM policies, and workload identity pools in the project:

<Tabs>
  <Tab title="Terraform">
    1. Save the configuration in your infrastructure repository, and authenticate the Google provider, for example with `gcloud auth application-default login`.
    2. Turn on the APIs it uses if they are off: `gcloud services enable pubsub.googleapis.com logging.googleapis.com iam.googleapis.com sts.googleapis.com --project=<project-id>`.
    3. For an organization-wide sink, use `google_logging_organization_sink` with `include_children = true` instead of the project sink.
    4. Run `terraform init`, review `terraform plan`, then run `terraform apply`.
    5. `terraform output` prints `subscription` and `workload_identity_provider` for the connection form.
  </Tab>

  <Tab title="gcloud CLI">
    1. Save the script, and sign in with `gcloud auth login`.
    2. Run `bash cotool-log-ingest.sh`. It turns on the APIs it needs first.
    3. It prints `subscription` and `workload_identity_provider` for the connection form.
  </Tab>
</Tabs>

To create the resources in the console instead, expand **Setting it up in the console?** for the issuer, subject, and audience the provider must trust.

## Step 2: Connect the source

1. **Connect**: enter your project ID and choose the logs to route. Deploy the setup, then paste the subscription path, the workload identity provider resource name, and optionally a service account to impersonate.
2. **Test**: Cotool exchanges its token at your provider, checks that it can consume the subscription, and pulls up to ten messages. Tested messages are returned to the subscription, not acknowledged. If the subscription is empty, paste a sample log entry.
3. **Format**: name the source and choose **GCP Cloud Audit Logs**, **Generic JSON**, or **Custom format, normalized to OCSF**. Cotool previews the sampled entries parsed with that format. The Cloud Audit Logs format skips other entries, so choose Generic JSON or OCSF when the sink routes more than audit logs.
4. Select **Enable source**, or **Save paused** to enable it later. Enabling needs a passing test.

## Reliability

* Messages are acknowledged only after their entries are archived and indexed. If Cotool restarts mid-run, the ack deadline lapses and Pub/Sub redelivers them.
* A temporary failure delays redelivery with backoff and leaves the messages in the subscription. The dead-letter policy is what lets Pub/Sub report each message's delivery attempt: without one, the access test warns, Pub/Sub redelivers failing messages indefinitely, and entries Cotool Logs rejects are kept only in the raw archive after their first delivery.
* Undecodable messages are kept in the raw archive with their original bytes and the reason, and acknowledged so they cannot block the subscription.
* Redeliveries do not create duplicates: audit entries are keyed by log name, insert ID, and timestamp, and generic entries by message ID.

## Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, and recent errors.