Skip to main content
The GCP Pub/Sub integration reads log entries that a Cloud Logging sink publishes to a Pub/Sub topic in your project. Cotool pulls from a subscription dedicated to it and indexes the entries into Cotool Logs. You can connect several named sources, for example one per project or organization-wide sink, and pause, edit, or remove each independently.

​
Log types

Each source carries one log type.

​
Authentication

Cotool uses workload identity federation, so no service account key is created or stored. Cotool’s runtime identity presents a Google-signed token whose audience is unique to your Cotool organization; your workload identity provider accepts only that subject and audience, so no other Cotool customer can use your setup. You can grant the subscription to the federated principal directly, or have Cotool impersonate a service account of yours.
If your organization restricts workload identity pool providers with the iam.workloadIdentityPoolProviders policy, allow https://accounts.google.com.

​
Step 1: Create the sink, subscription, and federation

On the integration page, select Add log source and enter your project ID. The setup section fills it, the logs to route, and Cotool’s Subject and your Audience into a Terraform configuration and a gcloud script. Each creates:
  • a topic and a subscription dedicated to Cotool (cotool-logs-<suffix>; 120-second ack deadline, 7-day retention, no expiry),
  • a dead-letter topic and subscription (cotool-logs-<suffix>-dead-letter) that keep messages Pub/Sub could not deliver after 100 attempts, with the Pub/Sub service agent allowed to move them there,
  • a log sink with the filter you build under Logs to route, with permission to publish to the topic,
  • a workload identity pool (cotool-<suffix>) and OIDC provider that trust https://accounts.google.com only for Cotool’s subject and your audience,
  • roles/pubsub.subscriber on the subscription for the federated principal.
<suffix> is a short random value generated for each setup, so several sources can live in the same project. Under Logs to route, choose one of:
  • Log types: combine Cloud Audit Logs (the default), VPC flow logs, firewall rule logs, Cloud DNS queries, load balancer and Cloud Armor logs, GKE workloads, Cloud Run, and Compute Engine VMs.
  • All logs: every entry in the project.
  • Custom filter: any Cloud Logging query, for example resource.type="k8s_container" AND severity>=WARNING.
Cotool keeps the filter with the source, so the setup guide on its settings shows the same configuration later.
Use a subscription dedicated to Cotool. Cotool acknowledges messages once they are ingested, so sharing a subscription with another SIEM would take messages away from it. Other tools can have their own subscription on the same topic.
Deploy it signed in as someone who can manage Pub/Sub, log sinks, IAM policies, and workload identity pools in the project:
  1. Save the configuration in your infrastructure repository, and authenticate the Google provider, for example with gcloud auth application-default login.
  2. Turn on the APIs it uses if they are off: gcloud services enable pubsub.googleapis.com logging.googleapis.com iam.googleapis.com sts.googleapis.com --project=<project-id>.
  3. For an organization-wide sink, use google_logging_organization_sink with include_children = true instead of the project sink.
  4. Run terraform init, review terraform plan, then run terraform apply.
  5. terraform output prints subscription and workload_identity_provider for the connection form.
To create the resources in the console instead, expand Setting it up in the console? for the issuer, subject, and audience the provider must trust.

​
Step 2: Connect the source

  1. Connect: enter your project ID and choose the logs to route. Deploy the setup, then paste the subscription path, the workload identity provider resource name, and optionally a service account to impersonate.
  2. Test: Cotool exchanges its token at your provider, checks that it can consume the subscription, and pulls up to ten messages. Tested messages are returned to the subscription, not acknowledged. If the subscription is empty, paste a sample log entry.
  3. Format: name the source and choose GCP Cloud Audit Logs, Generic JSON, or Custom format, normalized to OCSF. Cotool previews the sampled entries parsed with that format. The Cloud Audit Logs format skips other entries, so choose Generic JSON or OCSF when the sink routes more than audit logs.
  4. Select Enable source, or Save paused to enable it later. Enabling needs a passing test.

​
Reliability

  • Messages are acknowledged only after their entries are archived and indexed. If Cotool restarts mid-run, the ack deadline lapses and Pub/Sub redelivers them.
  • A temporary failure delays redelivery with backoff and leaves the messages in the subscription. The dead-letter policy is what lets Pub/Sub report each message’s delivery attempt: without one, the access test warns, Pub/Sub redelivers failing messages indefinitely, and entries Cotool Logs rejects are kept only in the raw archive after their first delivery.
  • Undecodable messages are kept in the raw archive with their original bytes and the reason, and acknowledged so they cannot block the subscription.
  • Redeliveries do not create duplicates: audit entries are keyed by log name, insert ID, and timestamp, and generic entries by message ID.

​
Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, and recent errors.