Skip to main content
The GCP Pub/Sub integration reads log entries that a Cloud Logging sink publishes to a Pub/Sub topic in your project, or the findings a Security Command Center notification config publishes there. Cotool pulls from a subscription dedicated to it and indexes the entries into Cotool Logs. You can connect several named sources, for example one per project or organization-wide sink, and pause, edit, or remove each independently.

Log types

Each source carries one log type.

Authentication

Cotool uses workload identity federation, so no service account key is created or stored. Cotool’s runtime identity presents a Google-signed token whose audience is unique to your Cotool organization; your workload identity provider accepts only that subject and audience, so no other Cotool customer can use your setup. You can grant the subscription to the federated principal directly, or have Cotool impersonate a service account of yours.
If your organization restricts workload identity pool providers with the iam.workloadIdentityPoolProviders policy, allow https://accounts.google.com.

Step 1: Create the sink, subscription, and federation

On the integration page, select Add log source, enter your project ID, and choose what to ingest: Cloud Logging through a log sink, or Security Command Center findings through a notification config. The setup section fills these, the logs to route, and Cotool’s Subject and your Audience into a Terraform configuration and a gcloud script. Each creates:
  • a topic and a subscription dedicated to Cotool (cotool-logs-<suffix>; 120-second ack deadline, 7-day retention, no expiry),
  • a dead-letter topic and subscription (cotool-logs-<suffix>-dead-letter) that keep messages Pub/Sub could not deliver after 100 attempts, with the Pub/Sub service agent allowed to move them there,
  • for Cloud Logging, a log sink with the filter you build under Logs to route, with permission to publish to the topic,
  • for Security Command Center, a notification config on your organization (or the project, if you leave Organization ID blank) that publishes findings matching its filter to the topic,
  • a workload identity pool (cotool-<suffix>) and OIDC provider that trust https://accounts.google.com only for Cotool’s subject and your audience,
  • roles/pubsub.subscriber on the subscription for the federated principal.
<suffix> is a short random value generated for each setup, so several sources can live in the same project. Under Logs to route, choose one of:
  • Log types: combine Cloud Audit Logs (the default), VPC flow logs, firewall rule logs, Cloud DNS queries, load balancer and Cloud Armor logs, GKE workloads, Cloud Run, and Compute Engine VMs.
  • All logs: every entry in the project.
  • Custom filter: any Cloud Logging query, for example resource.type="k8s_container" AND severity>=WARNING.
VPC flow logs, firewall rule logs, Cloud DNS queries, and load balancer logs are off in GCP until you turn them on for each subnet, rule, network, or backend service. Expand Turn on logging in GCP for the gcloud commands. For Security Command Center, the default finding filter, state = "ACTIVE" OR state = "INACTIVE", sends every finding create and update. Security Command Center only publishes an update that matches the filter, so a filter such as state = "ACTIVE" never sends the update that resolves a finding. Security Command Center must be activated on the organization or project, and an organization-wide config needs the Security Center Notification Configurations Editor role (roles/securitycenter.notificationConfigEditor) on the organization. Cotool keeps the filter with the source, so the setup guide on its settings shows the same configuration later.
Use a subscription dedicated to Cotool. Cotool acknowledges messages once they are ingested, so sharing a subscription with another SIEM would take messages away from it. Other tools can have their own subscription on the same topic.
Deploy it signed in as someone who can manage Pub/Sub, log sinks or Security Command Center notifications, IAM policies, and workload identity pools in the project:
  1. Save the configuration in your infrastructure repository, and authenticate the Google provider, for example with gcloud auth application-default login.
  2. Turn on the APIs it uses if they are off: gcloud services enable pubsub.googleapis.com logging.googleapis.com iam.googleapis.com sts.googleapis.com --project=<project-id>.
  3. For an organization-wide sink, use google_logging_organization_sink with include_children = true instead of the project sink.
  4. Run terraform init, review terraform plan, then run terraform apply.
  5. terraform output prints subscription and workload_identity_provider for the connection form.
To create the resources in the console instead, expand Setting it up in the console? for the issuer, subject, and audience the provider must trust.

Step 2: Connect the source

  1. Connect: enter your project ID and choose the logs to route or the findings to export. Deploy the setup, then paste the subscription path, the workload identity provider resource name, and optionally a service account to impersonate.
  2. Test: Cotool exchanges its token at your provider, checks that it can consume the subscription, and pulls up to ten messages. Tested messages are returned to the subscription, not acknowledged. If the subscription is empty, paste a sample log entry.
  3. Format: name the source and choose a format. Cotool previews the sampled entries parsed with it. A sink that routes a single log type reads with that type’s format, and Security Command Center sources use Security Command Center findings. Each of these formats rejects other entries, so choose Generic JSON or Custom format, normalized to OCSF when the sink routes several log types. To keep each log type in its own table, connect one source per log type.
  4. Select Enable source, or Save paused to enable it later. Enabling needs a passing test.

Reliability

  • Messages are acknowledged only after their entries are archived and indexed. If Cotool restarts mid-run, the ack deadline lapses and Pub/Sub redelivers them.
  • A temporary failure delays redelivery with backoff and leaves the messages in the subscription. The dead-letter policy is what lets Pub/Sub report each message’s delivery attempt: without one, the access test warns, Pub/Sub redelivers failing messages indefinitely, and entries Cotool Logs rejects are kept only in the raw archive after their first delivery.
  • Undecodable messages are kept in the raw archive with their original bytes and the reason, and acknowledged so they cannot block the subscription.
  • Redeliveries do not create duplicates: Cloud Logging entries are keyed by log name, insert ID, and timestamp, Security Command Center findings by finding name, event time, state, and mute, and generic entries by message ID.

Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, and recent errors.