Skip to main content
The Azure Blob Storage integration reads the flow logs Network Watcher writes to your storage accounts. Event Grid sends each new block of a log blob to a Storage queue dedicated to Cotool; Cotool drains the queue every minute, reads only the bytes it has not read yet, and indexes the flow tuples into Cotool Logs. You can connect several named sources, for example one per subscription or region, and pause, edit, or remove each independently.

​
Log types

By default Cotool indexes denied flows, and flow begin and end tuples that involve a public IP address. Continuation tuples and allowed traffic between private addresses are kept in the raw archive only, so byte totals in azure_flow_logs cover the indexed flows. The Azure detections in the detection library read azure_flow_logs: internet access to management and database ports, scans followed by an allowed connection, internal and outbound denied bursts, traffic with threat-listed addresses, outbound connections to restricted countries, and DNS sent to unapproved resolvers. They are written to work with this default filter.

​
Authentication

Cotool signs in as a user-assigned managed identity in your tenant through a federated identity credential, so no client secret or storage key is created or stored. Cotool’s runtime identity presents a Google-signed token (issuer https://accounts.google.com) whose audience is unique to your Cotool organization; the federated credential accepts only Cotool’s subject and that audience, so no other Cotool customer can use your identity. The identity holds two roles, each scoped as narrowly as Azure allows:
  • Storage Queue Data Message Processor on the Cotool queue, to read and delete its messages.
  • Storage Blob Data Reader on each log container, to read the flow log blobs.

​
Step 1: Create the queue, Event Grid subscription, and identity

On the integration page, select Add log source and enter the storage account and container that hold the flow logs. Network Watcher writes VNet flow logs to the insights-logs-flowlogflowevent container and legacy NSG flow logs to insights-logs-networksecuritygroupflowevent. Add a prefix to limit Cotool to part of a container. VNet flow logs are stored under flowLogResourceID=/<SUBSCRIPTION-ID>_<NETWORK-WATCHER-RESOURCE-GROUP>/<NETWORK-WATCHER>_<FLOW-LOG-NAME>/, so flowLogResourceID=/<SUBSCRIPTION-ID>_ limits a source to one subscription. NSG flow logs are stored under resourceId=/SUBSCRIPTIONS/<SUBSCRIPTION-ID>/. Azure writes these folder names in upper case; copy the prefix from the container in the Azure portal. The setup section fills the containers and Cotool’s Subject and your Audience into a Terraform configuration and an Azure CLI script. Each creates:
  • a Storage queue dedicated to Cotool (cotool-logs-<suffix>) in the first log storage account,
  • an Event Grid system topic for each log storage account, or reuses the one it already has (a storage account can have only one),
  • an event subscription for each container and prefix that sends Microsoft.Storage.BlobCreated events to the queue, with the queue’s 7-day maximum message lifetime,
  • a user-assigned managed identity (cotool-logs-<suffix>) with a federated credential that trusts only Cotool’s subject and your audience,
  • the two role assignments above,
  • Storage Queue Data Message Sender on the queue for each system topic’s system-assigned managed identity, which Event Grid delivers as, so delivery works from every log storage account and with shared-key access disabled.
<suffix> is a short random value generated for each setup, so several sources can live in the same subscription.
Use a queue dedicated to Cotool. Cotool deletes messages once their blobs are ingested, so sharing the queue with another tool would take messages away from it.
Deploy it signed in as someone who can create managed identities, role assignments, Storage queues, and Event Grid subscriptions in the subscription:
  1. Save the configuration in your infrastructure repository, and sign in with az login.
  2. Run terraform init, review terraform plan, then run terraform apply. It asks for the subscription that holds the storage accounts and a resource group for the managed identity. If a storage account already has an Event Grid system topic, name it in existing_system_topics and enable its system-assigned managed identity.
  3. terraform output prints queue_url, tenant_id, and client_id for the connection form.
To create the resources in the portal instead, expand Setting it up in the console? for the issuer, subject, and audience the federated credential must trust.
The storage accounts must allow access from public networks. Cotool cannot reach storage accounts that only accept private endpoints.

​
Step 2: Connect the source

  1. Connect: enter the storage accounts, containers, and prefixes. Deploy the setup, then paste the queue URL, tenant ID, and client ID.
  2. Test: Cotool signs in as the managed identity, checks that it can read the queue, and peeks at up to ten messages without removing them. It reads the start of the first blob they announce to check blob access. If the queue is empty, paste a sample flow log record.
  3. Format: name the source. Cotool previews the sampled flow tuples.
  4. Select Enable source, or Save paused to enable it later. Enabling needs a passing test.

​
Reliability

  • Network Watcher appends a block to each hourly PT1H.json blob about once a minute, and Event Grid announces every block. Cotool stores how far it has read each blob, so each run reads only the new bytes, and several announcements of the same blob in a run become one read.
  • Messages are deleted only after their bytes are archived and indexed. If Cotool restarts mid-run, or a run ends before reaching every message it received, the messages become visible again when the run’s time budget ends and are read again.
  • Rereading a blob does not create duplicates: flow tuples are keyed by their blob and byte position.
  • A blob that is deleted, or a message for a blob outside the allowed containers, is skipped and reported in the source’s recent errors.
  • Permission errors leave the messages on the queue, so they are read once access is fixed, until the queue’s 7-day message lifetime expires.

​
Status

Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, the number of queued messages, and recent errors.