> ## Documentation Index > Fetch the complete documentation index at: https://docs.cotool.ai/llms.txt > Use this file to discover all available pages before exploring further. # Azure Blob Storage Log Ingestion > Ingest Azure Network Watcher VNet flow logs from Blob Storage as Event Grid announces them The Azure Blob Storage integration reads the flow logs Network Watcher writes to your storage accounts. Event Grid sends each new block of a log blob to a Storage queue dedicated to Cotool; Cotool drains the queue every minute, reads only the bytes it has not read yet, and indexes the flow tuples into Cotool Logs. ```mermaid theme={null} flowchart LR Watcher["Network Watcher flow logs"] --> Blobs["Storage container"] Blobs --> Grid["Event Grid BlobCreated"] Grid --> Queue["Dedicated Storage queue"] Queue --> Cotool["Cotool (federated managed identity)"] Blobs --> Cotool ``` You can connect several named sources, for example one per subscription or region, and pause, edit, or remove each independently. ## Log types | Log type | Cotool Logs table | What Cotool does | | - | - | - | | Azure VNet flow logs | `azure_flow_logs` | Reads virtual network flow logs (and legacy NSG flow logs) into one row per flow tuple: source and destination IP and port, protocol, direction, flow state, encryption, packet and byte counters, the matched NSG and rule, the covered resource, and IP geolocation and threat-feed labels for the remote peer (the source of an inbound flow, the destination of an outbound one). | By default Cotool indexes denied flows, and flow begin and end tuples that involve a public IP address. Continuation tuples and allowed traffic between private addresses are kept in the raw archive only, so byte totals in `azure_flow_logs` cover the indexed flows. The Azure detections in the detection library read `azure_flow_logs`: internet access to management and database ports, scans followed by an allowed connection, internal and outbound denied bursts, traffic with threat-listed addresses, outbound connections to restricted countries, and DNS sent to unapproved resolvers. They are written to work with this default filter. ## Authentication Cotool signs in as a **user-assigned managed identity** in your tenant through a federated identity credential, so no client secret or storage key is created or stored. Cotool's runtime identity presents a Google-signed token (issuer `https://accounts.google.com`) whose audience is unique to your Cotool organization; the federated credential accepts only Cotool's subject and that audience, so no other Cotool customer can use your identity. The identity holds two roles, each scoped as narrowly as Azure allows: * **Storage Queue Data Message Processor** on the Cotool queue, to read and delete its messages. * **Storage Blob Data Reader** on each log container, to read the flow log blobs. ## Step 1: Create the queue, Event Grid subscription, and identity On the integration page, select **Add log source** and enter the storage account and container that hold the flow logs. Network Watcher writes VNet flow logs to the `insights-logs-flowlogflowevent` container and legacy NSG flow logs to `insights-logs-networksecuritygroupflowevent`. Add a prefix to limit Cotool to part of a container. VNet flow logs are stored under `flowLogResourceID=/_/_/`, so `flowLogResourceID=/_` limits a source to one subscription. NSG flow logs are stored under `resourceId=/SUBSCRIPTIONS//`. Azure writes these folder names in upper case; copy the prefix from the container in the Azure portal. The setup section fills the containers and Cotool's **Subject** and your **Audience** into a Terraform configuration and an Azure CLI script. Each creates: * a Storage queue dedicated to Cotool (`cotool-logs-`) in the first log storage account, * an Event Grid system topic for each log storage account, or reuses the one it already has (a storage account can have only one), * an event subscription for each container and prefix that sends `Microsoft.Storage.BlobCreated` events to the queue, with the queue's 7-day maximum message lifetime, * a user-assigned managed identity (`cotool-logs-`) with a federated credential that trusts only Cotool's subject and your audience, * the two role assignments above, * **Storage Queue Data Message Sender** on the queue for each system topic's system-assigned managed identity, which Event Grid delivers as, so delivery works from every log storage account and with shared-key access disabled. `` is a short random value generated for each setup, so several sources can live in the same subscription. Use a queue dedicated to Cotool. Cotool deletes messages once their blobs are ingested, so sharing the queue with another tool would take messages away from it. Deploy it signed in as someone who can create managed identities, role assignments, Storage queues, and Event Grid subscriptions in the subscription: 1. Save the configuration in your infrastructure repository, and sign in with `az login`. 2. Run `terraform init`, review `terraform plan`, then run `terraform apply`. It asks for the subscription that holds the storage accounts and a resource group for the managed identity. If a storage account already has an Event Grid system topic, name it in `existing_system_topics` and enable its system-assigned managed identity. 3. `terraform output` prints `queue_url`, `tenant_id`, and `client_id` for the connection form. 1. Save the script, sign in with `az login`, and select the subscription with `az account set --subscription `. 2. Run `RESOURCE_GROUP= bash cotool-log-ingest.sh`. It reuses a storage account's existing Event Grid system topic, enabling its system-assigned managed identity if it has none. 3. It prints `queue_url`, `tenant_id`, and `client_id` for the connection form. To create the resources in the portal instead, expand **Setting it up in the console?** for the issuer, subject, and audience the federated credential must trust. The storage accounts must allow access from public networks. Cotool cannot reach storage accounts that only accept private endpoints. ## Step 2: Connect the source 1. **Connect**: enter the storage accounts, containers, and prefixes. Deploy the setup, then paste the queue URL, tenant ID, and client ID. 2. **Test**: Cotool signs in as the managed identity, checks that it can read the queue, and peeks at up to ten messages without removing them. It reads the start of the first blob they announce to check blob access. If the queue is empty, paste a sample flow log record. 3. **Format**: name the source. Cotool previews the sampled flow tuples. 4. Select **Enable source**, or **Save paused** to enable it later. Enabling needs a passing test. ## Reliability * Network Watcher appends a block to each hourly `PT1H.json` blob about once a minute, and Event Grid announces every block. Cotool stores how far it has read each blob, so each run reads only the new bytes, and several announcements of the same blob in a run become one read. * Messages are deleted only after their bytes are archived and indexed. If Cotool restarts mid-run, or a run ends before reaching every message it received, the messages become visible again when the run's time budget ends and are read again. * Rereading a blob does not create duplicates: flow tuples are keyed by their blob and byte position. * A blob that is deleted, or a message for a blob outside the allowed containers, is skipped and reported in the source's recent errors. * Permission errors leave the messages on the queue, so they are read once access is fixed, until the queue's 7-day message lifetime expires. ## Status Each source shows its health, last successful ingestion, events and bytes read in the last 24 hours, the age of the oldest message received, the number of queued messages, and recent errors. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.