# Introduction to BigQuery monitoring

BigQuery monitoring lets you track resource usage, job
execution, and operational health across your workloads to identify
bottlenecks and maintain reliability.

Monitoring and logging are crucial for running reliable applications in the
cloud. BigQuery workloads are no exception, especially if your workload
has high volumes or is mission critical.

Monitoring sources vary by sampling frequency and aggregation level. For
example, information schema views often provide higher granularity than
Cloud Monitoring metrics. As a result, metric charts might temporarily
diverge from information schema data until aggregation smooths out
differences. When designing a monitoring strategy, assess latency, precision,
and accuracy against your requirements.

## Metrics

Metrics are numerical values that are collected at regular
intervals and made available for analysis. You can use metrics to:

- Create charts and dashboards.
- Trigger alerts for conditions or situations that need human intervention.
- Analyze historical performance.

In the case of BigQuery, the available metrics include the number
of jobs that are running, how many bytes were scanned during a query, and the
distribution of query times. The metrics for a query only
become available after the query succeeds and can take up to seven minutes to be
reported. Metrics for queries that fail are not reported. For a complete list of
available metrics including their sample rates, visibility, and limitations, see
[`bigquery`](https://docs.cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-bigquery) under
[Google Cloud metrics](https://docs.cloud.google.com/monitoring/api/metrics_gcp).

Use [Cloud Monitoring](https://docs.cloud.google.com/monitoring/docs) to view BigQuery
metrics and create charts and alerts. Each metric has a resource type, either
`bigquery_dataset`, `bigquery_project`, or `global`, and a set of labels. You
can group or filter each metric by using the labels.

For example, to chart the number of interactive queries in flight, use the
following PromQL statement, which filters by `priority` equal to `interactive`:

    {"bigquery.googleapis.com/query/count", monitored_resource="global", priority="interactive"}

The next example gets the number of load jobs in flight, grouped into 10-minute
intervals:

    avg_over_time({"bigquery.googleapis.com/job/num_in_flight",
      monitored_resource="bigquery_project",
      job_type="load"
    }[10m])

For more information, see
[Creating charts and alerts for BigQuery](https://docs.cloud.google.com/bigquery/docs/monitoring-dashboard).

## Logs

Logs are text records that are generated in response to particular events or
actions. BigQuery creates log entries for actions such as
creating or deleting a table, purchasing slots, or running a load job. For more
information about logging in Google Cloud, see
[Cloud Logging](https://docs.cloud.google.com/logging/docs).

A log is an append-only collection of log entries. For example, you could write
your own log entries to a log named
`projects/PROJECT_ID/logs/my-test-log`. Many
Google Cloud services, including BigQuery, create a type of
log called [audit logs](https://docs.cloud.google.com/logging/docs/audit). These logs record:

- Administrative activity, such as creating or modifying resources.
- Data access, such as reading user-provided data from a resource.
- System events that are generated by Google systems, rather than by user actions.

Audit logs are written in a structured JSON format. The base data type for
Google Cloud log entries is the
[`LogEntry`](https://docs.cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) structure. This
structure contains the name of the log, the resource that generated the log
entry, the timestamp (UTC), and other basic information.

The details of the logged event are contained in a subfield called the *payload*
field. For audit logs, the payload field is named `protoPayload`. The value of
this field is an
[`AuditLog`](https://docs.cloud.google.com/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog)
structure, indicated by the value of the `protoPayload.@type` field, which is
set to `type.googleapis.com/google.cloud.audit.AuditLog`.

For operations on datasets, tables, and jobs, BigQuery writes
audit logs in two different formats, although both share the `AuditLog` base
type.

In the older format:

- The `resource.type` field is `bigquery_resource`.
- Details about the operation are written to the `protoPayload.serviceData` field. The value of this field is an [`AuditData`](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/AuditData) structure.

In the newer format:

- The `resource.type` field is either `bigquery_project` or `bigquery_dataset`. The `bigquery_project` resource has log entries about jobs, while the `bigquery_dataset` resource has log entries about storage.
- Details about the operation are written to the `protoPayload.metadata` field. The value of this field is a [`BigQueryAuditMetadata`](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/BigQueryAuditMetadata) structure.

We recommend consuming logs in the newer format. For more information, see
[Audit logs migration guide](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/migration).

Here is an abbreviated example of a log entry that shows a failed operation:

    {
      "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "status": {
          "code": 5,
          "message": "Not found: Dataset my-project:my-dataset was not found in location US"
        },
        "authenticationInfo": { ... },
        "requestMetadata":  { ... },
        "serviceName": "bigquery.googleapis.com",
        "methodName": "google.cloud.bigquery.v2.JobService.InsertJob",
        "metadata": {
      },
      "resource": {
        "type": "bigquery_project",
        "labels": { .. },
      },
      "severity": "ERROR",
      "logName": "projects/my-project/logs/cloudaudit.googleapis.com%2Fdata_access",
      ...
    }

For operations on BigQuery Reservations, the `protoPayload` is an
[`AuditLog`](https://docs.cloud.google.com/logging/docs/reference/audit/auditlog/rest/Shared.Types/AuditLog)
structure, and the `protoPayload.request` and `protoPayload.response` fields
contain more information. You can find the field definitions in
[BigQuery Reservation API](https://docs.cloud.google.com/bigquery/docs/reference/reservations/rpc). For more
information, see
[Monitoring BigQuery Reservations](https://docs.cloud.google.com/bigquery/docs/reservations-monitoring).

## BigQuery information schema views

[Information schema views](https://docs.cloud.google.com/bigquery/docs/information-schema-intro) are
another source of insights in BigQuery, which you can use in
conjunction with metrics and logs.

These views contain metadata about jobs, datasets, tables, and other
BigQuery entities. For example, you can get real-time metadata
about which BigQuery jobs ran over a specified time period, and
then group or filter the results by project, user, tables referenced, and other
dimensions.

You can use this information to perform more detailed analysis about your
BigQuery workloads, and answer questions like:

- What is the average slot utilization for all queries over the past 7 days for a given project?
- Which users submitted a batch load job for a given project?
- What streaming errors occurred in the past 30 minutes, grouped by error code?

In particular, look at [jobs metadata](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs),
[streaming metadata](https://docs.cloud.google.com/bigquery/docs/information-schema-streaming), and
[reservations metadata](https://docs.cloud.google.com/bigquery/docs/information-schema-reservations) to
get insights into the performance of your BigQuery workloads.

You can find [example information schema queries on
GitHub](https://github.com/GoogleCloudPlatform/bigquery-utils/tree/master/dashboards/system_tables)
that show an organization's slot and reservation utilization, job execution,
and job errors. For example, the following query provides a list of
queries that are either pending or running. These queries are ordered
by the length of time since they were created in the `us` region:

```
SELECT
    creation_time,
    project_id,
    user_email,
    job_id,
    job_type,
    priority,
    state,
    TIMESTAMP_DIFF(CURRENT_TIMESTAMP(), start_time,second) as running_time_sec
 FROM
   `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
 WHERE
    creation_time BETWEEN TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY) AND CURRENT_TIMESTAMP()
    AND state != "DONE"
ORDER BY
    running_time_sec DESC
```

For more information, see [Troubleshoot BigQuery performance with
these dashboards](https://cloud.google.com/blog/products/data-analytics/troubleshoot-bigquery-performance-with-these-dashboards).

In addition to writing your own queries, you can use BigQuery
monitoring tools in the Google Cloud console:

- **[Monitor health, resource utilization, and jobs](https://docs.cloud.google.com/bigquery/docs/admin-resource-charts).** View operational health dashboards and BigQuery Admin Resource Charts that track slot usage, job concurrency, and job run time across your organization and reservations.
- **[Monitor jobs](https://docs.cloud.google.com/bigquery/docs/admin-jobs-explorer).** Search, filter, and group individual query executions across your organization to inspect query execution graphs, identify top slot consumers, and troubleshoot problematic jobs.

## Telemetry map

The following table maps BigQuery features and resources to
their operational monitoring telemetry, audited API services and resources,
and audit log reference documentation. You can use it to find the telemetry
that you need to troubleshoot issues, optimize performance, control costs,
audit usage, and maintain compliance.

Unlike many Google Cloud services that write operational job logs to
[Cloud Logging](https://docs.cloud.google.com/logging/docs), BigQuery writes operational job
telemetry to
[information schema views](https://docs.cloud.google.com/bigquery/docs/information-schema-intro) such as
[`INFORMATION_SCHEMA.JOBS`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs). These
views capture query execution stages, slot usage, bytes billed, and all
submitted jobs, including in-flight and failed jobs.

BigQuery uses Logging primarily to emit
[Cloud Audit Logs](https://docs.cloud.google.com/logging/docs/audit), which record administrative and
data access activity. When you audit this activity, you can determine who
accessed resources or data, where, and when.

Audit logs aren't emitted for every job disruption or failure. For example,
failed jobs don't generate `TableDataRead` or `TableDataChange` events, and
`jobCompletedEvent` only records successful job completions. For details about
scenarios that don't emit audit logs, see
[Data access](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs#data_access) in the
[BigQuery audit logs
overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs).

In the following table, an asterisk (`*`) in an `INFORMATION_SCHEMA` view name
indicates scope-specific view variants, such as `_BY_PROJECT`, `_BY_USER`,
`_BY_FOLDER`, and `_BY_ORGANIZATION`.

| Feature or resource | Operational monitoring | Cloud Audit Logs |
| Queries and jobs | **Information schema views:** - [`JOBS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs): real-time and historical job execution, slot milliseconds, bytes billed, execution stages, and all submitted jobs regardless of completion status - [`JOBS_TIMELINE*`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs-timeline): per-timeslice job state and slot usage **Console:** - [Jobs explorer](https://docs.cloud.google.com/bigquery/docs/admin-jobs-explorer) in the Google Cloud console - [Administrative resource charts](https://docs.cloud.google.com/bigquery/docs/admin-resource-charts) **Metrics:** [Cloud Monitoring metrics](https://docs.cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-bigquery), such as `query/count`, `query/execution_times`, and `job/num_in_flight`. | **API service:** `bigquery.googleapis.com`. **Audited resources:** jobs in `JobService` (monitored resource: `bigquery_project`). **Audit logs:** Data Access logs using [`BigQueryAuditMetadata`](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/rest/Shared.Types/BigQueryAuditMetadata) (`JobInsertion` and `JobChange` events), recording who submitted or canceled a job, job status, statement type, and referenced tables or views. See [BigQuery audit logs overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs#data_access). |
| Tables and views | **Information schema views:** - [`TABLES`](https://docs.cloud.google.com/bigquery/docs/information-schema-tables): table metadata - [`VIEWS`](https://docs.cloud.google.com/bigquery/docs/information-schema-views): view definitions and metadata - [`TABLE_STORAGE*`](https://docs.cloud.google.com/bigquery/docs/information-schema-table-storage): logical and physical bytes and storage billing - [`PARTITIONS`](https://docs.cloud.google.com/bigquery/docs/information-schema-partitions): partition row counts and storage size - [`STREAMING_TIMELINE*`](https://docs.cloud.google.com/bigquery/docs/information-schema-streaming): streaming insert throughput and errors - [`WRITE_API_TIMELINE*`](https://docs.cloud.google.com/bigquery/docs/information-schema-write-api): write stream throughput and errors **Metrics:** [Monitoring metrics](https://docs.cloud.google.com/monitoring/api/metrics_gcp_a_b#gcp-bigquery), such as `storage/stored_bytes` and `storage/table_count`. | **API services:** `bigquery.googleapis.com` and `bigquerystorage.googleapis.com`. Data Access logs for BigQuery Storage API use `bigquery.googleapis.com`, while Policy Denied logs use `bigquerystorage.googleapis.com`. **Audited resources:** tables, views, and materialized views in `TableService` and `TableDataService`; and storage read sessions and write streams in `BigQueryRead` and `BigQueryWrite` (monitored resources: `bigquery_dataset` and `bigquery_table`). **Audit logs:** Data Access logs using `BigQueryAuditMetadata` (`TableDataRead` and `TableDataChange` events for table reads, modifications, referenced fields, source job, and BigQuery Storage API stream reads and writes), plus Admin Activity, Data Access, and System Event logs for `TableCreation`, `TableChange`, and `TableDeletion` events, including table Identity and Access Management (IAM) policy changes and table expirations. See [BigQuery audit logs overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs). |
| Datasets | **Information schema views:** - [`SCHEMATA`](https://docs.cloud.google.com/bigquery/docs/information-schema-datasets-schemata): dataset metadata and default expiration - [`SCHEMATA_OPTIONS`](https://docs.cloud.google.com/bigquery/docs/information-schema-datasets-schemata-options): dataset options and labels - [`SCHEMATA_REPLICAS`](https://docs.cloud.google.com/bigquery/docs/information-schema-schemata-replicas): cross-region replication status | **API service:** `bigquery.googleapis.com`. **Audited resources:** datasets, dataset IAM policies, and dataset ACLs in `DatasetService` and `IAMPolicy` (monitored resource: `bigquery_dataset`). **Audit logs:** Admin Activity and Data Access logs using `BigQueryAuditMetadata` (`DatasetCreation`, `DatasetChange`, and `DatasetDeletion` events), including dataset IAM policy changes in `bindingDeltas` and ACL updates in `accessChanges`. See [BigQuery audit logs overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs). |
| Slots and reservations | **Information schema views:** - [`RESERVATIONS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-reservations): current reservation configurations - [`RESERVATIONS_TIMELINE*`](https://docs.cloud.google.com/bigquery/docs/information-schema-reservation-timeline): per-minute reservation metadata and autoscale details - [`ASSIGNMENTS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-assignments): project, folder, and organization assignments - [`CAPACITY_COMMITMENTS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-capacity-commitments): slot capacity commitments **Console:** - [Administrative resource charts](https://docs.cloud.google.com/bigquery/docs/admin-resource-charts) in the Google Cloud console - [Slot estimator](https://docs.cloud.google.com/bigquery/docs/slot-estimator) **Metrics:** [Monitoring metrics](https://docs.cloud.google.com/bigquery/docs/reservations-monitoring), such as `slots/allocated` and `slots/total_available`. | **API service:** `bigqueryreservation.googleapis.com` for the BigQuery Reservation API. **Audited resources:** reservations, capacity commitments, and reservation assignments in `ReservationService` (monitored resource: `bigquery_project`). **Audit logs:** Admin Activity and Data Access logs for creating, modifying, deleting, and listing reservations, capacity commitments, and project or folder assignments. See [BigQuery Reservation API audit logging](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/index-reservation). |
| Column-level security and data masking | **Information schema views:** - [`COLUMN_FIELD_PATHS`](https://docs.cloud.google.com/bigquery/docs/information-schema-column-field-paths): nested column paths, policy tags, and data policy assignments - [`COLUMNS`](https://docs.cloud.google.com/bigquery/docs/information-schema-columns): column schemas and data types | **API services:** `datacatalog.googleapis.com` for Data Catalog policy tags, `bigquerydatapolicy.googleapis.com` for the BigQuery Data Policy API, and `bigquery.googleapis.com`. **Audited resources:** taxonomies and policy tags in `PolicyTagManager`, data policies in `DataPolicyService`, and policy-tagged table columns (monitored resources: `audited_resource` and `bigquery_dataset`). **Audit logs:** Admin Activity and Data Access logs for taxonomy and policy tag management and IAM policy changes ([Audit policy tags](https://docs.cloud.google.com/bigquery/docs/auditing-policy-tags)); data policy management and grantee updates ([BigQuery Data Policy audit logging](https://docs.cloud.google.com/bigquery/docs/column-data-masking-audit-logging)); and `TableDataRead.fields` logs for reads on protected columns ([BigQuery audit logs overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs)). |
| Routines, including UDFs and stored procedures | **Information schema views:** - [`ROUTINES`](https://docs.cloud.google.com/bigquery/docs/information-schema-routines): routine definitions and metadata - [`ROUTINE_OPTIONS`](https://docs.cloud.google.com/bigquery/docs/information-schema-routine-options): routine options - [`PARAMETERS`](https://docs.cloud.google.com/bigquery/docs/information-schema-parameters): routine input and output parameters - [`JOBS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs): routine execution jobs | **API service:** `bigquery.googleapis.com`. **Audited resources:** user-defined functions (UDFs), table functions, and stored procedures in `RoutineService` (monitored resources: `bigquery_dataset` and `bigquery_project`). **Audit logs:** Admin Activity and Data Access logs using `BigQueryAuditMetadata` (`RoutineCreation`, `RoutineChange`, and `RoutineDeletion` events), plus child job events for stored procedure runs. See [BigQuery audit logs overview](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs). |
| External connections | **Information schema views:** - [`OBJECT_PRIVILEGES`](https://docs.cloud.google.com/bigquery/docs/information-schema-object-privileges): connection privileges - [`JOBS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs): federated queries and remote function jobs that use connections | **API service:** `bigqueryconnection.googleapis.com` for the BigQuery Connection API. **Audited resources:** connections and connection IAM policies in `ConnectionService` (monitored resource: `audited_resource`). **Audit logs:** Admin Activity and Data Access logs for connection creation, credential updates, deletion, listing, and IAM policy changes. See [BigQuery Connection API audit logging](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/index-connection). |
| BigQuery sharing | **Information schema views:** - [`SHARED_DATASET_USAGE`](https://docs.cloud.google.com/bigquery/docs/information-schema-shared-dataset-usage): subscriber usage of shared dataset tables - [`SCHEMATA_LINKS`](https://docs.cloud.google.com/bigquery/docs/information-schema-datasets-schemata-links): linked datasets and shared resources **Dashboard:** [sharing usage metrics dashboard](https://docs.cloud.google.com/bigquery/docs/analytics-hub-monitor-listings). | **API service:** `analyticshub.googleapis.com`. **Audited resources:** data exchanges, listings, subscriptions, and exchange or listing IAM policies in `AnalyticsHubService` (monitored resource: `audited_resource`). **Audit logs:** Admin Activity and Data Access logs for data exchange and listing creation, updates, deletion, IAM policy changes, and subscriber operations. See [Sharing audit logging](https://docs.cloud.google.com/bigquery/docs/analytics-hub-audit-logging). |
| BigQuery Data Transfer Service | **Metrics and logs:** - [Monitoring transfer metrics](https://docs.cloud.google.com/bigquery/docs/dts-monitor) - Operational transfer run logs in Logging under `resource.type="bigquery_dts_config"` | **API service:** `bigquerydatatransfer.googleapis.com` for the BigQuery Data Transfer Service API. **Audited resources:** transfer configurations, transfer runs, and data sources in `DataTransferService` (monitored resource: `audited_resource`). **Audit logs:** Admin Activity and Data Access logs for creating, updating, deleting, listing, and starting transfer configurations and runs. See [BigQuery Data Transfer Service audit logging](https://docs.cloud.google.com/bigquery/docs/audit-logging). |
| BigLake catalogs and tables | **Information schema views:** - [`TABLES`](https://docs.cloud.google.com/bigquery/docs/information-schema-tables): table metadata - [`JOBS*`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs): external object storage query statistics (`metadata_cache_statistics` and `object_storage_stats`) | **API services:** `biglake.googleapis.com` and `bigquery.googleapis.com`. **Audited resources:** metastore catalogs, databases, tables, and partitions in `MetastoreService` and `HiveMetastoreService`, and Apache Iceberg catalogs, namespaces, and tables in `IcebergCatalogService` (monitored resources: `audited_resource` and `bigquery_dataset`). **Audit logs:** Admin Activity and Data Access logs for catalog, database, namespace, table, partition, and catalog IAM policy operations. See [BigLake audit logging](https://docs.cloud.google.com/bigquery/docs/biglake-audit-logging). |
| BigQuery Migration Service | **Status and reports:** - [Batch SQL translation](https://docs.cloud.google.com/bigquery/docs/batch-sql-translator) job status in the Google Cloud console - Workflow and subtask status in the BigQuery Migration API - Translation summary reports in Cloud Storage | **API service:** `bigquerymigration.googleapis.com` for the BigQuery Migration API. **Audited resources:** migration workflows, migration subtasks, and SQL translation requests in `MigrationService` and `SqlTranslationService` (monitored resource: `audited_resource`). **Audit logs:** Data Access logs for creating, starting, listing, viewing, and deleting migration workflows and subtasks, and translating SQL queries. See [BigQuery Migration Service audit logging](https://docs.cloud.google.com/bigquery/docs/reference/auditlogs/audit-logging-bq-migration). |
|---|---|---|

## What's next

- Learn how to [monitor and filter jobs with the jobs explorer](https://docs.cloud.google.com/bigquery/docs/admin-jobs-explorer).
- Learn how to [monitor health and resource utilization](https://docs.cloud.google.com/bigquery/docs/admin-resource-charts).
- Learn how to [create charts and alerts for BigQuery](https://docs.cloud.google.com/bigquery/docs/monitoring-dashboard).