This document shows you how to do the following:
- [Monitor execution logs.](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#monitor-execution-logs)
- [View Logging logs for Dataform.](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#view-cloud-logging-logs-for-dataform)
- [Correlate BigQuery jobs to workflow invocations.](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#correlate-bigquery-audit-logs)
- [Configure alerts for failed workflow invocations.](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#configure-alerts-failed-workflow-invocations)
## Before you begin
To monitor execution logs, ensure you do the following:
1. In the Google Cloud console, go to the **Dataform** page.
[Go to Dataform](https://console.cloud.google.com/bigquery/dataform)
2. Select or [create a repository](https://docs.cloud.google.com/dataform/docs/create-repository.md.txt).
3. Select or [create a development workspace](https://docs.cloud.google.com/dataform/docs/create-workspace.md.txt).
4. [Trigger a run](https://docs.cloud.google.com/dataform/docs/trigger-execution.md.txt) at least once.
### Required roles
To get the permissions that
you need to view execution logs,
ask your administrator to grant you the
following IAM roles:
- [Dataform Editor](https://docs.cloud.google.com/iam/docs/roles-permissions/dataform.md.txt#dataform.editor) (`roles/dataform.editor`) on the Dataform repository where workflow executions are being tracked
- Verifying the setup of an execution: [Service Account User](https://docs.cloud.google.com/iam/docs/roles-permissions/iam.md.txt#iam.serviceAccountUser) (`roles/iam.serviceAccountUser`) on the custom service account
For more information about granting roles, see [Manage access to projects, folders, and organizations](https://docs.cloud.google.com/iam/docs/granting-changing-revoking-access.md.txt).
You might also be able to get
the required permissions through [custom
roles](https://docs.cloud.google.com/iam/docs/creating-custom-roles) or other [predefined
roles](https://docs.cloud.google.com/iam/docs/roles-overview#predefined).
The [Logs Viewer](https://docs.cloud.google.com/iam/docs/roles-permissions/logging.md.txt#logging.viewer)
(`roles/logging.viewer`) IAM role gives you read-only access
to all the features of Cloud Logging. For more information on the Identity and Access Management
permissions and roles that apply to Logging data, see
[Access control with IAM](https://docs.cloud.google.com/logging/docs/access-control.md.txt).
To create and manage log-based alerts, ensure that your IAM role
includes the permissions that are described in
[Roles required to create and use log-based alerting policies](https://docs.cloud.google.com/logging/docs/access-control.md.txt#lba-permissions).
## Monitor execution logs
This section shows you how to view execution logs in Dataform.
You can view the logs of all the executions that are triggered in a repository,
and the logs of executions that were triggered in a workspace in the last
24 hours.
### Inspect repository execution logs
To view the logs of all the executions triggered in a repository, follow these
steps:
1. In the Google Cloud console, go to the **Dataform** page.
[Go to Dataform](https://console.cloud.google.com/bigquery/dataform)
2. Go to your repository.
3. Click **Workflow Execution Logs**.
4. To view the details of an execution, click the execution.
5. To view the details of an action within the execution, click **View details**.
### Inspect workspace execution logs
To view the logs of the 1,000 most-recent executions that were triggered in a
workspace in the last 24 hours, follow these steps:
1. In the Google Cloud console, go to the **Dataform** page.
[Go to Dataform](https://console.cloud.google.com/bigquery/dataform)
2. Go to your repository and select your workspace.
3. In your workspace, click **Executions**.
4. To view details of an execution, click the execution.
5. To view details of an action within the execution, click **View details**.
## View Logging logs for Dataform
This section shows you how to view Cloud Logging logs for Dataform
workflow invocations.
Each Dataform workflow invocation is logged using
[Logging](https://docs.cloud.google.com/logging/docs.md.txt). Logging is
automatically enabled for Dataform workflow
invocations, which can incur Logging billing charges.
For more information, see [Dataform pricing](https://cloud.google.com/dataform/pricing).
### View logs
- To view logs, go to the **Logs Explorer** page.
[Go to Logs Explorer](https://console.cloud.google.com/logs)
For information about filtering Logging logs in
Logs Explorer, see
[View logs by using the Logs Explorer](https://docs.cloud.google.com/logging/docs/view/logs-explorer-interface.md.txt).
Dataform logs are indexed by Dataform repository.
### Log format
Dataform sends workflow invocation log messages in the following format:
```json
{
"insertId": "14ip1tza5",
"jsonPayload": {
"terminalState": "SUCCEEDED",
"workflowInvocationId": "1678383230-ac4ed48c-eb70-4555-b2b0-cda54d13edfa",
"@type": "type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
},
"resource": {
"type": "dataform.googleapis.com/Repository",
"labels": {
"repository_id": "repository-name-example",
"resource_container": "PROJECT_ID",
"location": "us-central1"
}
},
"timestamp": "2023-03-09T17:34:06.965981805Z",
"severity": "INFO",
"logName": "projects/project-name-example/logs/dataform.googleapis.com%2Fworkflow_invocation_completion",
"receiveTimestamp": "2023-03-09T17:34:07.362371980Z"
}
```
### What is logged
Dataform log entries contain information that is useful for
monitoring and debugging your workflow invocations.
Log entries contain the following types of information:
- `receiveTimestamp`
- `release_config_id`
- `repository_id`
- `resource_container`
- `workflow_invocation_id`
- `workflow_config_id`
- `severity`: can be `INFO`, `WARNING`, or `ERROR`
- `terminalState`: can be `SUCCEEDED`, `CANCELED`, or `FAILED`
- `timestamp`
- `@type`
Dataform service logs provide high-level invocation events,
such as workflow invocation completion logs. However, these service logs
don't include individual BigQuery job IDs or granular
table-level action executions. To retrieve granular action-level execution
details and the corresponding BigQuery job IDs for a workflow
invocation, see
[Correlate BigQuery jobs to workflow invocations](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#correlate-bigquery-audit-logs).
### Configure metrics and alerts with Cloud Monitoring
You can use Monitoring to observe trends in your logs and to notify
you when conditions that you describe occur.
To provide Monitoring with data from your logs,
Logging offers you the following:
- Log-based metrics, which you can use as follows:
- To create alerting policies that notify you of changes over time.
- To create charts that display changes over time.
- Log-based alerts, which notify you anytime a specific event appears in a log.
For more information about log-based metrics and log-based alerts, see
[Monitor your logs](https://docs.cloud.google.com/logging/docs/alerting/monitoring-logs.md.txt).
To learn how to configure log-based alerts for Dataform workflow
invocations, see
[Configure alerts for failed workflow invocations](https://docs.cloud.google.com/dataform/docs/monitor-runs.md.txt#configure-alerts-failed-workflow-invocations).
## Correlate BigQuery jobs to workflow invocations
Dataform service logs under the `dataform.googleapis.com/Repository`
resource type provide high-level workflow invocation events, such as
workflow invocation completion logs. However, these service logs don't include
granular table-level action executions or specific BigQuery job
IDs.
When Dataform runs actions such as creating or updating tables,
views, or assertions, it submits jobs in BigQuery and
automatically attaches labels to each job. You can also retrieve the
BigQuery job ID for each action directly from the Dataform API
by calling the
[`workflowInvocations.query` method](https://docs.cloud.google.com/dataform/reference/rest/v1/projects.locations.repositories.workflowInvocations/query.md.txt)
and inspecting the `bigqueryAction.jobId` field.
The following table lists the key labels that Dataform attaches
to BigQuery jobs:
| Label key | Description |
|---|---|
| `dataform_workflow_execution_id` | The sanitized ID of the Dataform workflow invocation. |
| `dataform_repository_id` | The sanitized ID of the Dataform repository. |
| `dataform_workflow_execution_action_id_database` | The target project (database) of the action. |
| `dataform_workflow_execution_action_id_schema` | The target dataset (schema) of the action. |
| `dataform_workflow_execution_action_id_name` | The target table, view, or assertion name of the action. |
To correlate BigQuery jobs and action-level execution details to
a workflow invocation, do the following:
1. Get the workflow invocation ID for your run. To look up the ID in
[Logs Explorer](https://console.cloud.google.com/logs),
query one of the following Dataform log types:
- **`workflow_invocation_completion` logs** (enabled by default): copy the
value of the `jsonPayload.workflowInvocationId` field from the log entry:
```
resource.type="dataform.googleapis.com/Repository"
resource.labels.repository_id="REPOSITORY_ID"
log_id("dataform.googleapis.com/workflow_invocation_completion")
```
- **`CreateWorkflowInvocation` Data Access audit logs** (if
[Data Access audit logs](https://docs.cloud.google.com/dataform/docs/audit-logging.md.txt) are enabled for
`DATA_WRITE` operations): copy the
`WORKFLOW_INVOCATION_ID` segment at the end of the
`protoPayload.resourceName` field:
```
resource.type="dataform.googleapis.com/Repository"
resource.labels.repository_id="REPOSITORY_ID"
log_id("cloudaudit.googleapis.com/data_access")
protoPayload.methodName:"Dataform.CreateWorkflowInvocation"
```
Replace `REPOSITORY_ID` with the ID of your
Dataform repository.
2. To find the correlated BigQuery jobs by querying either
BigQuery audit logs in Logs Explorer or the
[`INFORMATION_SCHEMA.JOBS`](https://docs.cloud.google.com/bigquery/docs/information-schema-jobs.md.txt) view in
BigQuery, select one of the following options:
### Logs Explorer
1. In the Google Cloud console, go to the **Logs Explorer** page.
[Go to Logs Explorer](https://console.cloud.google.com/logs)
2. In the **Query** pane, enter the following query:
```
resource.type="bigquery_project"
protoPayload.metadata.jobChange.job.jobConfig.labels.dataform_workflow_execution_id="WORKFLOW_INVOCATION_ID"
```
Replace `WORKFLOW_INVOCATION_ID` with
your workflow invocation ID.
3. Click **Run query**.
The query returns BigQuery job log entries for that
workflow invocation. In each log entry, the
`protoPayload.metadata.jobChange.job.jobName` field contains the
BigQuery job resource name, and the `jobStats` and
`jobStatus` fields provide execution statistics and status details.
### BigQuery Studio
1. In the Google Cloud console, go to the **BigQuery Studio** page.
[Go to BigQuery Studio](https://console.cloud.google.com/bigquery)
2. In the query editor, run the following query:
```sql
SELECT
job_id,
creation_time,
state,
error_result,
total_bytes_billed,
(SELECT value FROM UNNEST(labels) WHERE key = 'dataform_workflow_execution_action_id_schema') AS target_schema,
(SELECT value FROM UNNEST(labels) WHERE key = 'dataform_workflow_execution_action_id_name') AS target_name,
query
FROM
`region-LOCATION`.INFORMATION_SCHEMA.JOBS
WHERE
DATE(creation_time) = CURRENT_DATE()
AND EXISTS (
SELECT 1
FROM UNNEST(labels)
WHERE
key = 'dataform_workflow_execution_id'
AND value = 'WORKFLOW_INVOCATION_ID'
);
```
Replace the following:
- `LOCATION`: the BigQuery location where the jobs ran, such as `us` or `eu`.
- `WORKFLOW_INVOCATION_ID`: your workflow invocation ID.
## Configure alerts for failed workflow invocations
This section shows you how to configure log-based alerts for
failed Dataform workflow invocations using the Logs Explorer.
You can use Monitoring to observe trends in
Logging logs for Dataform workflow invocations
and to notify you when conditions that you describe occur.
Each Dataform workflow invocation is logged using
[Logging](https://docs.cloud.google.com/logging/docs.md.txt).
Logging is automatically enabled for Dataform workflow
invocations, which can incur Logging billing charges.
For more information, see [Dataform pricing](https://cloud.google.com/dataform/pricing).
To receive alerts when a Dataform workflow invocation fails,
you can [create log-based alerts](https://docs.cloud.google.com/logging/docs/alerting/log-based-alerts.md.txt#lba-definition).
You can create log-based alerts from the
**Logs Explorer** page in the Google Cloud console or by using the
Monitoring API. This section describes how to create log-based alerts for Dataform workflow invocation failure using
Logs Explorer.
To configure log-based alerts for failed Dataform workflow
invocations, follow these steps:
1. In the Google Cloud console, select **Logging** , and then select
**Logs Explorer**.
[Go to Logs Explorer](https://console.cloud.google.com/logs)
2. In the **Query** pane, select **Show query** and enter the following query:
```
resource.type="dataform.googleapis.com/Repository"
jsonPayload.@type="type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
jsonPayload.terminalState="FAILED"
```
This query alerts you of all failed Dataform workflow invocations.
1. Optional: To filter failed workflow invocations by a release configuration,
append the following to the query:
```
jsonPayload.releaseConfigId="RELEASE_CONFIGURATION_ID"
```
Replace `RELEASE_CONFIGURATION_ID` with the ID of the release configuration.
2. Optional: To filter failed workflow invocations by a workflow
configuration, append the following to the query:
```
jsonPayload.workflowConfigId="WORKFLOW_CONFIGURATION_ID"
```
Replace `WORKFLOW_CONFIGURATION_ID` with the ID of the workflow configuration.
The following query alerts you when a Dataform workflow invocation
that's related to both the `daily` release configuration and the `production`
workflow configuration fails:
resource.type="dataform.googleapis.com/Repository"
jsonPayload.@type="type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
jsonPayload.terminalState="FAILED"
jsonPayload.releaseConfigId="daily"
jsonPayload.workflowConfigId="production"
3. Optional: To validate the query, use **Run query** in the **Query results** pane.
4. In the header of the **Query results** pane, click **Create alert** . When your
window is narrow, the **Create alert** option might appear on the **Actions**
menu instead.
5. In the **Alert details** pane, give the alert a name and description:
1. Enter a name for your alert in the **Alert Name** field. For example:
`Dataform: workflow failure`.
2. Enter a description of this alert. You can also include information
that might help the recipient of a notification diagnose the problem.
The following string summarizes the reason for the alert:
```
Log-based alert in project ${project} detected a failed Dataform workflow.
```
For information about how you can format and tailor the content
of this field, see
[Annotate notifications with user-defined documentation](https://docs.cloud.google.com/monitoring/alerts/doc-variables.md.txt).
6. To advance to the next step, click **Next**.
7. In the **Choose logs to include in the alert** pane, check the query
and results by clicking **Preview logs**.
We recommend building the query in the Logs Explorer **Query** pane.
The query that you built in the **Query** pane is also displayed on this
pane.
You can edit the query in this pane, if necessary. If you edit the
query, then check the results by clicking **Preview logs**.
8. Click **Next**.
9. Select the minimum time between notifications. This value
lets you control the number of notifications that you get from this alert
if it is triggered multiple times. For this example, select **5 min**
from the options.
10. Optional: Select the incident autoclose duration. By default, the incident
autoclose duration is set to 7 days.
11. Click **Next**.
12. Select one or more notification channels for your alert. For this example,
select an email notification channel.
If you already have an email notification channel configured, then you can
select it from the list. If not, click **Manage notification channels** and
add an email channel. For information about creating notification
channels, see
[Create and manage notification channels](https://docs.cloud.google.com/monitoring/support/notification-options.md.txt).
13. Click **Save**.
Your log-based alert is now ready to test. For instructions to test the alert,
see [Test the example log-based alerting policy](https://docs.cloud.google.com/logging/docs/alerting/log-based-alerts.md.txt#lba-test).
## What's next
- Learn about [manually triggering runs](https://docs.cloud.google.com/dataform/docs/trigger-execution.md.txt).
- Learn how to [schedule runs with Workflows and Cloud Scheduler](https://docs.cloud.google.com/dataform/docs/schedule-runs.md.txt#workflows-cloud-scheduler).
- Learn how to [configure compilation overrides with the Dataform API](https://docs.cloud.google.com/dataform/docs/configure-compilation.md.txt#configure-compilation-overrides-with-api).
- Learn about [Cloud Audit Logs](https://docs.cloud.google.com/logging/docs/audit.md.txt) and [Monitoring](https://docs.cloud.google.com/monitoring).
- Learn about [Logging pricing](https://cloud.google.com/stackdriver/pricing#google-clouds-operations-suite-pricing).