Monitor runs

This document shows you how to do the following:

Before you begin

To monitor execution logs, ensure you do the following:

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Select or create a repository.

  3. Select or create a development workspace.

  4. Trigger a run at least once.

Required roles

To get the permissions that you need to view execution logs, ask your administrator to grant you the following IAM roles:

  • Dataform Editor (roles/dataform.editor) on the Dataform repository where workflow executions are being tracked
  • Verifying the setup of an execution: Service Account User (roles/iam.serviceAccountUser) on the custom service account

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

The Logs Viewer (roles/logging.viewer) IAM role gives you read-only access to all the features of Cloud Logging. For more information on the Identity and Access Management permissions and roles that apply to Logging data, see Access control with IAM.

To create and manage log-based alerts, ensure that your IAM role includes the permissions that are described in Roles required to create and use log-based alerting policies.

Monitor execution logs

This section shows you how to view execution logs in Dataform.

You can view the logs of all the executions that are triggered in a repository, and the logs of executions that were triggered in a workspace in the last 24 hours.

Inspect repository execution logs

To view the logs of all the executions triggered in a repository, follow these steps:

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Go to your repository.

  3. Click Workflow Execution Logs.

  4. To view the details of an execution, click the execution.

  5. To view the details of an action within the execution, click View details.

Inspect workspace execution logs

To view the logs of the 1,000 most-recent executions that were triggered in a workspace in the last 24 hours, follow these steps:

  1. In the Google Cloud console, go to the Dataform page.

    Go to Dataform

  2. Go to your repository and select your workspace.

  3. In your workspace, click Executions.

  4. To view details of an execution, click the execution.

  5. To view details of an action within the execution, click View details.

View Logging logs for Dataform

This section shows you how to view Cloud Logging logs for Dataform workflow invocations.

Each Dataform workflow invocation is logged using Logging. Logging is automatically enabled for Dataform workflow invocations, which can incur Logging billing charges. For more information, see Dataform pricing.

View logs

For information about filtering Logging logs in Logs Explorer, see View logs by using the Logs Explorer.

Dataform logs are indexed by Dataform repository.

Log format

Dataform sends workflow invocation log messages in the following format:

{
  "insertId": "14ip1tza5",
  "jsonPayload": {
    "terminalState": "SUCCEEDED",
    "workflowInvocationId": "1678383230-ac4ed48c-eb70-4555-b2b0-cda54d13edfa",
    "@type": "type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
  },
  "resource": {
    "type": "dataform.googleapis.com/Repository",
    "labels": {
      "repository_id": "repository-name-example",
      "resource_container": "PROJECT_ID",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-03-09T17:34:06.965981805Z",
  "severity": "INFO",
  "logName": "projects/project-name-example/logs/dataform.googleapis.com%2Fworkflow_invocation_completion",
  "receiveTimestamp": "2023-03-09T17:34:07.362371980Z"
}

What is logged

Dataform log entries contain information that is useful for monitoring and debugging your workflow invocations.

Log entries contain the following types of information:

  • receiveTimestamp
  • release_config_id
  • repository_id
  • resource_container
  • workflow_invocation_id
  • workflow_config_id
  • severity: can be INFO, WARNING, or ERROR
  • terminalState: can be SUCCEEDED, CANCELED, or FAILED
  • timestamp
  • @type

Dataform service logs provide high-level invocation events, such as workflow invocation completion logs. However, these service logs don't include individual BigQuery job IDs or granular table-level action executions. To retrieve granular action-level execution details and the corresponding BigQuery job IDs for a workflow invocation, see Correlate BigQuery jobs to workflow invocations.

Configure metrics and alerts with Cloud Monitoring

You can use Monitoring to observe trends in your logs and to notify you when conditions that you describe occur.

To provide Monitoring with data from your logs, Logging offers you the following:

  • Log-based metrics, which you can use as follows:
    • To create alerting policies that notify you of changes over time.
    • To create charts that display changes over time.
  • Log-based alerts, which notify you anytime a specific event appears in a log.

For more information about log-based metrics and log-based alerts, see Monitor your logs.

To learn how to configure log-based alerts for Dataform workflow invocations, see Configure alerts for failed workflow invocations.

Correlate BigQuery jobs to workflow invocations

Dataform service logs under the dataform.googleapis.com/Repository resource type provide high-level workflow invocation events, such as workflow invocation completion logs. However, these service logs don't include granular table-level action executions or specific BigQuery job IDs.

When Dataform runs actions such as creating or updating tables, views, or assertions, it submits jobs in BigQuery and automatically attaches labels to each job. You can also retrieve the BigQuery job ID for each action directly from the Dataform API by calling the workflowInvocations.query method and inspecting the bigqueryAction.jobId field.

The following table lists the key labels that Dataform attaches to BigQuery jobs:

Label key Description
dataform_workflow_execution_id The sanitized ID of the Dataform workflow invocation.
dataform_repository_id The sanitized ID of the Dataform repository.
dataform_workflow_execution_action_id_database The target project (database) of the action.
dataform_workflow_execution_action_id_schema The target dataset (schema) of the action.
dataform_workflow_execution_action_id_name The target table, view, or assertion name of the action.

To correlate BigQuery jobs and action-level execution details to a workflow invocation, do the following:

  1. Get the workflow invocation ID for your run. To look up the ID in Logs Explorer, query one of the following Dataform log types:

    • workflow_invocation_completion logs (enabled by default): copy the value of the jsonPayload.workflowInvocationId field from the log entry:

      resource.type="dataform.googleapis.com/Repository"
resource.labels.repository_id="REPOSITORY_ID"
log_id("dataform.googleapis.com/workflow_invocation_completion")

    • CreateWorkflowInvocation Data Access audit logs (if Data Access audit logs are enabled for DATA_WRITE operations): copy the WORKFLOW_INVOCATION_ID segment at the end of the protoPayload.resourceName field:

      resource.type="dataform.googleapis.com/Repository"
resource.labels.repository_id="REPOSITORY_ID"
log_id("cloudaudit.googleapis.com/data_access")
protoPayload.methodName:"Dataform.CreateWorkflowInvocation"

    Replace REPOSITORY_ID with the ID of your Dataform repository.

  2. To find the correlated BigQuery jobs by querying either BigQuery audit logs in Logs Explorer or the INFORMATION_SCHEMA.JOBS view in BigQuery, select one of the following options:

    Logs Explorer

    1. In the Google Cloud console, go to the Logs Explorer page.

      Go to Logs Explorer

    2. In the Query pane, enter the following query:

      resource.type="bigquery_project"
protoPayload.metadata.jobChange.job.jobConfig.labels.dataform_workflow_execution_id="WORKFLOW_INVOCATION_ID"

      Replace WORKFLOW_INVOCATION_ID with your workflow invocation ID.

    3. Click Run query.

      The query returns BigQuery job log entries for that workflow invocation. In each log entry, the protoPayload.metadata.jobChange.job.jobName field contains the BigQuery job resource name, and the jobStats and jobStatus fields provide execution statistics and status details.

    BigQuery Studio

    1. In the Google Cloud console, go to the BigQuery Studio page.

      Go to BigQuery Studio

    2. In the query editor, run the following query:

      SELECT
  job_id,
  creation_time,
  state,
  error_result,
  total_bytes_billed,
  (SELECT value FROM UNNEST(labels) WHERE key = 'dataform_workflow_execution_action_id_schema') AS target_schema,
  (SELECT value FROM UNNEST(labels) WHERE key = 'dataform_workflow_execution_action_id_name') AS target_name,
  query
FROM
  `region-LOCATION`.INFORMATION_SCHEMA.JOBS
WHERE
  DATE(creation_time) = CURRENT_DATE()
  AND EXISTS (
    SELECT 1
    FROM UNNEST(labels)
    WHERE
      key = 'dataform_workflow_execution_id'
      AND value = 'WORKFLOW_INVOCATION_ID'
  );

      Replace the following:

      • LOCATION: the BigQuery location where the jobs ran, such as us or eu.
      • WORKFLOW_INVOCATION_ID: your workflow invocation ID.

Configure alerts for failed workflow invocations

This section shows you how to configure log-based alerts for failed Dataform workflow invocations using the Logs Explorer.

You can use Monitoring to observe trends in Logging logs for Dataform workflow invocations and to notify you when conditions that you describe occur.

Each Dataform workflow invocation is logged using Logging. Logging is automatically enabled for Dataform workflow invocations, which can incur Logging billing charges. For more information, see Dataform pricing.

To receive alerts when a Dataform workflow invocation fails, you can create log-based alerts.

You can create log-based alerts from the Logs Explorer page in the Google Cloud console or by using the Monitoring API. This section describes how to create log-based alerts for Dataform workflow invocation failure using Logs Explorer.

To configure log-based alerts for failed Dataform workflow invocations, follow these steps:

  1. In the Google Cloud console, select Logging, and then select Logs Explorer.

    Go to Logs Explorer

  2. In the Query pane, select Show query and enter the following query:

        resource.type="dataform.googleapis.com/Repository"
    jsonPayload.@type="type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
    jsonPayload.terminalState="FAILED"

    This query alerts you of all failed Dataform workflow invocations.

    1. Optional: To filter failed workflow invocations by a release configuration, append the following to the query:

          jsonPayload.releaseConfigId="RELEASE_CONFIGURATION_ID"

      Replace RELEASE_CONFIGURATION_ID with the ID of the release configuration.

    2. Optional: To filter failed workflow invocations by a workflow configuration, append the following to the query:

          jsonPayload.workflowConfigId="WORKFLOW_CONFIGURATION_ID"

      Replace WORKFLOW_CONFIGURATION_ID with the ID of the workflow configuration.

    The following query alerts you when a Dataform workflow invocation that's related to both the daily release configuration and the production workflow configuration fails:

     resource.type="dataform.googleapis.com/Repository"
 jsonPayload.@type="type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
 jsonPayload.terminalState="FAILED"
 jsonPayload.releaseConfigId="daily"
 jsonPayload.workflowConfigId="production"

  3. Optional: To validate the query, use Run query in the Query results pane.

  4. In the header of the Query results pane, click Create alert. When your window is narrow, the Create alert option might appear on the Actions menu instead.

  5. In the Alert details pane, give the alert a name and description:

    1. Enter a name for your alert in the Alert Name field. For example: Dataform: workflow failure.

    2. Enter a description of this alert. You can also include information that might help the recipient of a notification diagnose the problem. The following string summarizes the reason for the alert:

      Log-based alert in project ${project} detected a failed Dataform workflow.

      For information about how you can format and tailor the content of this field, see Annotate notifications with user-defined documentation.

  6. To advance to the next step, click Next.

  7. In the Choose logs to include in the alert pane, check the query and results by clicking Preview logs.

    We recommend building the query in the Logs Explorer Query pane. The query that you built in the Query pane is also displayed on this pane.

    You can edit the query in this pane, if necessary. If you edit the query, then check the results by clicking Preview logs.

  8. Click Next.

  9. Select the minimum time between notifications. This value lets you control the number of notifications that you get from this alert if it is triggered multiple times. For this example, select 5 min from the options.

  10. Optional: Select the incident autoclose duration. By default, the incident autoclose duration is set to 7 days.

  11. Click Next.

  12. Select one or more notification channels for your alert. For this example, select an email notification channel.

    If you already have an email notification channel configured, then you can select it from the list. If not, click Manage notification channels and add an email channel. For information about creating notification channels, see Create and manage notification channels.

  13. Click Save.

Your log-based alert is now ready to test. For instructions to test the alert, see Test the example log-based alerting policy.

What's next