This document describes how to use the Cloud Monitoring console to monitor
your Spanner instances.
The Cloud Monitoring console provides several monitoring tools for
Spanner:
- A *curated dashboard*, which shows pre-made charts for your Spanner resources
- *Custom charts* , including [ad-hoc charts in the Metrics Explorer](https://docs.cloud.google.com/monitoring/charts/metrics-explorer) as well as [charts in custom dashboards](https://docs.cloud.google.com/monitoring/charts)
- *Alerts*, which notify you if a metric exceeds a threshold that you specify
If you prefer to monitor Spanner programmatically, use the [Cloud
Client Libraries for Cloud Monitoring](https://docs.cloud.google.com/monitoring/docs/reference/libraries) to
retrieve metrics.
> [!NOTE]
> **Note:** You can also monitor your instances by [viewing charts in the
> Google Cloud console](https://docs.cloud.google.com/spanner/docs/monitoring-console). Use the Google Cloud console to get a quick view of the most important metrics for your instance.
## Use the Cloud Monitoring curated dashboard
Cloud Monitoring provides you with a curated dashboard that summarizes key
information about your Spanner instances, including:
- *Incidents*: User-created monitoring alerts that are open, active, or resolved
- *Events* : A list of Spanner [audit logs](https://docs.cloud.google.com/spanner/docs/logs) (if enabled and available)
- *Instances* : A high-level summary of your Spanner instances, including [compute capacity](https://docs.cloud.google.com/spanner/docs/compute-capacity), database count, and instance health
- *Aggregated charts* of throughput and storage use
To view the Spanner dashboard, do the following:
1. In the Google Cloud console, select **Monitoring**, or use the following button:
[Go to Monitoring](https://console.cloud.google.com/monitoring)
2. If **Resources** is shown in the navigation pane, then select **Resources**
and then select **Cloud Spanner** . Otherwise, select **Dashboards** and then
select the dashboard named **Cloud Spanner**.
### View instance and database details
When you open the curated dashboard for Spanner, it shows
aggregated data for all of your instances. You can view more details about a
specific instance by clicking the instance's name under **Instances**.
The dashboard displays information such as instance metadata, databases in the
instance, and charts of various metrics broken down by region.
From the instance dashboard page, you can also see charts for a specific
database in the instance:
1. On the right-hand side, above the instance metrics charts, click **Database
metrics**.
2. In the **Select a breakdown** drop-down list, select the database that you
want to examine.
The Cloud Monitoring console displays charts for the database.
## Create custom charts for Spanner metrics
You can use Cloud Monitoring to create custom charts for
Spanner metrics. You can use the Metrics Explorer to create
temporary, ad-hoc charts, or you can create charts that appear on custom
dashboards.
In particular, Cloud Monitoring lets you create a custom chart that shows
whether two or more metrics are correlated with each other. For example, you can
check for a correlation between [CPU utilization](https://docs.cloud.google.com/spanner/docs/cpu-utilization) and
[latency](https://docs.cloud.google.com/spanner/docs/latency-guide) in a Spanner instance, which might indicate
that your instance needs more [compute capacity](https://docs.cloud.google.com/spanner/docs/compute-capacity) or that some of your
queries are causing high CPU utilization.
To get started with this example, follow these steps:
1. In the Google Cloud console, select **Monitoring**, or use the following button:
[Go to Monitoring](https://console.cloud.google.com/monitoring)
2. If **Metrics Explorer** is shown in the navigation pane, select it.
Otherwise, select **Resources** and then select **Metrics Explorer**.
3. Click the **View options** tab, then select the **Log scale on Y-axis**
checkbox. This option helps you compare multiple metrics when one metric has
much larger values than the others.
4. In the drop-down list above the right pane, select **Line**.
5. Click the **Metrics** tab. You can now add metrics to the chart.
To add latency metrics to the chart, follow these steps:
1. In the **Find resource type and metric** box, enter the value `spanner.googleapis.com/api/request_latencies`, then click the row that appears below the box.
2. In the **Filter** box, enter the value `instance_id`, then enter the instance ID you want to examine and click **Apply**.
3. In the **Aggregator** drop-down list, click **max**.
4. Optional: Change the latency percentile:
1. Click **Show advanced options**.
2. Click the **Aligner** drop-down list, then click the latency percentile that you want to view.
In most cases, you should look at either the 50th percentile latency, to
understand the typical amount of latency, or the 99th percentile
latency, to understand the latency for the slowest 1% of requests.
To add CPU utilization metrics to the chart, follow these steps:
1. Click **Add metric**.
2. In the **Find resource type and metric** box, enter the value `spanner.googleapis.com/instance/cpu/utilization`, then click the row that appears below the box.
3. In the **Filter** box, enter the value `instance_id`, then enter the instance ID you want to examine and click **Apply**.
4. In the **Aggregator** drop-down list, click **max**.
You now have a chart that shows the CPU utilization and latency metrics for a
Spanner instance. If both metrics are higher than expected at the
same time, you can
[take additional steps to correct the issue](https://docs.cloud.google.com/spanner/docs/latency-metrics).
For more information about creating custom charts, see the [Cloud Monitoring
documentation](https://docs.cloud.google.com/monitoring/charts).
## Create alerts for Spanner metrics
When you create a Spanner [instance](https://docs.cloud.google.com/spanner/docs/instances), you choose the
[compute capacity](https://docs.cloud.google.com/spanner/docs/compute-capacity) for the instance. As the instance's workload changes,
Spanner does not automatically adjust compute capacity of the
instance. As a result, you need to set up several alerts to ensure that the
instance stays within the
[recommended maximums for CPU utilization](https://docs.cloud.google.com/spanner/docs/cpu-utilization#recommended-max) and the
[recommended limit for storage](https://docs.cloud.google.com/spanner/docs/limits).
The following examples show how to set up alerting policies for some
Spanner metrics. For a full list of available metrics, see [metrics
list for Spanner](https://docs.cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-spanner).
### High-priority CPU
To create an alerting policy that triggers when your high priority cpu utilization for
[Spanner](https://docs.cloud.google.com/spanner/docs) is above a recommended threshold,
use the following settings.
#### Steps to create an [alerting policy](https://docs.cloud.google.com/monitoring/alerts/using-alerting-ui#create-policy).
To create an alerting policy, do the following:
1. In the Google Cloud console, go to the
**Alerting** page:
[Go to **Alerting**](https://console.cloud.google.com/monitoring/alerting)
If you use the search bar to find this page, then select the result whose subheading is
**Monitoring**.
2. If you haven't created your notification channels and if you want to be notified, then click **Edit Notification Channels** and add your notification channels. Return to the **Alerting** page after you add your channels.
3. From the **Alerting** page, select **Create policy**.
4. To select the resource, metric, and filters, expand the **Select a metric** menu and then use the values in the **New condition** table:
1. Optional: To limit the menu to relevant entries, enter the resource or metric name in the filter bar.
2. Select a **Resource type** . For example, select **VM instance**.
3. Select a **Metric category** . For example, select **instance**.
4. Select a **Metric** . For example, select **CPU Utilization**.
5. Select **Apply**.
5. Click **Next** and then configure the alerting policy trigger. To complete these fields, use the values in the **Configure alert trigger** table.
6. Click **Next**.
7. Optional: To add notifications to your alerting policy, click
**Notification channels** . In the dialog, select one or more notification
channels from the menu, and then click **OK**.
To be notified when alerts are opened and closed, check
**Notify on alert closure**. By default, notifications are sent only when
alerts are opened.
8. Optional: Update the **Alert autoclose duration**. This field determines when Monitoring closes alerts in the absence of metric data.
9. Optional: Click **Documentation**, and then add any information that you want included in a notification message.
10. Click **Alert name** and enter a name for the alerting policy.
11. Click **Create Policy**.
| **New condition** Field | Value |
|---|---|
| **Resource and Metric** | In the **Resources** menu, select **Spanner Instance** . In the **Metric categories** menu, select **Instance** . In the **Metrics** menu, select **CPU Utilization by priority** . (The metric.type is `spanner.googleapis.com/instance/cpu/utilization_by_priority`). |
| **Filter** | `instance_id = YOUR_INSTANCE_ID` `priority = high` |
| **Across time series Time series group by** | `location` for multi-region instances; leave it blank for regional instances. |
| **Across time series Time series aggregation** | `sum` |
| **Rolling window** | `10 m` |
| **Rolling window function** | `mean` |
| **Configure alert trigger** Field | Value |
|---|---|
| **Condition type** | `Threshold` |
| **Alert trigger** | `Any time series violates` |
| **Threshold position** | `Above threshold` |
| **Threshold value** | `45%` for multi-region instances; `65%` for regional instances. |
| **Retest window** | `10 minutes` |
### 24 hour rolling average CPU
To create an alerting policy that triggers when the 24 hour rolling average of your cpu
utilization for [Spanner](https://docs.cloud.google.com/spanner/docs) is above a recommended threshold,
use the following settings.
#### Steps to create an [alerting policy](https://docs.cloud.google.com/monitoring/alerts/using-alerting-ui#create-policy).
To create an alerting policy, do the following:
1. In the Google Cloud console, go to the
**Alerting** page:
[Go to **Alerting**](https://console.cloud.google.com/monitoring/alerting)
If you use the search bar to find this page, then select the result whose subheading is
**Monitoring**.
2. If you haven't created your notification channels and if you want to be notified, then click **Edit Notification Channels** and add your notification channels. Return to the **Alerting** page after you add your channels.
3. From the **Alerting** page, select **Create policy**.
4. To select the resource, metric, and filters, expand the **Select a metric** menu and then use the values in the **New condition** table:
1. Optional: To limit the menu to relevant entries, enter the resource or metric name in the filter bar.
2. Select a **Resource type** . For example, select **VM instance**.
3. Select a **Metric category** . For example, select **instance**.
4. Select a **Metric** . For example, select **CPU Utilization**.
5. Select **Apply**.
5. Click **Next** and then configure the alerting policy trigger. To complete these fields, use the values in the **Configure alert trigger** table.
6. Click **Next**.
7. Optional: To add notifications to your alerting policy, click
**Notification channels** . In the dialog, select one or more notification
channels from the menu, and then click **OK**.
To be notified when alerts are opened and closed, check
**Notify on alert closure**. By default, notifications are sent only when
alerts are opened.
8. Optional: Update the **Alert autoclose duration**. This field determines when Monitoring closes alerts in the absence of metric data.
9. Optional: Click **Documentation**, and then add any information that you want included in a notification message.
10. Click **Alert name** and enter a name for the alerting policy.
11. Click **Create Policy**.
| **New condition** Field | Value |
|---|---|
| **Resource and Metric** | In the **Resources** menu, select **Spanner Instance** . In the **Metric categories** menu, select **Instance** . In the **Metrics** menu, select **Smoothed CPU utilization** . (The metric.type is `spanner.googleapis.com/instance/cpu/smoothed_utilization`). |
| **Filter** | `instance_id = YOUR_INSTANCE_ID` |
| **Across time series Time series aggregation** | `sum` |
| **Rolling window** | `10 m` |
| **Rolling window function** | `mean` |
| **Configure alert trigger** Field | Value |
|---|---|
| **Condition type** | `Threshold` |
| **Alert trigger** | `Any time series violates` |
| **Threshold position** | `Above threshold` |
| **Threshold** | `90%` |
| **Retest window** | `10 minutes` |
### Storage
To create an alerting policy that triggers when your storage for your
[Spanner](https://docs.cloud.google.com/spanner/docs) instance is above a recommended threshold,
use the following settings.
#### Steps to create an [alerting policy](https://docs.cloud.google.com/monitoring/alerts/using-alerting-ui#create-policy).
To create an alerting policy, do the following:
1. In the Google Cloud console, go to the
**Alerting** page:
[Go to **Alerting**](https://console.cloud.google.com/monitoring/alerting)
If you use the search bar to find this page, then select the result whose subheading is
**Monitoring**.
2. If you haven't created your notification channels and if you want to be notified, then click **Edit Notification Channels** and add your notification channels. Return to the **Alerting** page after you add your channels.
3. From the **Alerting** page, select **Create policy**.
4. To select the resource, metric, and filters, expand the **Select a metric** menu and then use the values in the **New condition** table:
1. Optional: To limit the menu to relevant entries, enter the resource or metric name in the filter bar.
2. Select a **Resource type** . For example, select **VM instance**.
3. Select a **Metric category** . For example, select **instance**.
4. Select a **Metric** . For example, select **CPU Utilization**.
5. Select **Apply**.
5. Click **Next** and then configure the alerting policy trigger. To complete these fields, use the values in the **Configure alert trigger** table.
6. Click **Next**.
7. Optional: To add notifications to your alerting policy, click
**Notification channels** . In the dialog, select one or more notification
channels from the menu, and then click **OK**.
To be notified when alerts are opened and closed, check
**Notify on alert closure**. By default, notifications are sent only when
alerts are opened.
8. Optional: Update the **Alert autoclose duration**. This field determines when Monitoring closes alerts in the absence of metric data.
9. Optional: Click **Documentation**, and then add any information that you want included in a notification message.
10. Click **Alert name** and enter a name for the alerting policy.
11. Click **Create Policy**.
| **New condition** Field | Value |
|---|---|
| **Resource and Metric** | In the **Resources** menu, select **Spanner Instance** . In the **Metric categories** menu, select **Instance** . In the **Metrics** menu, select **Storage used** . (The metric.type is `spanner.googleapis.com/instance/storage/utilization`). |
| **Filter** | `instance_id = YOUR_INSTANCE_ID` |
| **Across time series Time series aggregation** | `sum` |
| **Rolling window** | `10 m` |
| **Rolling window function** | `max` |
| **Configure alert trigger** Field | Value |
|---|---|
| **Condition type** | `Threshold` |
| **Condition triggers if** | `Any time series violates` |
| **Threshold position** | `Above threshold` |
| **Threshold value** | You don't need to set a specific threshold for the maximum storage per node. However, we recommended that you set up an alert for when you are approaching the maximum storage limit. To learn more, see [Storage utilization metrics](https://docs.cloud.google.com/spanner/docs/storage-utilization#recommended-max). |
| **Retest window** | `10 minutes` |
> [!NOTE]
> **Note:** Spanner usage amounts are calculated in *binary terabytes* , where 1 TB is 2^40^ bytes. This unit of measurement is also known as a [tebibyte (TiB)](https://en.wikipedia.org/wiki/Tebibyte).
## What's next
- Understand the [CPU utilization](https://docs.cloud.google.com/spanner/docs/cpu-utilization) and [latency](https://docs.cloud.google.com/spanner/docs/latency-guide) metrics for Spanner.
- [Use the Google Cloud console](https://docs.cloud.google.com/spanner/docs/monitoring-console) to get a quick view of the most important metrics for your instance.
- Learn more about [Cloud Monitoring](https://docs.cloud.google.com/monitoring/docs).