Manage application backups using snapshots

A snapshot is a backup of the state of a Managed Service for Apache Flink application. Each snapshot is an Apache Flink savepoint. For more information, see Savepoints in the Apache Flink Documentation.

Restore an application from a snapshot to resume processing from the saved state.

Managed Service for Apache Flink supports two types of snapshots:

An application uses one type of snapshot at a time. New applications use application snapshots by default. To use enhanced snapshots, create the application in enhanced snapshot mode, or upgrade an existing application to enhanced snapshot mode.

Use the following topics to work with enhanced snapshots:

Application snapshots (legacy)

You manage application snapshots with the following API actions:

After the upgrade to enhanced snapshot mode, these rules apply to the application snapshots (legacy) of the application:

For the quota on application snapshots (legacy), see Managed Service for Apache Flink and Studio notebook quota. If your application reaches the limit on snapshots, then manually creating a snapshot fails with a LimitExceededException.

While the application exists, Managed Service for Apache Flink does not delete its application snapshots (legacy). To delete one, use the DeleteApplicationSnapshot action. When you delete the application, Managed Service for Apache Flink also deletes its application snapshots (legacy). DeleteApplication does not delete enhanced snapshots.

To load a saved snapshot of application state when starting an application, use the ApplicationRestoreConfiguration parameter of the StartApplication or UpdateApplication action.

System snapshots

A system snapshot is a snapshot that Managed Service for Apache Flink creates when you update or stop the application. Depending on the snapshot type, other application actions also create a system snapshot, as this section lists. The application uses the system snapshot to keep exactly-once processing.

To turn on system snapshots for either snapshot type, set SnapshotsEnabled to true in the ApplicationSnapshotConfiguration for the application.

The operations that create a system snapshot depend on whether the application uses enhanced snapshot mode or application snapshots (legacy):

In enhanced snapshot mode, system snapshots have the following properties:

For application snapshots (legacy), system snapshots have the following properties:

Restore from a snapshot that contains incompatible state data

A snapshot contains state data for each operator. If an operator changed after the previous application version, a restore of its state data can have unexpected results. An application faults if it restores state data that does not match the current operator. The faulted application stays in the STARTING or UPDATING status.

The guidance in this section applies to enhanced snapshots and to application snapshots (legacy). The ApplicationRestoreConfiguration parameter identifies each type differently:

For more information, see Restore an application from an enhanced snapshot.

You can restore an application from an enhanced snapshot that a different application created. Before you restore, confirm that the snapshot state is compatible with the operators in the target application. The target application must be in the same account and Region as the enhanced snapshot. For the other conditions, see Restore to a different application.

To restore from a snapshot that contains incompatible state data, set AllowNonRestoredState to true. Set this parameter of the FlinkRunConfiguration object with the UpdateApplication action.

A restore from a snapshot that an earlier application version created has the following results:

To resume an application that is incompatible with the existing savepoint data, we recommend that you skip the restore. Set the ApplicationRestoreType parameter of the StartApplication action to SKIP_RESTORE_FROM_SNAPSHOT.

For more information about how Apache Flink deals with incompatible state data, see State Schema Evolution in the Apache Flink Documentation.

Review snapshot API examples

Each example in this section shows the request for application snapshots (legacy). Where the API differs, the example also shows the equivalent request for an enhanced snapshot. For more enhanced snapshot examples and their responses, see Enhanced snapshot API examples.

This section includes example requests for API actions for using snapshots with an application. For information about how to use a JSON file for input for an API action, see Managed Service for Apache Flink API example code.

Enable system snapshots

The following example request for the UpdateApplication action enables snapshots for an application:

{
   "ApplicationName": "MyApplication",
   "CurrentApplicationVersionId": 1,
   "ApplicationConfigurationUpdate": { 
      "ApplicationSnapshotConfigurationUpdate": { 
         "SnapshotsEnabledUpdate": true
       }
    }
}

SnapshotsEnabledUpdate turns on system snapshots for the application. The same ApplicationSnapshotConfigurationUpdate object also accepts SnapshotModeUpdate. To upgrade to enhanced snapshot mode, see Upgrade an existing application to enhanced snapshots.

The following example request upgrades the application to enhanced snapshot mode. The request sets SnapshotModeUpdate and SnapshotsEnabledUpdate together. For the limits on a rollback to application snapshots (legacy), see Roll back after the upgrade.

{
   "ApplicationName": "MyApplication",
   "CurrentApplicationVersionId": 1,
   "ApplicationConfigurationUpdate": {
      "ApplicationSnapshotConfigurationUpdate": {
         "SnapshotsEnabledUpdate": true,
         "SnapshotModeUpdate": "ENHANCED_SNAPSHOTS"
      }
   }
}

Create a snapshot

The following example request for the CreateApplicationSnapshot action creates a snapshot of the current application state:

{
   "ApplicationName": "MyApplication",
   "SnapshotName": "MyCustomSnapshot"
}

For an application in enhanced snapshot mode, use the CreateSnapshot action. The following example request creates an enhanced snapshot of a running application:

{
   "ApplicationName": "MyApplication",
   "SnapshotName": "MyApplication-snapshot-1",
   "SnapshotDescription": "Before code update"
}

List snapshots

The following example request for the ListApplicationSnapshots action lists the first 50 snapshots for the current application state:

{
   "ApplicationName": "MyApplication",
   "Limit": 50
}

For enhanced snapshots, use the ListSnapshots action. The following example request lists the first 50 enhanced snapshots in the account and Region. To list the enhanced snapshots of one application, add the optional SourceApplicationName filter.

{
   "Limit": 50
}

Describe a snapshot

The following example request for the DescribeApplicationSnapshot action lists details for a specific application snapshot:

{
   "ApplicationName": "MyApplication",
   "SnapshotName": "MyCustomSnapshot"
}

For an enhanced snapshot, use the DescribeSnapshot action. The request identifies the enhanced snapshot by name only, because the name is unique in the account and Region:

{
   "SnapshotName": "MyApplication-snapshot-1"
}

Delete a snapshot

The following example request for the DeleteApplicationSnapshot action deletes a previously saved snapshot. You can get the SnapshotCreationTimestamp value using either ListApplicationSnapshots or DescribeApplicationSnapshot:

{
   "ApplicationName": "MyApplication",
   "SnapshotName": "MyCustomSnapshot",
   "SnapshotCreationTimestamp": 12345678901.0
}

For an enhanced snapshot, use the DeleteSnapshot action. Copy the SnapshotCreationTimestamp value from the DescribeSnapshot response:

{
   "SnapshotName": "MyApplication-snapshot-1",
   "SnapshotCreationTimestamp": 1767225600
}

Restart an application from a specific snapshot

The following example request for the StartApplication action starts the application using the saved state from a specific snapshot:

{
   "ApplicationName": "MyApplication",
   "RunConfiguration": { 
      "ApplicationRestoreConfiguration": { 
         "ApplicationRestoreType": "RESTORE_FROM_CUSTOM_SNAPSHOT",
         "SnapshotName": "MyCustomSnapshot"
      }
   }
}

For an application in enhanced snapshot mode, identify the enhanced snapshot with SnapshotArn, not SnapshotName. The caller needs kinesisanalytics:RestoreFromSnapshot on the enhanced snapshot. For more information, see Restore an application from an enhanced snapshot.

{
   "ApplicationName": "MyApplication",
   "RunConfiguration": {
      "ApplicationRestoreConfiguration": {
         "ApplicationRestoreType": "RESTORE_FROM_CUSTOM_SNAPSHOT",
         "SnapshotArn": "arn:aws:kinesisanalytics:aa-example-1:111122223333:snapshot/MyApplication-snapshot-1"
      }
   }
}

Restart an application from the latest snapshot

The following example request for the StartApplication action starts the application using the most recent snapshot:

{
   "ApplicationName": "MyApplication",
   "RunConfiguration": { 
      "ApplicationRestoreConfiguration": { 
         "ApplicationRestoreType": "RESTORE_FROM_LATEST_SNAPSHOT"
      }
   }
}

The same request works in enhanced snapshot mode. In that mode, Managed Service for Apache Flink restores the newest enhanced snapshot in the READY status that the application created. For more information, see Restore from the latest snapshot.

Restart an application without a snapshot

The following example request for the StartApplication action starts the application without loading application state, even if a snapshot is present:

{
   "ApplicationName": "MyApplication",
   "RunConfiguration": { 
      "ApplicationRestoreConfiguration": { 
         "ApplicationRestoreType": "SKIP_RESTORE_FROM_SNAPSHOT"
      }
   }
}

The same request works in enhanced snapshot mode.