Order of Operations for Updating an Avro Schema and its Associated Database Objects

When you update an Avro schema, follow the steps below (in order) to modify the Pipeline and the Pipeline’s target table to match the updated schema. You can modify the schema (which resides in the schema registry) at any time during this process.

1. Run STOP PIPELINE to detach the Pipeline. If you are are adding a field to the schema, this step is not required.

2. Using ALTER TABLE, add or remove fields from the Pipeline’s target tables.

3. Using ALTER PIPELINE or CREATE OR REPLACE PIPELINE, add or remove fields from the Pipeline.

4. Run START PIPELINE to start the Pipeline. If you are adding a field to the schema and have not stopped the Pipeline, this step is not required.

If you add a field to the schema prior to adding the field to the Pipeline and its target table, and have not stopped the Pipeline, the Pipeline continues to run. The data from the new field in the data source is not populated in the Pipeline’s target table, until the field is added to the Pipeline and its target table.

If you add a field to the schema after adding the field to the Pipeline and its target table, and have not stopped the Pipeline, the Pipeline continues to run. Before the new field is added to the schema, a default value for the field is populated in the Pipeline’s target table. See Example 1: Adding a New Field to an Avro Schema below for details.