Schema management in Stream Connect

Import and create schemas to send and receive messages in an Apache Avro format. Using an Avro format can reduce the size of the payload and simplify your integration to your local Kafka instance.

Avro is an open-source data-serialization system that uses schemas to structure encoded data. With an Avro schema, data can be converted from plain-text JSON to an Avro binary format and back. You can store schemas in ServiceNow, so your Stream Connect producers and consumers can use the schemas to serialize Avro messages.

The following image shows an overview of schema management in Stream Connect. Schemas, stored in schema registries, enable messages in producers and consumers to be converted from plain text to an Avro format and back.

[Omitted image “stream-connect-schema-diagram.png”] Alt text: Diagram showing how Stream Connect uses schemas stored in the schema registries to convert Kafka messages into different formats.

Schemas

You can import a schema from the Confluent registry or create your own standalone schema by uploading a JSON file or entering a schema directly as a JSON-formatted string.

After your schema is imported or created, you can see it on the Stream Connect Schemas [stream_connect_schema] table, which stores both Confluent and standalone schemas. Additionally, Confluent schemas are visible on the Confluent Stream Connect Schema [confluent_stream_connect_schema] table. Standalone schemas are on the Standalone Stream Connect Schema [standalone_stream_connect_schema] table.

All schemas have a schema ID, a globally unique identifier of the schema. For Confluent schemas, the schema ID is imported from the Confluent registry. For standalone schemas, the schema ID is generated locally and is unique on the instance. By default, the generated schema ID value is the next highest available schema ID on the instance. For example, if your schemas have ID numbers one through five, the next schema you create will have a schema ID of six. You can change the default value.

Schema IDs are unique per registry. For example, two schemas can both have an ID of one as long as they’re in different registries.

Schema registries

Every schema belongs to a registry. There are two types of schema registries in ServiceNow: the Confluent Schema Registry and the Standalone Schema Registry.

Both schema registries have an option to Track in the update set. When this option is enabled, the schemas in that registry are saved to the update set. Saving the schemas to the update set makes it possible to move them from one environment to another. By default, this option is turned off for the Confluent Schema Registry because schema IDs may change from one environment to another. This option is enabled for the Standalone Schema Registry, because if you’re creating schemas manually, the schema ID is less likely to change from one environment to another. To change the default setting for either registry, navigate to All > IntegrationHub > Schema Registries, select the registry, and change the Track in the update set option.

When you create or update a Stream Producer record with the serialization format set to Avro, ServiceNow automatically generates and maintains an Avro schema for the associated table.

If a schema already exists for the table when the Stream Producer record is saved, that schema is reused. If no schema exists, a new one is generated based on the table’s field layout. Table metadata is used to detect whether a schema needs to be regenerated. A new schema is only generated when the table structure has changed.

Schemas used by Stream Producer are cached in memory. If a message payload contains a field that the cached schema doesn’t recognize, a new schema is generated and stored in the schema tables. The updated schema is then used to convert the change data capture (CDC) payload to Avro format before the message is sent to Kafka.

Stream Producer schemas are stored across two tables in the ServiceNow IntegrationHub Stream Connect Schema [com.glide.hub.stream_connect.schema] plugin. These tables are read-only for all users. No one has create, update, or delete access.

<table id=“table_schema_tables”><thead><tr><th>

Table

</th><th>

Description

</th></tr></thead><tbody><tr><td>

Table Message Schemas [sys_table_message_schema]

</td><td>

Stores the Avro schema content for a table. Extends stream_connect_schema. Inherits all fields from the parent table, including the schema content field.

</td></tr><tr><td>

Table Message Schema Evolutions [sys_table_message_schema_evolution]

</td><td>

Tracks schema versions for a table. Each record has the following.- A subject in the format sn_*table_name* (for example, sn_incident). - The Table Message Schema, which is a reference to the corresponding sys_table_message_schema record. - The name of the ServiceNow table this schema is associated with. - The version number of the schema. The version increments each time a new schema is generated for the table.

</td></tr></tbody> </table>For reference fields in the Avro payload, both the sys_id and display value are included.

Role Description message_schema_viewer Grants read, range, and report_view access to the schema tables (sys_table_message_schema and sys_table_message_schema_evolution). No user has create, update, or delete access to these tables. If the Stream Connect Core [com.glide.hub.stream_connect.common.core] plugin is active, this role is automatically included in the stream_connect_viewer role.

Wire-level message format

For interoperability, ServiceNow uses a wire-level message format similar to the ones used by other systems. The first byte is set to 0. The next 4 bytes are used for the schema ID. The remaining bytes are used for the data, serialized in an Avro format.

<table id=“table_rhd_vjy_jbc”><tbody><tr><td>

Byte 0

</td><td>

Magic byte.

</td></tr><tr><td>

Byte 1–4

</td><td>

Schema ID.

</td></tr><tr><td>

Remaining bytes

</td><td>

Data, serialized in an Avro format.

</td></tr></tbody> </table>## Producers and consumers

Stream Connect producers and consumers can be configured to use an Avro format.

When configuring a producer, simply specify which schema you want to use. Then when you run the producer, the message payload is generated in JSON and automatically converted to an Avro format using the specified schema. For more information on producers, see the or the ProducerV2 API.

You can also configure a Stream Producer to send messages in an Avro format. When the serialization format is set to Avro, the Stream Producer uses the auto-generated schema for the selected table to convert CDC payloads to Avro before sending them to Kafka.

Configuring a consumer is similar. Specify the serialization format as Encoded and select a schema registry. When the consumer receives a message in an Avro format, it’s automatically converted to JSON according to the schema for the schema ID received in the message. For more information, see the Kafka Message trigger or the ETL, Transform Map, or Script consumers.

The Schema Registry REST API exposes your Stream Producer Avro schemas to external systems and consumers. External applications can use the API to retrieve schemas and decode Avro-encoded messages received from Kafka topics.

The API is available at the base URI /api/now/schemaregistry. It uses standard ServiceNow authentication. ACLs control access to the scripted REST API endpoints.

Plugin

Schema management features require the ServiceNow Stream Connect Installer [com.glide.hub.stream_connect.installer] plugin.

Stream Producer schema evolution features requires either the:

• ServiceNow Integration Hub Stream Connect Schema [com.glide.hub.stream_connect.schema] plugin, or the

• ServiceNow Stream Connect Installer (Direct Kafka) [com.glide.hub.stream_connect.onprem_installer] plugin.

If the Stream Connect Core [com.glide.hub.stream_connect.common.core] plugin is installed, the message_schema_viewer role is automatically added to the stream_connect_viewer role.

• Import a schema from the Confluent Registry
  Import a Schema from the Confluent Registry to enable your Stream Connect producers and consumers to send and receive Kafka messages in an Apache Avro format.

• Create a standalone schema in Stream Connect
  Create a schema to enable your Stream Connect producers and consumers to send and receive Kafka messages in an Apache Avro format.

Parent Topic:Using Stream Connect for Apache Kafka