Skip to content
javahacks.dev
Microservices

How to Version Event Schemas With Avro and Schema Registry

Version event schemas with Avro and a schema registry in Java: define a schema, set compatibility modes, evolve fields safely, use Spring Kafka.

6 min read
Summarize in AI
Flat illustration of a database beside the table it holds, with the Redis and Hibernate logos.

Services that communicate through events need a contract that can change without breaking the readers. Avro with a schema registry provides a compact binary format and, more importantly, enforces compatibility when a schema changes, so a producer cannot publish something that existing consumers would choke on. The examples use Avro 1.12, Confluent Schema Registry and Spring Kafka.

A schema

JSON
{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.events",
  "fields": [
    {"name": "orderId", "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "totalCents", "type": "long"},
    {"name": "currency", "type": "string", "default": "EUR"}
  ]
}

Store the file in the repository next to the code that publishes it. Generate Java classes from it in the build, with the Avro Maven plugin:

XML
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.12.0</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals><goal>schema</goal></goals>
      <configuration>
        <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
        <stringType>String</stringType>
      </configuration>
    </execution>
  </executions>
</plugin>

Without stringType set to String, generated fields use CharSequence, which surprises everyone the first time they compare one. Where classes are not wanted, the generic GenericRecord API reads any schema without generated code.

Producing

properties
spring.kafka.producer.value-serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
spring.kafka.properties.schema.registry.url=http://registry:8081
spring.kafka.properties.auto.register.schemas=false
spring.kafka.properties.use.latest.version=true
java
kafkaTemplate.send("orders.placed", event.getOrderId(), event);

The serializer looks up the schema id in the registry and writes it as a short prefix on every message, and consumers fetch the schema by that id and decode. Turn auto.register.schemas off in production and register schemas from CI, so an application cannot publish an unreviewed change by accident. With auto registration off, the serializer must be told which registered version to use, which is what use.latest.version does. Keys and partitions are covered in Kafka partitions and keys.

Compatibility modes

Compatibility describes which readers can read which writers, and the mode you choose decides who has to upgrade first.

ModeAllowed changesDeploy in this order
BACKWARDDelete fields; add fields that have defaultsConsumers, then producers
FORWARDAdd fields; delete fields that have defaultsProducers, then consumers
FULLBoth of the aboveAny order
The _TRANSITIVE formsChecked against every earlier version, not only the lastAny order, and safe for replays

BACKWARD is the registry default. Choose the mode per subject to match how you deploy:

Shell
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://registry:8081/config/orders.placed-value

For a topic with long retention, where a new consumer may read the log from the beginning, use a transitive mode. A non-transitive check only compares with the previous version, so a field removed in version 3 and a consumer still holding version 1 can break.

Safe changes and unsafe ones

Safe: add a field with a default, such as {"name": "couponCode", "type": ["null", "string"], "default": null}. Add an enum symbol only when the enum declares a default symbol or every reader handles unknown values. Widen a type along the promotion paths Avro defines, such as int to long. Rename a field only by adding an alias, which readers use to match the old name.

Unsafe: remove a field that has no default, change a type in a way Avro does not promote, or reuse a field name for a different meaning. The last is the worst because the registry cannot see it; the types still match and the data is nonsense. When a breaking change is genuinely needed, publish a new event type or a new topic such as OrderPlacedV2, run both for a while, and retire the old one, which is the same discipline as REST versions in API versioning strategies.

Check compatibility in CI

Shell
mvn io.confluent:kafka-schema-registry-maven-plugin:test-compatibility

Configure the plugin with the registry URL and a map from subject to schema file. It tests the local file against the registry and fails the build for an incompatible change, which turns a production incident into a red pull request. Register on release with the plugin's register goal, from the pipeline, not from a laptop.

The consumer side

properties
spring.kafka.consumer.value-deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
spring.kafka.properties.specific.avro.reader=true

A consumer uses its own compiled version of the schema as the reader schema. Avro resolves differences between the writer's schema, fetched by id, and the reader's: it fills in defaults for fields the writer did not know and ignores fields the reader does not use. This is why defaults carry so much weight. A field without one cannot be added safely, because old messages cannot supply a value for it. Replays and duplicates need their own handling, described in duplicate messages.

Check evolution in a unit test, before CI

The registry check needs a running registry. A plain unit test needs nothing, and Avro ships the logic it uses. SchemaCompatibility compares a reader schema against a writer schema and reports whether data written with one can be read with the other:

java
@Test
void newReaderCanReadOldEvents() throws IOException {
    Schema writer = new Schema.Parser().parse(resource("/avro/OrderPlaced-v1.avsc"));
    Schema reader = OrderPlaced.getClassSchema();

    var result = SchemaCompatibility.checkReaderWriterCompatibility(reader, writer);

    assertEquals(SchemaCompatibilityType.COMPATIBLE, result.getType(), result.getDescription());
}

Keep every schema that has ever been released in the test resources, named by version, and loop over all of them. That is the transitive check done locally, in milliseconds, on a laptop. When a developer removes a field without a default, the failing test names the field and the reason, which is a far better review comment than a rejected registry call during a release.

Moving an existing topic onto Avro

Topics that already carry JSON cannot switch format in place. Consumers that expect JSON would receive binary bytes with a registry prefix and fail on the first message. The safe route is to introduce a new topic, publish to both for a period, and move consumers one at a time. Name the new topic for its format or version only if you expect to run both for a long time; otherwise reuse the logical name with a suffix you can drop later. While both exist, put the conversion in one place, a small bridge that reads the old topic and writes the new one, so no producer has two code paths to keep consistent.

Decide the subject naming strategy at the start. The default derives the subject from the topic name, which gives one schema history per topic and works while a topic holds one event type. If a topic mixes several event types, as an ordered stream of an aggregate's events does, switch to the record name strategy, so each type has its own subject and its own compatibility rules. Changing the strategy later is painful, because existing subjects no longer match.

Other formats

The registry also supports Protobuf and JSON Schema. Protobuf's numbered fields give comparable evolution rules, discussed in Protobuf and gRPC. Avro is the common choice on Kafka because its tooling is the most mature there.

When you do not need a registry

A single producer and consumer deployed together as one unit have no independent upgrade problem, and a registry adds an operational dependency for nothing. The same applies to events that never leave one service. The format is defined in the Avro specification, and the registry is documented in the Schema Registry guide.

Taggedhow to
Share

Keep reading

Get new posts by email

We send one email when a new post is published. No spam.

Unsubscribe in one click. No spam, ever.