Producer fails to use existing schema unless auto-registration creates a new version

[aislinggallagher]

Hi,

I have a query that I hope you could help me with.

• I pre-registered a Protobuf schema using the APIcurio V3 API with the artifact ID set to -value. This created the schema with global ID = 1 and content ID = 1.

• When I try to produce a message to Kafka with auto-registration disabled, the operation fails, even though the schema is already registered.

• If I enable auto-registration, the message is produced successfully, but the registry creates a new version of the schema with a different global ID and content ID, instead of reusing the existing one.

.put(SerdeConfig.ARTIFACT_RESOLVER_STRATEGY, TopicIdStrategy.class.getName())
.put(SerdeConfig.REGISTRY_URL, schemaRegistryUrl)
.put(SerdeConfig.EXPLICIT_ARTIFACT_GROUP_ID, "default")
.put(SerdeConfig.EXPLICIT_ARTIFACT_ID, "-value")
.put(SerdeConfig.AUTO_REGISTER_ARTIFACT, "false")
.put(SerdeConfig.SEND_TYPE_REF, false)
.put(SerdeConfig.SEND_INDEXES, true)

Could you explain why this happens?

Thanks in advance!

[Rahul-20]

Hi @EricWittmann,

I've included the class below where I can reproduce the issue. In my test class, I registered the schema and then produced a message with auto-register set to false. In that case, it fails to find the schema. When I enable auto-register, it registers a new schema version. I also noticed that even a small change, such as adding a space or a comment, causes it to register a new schema version.

<dependency> <groupId>io.apicurio</groupId> <artifactId>apicurio-registry-protobuf-serde-kafka</artifactId> <version>3.0.15</version> </dependency> <dependency> <groupId>io.apicurio</groupId> <artifactId>apicurio-registry-java-sdk</artifactId> <version>3.0.15</version> </dependency>

Test class -
APICurioRegisterAndser.java

Proto Schema -
person.txt

The schema that is registered by the test class is version 1 -

Enabling auto register is creating a new version 2 schema -

Thanks in advance!

[EricWittmann]

OK I have added a unit test for this, just to make sure I understood what happens. And more tests is always better. Here is the PR:

#6680

I'll try to explain what's happening.

When the serializer runs, what is MUST do is ensure that the schema it's using for serialization exists in the registry so that it can include a link to that schema in the message payload (or headers) - the link is used by consumers when deserializing. It can either register the schema itself, or it can verify that the schema exists already.

If autoRegister is configured to be true, then the serializer will extract the raw schema (e.g. the .proto file) from the generated sources and register it at the appropriate registry coordinates. When registering, it uses the content of the raw schema to determine if a new version should be created. If the content of the raw schema matches any version of the artifact, then nothing is created and the existing matching version is used. If the content differs (in any way), then a new version is created and used.

If autoRegister is configured to be false, then the serializer will try to locate a schema by some criteria. That criteria can be either explicit coordinates or by content. When using explicit coordinates (as in your example) you MUST provide the following:

• groupId
• artifactId
• version or "find latest"

If you know the version you want to use, then configure the version number explicitly using apicurio.registry.artifact.version aka SerdeConfig.EXPLICIT_ARTIFACT_VERSION.

If you just want to use the latest version, then configure SerdeConfig.FIND_LATEST_ARTIFACT to be true.

If some or all of these options are not configured, then the schema resolver will try to find a version of an artifact that matches the content of the raw schema.

For protobuf, the latter option often does not work because the raw schema will typically be a little bit different when extracted from the generated source files vs. the actual .proto file used to generate those sources. The "find by content" feature is very sensitive to slight changes in the content, even when searching by "canonical" content, especially for protobuf.

This is an area we can improve (protobuf content canonicalization).