Defining reusable schemas in Avro

In one of my project Avro schemas was very complex and they will run with 50 or more fields. Lots of schemas will share the fields and less than 10 fields will be unique for a individual schemas.

It was quite exhausting to maintain different schemas when one or more fields will change.

That’s where I figured out that you can divide the schemas into multiple and make them reusable across multiple schemas.

How to define multiple Avro schema

I have defined a very short schema to show the syntax where one complex type uses another one.

Here is Author schema below:

{
  "namespace": "com.deepti.sample.avro.schema",
  "type": "record",
  "name": "Author",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name":"publication",
      "type":"string"
    }
  ]
}

Now lets define Book Schema which used Author schema:

{
  "namespace": "com.deepti.sample.avro.schema",
  "type": "record",
  "name": "Book",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name":"author",
      "type":"com.deepti.sample.avro.schema.Author"
    },
    {
      "name":"publication",
      "type":"string"
    }
  ]
}

Now we will define library schema which will use both Author and Book schema:

{
  "namespace": "com.deepti.sample.avro.schema",
  "type": "record",
  "name": "Library",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "owner",
      "type": "string"
    },
    {
      "name":"books",
      "type":"com.deepti.sample.avro.schema.Book"
    },
    {
      "name":"authors",
      "type":"com.deepti.sample.avro.schema.Author"
    },
    {
      "name":"address",
      "type":"string"
    }
  ]
}

So you can see how type will point to type of other schema.

Gradle tasks for generating Avro classes

In Avro where one file is depended on other we have to make sure that dependencies are met before generating classes for dependent schemas. For above example, below could be the gradle tasks:

import com.commercehub.gradle.plugin.avro.GenerateAvroJavaTask
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath "com.commercehub.gradle.plugin:gradle-avro-plugin:0.20.0"
    }
}
tasks.register("generateAvro", GenerateAvroJavaTask) {
    source("src/main/resources/avro/author.avsc")
    source("src/main/resources/avro/book.avsc")
    source("src/main/resources/avro/library.avsc")
    outputDir = file("src/main/java/")
}