Skip to main content

Apache Avro

This Apache Druid extension enables Druid to ingest and parse the Apache Avro data format as follows:

The Avro Stream Parser is deprecated.

Load the Avro extension​

To use the Avro extension, add the druid-avro-extensions to the list of loaded extensions. See Loading extensions for more information.

Avro types​

Druid supports most Avro types natively. This section describes some exceptions.

Unions​

Druid has two modes for supporting union types.

The default mode treats unions as a single value regardless of the type of data populating the union.

If you want to operate on individual members of a union, set extractUnionsByType on the Avro parser. This configuration expands union values into nested objects according to the following rules:

  • Primitive types and unnamed complex types are keyed by their type name, such as int and string.
  • Complex named types are keyed by their names, this includes record, fixed, and enum.
  • The Avro null type is elided as its value can only ever be null.

This is safe because an Avro union can only contain a single member of each unnamed type and duplicates of the same named type are not allowed. For example, only a single array is allowed, multiple records (or other named types) are allowed as long as each has a unique name.

You can then access the members of the union with a flattenSpec like you would for other nested types.

Binary types​

The extension returns bytes and fixed Avro types as base64 encoded strings by default. To decode these types as UTF-8 strings, enable the binaryAsString option on the Avro parser.

Enums​

The extension returns enum types as string of the enum symbol.

Complex types​

You can ingest record and map types representing nested data with a flattenSpec on the parser.

Logical types​

Druid does not currently support Avro logical types. It ignores them and handles fields according to the underlying primitive type.