Table of Contents

Ingestion Spec

An Apache Druid (incubating) ingestion spec consists of 3 components:

{
  "dataSchema" : {...},
  "ioConfig" : {...},
  "tuningConfig" : {...}
}

DataSchema

An example dataSchema is shown below:

"dataSchema" : {
  "dataSource" : "wikipedia",
  "parser" : {
    "type" : "string",
    "parseSpec" : {
      "format" : "json",
      "timestampSpec" : {
        "column" : "timestamp",
        "format" : "auto"
      },
      "dimensionsSpec" : {
        "dimensions": [
          "page",
          "language",
          "user",
          "unpatrolled",
          "newPage",
          "robot",
          "anonymous",
          "namespace",
          "continent",
          "country",
          "region",
          "city",
          {
            "type": "long",
            "name": "countryNum"
          },
          {
            "type": "float",
            "name": "userLatitude"
          },
          {
            "type": "float",
            "name": "userLongitude"
          }
        ],
        "dimensionExclusions" : [],
        "spatialDimensions" : []
      }
    }
  },
  "metricsSpec" : [{
    "type" : "count",
    "name" : "count"
  }, {
    "type" : "doubleSum",
    "name" : "added",
    "fieldName" : "added"
  }, {
    "type" : "doubleSum",
    "name" : "deleted",
    "fieldName" : "deleted"
  }, {
    "type" : "doubleSum",
    "name" : "delta",
    "fieldName" : "delta"
  }],
  "granularitySpec" : {
    "segmentGranularity" : "DAY",
    "queryGranularity" : "NONE",
    "intervals" : [ "2013-08-31/2013-09-01" ]
  },
  "transformSpec" : null
}

Parser

If type is not included, the parser defaults to string. For additional data formats, please see our extensions list.

String Parser

ParseSpec

ParseSpecs serve two purposes:

• The String Parser use them to determine the format (i.e. JSON, CSV, TSV) of incoming rows.
• All Parsers use them to determine the timestamp and dimensions of incoming rows.

If format is not included, the parseSpec defaults to tsv.

JSON ParseSpec

Use this with the String Parser to load JSON.

JSON Lowercase ParseSpec

This is a special variation of the JSON ParseSpec that lower cases all the column names in the incoming JSON data. This parseSpec is required if you are updating to Druid 0.7.x from Druid 0.6.x, are directly ingesting JSON with mixed case column names, do not have any ETL in place to lower case those column names, and would like to make queries that include the data you created using 0.6.x and 0.7.x.

CSV ParseSpec

Use this with the String Parser to load CSV. Strings are parsed using the com.opencsv library.

TSV / Delimited ParseSpec

Use this with the String Parser to load any delimited text that does not require special escaping. By default, the delimiter is a tab, so this will load TSV.

TimeAndDims ParseSpec

Use this with non-String Parsers to provide them with timestamp and dimensions information. Non-String Parsers handle all formatting decisions on their own, without using the ParseSpec.

TimestampSpec

"dimensionsSpec" : {
  "dimensions": [
    "page",
    "language",
    "user",
    "unpatrolled",
    "newPage",
    "robot",
    "anonymous",
    "namespace",
    "continent",
    "country",
    "region",
    "city",
    {
      "type": "string",
      "name": "comment",
      "createBitmapIndex": false
    },
    {
      "type": "long",
      "name": "countryNum"
    },
    {
      "type": "float",
      "name": "userLatitude"
    },
    {
      "type": "float",
      "name": "userLongitude"
    }
  ],
  "dimensionExclusions" : [],
  "spatialDimensions" : []
}

The metricsSpec is a list of aggregators. If rollup is false in the granularity spec, the metrics spec should be an empty list and all columns should be defined in the dimensionsSpec instead (without rollup, there isn't a real distinction between dimensions and metrics at ingestion time). This is optional, however.

GranularitySpec

GranularitySpec is to define how to partition a dataSource into time chunks. The default granularitySpec is uniform, and can be changed by setting the type field. Currently, uniform and arbitrary types are supported.

Uniform Granularity Spec

This spec is used to generated segments with uniform intervals.

Arbitrary Granularity Spec

This spec is used to generate segments with arbitrary intervals (it tries to create evenly sized segments). This spec is not supported for real-time processing.

Transform Spec

Transform specs allow Druid to transform and filter input data during ingestion. See Transform specs

IO Config

The IOConfig spec differs based on the ingestion task type.

• Native Batch ingestion: See Native Batch IOConfig
• Hadoop Batch ingestion: See Hadoop Batch IOConfig
• Kafka Indexing Service: See Kafka Supervisor IOConfig
• Stream Push Ingestion: Stream push ingestion with Tranquility does not require an IO Config.
• Stream Pull Ingestion (Deprecated): See Stream pull ingestion.

Tuning Config

The TuningConfig spec differs based on the ingestion task type.

• Native Batch ingestion: See Native Batch TuningConfig
• Hadoop Batch ingestion: See Hadoop Batch TuningConfig
• Kafka Indexing Service: See Kafka Supervisor TuningConfig
• Stream Push Ingestion (Tranquility): See Tranquility TuningConfig.
• Stream Pull Ingestion (Deprecated): See Stream pull ingestion.

Evaluating Timestamp, Dimensions and Metrics

Druid will interpret dimensions, dimension exclusions, and metrics in the following order:

• Any column listed in the list of dimensions is treated as a dimension.
• Any column listed in the list of dimension exclusions is excluded as a dimension.
• The timestamp column and columns/fieldNames required by metrics are excluded by default.
• If a metric is also listed as a dimension, the metric must have a different name than the dimension name.