Apache Druid
  • Technology
  • Use Cases
  • Powered By
  • Docs
  • Community
  • Apache
  • Download

โ€บMisc

Getting started

  • Introduction to Apache Druid
  • Quickstart (local)
  • Single server deployment
  • Clustered deployment

Tutorials

  • Load files natively
  • Load files using SQL ๐Ÿ†•
  • Load from Apache Kafka
  • Load from Apache Hadoop
  • Querying data
  • Roll-up
  • Theta sketches
  • Configuring data retention
  • Updating existing data
  • Compacting segments
  • Deleting data
  • Writing an ingestion spec
  • Transforming input data
  • Tutorial: Run with Docker
  • Kerberized HDFS deep storage
  • Convert ingestion spec to SQL
  • Jupyter Notebook tutorials

Design

  • Design
  • Segments
  • Processes and servers
  • Deep storage
  • Metadata storage
  • ZooKeeper

Ingestion

  • Ingestion
  • Data formats
  • Data model
  • Data rollup
  • Partitioning
  • Ingestion spec
  • Schema design tips
  • Stream ingestion

    • Apache Kafka ingestion
    • Apache Kafka supervisor
    • Apache Kafka operations
    • Amazon Kinesis

    Batch ingestion

    • Native batch
    • Native batch: input sources
    • Migrate from firehose
    • Hadoop-based

    SQL-based ingestion ๐Ÿ†•

    • Overview
    • Key concepts
    • API
    • Security
    • Examples
    • Reference
    • Known issues
  • Task reference
  • Troubleshooting FAQ

Data management

  • Overview
  • Data updates
  • Data deletion
  • Schema changes
  • Compaction
  • Automatic compaction

Querying

    Druid SQL

    • Overview and syntax
    • SQL data types
    • Operators
    • Scalar functions
    • Aggregation functions
    • Multi-value string functions
    • JSON functions
    • All functions
    • Druid SQL API
    • JDBC driver API
    • SQL query context
    • SQL metadata tables
    • SQL query translation
  • Native queries
  • Query execution
  • Troubleshooting
  • Concepts

    • Datasources
    • Joins
    • Lookups
    • Multi-value dimensions
    • Nested columns
    • Multitenancy
    • Query caching
    • Using query caching
    • Query context

    Native query types

    • Timeseries
    • TopN
    • GroupBy
    • Scan
    • Search
    • TimeBoundary
    • SegmentMetadata
    • DatasourceMetadata

    Native query components

    • Filters
    • Granularities
    • Dimensions
    • Aggregations
    • Post-aggregations
    • Expressions
    • Having filters (groupBy)
    • Sorting and limiting (groupBy)
    • Sorting (topN)
    • String comparators
    • Virtual columns
    • Spatial filters

Configuration

  • Configuration reference
  • Extensions
  • Logging

Operations

  • Web console
  • Java runtime
  • Security

    • Security overview
    • User authentication and authorization
    • LDAP auth
    • Password providers
    • Dynamic Config Providers
    • TLS support

    Performance tuning

    • Basic cluster tuning
    • Segment size optimization
    • Mixed workloads
    • HTTP compression
    • Automated metadata cleanup

    Monitoring

    • Request logging
    • Metrics
    • Alerts
  • API reference
  • High availability
  • Rolling updates
  • Using rules to drop and retain data
  • Working with different versions of Apache Hadoop
  • Misc

    • dump-segment tool
    • reset-cluster tool
    • insert-segment-to-db tool
    • pull-deps tool
    • Deep storage migration
    • Export Metadata Tool
    • Metadata Migration
    • Content for build.sbt

Development

  • Developing on Druid
  • Creating extensions
  • JavaScript functionality
  • Build from source
  • Versioning
  • Experimental features

Misc

  • Papers

Hidden

  • Apache Druid vs Elasticsearch
  • Apache Druid vs. Key/Value Stores (HBase/Cassandra/OpenTSDB)
  • Apache Druid vs Kudu
  • Apache Druid vs Redshift
  • Apache Druid vs Spark
  • Apache Druid vs SQL-on-Hadoop
  • Authentication and Authorization
  • Broker
  • Coordinator Process
  • Historical Process
  • Indexer Process
  • Indexing Service
  • MiddleManager Process
  • Overlord Process
  • Router Process
  • Peons
  • Approximate Histogram aggregators
  • Apache Avro
  • Microsoft Azure
  • Bloom Filter
  • DataSketches extension
  • DataSketches HLL Sketch module
  • DataSketches Quantiles Sketch module
  • DataSketches Theta Sketch module
  • DataSketches Tuple Sketch module
  • Basic Security
  • Kerberos
  • Cached Lookup Module
  • Apache Ranger Security
  • Google Cloud Storage
  • HDFS
  • Apache Kafka Lookups
  • Globally Cached Lookups
  • MySQL Metadata Store
  • ORC Extension
  • Druid pac4j based Security extension
  • Apache Parquet Extension
  • PostgreSQL Metadata Store
  • Protobuf
  • S3-compatible
  • Simple SSLContext Provider Module
  • Stats aggregator
  • Test Stats Aggregators
  • Druid AWS RDS Module
  • Kubernetes
  • Ambari Metrics Emitter
  • Apache Cassandra
  • Rackspace Cloud Files
  • DistinctCount Aggregator
  • Graphite Emitter
  • InfluxDB Line Protocol Parser
  • InfluxDB Emitter
  • Kafka Emitter
  • Materialized View
  • Moment Sketches for Approximate Quantiles module
  • Moving Average Query
  • OpenTSDB Emitter
  • Druid Redis Cache
  • Microsoft SQLServer
  • StatsD Emitter
  • T-Digest Quantiles Sketch module
  • Thrift
  • Timestamp Min/Max aggregators
  • GCE Extensions
  • Aliyun OSS
  • Prometheus Emitter
  • kubernetes
  • Cardinality/HyperUnique aggregators
  • Select
  • Firehose (deprecated)
  • Native batch (simple)
  • Realtime Process
Edit

dump-segment tool

The DumpSegment tool can be used to dump the metadata or contents of an Apache Druid segment for debugging purposes. Note that the dump is not necessarily a full-fidelity translation of the segment. In particular, not all metadata is included, and complex metric values may not be complete.

To run the tool, point it at a segment directory and provide a file for writing output:

java -classpath "/my/druid/lib/*" -Ddruid.extensions.loadList="[]" org.apache.druid.cli.Main \
  tools dump-segment \
  --directory /home/druid/path/to/segment/ \
  --out /home/druid/output.txt

Output format

Data dumps

By default, or with --dump rows, this tool dumps rows of the segment as newline-separate JSON objects, with one object per line, using the default serialization for each column. Normally all columns are included, but if you like, you can limit the dump to specific columns with --column name.

For example, one line might look like this when pretty-printed:

{
  "__time": 1442018818771,
  "added": 36,
  "channel": "#en.wikipedia",
  "cityName": null,
  "comment": "added project",
  "count": 1,
  "countryIsoCode": null,
  "countryName": null,
  "deleted": 0,
  "delta": 36,
  "isAnonymous": "false",
  "isMinor": "false",
  "isNew": "false",
  "isRobot": "false",
  "isUnpatrolled": "false",
  "iuser": "00001553",
  "metroCode": null,
  "namespace": "Talk",
  "page": "Talk:Oswald Tilghman",
  "regionIsoCode": null,
  "regionName": null,
  "user": "GELongstreet"
}

Metadata dumps

With --dump metadata, this tool dumps metadata instead of rows. Metadata dumps generated by this tool are in the same format as returned by the SegmentMetadata query.

Bitmap dumps

With --dump bitmaps, this tool will dump bitmap indexes instead of rows. Bitmap dumps generated by this tool include dictionary-encoded string columns only. The output contains a field "bitmapSerdeFactory" describing the type of bitmaps used in the segment, and a field "bitmaps" containing the bitmaps for each value of each column. These are base64 encoded by default, but you can also dump them as lists of row numbers with --decompress-bitmaps.

Normally all columns are included, but if you like, you can limit the dump to specific columns with --column name.

Sample output:

{
  "bitmapSerdeFactory": {
    "type": "roaring",
    "compressRunOnSerialization": true
  },
  "bitmaps": {
    "isRobot": {
      "false": "//aExfu+Nv3X...",
      "true": "gAl7OoRByQ..."
    }
  }
}

Nested column dumps

With --dump nested, this tool can be used to examine Druid nested columns. Using nested always requires exactly one --column name argument, and takes an optional argument to specify a specific nested field in JSONPath syntax, --nested-path $.path.to.field. If --nested-path is not specified, the output will contain the list of nested fields and their types, the global value dictionaries, and the list of null rows.

Sample output:

{
  "nest": {
    "fields": [
      {
        "path": "$.x",
        "types": [
          "LONG"
        ]
      },
      {
        "path": "$.y",
        "types": [
          "DOUBLE"
        ]
      },
      {
        "path": "$.z",
        "types": [
          "STRING"
        ]
      }
    ],
    "dictionaries": {
      "strings": [
        {
          "globalId": 0,
          "value": null
        },
        {
          "globalId": 1,
          "value": "a"
        },
        {
          "globalId": 2,
          "value": "b"
        }
      ],
      "longs": [
        {
          "globalId": 3,
          "value": 100
        },
        {
          "globalId": 4,
          "value": 200
        },
        {
          "globalId": 5,
          "value": 400
        }
      ],
      "doubles": [
        {
          "globalId": 6,
          "value": 1.1
        },
        {
          "globalId": 7,
          "value": 2.2
        },
        {
          "globalId": 8,
          "value": 3.3
        }
      ],
      "nullRows": []
    }
  }
}

If --nested-path is specified, the output will instead contain the types of the nested field, the local value dictionary, including the 'global' dictionary id and value, the uncompressed bitmap index for each value (list of row numbers which contain the value), and a dump of the column itself, which contains the row number, raw JSON form of the nested column itself, the local dictionary id of the field for that row, and the value for the field for the row.

Sample output:

{
  "bitmapSerdeFactory": {
    "type": "roaring",
    "compressRunOnSerialization": true
  },
  "nest": {
    "$.x": {
      "types": [
        "LONG"
      ],
      "dictionary": [
        {
          "localId": 0,
          "globalId": 0,
          "value": null,
          "rows": [
            4
          ]
        },
        {
          "localId": 1,
          "globalId": 3,
          "value": "100",
          "rows": [
            3
          ]
        },
        {
          "localId": 2,
          "globalId": 4,
          "value": "200",
          "rows": [
            0,
            2
          ]
        },
        {
          "localId": 3,
          "globalId": 5,
          "value": "400",
          "rows": [
            1
          ]
        }
      ],
      "column": [
        {
          "row": 0,
          "raw": {
            "x": 200,
            "y": 2.2
          },
          "fieldId": 2,
          "fieldValue": "200"
        },
        {
          "row": 1,
          "raw": {
            "x": 400,
            "y": 1.1,
            "z": "a"
          },
          "fieldId": 3,
          "fieldValue": "400"
        },
        {
          "row": 2,
          "raw": {
            "x": 200,
            "z": "b"
          },
          "fieldId": 2,
          "fieldValue": "200"
        },
        {
          "row": 3,
          "raw": {
            "x": 100,
            "y": 1.1,
            "z": "a"
          },
          "fieldId": 1,
          "fieldValue": "100"
        },
        {
          "row": 4,
          "raw": {
            "y": 3.3,
            "z": "b"
          },
          "fieldId": 0,
          "fieldValue": null
        }
      ]
    }
  }
}

Command line arguments

argumentdescriptionrequired?
--directory fileDirectory containing segment data. This could be generated by unzipping an "index.zip" from deep storage.yes
--output fileFile to write to, or omit to write to stdout.yes
--dump TYPEDump either 'rows' (default), 'metadata', 'bitmaps', or 'nested' for examining nested columns.no
--column columnNameColumn to include. Specify multiple times for multiple columns, or omit to include all columns.no
--filter jsonJSON-encoded query filter. Omit to include all rows. Only used if dumping rows.no
--time-iso8601Format __time column in ISO8601 format rather than long. Only used if dumping rows.no
--decompress-bitmapsDump bitmaps as arrays rather than base64-encoded compressed bitmaps. Only used if dumping bitmaps.no
--nested-pathSpecify a specific nested column field using JSONPath syntax. Only used if dumping a nested column.no
โ† Working with different versions of Apache Hadoopreset-cluster tool โ†’

Technologyโ€‚ยทโ€‚Use Casesโ€‚ยทโ€‚Powered by Druidโ€‚ยทโ€‚Docsโ€‚ยทโ€‚Communityโ€‚ยทโ€‚Downloadโ€‚ยทโ€‚FAQ

โ€‚ยทโ€‚โ€‚ยทโ€‚โ€‚ยทโ€‚
Copyright ยฉ 2022 Apache Software Foundation.
Except where otherwise noted, licensed under CC BY-SA 4.0.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.