Apache Druid
  • Technology
  • Use Cases
  • Powered By
  • Docs
  • Community
  • Apache
  • Download

โ€บPerformance tuning

Getting started

  • Introduction to Apache Druid
  • Quickstart (local)
  • Single server deployment
  • Clustered deployment

Tutorials

  • Load files natively
  • Load files using SQL ๐Ÿ†•
  • Load from Apache Kafka
  • Load from Apache Hadoop
  • Querying data
  • Roll-up
  • Theta sketches
  • Configuring data retention
  • Updating existing data
  • Compacting segments
  • Deleting data
  • Writing an ingestion spec
  • Transforming input data
  • Tutorial: Run with Docker
  • Kerberized HDFS deep storage
  • Convert ingestion spec to SQL
  • Jupyter Notebook tutorials

Design

  • Design
  • Segments
  • Processes and servers
  • Deep storage
  • Metadata storage
  • ZooKeeper

Ingestion

  • Ingestion
  • Data formats
  • Data model
  • Data rollup
  • Partitioning
  • Ingestion spec
  • Schema design tips
  • Stream ingestion

    • Apache Kafka ingestion
    • Apache Kafka supervisor
    • Apache Kafka operations
    • Amazon Kinesis

    Batch ingestion

    • Native batch
    • Native batch: input sources
    • Migrate from firehose
    • Hadoop-based

    SQL-based ingestion ๐Ÿ†•

    • Overview
    • Key concepts
    • API
    • Security
    • Examples
    • Reference
    • Known issues
  • Task reference
  • Troubleshooting FAQ

Data management

  • Overview
  • Data updates
  • Data deletion
  • Schema changes
  • Compaction
  • Automatic compaction

Querying

    Druid SQL

    • Overview and syntax
    • SQL data types
    • Operators
    • Scalar functions
    • Aggregation functions
    • Multi-value string functions
    • JSON functions
    • All functions
    • Druid SQL API
    • JDBC driver API
    • SQL query context
    • SQL metadata tables
    • SQL query translation
  • Native queries
  • Query execution
  • Troubleshooting
  • Concepts

    • Datasources
    • Joins
    • Lookups
    • Multi-value dimensions
    • Nested columns
    • Multitenancy
    • Query caching
    • Using query caching
    • Query context

    Native query types

    • Timeseries
    • TopN
    • GroupBy
    • Scan
    • Search
    • TimeBoundary
    • SegmentMetadata
    • DatasourceMetadata

    Native query components

    • Filters
    • Granularities
    • Dimensions
    • Aggregations
    • Post-aggregations
    • Expressions
    • Having filters (groupBy)
    • Sorting and limiting (groupBy)
    • Sorting (topN)
    • String comparators
    • Virtual columns
    • Spatial filters

Configuration

  • Configuration reference
  • Extensions
  • Logging

Operations

  • Web console
  • Java runtime
  • Security

    • Security overview
    • User authentication and authorization
    • LDAP auth
    • Password providers
    • Dynamic Config Providers
    • TLS support

    Performance tuning

    • Basic cluster tuning
    • Segment size optimization
    • Mixed workloads
    • HTTP compression
    • Automated metadata cleanup

    Monitoring

    • Request logging
    • Metrics
    • Alerts
  • API reference
  • High availability
  • Rolling updates
  • Using rules to drop and retain data
  • Working with different versions of Apache Hadoop
  • Misc

    • dump-segment tool
    • reset-cluster tool
    • insert-segment-to-db tool
    • pull-deps tool
    • Deep storage migration
    • Export Metadata Tool
    • Metadata Migration
    • Content for build.sbt

Development

  • Developing on Druid
  • Creating extensions
  • JavaScript functionality
  • Build from source
  • Versioning
  • Experimental features

Misc

  • Papers

Hidden

  • Apache Druid vs Elasticsearch
  • Apache Druid vs. Key/Value Stores (HBase/Cassandra/OpenTSDB)
  • Apache Druid vs Kudu
  • Apache Druid vs Redshift
  • Apache Druid vs Spark
  • Apache Druid vs SQL-on-Hadoop
  • Authentication and Authorization
  • Broker
  • Coordinator Process
  • Historical Process
  • Indexer Process
  • Indexing Service
  • MiddleManager Process
  • Overlord Process
  • Router Process
  • Peons
  • Approximate Histogram aggregators
  • Apache Avro
  • Microsoft Azure
  • Bloom Filter
  • DataSketches extension
  • DataSketches HLL Sketch module
  • DataSketches Quantiles Sketch module
  • DataSketches Theta Sketch module
  • DataSketches Tuple Sketch module
  • Basic Security
  • Kerberos
  • Cached Lookup Module
  • Apache Ranger Security
  • Google Cloud Storage
  • HDFS
  • Apache Kafka Lookups
  • Globally Cached Lookups
  • MySQL Metadata Store
  • ORC Extension
  • Druid pac4j based Security extension
  • Apache Parquet Extension
  • PostgreSQL Metadata Store
  • Protobuf
  • S3-compatible
  • Simple SSLContext Provider Module
  • Stats aggregator
  • Test Stats Aggregators
  • Druid AWS RDS Module
  • Kubernetes
  • Ambari Metrics Emitter
  • Apache Cassandra
  • Rackspace Cloud Files
  • DistinctCount Aggregator
  • Graphite Emitter
  • InfluxDB Line Protocol Parser
  • InfluxDB Emitter
  • Kafka Emitter
  • Materialized View
  • Moment Sketches for Approximate Quantiles module
  • Moving Average Query
  • OpenTSDB Emitter
  • Druid Redis Cache
  • Microsoft SQLServer
  • StatsD Emitter
  • T-Digest Quantiles Sketch module
  • Thrift
  • Timestamp Min/Max aggregators
  • GCE Extensions
  • Aliyun OSS
  • Prometheus Emitter
  • kubernetes
  • Cardinality/HyperUnique aggregators
  • Select
  • Firehose (deprecated)
  • Native batch (simple)
  • Realtime Process
Edit

Automated cleanup for metadata records

Apache Druid relies on metadata storage to track information on data storage, operations, and system configuration. The metadata store includes the following:

  • Segment records
  • Audit records
  • Supervisor records
  • Rule records
  • Compaction configuration records
  • Datasource records created by supervisors
  • Indexer task logs

When you delete some entities from Apache Druid, records related to the entity may remain in the metadata store. If you have a high datasource churn rate, meaning you frequently create and delete many short-lived datasources or other related entities like compaction configuration or rules, the leftover records can fill your metadata store and cause performance issues. To maintain metadata store performance, you can configure Apache Druid to automatically remove records associated with deleted entities from the metadata store.

By default, Druid automatically cleans up metadata older than 90 days. This applies to all metadata entities in this topic except compaction configuration records and indexer task logs, for which cleanup is disabled by default. You can configure the retention period for each metadata type, when available, through the record's durationToRetain property. Certain records may require additional conditions be satisfied before clean up occurs.

See the example for how you can customize the automated metadata cleanup for a specific use case.

Automated cleanup strategies

There are several cases when you should consider automated cleanup of the metadata related to deleted datasources:

  • If you know you have many high-churn datasources, for example, you have scripts that create and delete supervisors regularly.
  • If you have issues with the hard disk for your metadata database filling up.
  • If you run into performance issues with the metadata database. For example, API calls are very slow or fail to execute.

If you have compliance requirements to keep audit records and you enable automated cleanup for audit records, use alternative methods to preserve audit metadata, for example, by periodically exporting audit metadata records to external storage.

Configure automated metadata cleanup

You can configure cleanup for each entity separately, as described in this section. Define the properties in the coordinator/runtime.properties file.

The cleanup of one entity may depend on the cleanup of another entity as follows:

  • You have to configure a kill task for segment records before you can configure automated cleanup for rules or compaction configuration.
  • You have to schedule the metadata management tasks to run at the same or higher frequency as your most frequent cleanup job. For example, if your most frequent cleanup job is every hour, set the metadata store management period to one hour or less: druid.coordinator.period.metadataStoreManagementPeriod=P1H.

For details on configuration properties, see Metadata management. If you want to skip the details, check out the example for configuring automated metadata cleanup.

Segment records and segments in deep storage (kill task)

The kill task is the only configuration in this topic that affects actual data in deep storage and not simply metadata or logs.

Segment records and segments in deep storage become eligible for deletion when both of the following conditions hold:

  • When they meet the eligibility requirement of kill task datasource configuration according to killDataSourceWhitelist set in the Coordinator dynamic configuration. See Dynamic configuration.
  • When the durationToRetain time has passed since their creation.

Kill tasks use the following configuration:

  • druid.coordinator.kill.on: When true, enables the Coordinator to submit a kill task for unused segments, which deletes them completely from metadata store and from deep storage. Only applies to the specified datasources in the dynamic configuration parameter killDataSourceWhitelist. If killDataSourceWhitelist is not set or empty, then kill tasks can be submitted for all datasources.
  • druid.coordinator.kill.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible segments. Defaults to P1D. Must be greater than druid.coordinator.period.indexingPeriod.
  • druid.coordinator.kill.durationToRetain: Defines the retention period in ISO 8601 format after creation that segments become eligible for deletion.
  • druid.coordinator.kill.maxSegments: Defines the maximum number of segments to delete per kill task.

Audit records

All audit records become eligible for deletion when the durationToRetain time has passed since their creation.

Audit cleanup uses the following configuration:

  • druid.coordinator.kill.audit.on: When true, enables cleanup for audit records.
  • druid.coordinator.kill.audit.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible audit records. Defaults to P1D.
  • druid.coordinator.kill.audit.durationToRetain: Defines the retention period in ISO 8601 format after creation that audit records become eligible for deletion.

Supervisor records

Supervisor records become eligible for deletion when the supervisor is terminated and the durationToRetain time has passed since their creation.

Supervisor cleanup uses the following configuration:

  • druid.coordinator.kill.supervisor.on: When true, enables cleanup for supervisor records.
  • druid.coordinator.kill.supervisor.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible supervisor records. Defaults to P1D.
  • druid.coordinator.kill.supervisor.durationToRetain: Defines the retention period in ISO 8601 format after creation that supervisor records become eligible for deletion.

Rules records

Rule records become eligible for deletion when all segments for the datasource have been killed by the kill task and the durationToRetain time has passed since their creation. Automated cleanup for rules requires a kill task.

Rule cleanup uses the following configuration:

  • druid.coordinator.kill.rule.on: When true, enables cleanup for rules records.
  • druid.coordinator.kill.rule.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible rules records. Defaults to P1D.
  • druid.coordinator.kill.rule.durationToRetain: Defines the retention period in ISO 8601 format after creation that rules records become eligible for deletion.

Compaction configuration records

Druid retains all compaction configuration records by default, which should be suitable for most use cases. If you create and delete short-lived datasources with high frequency, and you set auto compaction configuration on those datasources, then consider turning on automated cleanup of compaction configuration records.

With automated cleanup of compaction configuration records, if you create a compaction configuration for some datasource before the datasource exists, for example if initial ingestion is still ongoing, Druid may remove the compaction configuration. To prevent the configuration from being prematurely removed, wait for the datasource to be created before applying the compaction configuration to the datasource.

Unlike other metadata records, compaction configuration records do not have a retention period set by durationToRetain. Druid deletes compaction configuration records at every cleanup cycle for inactive datasources, which do not have segments either used or unused.

Compaction configuration records in the druid_config table become eligible for deletion after all segments for the datasource have been killed by the kill task. Automated cleanup for compaction configuration requires a kill task.

Compaction configuration cleanup uses the following configuration:

  • druid.coordinator.kill.compaction.on: When true, enables cleanup for compaction configuration records.
  • druid.coordinator.kill.compaction.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible compaction configuration records. Defaults to P1D.

If you already have an extremely large compaction configuration, you may not be able to delete compaction configuration due to size limits with the audit log. In this case you can set druid.audit.manager.maxPayloadSizeBytes and druid.audit.manager.skipNullField to avoid the auditing issue. See Audit logging.

Datasource records created by supervisors

Datasource records created by supervisors become eligible for deletion when the supervisor is terminated or does not exist in the druid_supervisors table and the durationToRetain time has passed since their creation.

Datasource cleanup uses the following configuration:

  • druid.coordinator.kill.datasource.on: When true, enables cleanup datasources created by supervisors.
  • druid.coordinator.kill.datasource.period: Defines the frequency in ISO 8601 format for the cleanup job to check for and delete eligible datasource records. Defaults to P1D.
  • druid.coordinator.kill.datasource.durationToRetain: Defines the retention period in ISO 8601 format after creation that datasource records become eligible for deletion.

Indexer task logs

You can configure the Overlord to delete indexer task log metadata and the indexer task logs from local disk or from cloud storage. Set these properties in the overlord/runtime.properties file.

Indexer task log cleanup on the Overlord uses the following configuration:

  • druid.indexer.logs.kill.enabled: When true, enables cleanup of task logs.
  • druid.indexer.logs.kill.durationToRetain: Defines the length of time in milliseconds to retain task logs.
  • druid.indexer.logs.kill.initialDelay: Defines the length of time in milliseconds after the Overlord starts before it executes its first job to kill task logs.
  • druid.indexer.logs.kill.delay: The length of time in milliseconds between jobs to kill task logs.

For more detail, see Task logging.

Disable automated metadata cleanup

Druid automatically cleans up metadata records, excluding compaction configuration records and indexer task logs. To disable automated metadata cleanup, set the following properties in the coordinator/runtime.properties file:

# Keep unused segments
druid.coordinator.kill.on=false

# Keep audit records
druid.coordinator.kill.audit.on=false

# Keep supervisor records
druid.coordinator.kill.supervisor.on=false

# Keep rules records
druid.coordinator.kill.rule.on=false

# Keep datasource records created by supervisors
druid.coordinator.kill.datasource.on=false

Example configuration for automated metadata cleanup

Consider a scenario where you have scripts to create and delete hundreds of datasources and related entities a day. You do not want to fill your metadata store with leftover records. The datasources and related entities tend to persist for only one or two days. Therefore, you want to run a cleanup job that identifies and removes leftover records that are at least four days old. The exception is for audit logs, which you need to retain for 30 days:

...
# Schedule the metadata management store task for every hour:
druid.coordinator.period.metadataStoreManagementPeriod=P1H

# Set a kill task to poll every day to delete Segment records and segments
# in deep storage > 4 days old. When druid.coordinator.kill.on is set to true,
# you can set killDataSourceWhitelist in the dynamic configuration to limit
# the datasources that can be killed.
# Required also for automated cleanup of rules and compaction configuration.

druid.coordinator.kill.on=true
druid.coordinator.kill.period=P1D 
druid.coordinator.kill.durationToRetain=P4D
druid.coordinator.kill.maxSegments=1000

# Poll every day to delete audit records > 30 days old
druid.coordinator.kill.audit.on=true
druid.coordinator.kill.audit.period=P1D
druid.coordinator.kill.audit.durationToRetain=P30D

# Poll every day to delete supervisor records > 4 days old
druid.coordinator.kill.supervisor.on=true
druid.coordinator.kill.supervisor.period=P1D
druid.coordinator.kill.supervisor.durationToRetain=P4D

# Poll every day to delete rules records > 4 days old
druid.coordinator.kill.rule.on=true
druid.coordinator.kill.rule.period=P1D
druid.coordinator.kill.rule.durationToRetain=P4D

# Poll every day to delete compaction configuration records
druid.coordinator.kill.compaction.on=true
druid.coordinator.kill.compaction.period=P1D

# Poll every day to delete datasource records created by supervisors > 4 days old
druid.coordinator.kill.datasource.on=true
druid.coordinator.kill.datasource.period=P1D
druid.coordinator.kill.datasource.durationToRetain=P4D
...

Learn more

See the following topics for more information:

  • Metadata management for metadata store configuration reference.
  • Metadata storage for an overview of the metadata storage database.
โ† HTTP compressionRequest logging โ†’
  • Automated cleanup strategies
  • Configure automated metadata cleanup
    • Segment records and segments in deep storage (kill task)
    • Audit records
    • Supervisor records
    • Rules records
    • Compaction configuration records
    • Datasource records created by supervisors
    • Indexer task logs
  • Disable automated metadata cleanup
  • Example configuration for automated metadata cleanup
  • Learn more

Technologyโ€‚ยทโ€‚Use Casesโ€‚ยทโ€‚Powered by Druidโ€‚ยทโ€‚Docsโ€‚ยทโ€‚Communityโ€‚ยทโ€‚Downloadโ€‚ยทโ€‚FAQ

โ€‚ยทโ€‚โ€‚ยทโ€‚โ€‚ยทโ€‚
Copyright ยฉ 2022 Apache Software Foundation.
Except where otherwise noted, licensed under CC BY-SA 4.0.
Apache Druid, Druid, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.