Edit

SQL-based ingestion security

This page describes SQL-based batch ingestion using the druid-multi-stage-query extension, new in Druid 24.0. Refer to the ingestion methods table to determine which ingestion method is right for you.

All authenticated users can use the multi-stage query task engine (MSQ task engine) through the UI and API if the extension is loaded. However, without additional permissions, users are not able to issue queries that read or write Druid datasources or external data. The permission needed depends on what the user is trying to do.

To submit a query:

  • SELECT from a Druid datasource requires the READ DATASOURCE permission on that datasource.
  • INSERT or REPLACE into a Druid datasource requires the WRITE DATASOURCE permission on that datasource.
  • EXTERN and the input-source-specific table functions require READ permission on a resource named "EXTERNAL" with type "EXTERNAL". Users without the correct permission encounter a 403 error when trying to run queries that include EXTERN.

Once a query is submitted, it executes as a query_controller task. Query tasks that users submit to the MSQ task engine are Overlord tasks, so they follow the Overlord's security model. This means that users with access to the Overlord API can perform some actions even if they didn't submit the query, including retrieving status or canceling a query. For more information about the Overlord API and the task API, see APIs for SQL-based ingestion.

To interact with a query through the Overlord API, users need the following permissions:

  • INSERT or REPLACE queries: Users must have READ DATASOURCE permission on the output datasource.
  • SELECT queries: Users must have read permissions on the __query_select datasource, which is a stub datasource that gets created.

S3

The MSQ task engine can use S3 to store intermediate files when running queries. This can increase its reliability but requires certain permissions in S3. These permissions are required if you configure durable storage.

Permissions for pushing and fetching intermediate stage results to and from S3:

  • s3:GetObject
  • s3:PutObject
  • s3:AbortMultipartUpload

Permissions for removing intermediate stage results:

  • s3:DeleteObject
← APIExamples →