Apache Druid uses Apache ZooKeeper (ZK) for management of current cluster state. The operations that happen over ZK are
- Coordinator leader election
- Segment "publishing" protocol from Historical
- Segment load/drop protocol between Coordinator and Historical
- Overlord leader election
- Overlord and MiddleManager task management
Coordinator Leader Election
We use the Curator LeadershipLatch recipe to do leader election at path
Segment "publishing" protocol from Historical and Realtime
servedSegmentsPath are used for this.
All Historical processes publish themselves on the
announcementsPath, specifically, they will create an ephemeral znode at
Which signifies that they exist. They will also subsequently create a permanent znode at
And as they load up segments, they will attach ephemeral znodes that look like
Segment load/drop protocol between Coordinator and Historical
loadQueuePath is used for this.
This znode will contain a payload that indicates to the Historical process what it should do with the given segment. When the Historical process is done with the work, it will delete the znode in order to signify to the Coordinator that it is complete.