Zookeeper Partitioner Implementation


SetPartitioner implements a partitioning scheme using Zookeeper for dividing up resources amongst members of a party.

This is useful when there is a set of resources that should only be accessed by a single process at a time that multiple processes across a cluster might want to divide up.

Example Use-Case

  • Multiple workers across a cluster need to divide up a list of queues so that no two workers own the same queue.

Public API

class kazoo.recipe.partitioner.SetPartitioner(client, path, set, partition_func=None, identifier=None, time_boundary=30, max_reaction_time=1, state_change_event=None)[source]

Partitions a set amongst members of a party

This class will partition a set amongst members of a party such that each member will be given zero or more items of the set and each set item will be given to a single member. When new members enter or leave the party, the set will be re-partitioned amongst the members.

When the SetPartitioner enters the FAILURE state, it is unrecoverable and a new SetPartitioner should be created.


from kazoo.client import KazooClient
client = KazooClient()

qp = client.SetPartitioner(
    path='/work_queues', set=('queue-1', 'queue-2', 'queue-3'))

while 1:
    if qp.failed:
        raise Exception("Lost or unable to acquire partition")
    elif qp.release:
    elif qp.acquired:
        for partition in qp:
            # Do something with each partition
    elif qp.allocating:

State Transitions

When created, the SetPartitioner enters the PartitionState.ALLOCATING state.


Set was partitioned successfully, the partition list assigned is accessible via list/iter methods or calling list() on the SetPartitioner instance.


Allocating the set failed either due to a Zookeeper session expiration, or failure to acquire the items of the set within the timeout period.


The members of the party have changed, and the set needs to be repartitioned. SetPartitioner.release() should be called as soon as possible.


The current partition was lost due to a Zookeeper session expiration.


The current partition was released and is being re-allocated.
__init__(client, path, set, partition_func=None, identifier=None, time_boundary=30, max_reaction_time=1, state_change_event=None)[source]

Create a SetPartitioner instance

  • client – A KazooClient instance.
  • path – The partition path to use.
  • set – The set of items to partition.
  • partition_func – A function to use to decide how to partition the set.
  • identifier – An identifier to use for this member of the party when participating. Defaults to the hostname + process id.
  • time_boundary – How long the party members must be stable before allocation can complete.
  • max_reaction_time – Maximum reaction time for party members change.
  • state_change_event – An optional Event object that will be set on every state change.

Corresponds to the PartitionState.ACQUIRED state


Corresponds to the PartitionState.ALLOCATING state


Corresponds to the PartitionState.FAILURE state


Call to release the set and leave the party


Corresponds to the PartitionState.RELEASE state


Call to release the set

This method begins the step of allocating once the set has been released.


Wait for the set to be partitioned and acquired

Parameters:timeout (int) – How long to wait before returning.
class kazoo.recipe.partitioner.PartitionState[source]

High level partition state values


The set needs to be partitioned, and may require an existing partition set to be released before acquiring a new partition of the set.


The set has been partitioned and acquired.


The set needs to be repartitioned, and the current partitions must be released before a new allocation can be made.


The set partition has failed. This occurs when the maximum time to partition the set is exceeded or the Zookeeper session is lost. The partitioner is unusable after this state and must be recreated.