Zookeeper Partitioner Implementation
SetPartitioner implements a partitioning scheme using Zookeeper for dividing up resources amongst members of a party.
This is useful when there is a set of resources that should only be accessed by a single process at a time that multiple processes across a cluster might want to divide up.
- Multiple workers across a cluster need to divide up a list of queues so that no two workers own the same queue.
- class kazoo.recipe.partitioner.SetPartitioner(client, path, set, partition_func=None, identifier=None, time_boundary=30)¶
Partitions a set amongst members of a party
This class will partition a set amongst members of a party such that each member will be given zero or more items of the set and each set item will be given to a single member. When new members enter or leave the party, the set will be re-partitioned amongst the members.
Example:from kazoo.client import KazooClient client = KazooClient() qp = client.SetPartitioner( path='/work_queues', set=('queue-1', 'queue-2', 'queue-3')) while 1: if qp.failed: raise Exception("Lost or unable to acquire partition") elif qp.release: qp.release_set() elif qp.acquired: for partition in qp: # Do something with each partition elif qp.allocating: qp.wait_for_acquire()
State TransitionsSet was partitioned successfully, the partition list assigned is accessible via list/iter methods or calling list() on the SetPartitioner instance.Allocating the set failed either due to a Zookeeper session expiration, or failure to acquire the items of the set within the timeout period.The members of the party have changed, and the set needs to be repartitioned. SetPartitioner.release() should be called as soon as possible.The current partition was lost due to a Zookeeper session expiration.The current partition was released and is being re-allocated.
- __init__(client, path, set, partition_func=None, identifier=None, time_boundary=30)¶
Create a SetPartitioner instance
- client – A KazooClient instance.
- path – The partition path to use.
- set – The set of items to partition.
- partition_func – A function to use to decide how to partition the set.
- identifier – An identifier to use for this member of the party when participating. Defaults to the hostname + process id.
- time_boundary – How long the party members must be stable before allocation can complete.
Call to release the set and leave the party
Call to release the set
This method begins the step of allocating once the set has been released.
Wait for the set to be partitioned and acquired
Parameters: timeout (int) – How long to wait before returning.
- class kazoo.recipe.partitioner.PartitionState¶
High level partition state values
The set needs to be partitioned, and may require an existing partition set to be released before acquiring a new partition of the set.
The set has been partitioned and acquired.
The set needs to be repartitioned, and the current partitions must be released before a new allocation can be made.
The set partition has failed. This occurs when the maximum time to partition the set is exceeded or the Zookeeper session is lost. The partitioner is unusable after this state and must be recreated.