kazoo.recipe.partitioner
¶
Zookeeper Partitioner Implementation
- Maintainer:
None
- Status:
Unknown
SetPartitioner
implements a partitioning scheme using
Zookeeper for dividing up resources amongst members of a party.
This is useful when there is a set of resources that should only be accessed by a single process at a time that multiple processes across a cluster might want to divide up.
Example Use-Case¶
Multiple workers across a cluster need to divide up a list of queues so that no two workers own the same queue.
Public API¶
- class kazoo.recipe.partitioner.SetPartitioner(client, path, set, partition_func=None, identifier=None, time_boundary=30, max_reaction_time=1, state_change_event=None)[source]¶
Partitions a set amongst members of a party
This class will partition a set amongst members of a party such that each member will be given zero or more items of the set and each set item will be given to a single member. When new members enter or leave the party, the set will be re-partitioned amongst the members.
When the
SetPartitioner
enters theFAILURE
state, it is unrecoverable and a newSetPartitioner
should be created.Example:
from kazoo.client import KazooClient client = KazooClient() client.start() qp = client.SetPartitioner( path='/work_queues', set=('queue-1', 'queue-2', 'queue-3')) while 1: if qp.failed: raise Exception("Lost or unable to acquire partition") elif qp.release: qp.release_set() elif qp.acquired: for partition in qp: # Do something with each partition elif qp.allocating: qp.wait_for_acquire()State Transitions
When created, the
SetPartitioner
enters thePartitionState.ALLOCATING
state.Set was partitioned successfully, the partition list assigned is accessible via list/iter methods or calling list() on the
SetPartitioner
instance.Allocating the set failed either due to a Zookeeper session expiration, or failure to acquire the items of the set within the timeout period.
The members of the party have changed, and the set needs to be repartitioned.
SetPartitioner.release()
should be called as soon as possible.The current partition was lost due to a Zookeeper session expiration.
The current partition was released and is being re-allocated.
- __init__(client, path, set, partition_func=None, identifier=None, time_boundary=30, max_reaction_time=1, state_change_event=None)[source]¶
Create a
SetPartitioner
instance
- Parameters:
client – A
KazooClient
instance.path – The partition path to use.
set – The set of items to partition.
partition_func – A function to use to decide how to partition the set.
identifier – An identifier to use for this member of the party when participating. Defaults to the hostname + process id.
time_boundary – How long the party members must be stable before allocation can complete.
max_reaction_time – Maximum reaction time for party members change.
state_change_event – An optional Event object that will be set on every state change.
- property acquired¶
Corresponds to the
PartitionState.ACQUIRED
state
- property allocating¶
Corresponds to the
PartitionState.ALLOCATING
state
- property failed¶
Corresponds to the
PartitionState.FAILURE
state
- property release¶
Corresponds to the
PartitionState.RELEASE
state
- class kazoo.recipe.partitioner.PartitionState[source]¶
High level partition state values
- ALLOCATING¶
The set needs to be partitioned, and may require an existing partition set to be released before acquiring a new partition of the set.
- ACQUIRED¶
The set has been partitioned and acquired.
- RELEASE¶
The set needs to be repartitioned, and the current partitions must be released before a new allocation can be made.
- FAILURE¶
The set partition has failed. This occurs when the maximum time to partition the set is exceeded or the Zookeeper session is lost. The partitioner is unusable after this state and must be recreated.