kazoo.recipe.partitioner

Zookeeper Partitioner Implementation

Maintainer:None
Status:Unknown

SetPartitioner implements a partitioning scheme using Zookeeper for dividing up resources amongst members of a party.

This is useful when there is a set of resources that should only be accessed by a single process at a time that multiple processes across a cluster might want to divide up.

Example Use-Case

  • Multiple workers across a cluster need to divide up a list of queues so that no two workers own the same queue.

Public API

class kazoo.recipe.partitioner.SetPartitioner(client, path, set, partition_func=None, identifier=None, time_boundary=30)[source]

Partitions a set amongst members of a party

This class will partition a set amongst members of a party such that each member will be given zero or more items of the set and each set item will be given to a single member. When new members enter or leave the party, the set will be re-partitioned amongst the members.

When the SetPartitioner enters the FAILURE state, it is unrecoverable and a new SetPartitioner should be created.

Example:

from kazoo.client import KazooClient
client = KazooClient()

qp = client.SetPartitioner(
    path='/work_queues', set=('queue-1', 'queue-2', 'queue-3'))

while 1:
    if qp.failed:
        raise Exception("Lost or unable to acquire partition")
    elif qp.release:
        qp.release_set()
    elif qp.acquired:
        for partition in qp:
            # Do something with each partition
    elif qp.allocating:
        qp.wait_for_acquire()

State Transitions

When created, the SetPartitioner enters the PartitionState.ALLOCATING state.

ALLOCATING -> ACQUIRED

Set was partitioned successfully, the partition list assigned is accessible via list/iter methods or calling list() on the SetPartitioner instance.

ALLOCATING -> FAILURE

Allocating the set failed either due to a Zookeeper session expiration, or failure to acquire the items of the set within the timeout period.

ACQUIRED -> RELEASE

The members of the party have changed, and the set needs to be repartitioned. SetPartitioner.release() should be called as soon as possible.

ACQUIRED -> FAILURE

The current partition was lost due to a Zookeeper session expiration.

RELEASE -> ALLOCATING

The current partition was released and is being re-allocated.
__init__(client, path, set, partition_func=None, identifier=None, time_boundary=30)[source]

Create a SetPartitioner instance

Parameters:
  • client – A KazooClient instance.
  • path – The partition path to use.
  • set – The set of items to partition.
  • partition_func – A function to use to decide how to partition the set.
  • identifier – An identifier to use for this member of the party when participating. Defaults to the hostname + process id.
  • time_boundary – How long the party members must be stable before allocation can complete.
acquired[source]

Corresponds to the PartitionState.ACQUIRED state

allocating[source]

Corresponds to the PartitionState.ALLOCATING state

failed[source]

Corresponds to the PartitionState.FAILURE state

finish()[source]

Call to release the set and leave the party

release[source]

Corresponds to the PartitionState.RELEASE state

release_set()[source]

Call to release the set

This method begins the step of allocating once the set has been released.

wait_for_acquire(timeout=30)[source]

Wait for the set to be partitioned and acquired

Parameters:timeout (int) – How long to wait before returning.
class kazoo.recipe.partitioner.PartitionState[source]

High level partition state values

ALLOCATING

The set needs to be partitioned, and may require an existing partition set to be released before acquiring a new partition of the set.

ACQUIRED

The set has been partitioned and acquired.

RELEASE

The set needs to be repartitioned, and the current partitions must be released before a new allocation can be made.

FAILURE

The set partition has failed. This occurs when the maximum time to partition the set is exceeded or the Zookeeper session is lost. The partitioner is unusable after this state and must be recreated.