AWS Kinesis – Shard Splitting – Calculate Fulcrum

When you split a shard in Kinesis, the AWS CLI tool wants to know where you want the hash key fulcrum of the split to be.  Then it closes down the old shard, and creates two new shards whose ranges correspond to the ‘halves’ bounded by the fulcrum.

I assume you can provide this (rather than just having the system automatically pick the middle value) if you’re manually managing hash keys and you want to purposefully create shards (whose ingress/egress rate is strictly limited) whose distributions are unbalanced.

If you just want to split a shard in two exact halves, you need to use

aws kinesis describe-stream --stream-name my_stream

… to get your shard ID and the (huge) start and end hash key ranges for that shard.  Then you can use this groovy script to work out the middle of those numbers:

groovy -e "println ((((args[0] as BigInteger) + \
  (args[1] as BigInteger)) /2 ) as BigInteger)" LOW_HASH HIGH_HASH

… where LOW_HASH and HIGH_HASH correspond to the low and high hash keys.  The output of this (which is the middle value) can be used to split the shard:

aws kinesis split-shard --stream-name my_stream \
   --shard-to-split shardId-000000000000 
   --new-starting-hash-key MIDDLE_VALUE

… where MIDDLE_VALUE was the value emitted by the script above.

Simple.