Saturday, June 24, 2017

AWS : S3 : Object Distribution

In backend of S3 are partitioned disks. You should avoid sequential keys or very similar keys to avoid overloading single partition as soon as you get to some significant load.

Generally each object has an id , generally sequential…. But it will be in sequence hence overwhelm a partition….
  1. Good & easy solution is to reverse the number. The first digit alone will help distribute
  2. Another solution prepend a hash ( 4-8 chars long )

What is Significant Load : according to AWS as soon as you exceed 100 put / list / delete requests per second … or more than 300 GET requests per second… as a developer you need to be careful with key strategy.

No comments: