AWS and other interesting stuff



A fully managed, NoSQL Database services

  • Predictable fully manageable performance, with seamless scalability
  • No visible servers
  • No practical storage limitations
  • Fully resilient and highly available
    • DynamoDB synchronously replicates data across three facilities in an AWS Region
  • Performance scales in a linear way
  • Fully integrated with IAM e.g. grant permissions to users authenticated with Web Identity Federation



  • The highest level structure within a Database
  • You specify performance requirements at a table level
    • RCU - Read capacity units - 4KB strongly consistent reads per second
    • WCU - Write capacity units - 1KB blocks per second
  • Charges are based on storage capacity used, RCU and WCU


  • The equivalent of a row in a relational database


  • Items in the same table can have different attributes (different number or set or attributes)
    • Partition Key (Hash Key)
    • Sort Key (Range Key)
    • Types
    • Scalar - single value
      • String (S), Number (N), Binary (B), Boolean (BOOL), Null (NULL)
    • Set
      • A set of Strings (SS), Numbers (NS) or Binaries (BS)
    • DynamoDB data type - JSON
      • List (L), Map(M)


  • Returns all items in the table (paginated after 1MB by returning LastEvaluatedKey)
  • Eventually Consistent Reads are the default
  • A Read Capacity Unit (RCU) is 4KB strongly consistent reads, or 2 x 4KB eventually consistent
  • It can return up to 1MB of data
(1 MB page size / 4 KB item size) / 2 (eventually consistent reads) = 128 read operations consumed
(1 MB page size / 4 KB item size) = 256 read operations consumed


  • Query by partition key and optionally by sort key (Note, partition key must always be provided)


  • BatchGetItem - limit 100 items, 16MB
  • BatchWriteItem - limit 25 PutItem or DeleteItem requests, maximum 16MB


  • DynamoDB is eventually consistent by default.
  • A strongly consistent read returns a result that reflects all writes that received a successful response prior to the read.


  • The underlying storage and processing nodes of DynamoDB
  • Initially, one table equates to one partition
  • You don’t directly control the number of partitions
  • A partition can store 10GB of data - therefore each partition key is also limited to 10GB of data
  • A partition can handle 3000 RCU and 1000 WCU - therefore each partition key also has these limits
  • The allocated WCU and RCU is split between partitions. This split is not necessarily even
  • When >10GB or >3000 RCU or >1000 WCU a new partition is added and the data is spread between them over time
  • There is no automatic partition decrease
    • i.e. be careful when
      • you set large RCU or WCU units (you should do read leveling e.g. Redis)
      • you do bulk imports (you should do write leveling e.g. SQS buffer)
    • Otherwise when you decrease RCU or WCU at off-peak times to save money there will be a significant performance impact e.g. RCU divided over 1 partition versus RCU over 2 partitions etc.
  • The partition is selected by hashing the Partition Key


Without indexes, queries would be very limited

  • Queries

    • Scan - scan through the table and returns all items (paginated)
    • Query - search for a specific item (partition key) or set of items (partition key and sort key)
  • GSI (Global Secondary Index)

    • Can be created at any time
    • You can use an alternate partition and sort key
    • As with LSI you can project Keys Only, Include (a selection of attributes), or All (all attributes)
    • They have their own set of RCU and WCU
    • Changes are copied asynchronously
      • i.e. a GSI can only have eventually consistent reads
    • Unlike a table, items can have the same partition key:
  • LSI (Local Secondary Index)

    • Same partition key as the table and a different sort key
    • The table’s sort key is always projected into the index course example LSI course example LSI

    • By default, non key values are not stored in an LSI
    • Any other attributes can be added as projected values
    • Shares RCU and WCU with table
    • Is a sparse index (the item will only be in the index if it contains the sort key as an attribute)
    • Must be created at the time of table creation
    • Data is copied asynchronously from the main table
    • Storage and performance considerations
      • If you query an item for its projected attributes, you are just charged for the size of the item in the LSI. i.e. LSI can allow you to save on retrieval costs.
      • If you query an item for its non-projected attributes: you are charged for the entire item cost (i.e. all attributes) from pulling it from the main table there is additional query latency as the item is retrieved from the main table. course example LSI protected attributes course example LSI protected attributes

      • i.e. 4 elements are returned for each query, but the second one is much less efficient:
        • first query is 4 x 200 Bytes, rounded to 1 x 4KB read
        • second query has a sub query for each row, and each sub query is rounded to 4KB. So the lack of projection results in a 20KB instead of 4KB and performance/latency issue.
      • $ aws dynamodb query --table-name test_table --index-name id-size-index --key-condition-expression "id = :v1" --expression-attribute-values '{":v1": {"S": "one"}}' --return-consumed-capacity INDEXES --select ALL_PROJECTED_ATTRIBUTES
        "ConsumedCapacity": {
            "CapacityUnits": 0.5,
            "TableName": "test_table",
            "LocalSecondaryIndexes": {
                "id-size-index": {
                    "CapacityUnits": 0.5
            "Table": {
                "CapacityUnits": 0.0
      • $ aws dynamodb query --table-name test_table --index-name id-size-index --key-condition-expression "id = :v1" --expression-attribute-values '{":v1": {"S": "one"}}' --return-consumed-capacity INDEXES --select ALL_ATTRIBUTES
        "ConsumedCapacity": {
            "CapacityUnits": 2.5,
            "TableName": "test_table",
            "LocalSecondaryIndexes": {
                "id-size-index": {
                    "CapacityUnits": 0.5
            "Table": {
                "CapacityUnits": 2.0
      • Add and deletes cause writes to the LSI
      • An update could cause 2 writes to the LSI if you change the LSI’s sort key in the item: i.e. delete the old one and insert the new. Considerations
      • Be aware of ItemCollections (all data in a table and indexes that share the same partition key). They only apply to items with an LSI. The maximum size is 10GB (ItemCollectionSizeLimitExceededException)

Streams and Replication

  • Ordered record of updates to a table
  • Records changes to a table and stores those values for 24 hours
    • There are four different views
      • KEYS_ONLY
      • NEW_IMAGE
      • OLD_IMAGE
  • AWS guarantee:
    • each change occurs in the stream once and only once
    • All changes to the table occur in the steam in near realtime
  • Realworld use cases:
    • Replication
    • DR - replicate to another region
    • Multi-master model
    • Triggers
    • Lambda function triggered when items are added to aggregate data
    • Lambda function when a new user signup happens
  • The old way this was setup was using CloudFormation, Kinesis and Elastic Beanstalk. It also setup additional tables (Metadata table and KCL Checkpoint table [in each region]). The new way is to just enable the stream on the table.

Burst capacity

When you are not fully utilizing a partition’s throughput, DynamoDB retains a portion of your unused capacity for later bursts of throughput usage. DynamoDB currently retains up five minutes (300 seconds) of unused read and write capacity. During an occasional burst of read or write activity, these extra capacity units can be consumed very quickly—even faster than the per-second provisioned throughput capacity that you’ve defined for your table. However, do not design your application so that it depends on burst capacity being available at all times: DynamoDB can and does use burst capacity for background maintenance and other tasks without prior notice.

Cross-region Replication

Cross-region replication relies on reading data from DynamoDB Streams to keep the tables in sync.

It can be useful for:

  • Disaster Recovery
  • Faster Reads - data closer to the customer
  • Distribute load away from the master table

Performance i.e. Partitions, Partitions, Partitions

Performance is based on allocated performance, key structure, time and key distribution of reads and writes. To get the most out of DynamoDB throughput, create tables where the partition key has a large number of distinct values, and values are requested fairly uniformly, as randomly as possible.

A single partition can support a maximum of 3,000 read capacity units or 1,000 write capacity units.

Formulas for the approximate number of partitions:

CEIL( Desired RCU / 3000 RCU + Desired WCU / 1000 WCU )

e.g. 7500 RCU + 3000 WCU i.e 2.5 + 3 = 6 partitions

CEIL(Data Size in GB / 10 GB)

e.g. 65GB / 10GB = 6.5 = 7 partitions

Actual partitions:

MAX(capacity formula, storage formula)
  • You only get the full WCU / RCU for a table if your task is running on parallel on each partition the table is across. i.e. Hot Partition Keys are an issue.
  • You need to have a partition key that is evenly spread across partitions. Also including from a time perspective.
  • Requirements of <3000RCU or <1000WCU for a table will result in one partition anyway, so key selection isn’t as important.

Key criteria for an effective partition key

  • The attribute should have as many distinct values as possible
  • The attribute should have a uniform write pattern across all partition key values e.g. a student id when you expect 2 students to get 90% of the votes is not a good idea.
  • The attribute should have a uniform temporal write pattern across time (day, night, week, month, year etc)
  • If any of the above aren’t possible with an existing value, you should consider a synthetic/created/hybrid value.
  • You should not mix HOT and COLD key values within a table.

Example 1: Weather station that tracks failure due to intrusion by wildlife weather station example weather station example

  • The table has good key distribution, so it isn’t constrained (12,000 WCU will result in approximately 12 partitions)
  • The GSI IS constrained though: The intrusion field is either True or False which means a maximum of 2 partitions. A best case scenario would be 6000 WCU for each table, but we know that isn’t possible. In reality, Intrusion = False will be hotter than than True, so it’s even worse.
  • The throttled GSI also throttles the table writes!

Example 2: Write leveling write leveling write leveling

  • Use write leveling to handle bursts of writes e.g. use SQS as an intermediary

Example 3: Read leveling read leveling read leveling

  • Use Elasticache to cache reads
  • You could extend this example by invalidating objects in the cache using a DynamoDB event stream and trigger.

Example 4: Election votes

Setup 1: Votes table with candidate_id as the partition key and date+time as a range key

10 partitions based on the 10,000 writes (1,000 writes each)


  • You’ll have candidates spread over the partitions, where some will be “hot” and some will be not. hots won’t have enough capacity, nots will have too much (wasted).
  • Even if writes were uniform across the keys from a value perspective, it may not be true over time e.g. different candidates will be popular in states with different timezones.
  • These problems would be exacerbated when you move to the main election with even less candidates. e.g. 100,000 WCU with two candidates you get 100 partition, with two hot ones limited to 1,000 WCU. votes example votes example

Setup 2: Votes table with candidate_id+RAND([0,10]) as the partition key votes example votes example

Best Practices



Choose a partition key

Example: consider a table with a composite primary key. The partition key represents the item’s creation date, rounded to the nearest day. The sort key is an item identifier. On a given day, say 2014-07-09, all of the new items will be written to that same partition key value.

Strategy 1:

Randomizing Across Multiple Partition Key Values e.g. [DATE]+RAND([1, 200])

+ve: Randomising means writes to the table on each day are spread evenly across all of the partition key values; this will yield better parallelism and higher overall throughput.

-ve: To get items for a given day you need to iterate by querying through all the random suffixes then merging the results

Strategy 2:

Using A Calculated Suffix.

The random suffix option above greatly improves write throughput, but makes reading a specific item difficult as you don’t know which suffix was used. Instead of using a random number, use a number you are able to calculate based upon something that’s intrinsic to the item.

e.g. The partition key could be [DATE]+(OrderId % 200 + 1) (Continuing the example above, suppose each item has an OrderId)

+ve: Writes are spread evenly across the partition key values. You can easily perform a GetItem for a particular item when you want to retrieve a specific OrderId value.

-ve: To get all items for a given day you would still need to query the 200 partition keys an merge the result, but you avoid having a single hot partition key taking all the workload.


( RCU / 3,000 ) + ( WCU / 1,000 ) = initialPartitions (rounded up)

Subsequent Allocation of Partitions

A partition split can occur in response to:

  • Increased provisioned throughput settings
  • Increased storage requirements
  • the data, RCU and WCU are all split in two and evenly distributed across the 2 new partitions
  • other partitions are not affected

Increased Provisioned Throughput Settings

You can change the provisioned throughput of a particular DynamoDB table up to four times per day.

If you increase a table’s provisioned throughput and the table’s current partitioning scheme cannot accommodate your new requirements, DynamoDB will double the current number of partitions.

For example, initially 5,000 RCU and 2,000 WCU …

( 5,000 / 3,000 ) + ( 2,000 / 1,000 ) = 3.6667 --> 4

… then change to 8,000 RCU:

( 8,000 / 3,000 ) + ( 2,000 / 1,000 ) = 4.6667 --> 5

Increased Storage Requirements

If an existing partition fills up with data, DynamoDB will split that partition. The result will be two partitions, with the data from the old partition divided equally between the new partitions.

Throughput Capacity Per Partition Example
( 5,000 / 3,000 ) + ( 2,000 / 1,000 ) = 3.6667 --> 4


5,000 read capacity units / 4 partitions = 1,250 read capacity units per partition

2,000 write capacity units / 4 partitions = 500 write capacity units per partition

If one of these four partitions were to fill to capacity:

  • Three of the five partitions would each have 1,250 read capacity units and 500 write capacity units.
  • The other two partitions would each have 625 read capacity units and 250 write capacity units.

Distribute Write Activity During Data Upload

Source data is often sequential, so naturally puts load on partitions in sequence as you process one partition key at a time. Instead, you should upload one item for each partition key at the same time until you’ve uploaded all the data.

Understand Access Patterns for Time Series Data

In an example where latest data most relevant and most frequently accessed, you can store the data in multiple tables. e.g. tables to store monthly or weekly data.

  • The most recent tables have higher RCU and WCU
  • As the tables age you decrease their RCU and WCU
  • Old data can be moved to S3 and the table deleted

Use in-memory caching


Using a Forum, Thread, Reply example:

  • Use one-to-many relationships between tables using the primary keys has advantages over storing items in attributes:
    • Consumes less throughput (e.g. request for threads doesn’t return all replies too)
    • Store unlimited related items (i.e. you’re not limited by the size of a single item)
    • You can request a sub-set of items rather than all of them.
    • Adding a new related item is a more efficient write

There is a 400KB size limit for items

  • You can compress attribute values using GZIP et. al. to reduce the cost of storing and retrieving data.
  • You can store large attribute values in S3
  • You can break up large attributes across multiple items e.g. Reply and ReplyChunks

Notice how the Id is a concatenation of reply, date-time, message number and chunk.

  • The application needs to deal with failure scenarios with writing multiple items and with inconsistencies between items when reading multiple items.
  • A less optimal solution would be to store each chunk in a table with a composite key, with the partition key being the primary key of the parent item. This would allow you to receive all chunks from the same parent, but could cause uneven I/O distribution across partitions i.e. the primary key could be a hot key.

Query and Scan

Avoid Sudden Bursts of Read activity

  • e.g. a Scan returning 1M page of data, at 4KB item size is 256 strongly consistent reads. You will get a ProvisionedThroughputExceeded error when you exceed your RCU limit.
  • The scan example above is likely to consume all its reads from the same partition as the scan reads items that are next to each other on the partition.

Instead of a large scan you can use the following techniques:

  • Reduce Page Size
    • Use Limit Parameter e.g. if each item is 4K and you set a Limit of 40 you’d consume 40 strongly consistent RCU
    • A larger number of smaller Scan or Query operations would allow your other critical requests to succeed without throttling.
  • Isolate Scan Operations
    • You want to perform scans on a table that is not taking “mission-critical” traffic
    • e.g. rotate traffic hourly between 2 tables
    • e.g. perform every write on 2 tables, one a “mission-critical” table, the other a shadow table

You should configure your application to retry (with exponential backoff) any request that receives a response code that indicates you have exceeded your provisioned throughput (ProvisionedThroughputExceededException), or increase the provisioned throughput for your table using the UpdateTable operation

FYI: CreateTable; UpdateTable; DeleteTable may return a ThrottlingException exception.

Take Advantage of Parallel Scans

For example: 4 worker threads in a background “sweeper” process could scan a table at a low priority without affecting production traffic.

Scan(TotalSegments=4, Segment=0, …)
Scan(TotalSegments=4, Segment=1, …)
Scan(TotalSegments=4, Segment=2, …)
Scan(TotalSegments=4, Segment=3, …)

The TotalSegments and Segment parameters together limit the scan to a particular block of items in the table.

The best setting for TotalSegments depends on the context: data, provisioned throughput, performance requirements. It is recommended you begin with a simple ratio e.g. one segment per 2GB.

You need to monitor performance to make sure your parallel scan does not starve other processes of capacity.

Update Expressions

Atomic Counters and Conditional Writes

Atomic Counter:

UpdateItem operation to increment or decrement the value of an existing attribute without interfering with other write requests

SET Price = Price - :p

Writes are not idempotent: the same action applied twice will run twice.

Conditional Writes:

DynamoDB supports conditional writes for PutItem, DeleteItem, and UpdateItem operations. With a conditional write, an operation succeeds only if the item attributes meet one or more expected conditions; otherwise it returns an error.

  • Condition expression: Total = :oldtotal
  • Update expression: SET Total = Total + :amt

Writes are not idempotent: the same action applied twice will only run once.

Conditional Writes can be used to implement Optimistic Locking With Version Number.

With optimistic locking, each item has an attribute that acts as a version number. If you retrieve an item from a table, the application records the version number of that item. You can update the item, but only if the version number on the server side has not changed. If there is a version mismatch, it means that someone else has modified the item before you did; the update attempt fails, because you have a stale version of the item.

Python Table Aggregation Example

The following code does an “upsert” for a category value in a data aggregation table. It reads the keys for the updated category and increments a counter on insert and decrements a counter on remove. It does an “upsert” in that it doesn’t matter if the category has an entry in the aggregation table already or not

  • insert if
    • it doesn’t exist yet ‘attribute_not_exists(num)”
    • and count is going up ‘:val > :zero’
  • increment if num is greater than or equal to zero (i.e. we don’t want negative num) ‘num >= :zero’
def q_and_a_insert_trigger(event, dynamodb=None):
        if dynamodb is None:
                dynamodb=boto3.resource('dynamodb', region_name=AWS_REGION)

        table = dynamodb.Table(TABLE_QUESTIONS_CATEGORIES)

        for r in event[u'Records']:
                if r[u'eventName'] == u'MODIFY':
                        if r[u'eventName'] == u'INSERT':
                                counter_change = decimal.Decimal(1)
                        elif r[u'eventName'] == u'REMOVE':
                                counter_change = decimal.Decimal(-1)

                                category = r[u'dynamodb'][u'Keys'][u'category'][u'S']
                        except KeyError as e:
                                print "KeyError trying to get r[u'dynamodb'][u'Keys'][u'category'][u'S']"
                                print r[u'dynamodb']
                                response = table.update_item(
                                        Key={'category': category},
                                    UpdateExpression='ADD num :val',
                                                'attribute_not_exists(num) AND :val > :zero'
                                                ' '
                                                'OR num >= :zero'),
                                        ':val': counter_change,
                                        ':zero': decimal.Decimal(0)
                        except ClientError as e:
                                print 'q_and_a_insert_trigger failed'
                                # print 'Event: {}'.format(event)
                                print 'e1: {}'.format(e)

        return True

Here is another upsert example:

const docParams = { TableName: databaseConfig.table, UpdateExpression: 'SET #T = list_append(if_not_exists(#T, :empty), :t)', ExpressionAttributeNames: { '#T': 'trials' }, ExpressionAttributeValues: { ':t': [trial], ':empty': [] }, Key: { 'user_id': id } }

i.e. If the trials attribute does not exist, append to an empty list otherwise append to it.

Return Values

Deleting an Item:

In a DeleteItem operation, you can set ReturnValues to ALL_OLD. Doing this will cause DynamoDB to return the entire item, as it appeared before the delete operation occurred.

Updating an Item:

In an UpdateItem operation, you can set ReturnValues to one of the following:

  • ALL_OLD – The entire item is returned, as it appeared before the update occurred.
  • ALL_NEW – The entire item is returned, as it appears after the update.
  • UPDATED_OLD – Only the value(s) that you updated are returned, as they appear before the update occurred.
  • UPDATED_NEW – Only the value(s) that you updated are returned, as they appear after the update.


Use indexes sparingly

  • They consume storage and I/O
  • They impact data capture applications with heavy write activity
    • A shadow table is often the best option in this case
  • Allow you to retrieve attributes that are not projected into the index

LSIs contribute to the 10GB partition usage limit; they’re included in the ItemCollectionSizeLimitExceededException. Limit their use if possible.

Choose Projections Carefully

  • Project few attributes to keep the size under 1KB, so you only consume one WCU
    • Conversely, you can project more attributes for no extra cost if their sum is under 1KB
  • You’re better off keeping rarely used attributes not projected
    • You won’t incur the write to the index cost if you update the attribute value
    • You can still retrieve non-projected attributes in a Query, but at a higher provisioned throughput cost
  • Projecting ALL
    • Eliminates the need for table fetches (sub queries)
    • Increases costs for storage and write activity

Take Advantage of Sparse Indexes

For any item in a table, DynamoDB will only write a corresponding index entry if the index sort key value is present in the item. If the sort key does not appear in every table item, the index is said to be sparse.

e.g. have an IsOpen attribute for an Order item with an LSI using IsOpen for the sort key. You delete the IsOpen attribute for the Order item once it is complete, thereby removing the item from the LSI.

e.g. instead of IsOpen you could do the same with OrderOpenDate, thereby having a useful sort key rather than an arbitrary one like IsOpen.


  • Unlike LSI, GSI do not allow you to retrieve attributes that are not projected into the index

Choose a Key That Will Provide Uniform Workloads

  • Choose partition keys and sort keys that have a high number of values relative to the number of items in the index (same applies to tables) i.e. Understand the cardinality of your key attributes: the distinct number of values in a particular attribute, relative to the number of items you have.

For example, suppose you have an Employee table with attributes such as Name, Title, Address, PhoneNumber, Salary, and PayLevel. Now suppose that you had a global secondary index named PayLevelIndex, with PayLevel as the partition key. Many companies only have a very small number of pay codes, often fewer than ten, even for companies with hundreds of thousands of employees. Such an index would not provide much benefit, if any, for an application.

Another problem with PayLevelIndex is the uneven distribution of distinct values. For example, there may be only a few top executives in the company, but a very large number of hourly workers. Queries on PayLevelIndex will not be very efficient because the read activity will not be evenly distributed across partitions. -

Take Advantage of Sparse Indexes

For example, in the GameScores table, certain players might have earned a particular achievement for a game - such as “Champ” - but most players have not. Rather than scanning the entire GameScores table for Champs, you could create a global secondary index with a partition key of Champ and a sort key of UserId. This would make it easy to find all the Champs by querying the index instead of scanning the table.

Such a query can be very efficient, because the number of items in the index will be significantly fewer than the number of items in the table. In addition, the fewer table attributes you project into the index, the fewer read capacity units you will consume from the index.

Use a Global Secondary Index For Quick Lookups, Read Replica and Permissions Boundary

Quick Lookups:

If most of your queries do not require much data to be returned, you can create a global secondary index with a bare minimum of projected attributes - including no projected attributes at all, other than the table’s key. This lets you query a much smaller global secondary index, and if you really require the additional attributes, you can then query the table using the same key values.

Eventually Consistent Read Replica:

Like the above example, but project all attributes to a GSI and use that to offload work from the table.

You could have 2 GSI, one with high provisioned throughput for high priority apps, and one with low provisioned throughput for low priority apps.

Permissions Boundary:

You can restrict the applications that can read from the table, and create a GSI for other applications, projecting only the attributes they need.


  • ItemCollectionSizeLimitExceededException - For a table with a local secondary index, a group of items with the same partition key value has exceeded the maximum size limit of 10 GB
  • ThrottlingException - This exception might be returned if you perform any of the following operations too rapidly: CreateTable; UpdateTable; DeleteTable.
  • ProvisionedThroughputExceededException - You exceeded your maximum allowed provisioned throughput for a table or for one or more global secondary indexes.
    • If you get this exception your code should perform an Exponential Backoff e.g. try again in 50ms, then 100ms, then 200ms

Batch processes:

  • BatchGetItem invokes GetItem once for each item in the batch. If any sub request fail it returns a value in UnprocessedKeys
  • BatchWriteItem invokes WriteItem once for each item in the batch. If any sub request fail it returns a value in UnprocessedItems
  • If DynamoDB returns any unprocessed items, you should retry the batch operation on those items. However, AWS strongly recommend that you use an exponential backoff algorithm.