AWS and other interesting stuff

Simple Queue Service (SQS)

Simple Queue Service (SQS)

Amazon Simple Queue Service (SQS) is a fast, reliable, scalable, fully managed message queuing service. Amazon SQS makes it simple and cost-effective to decouple the components of a cloud application. You can use Amazon SQS to transmit any volume of data, without losing messages or requiring other services to be always available. Amazon SQS includes standard queues with high throughput and at-least-once processing, and FIFO queues that provide FIFO (first-in, first-out) delivery and exactly-once processing.

Concepts

  • Queue
    • Provides ability to store messages in a fail safe queue
      • Stores copies of the messages on multiple servers for redundancy and high availability.
    • Queue can be in any region
    • Any application can store in the queue
    • Any application can retrieve from the queue
    • Access Control
      • Access can be controlled for who can produce and consume messages to each queue
    • SQS is PCI Compliant
    • Queue is identified by a unique queue name within the same AWS account SQS assigns each queue with a Queue URL identifier for e.g. http://sqs.us-east-1.amazonaws.com/123456789012/queue2
      • Queue URL is needed to perform any operation on the Queue
    • Free tier provides 1 million request per month at no charge
    • Types
      • Standard queues are not guaranteed FIFO
      • FIFO queues are new as of Nov 2016. They are limited to 300 transactions per second.
        • They guarantee order and delivery only once.
        • Message groups within FIFO queues allow multiple separate ordered message streams within the same queue.
        • Standard queues are now $0.40 for every 1 million requests, and FIFO are $0.50.
    • SQS can delete a queue without notification if actions haven’t been performed on it for 30 consecutive days.
  • Message
    • Messages are stored in the queue
    • Batching
      • Messages can be sent or retrieved in batches of 10 while charging price for a single message
      • This lowers the cost and helps improve throughput
    • Ensures delivery of a message at least once
    • When a message is received, the receiving application gets a Receipt Handle. A message will stay in the queue until an application uses the Receipt Handle to delete it, that means the message may be delivered more than once. For that reason, applications should be designed to be idempotent.
    • Each message receives a system-assigned message ID that SQS returns to with the SendMessage response.
    • Supports multiple readers and writers
    • Messages can be sent and read simultaneously
    • The can be up to 256KB in size. If a message > 256KB it charged as multiple messages
      • CEIL(size / 256KB)
      • Messages greater than this size can use the SQS Extended Client Library for Java, which uses S3 for storage.
    • Messages can be retained for up to 14 days, or as little as 60 seconds. The default is 4 days.
    • Message Attributes
      • Up to 10 meta data attributes can be added to a message
      • Means meta data doesn’t have to be parsed from the message, speeding up processing
    • Message Sampling
      • Short polling - SQS samples only a subset of the servers (based on a weighted random distribution) and returns messages from just those servers. A receive request might not return all the messages in the queue. But a subsequent receive request would return the message
      • Long polling - request persists for the time specified and returns as soon as the message is available thereby reducing costs and time the message is idle in the queue
        • Long polling reduces extraneous polling and allows you to receive messages as quickly as possible.
        • Long polling can wait up to 20 seconds.
        • Each long poll request cost the same as any other request
  • Visibility timeout
    • Stops any other consumer getting the message
    • Defaults to 30 seconds
    • Clock starts ticking once SQS returns the message
    • Can be extended by the consumer
  • Dead-Letter Queue
    • Messages that have been received more than a set threshold can be sent to another queue called a Dead-Letter Queue.
  • Limits
    • Up to 120,000 in-flight messages per queue

Architectures Examples

Multi-Queue

  • Have different queues for different priorities of work
  • The workers process messages in the high-priority queue first, and when there aren’t any there they process the low priority queue messages

Fan Out

A single message sent to an SNS Topic is “fanned out” to multiple SQS queues for processing.

For example, a new image is added to a server which triggers an SNS Topic message. SNS creates a message in 3 SQS queues, with the each queue for a sub task:

  • Generate Thumbnail
  • Perform image recognition
  • Scan image meta data

Request Offloading

Move slow operations off of interactive request paths by enqueueing the request.

Auto Scaling

SQS queues can be used to determine the load on an application, and combined with Auto Scaling, the EC2 instances can be scaled in or out, depending on the volume of traffic

Add scalability and reliability to the architecture and smooth out temporary volume spikes without losing messages or increasing latency

Testing

Create A Queue

$ aws sqs create-queue --queue-name test-queue
{
    "QueueUrl": "https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue"
}

Send Messages

$ aws sqs send-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --message-body 1
{
    "MD5OfMessageBody": "c4ca4238a0b923820dcc509a6f75849b",
    "MessageId": "51b48ed3-d27e-4973-b088-e5d539a933f3"
}
$ aws sqs send-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --message-body 2
{
    "MD5OfMessageBody": "c81e728d9d4c2f636f067f89cc14862c",
    "MessageId": "702f67b5-5657-45fa-8cb4-1f08e97e54b2"
}

Receive Message

$ aws sqs receive-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue
{
    "Messages": [
        {
            "Body": "1",
            "ReceiptHandle": "AQEB4BS2TpFoWbYvq5NuMl3bFU17PU8rydr8Xapi7h7vtxKq0zOA9elIdR6BW38yKlbgm7L8QyPGK8HTl3F3qtCnUZFNTay8mLiE44sgOf34Wy4BJuFfXLim/YNEZV3YkGEQoLBoXPp320iIAt9FMZqatobF3fClftfxDu2W+O8SfGvWhXYhokx+wlsvzTDkjR6o4A5C4PoFfmqGrniuGJWowMQwOwpJnDa9S9bI1pC3J70gkjeMdqyaRH8ue+O5FCfPfqicsuwjT3uL5Lcol/e7R3lNQdt7uHtc0LjF2kD8V6FsUdtVsBFxshBFJVevzOqf1qFBP0IFxgURX744d5ju3lwm8Rcn5mfSuagm39B2OnQuqBOy/FZwscC3XYIUQ3kipvEukfJ/QF4zxSTUCFg+Qw==",
            "MD5OfBody": "c4ca4238a0b923820dcc509a6f75849b",
            "MessageId": "51b48ed3-d27e-4973-b088-e5d539a933f3"
        }
    ]
}

Delete Message With Receipt Handle

$ aws sqs delete-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --receipt-handle $RH

Change Message Visibility

$ aws sqs receive-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue
$ aws sqs change-message-visibility --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --receipt-handle $RH --visibility-timeout 10

Receive Message With Wait Time (Long Polling)

$ aws sqs receive-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --wait-time-seconds 20

Set Message Attributes

Set and receive messages with attributes

$ cat > tmp.json <<EOF
{
  "City": {
    "DataType": "String",
    "StringValue": "Any City"
  },
  "Greeting": {
    "DataType": "Binary",
    "BinaryValue": "Hello, World!"
  },
  "Population": {
    "DataType": "Number",
    "StringValue": "1250800"
  }
}
EOF
$ aws sqs send-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --message-body 4 --message-attributes file://tmp.json
{
    "MD5OfMessageBody": "a87ff679a2f3e71d9181a67b7542122c",
    "MD5OfMessageAttributes": "28eb0e573cf8e8a77e349a2f968eac4a",
    "MessageId": "40
}
$ aws sqs receive-message --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue --message-attribute-names City Greeting Population
{
    "Messages": [
        {
            "Body": "4",
            "ReceiptHandle": "AQEBmjIsFRUdZ/6USxmgQUUf8d8pjlpry42kO8SKCV9+BcNBVtBvlQ2FUQQ8Yucsbr0t5JdmwHZLAd/czy4Bo+DUzEH+tjfFtAlw2eAeAhXTgWZBMn+L2gF4vcX1YtVp2T0h6agqVw3UWgIXuqT/3Ths8RLi7d/Gl8xpGPVjgE6Z+JBoRUPuuQ5RH38pkeDaA9hds075a4Iyz4vWlKYTYNpfnSFsGcojIL+QG7Om+fQe5AXRmqfL1MELHLobMCkvxTKnOKgRScTIUV2i+1vxpiA7cgAqcUnB6JBTTlQcP1/jGc1QYg9J8xppdMjTOXli6u447Z95tWyduJkSji7rjb/nYF9FXuYmtc8rswnzlTuvnHCapPg0DThRQN//m6h445517Ur5nbUQxKCotrjXH6u1xg==",
            "MD5OfBody": "a87ff679a2f3e71d9181a67b7542122c",
            "MD5OfMessageAttributes": "28eb0e573cf8e8a77e349a2f968eac4a",
            "MessageId": "400b1ba1-f17d-4a70-9672-1d2895f3820a",
            "MessageAttributes": {
                "City": {
                    "DataType": "String",
                    "StringValue": "Any City"
                },
                "Greeting": {
                    "DataType": "Binary",
                    "BinaryValue": "SGVsbG8sIFdvcmxkIQ=="
                },
                "Population": {
                    "DataType": "Number",
                    "StringValue": "1250800"
                }
            }
        }
    ]
}

Delete The Queue

$ aws sqs delete-queue --queue-url https://ap-southeast-2.queue.amazonaws.com/012345678910/test-queue