Amazon SQS

**
**
Amazon SQS — how to describe it:

“SQS is a fully-managed message queue — producers push messages onto a queue, and one or more consumer workers pull messages off it to process asynchronously. It guarantees at-least-once delivery, and messages stay invisible to other consumers for a configurable ‘visibility timeout’ once a worker picks one up, so if that worker crashes before finishing, the message becomes visible again for another worker to retry.”

Where you already used this, concretely:

• Dropbox — S3 event notifications delivered via SQS, telling your backend an upload finished
• This News Feed problem, in your own answer — async workers reading from an SQS queue to compute/write precomputed feeds when a post is created, instead of blasting writes synchronously

Key mechanics worth being precise about:

• At-least-once delivery, not exactly-once — a message can be delivered more than once (e.g., a worker processes it but crashes before deleting it from the queue) — so your consumer logic needs to be idempotent (safe to process the same message twice without bad side effects)
• Visibility timeout — when a worker picks up a message, it becomes invisible to other workers for N seconds; if the worker doesn’t delete it (i.e., finish processing) within that window, it reappears for someone else to retry
• Standard vs. FIFO queues — Standard queues are high-throughput but don’t guarantee strict ordering; FIFO queues guarantee order and exactly-once processing within a message group, but at lower throughput