In this post i'll try to guide you through 5 easy steps to connect your java application to Amazon SQS ( Simple Queue Service). For this purpose i'll use Spring framework and maven for dependencies management.
What is AWS SQS ?
The simplest answer for this question is directly quoted from AWS :
"Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing."
Why to use queues at the first place ?
In web applications API calls might not be the answer for every case. Lets look at the following case :
Imagine we have 2 web services ( Cart, EventsRepo) and we want to keep track for all events resulting from customer cart ( ex : add item to cart, delete item, replace item .. etc ) . So for each request coming to the ( Cart service ) we want to create the responding event and send it to ( EventsRepo ) service for further processing.
Now imagine we have 30 other services that want to send events to our EventsRepo. Then we should be able to handle a s*** load of requests coming to the EventsRepo service and that means of course spawning more instances and paying more money.
In this case a messaging communication would come really handy, where all service put there events in a queue and our EventsRepo will consume the events one by one . With this approach even one instance of EventsRepo would be enough ( more instances means faster processing for events in the queue ).
What makes SQS so special ?
When it comes to queueing systems there is a lot of options out there ( rabbitMq, Redis, Kafka .. etc) and each with its own pros and cons, so before jumping to SQS i'd suggest more in depth reading about other tools to find out the best choice for your use case.
Thats said, i've found SQS so simple and easy to use . There is no prior setup or maintenance overhead needed, just AWS credentials and you are ready to go. So if you are looking for something simple with basic queuing functionalities then SQS is your friend.
How does it work ?
In SQS every thing is a request and you'll be billed on request base, we have 3 basic types of requests ( send message - receive message - delete message ).
For one message it'll take 3 requests in normal case to handle it :
Sender => 1 request to send the message .
Receiver => 1 request to receive the message and 1 request to delete it from the queue.
So keep in mind normally each message needs 3 requests and ( 1 mil requests cost 0.5$ ) .
Note1 : SQS follows (producer/consumer) model not (publish/subscribe) model.
i.e : only one consumer will receive the message at the same time.
Note2 : There is also batching possibilities for sending and
receiving but it is out of this article scope.
- Step (1) - Create the queue :
As you can see creating the queue is straight forward, lets go quickly through the important properties :
- Default visibility timeout : how long should SQS wait before making the message available again for other consumers. If we have 2 consumers reading from the queue ( consumer1, consumer2 ), and if consumer1 read a message but did't send the acknowledgment back ( i.e : delete request ), then the message won't available for consumer2 until default visibility timeout is over.
- Message Retention : how long the message should stays in the queue if its not consumed.
- Receive Message Wait Time : this is important to reduce SQS costs and it means how much time
the receive call should wait before returning empty result to consumer.
i.e : lets say the queue was empty for 2 minutes then during this time all receive calls made by consumers are useless, so long polling receive call will guarantee a reasonable waiting time before returning empty result and forcing the consumer to make a new call .
- Step (2) - Add maven dependencies :
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-messaging</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-aws-autoconfigure</artifactId>
</dependency>
for more information about these dependencies visit Spring cloud
- Step (3) - Create the beans:
As you can see here i'm using the ( DefaultAWSCredentialsProviderChain ) class which will scan my environment for valid AWS credentials, but feel free to use another provider. For more info about this topic AWS Credentials .
- Step (4) - Send the message :
Sending message is that simple, we just pass the queue name and the object we want to send to SQS ( it'd be converted to JSON by default).
- Step (5) - Consume the message:
Also consuming is a piece of cake ! We just need to add (@SqsListener) annotation before any function and it'd become our consumer. You can notice here we set our deletion policy to ( ON_SUCCESS ) and that means we will only send the delete request to SQS when the consuming is completed successfully.
Thats it ! now you can start using AWS SQS, of course there is more details and we just scratched the surface in this article, but this is the basic and simplest steps to start production using SQS.
Note 1: To write integration tests for SQS we can use elasticMq it might be little tricky to configure it especially with spring auto configuration around, but i've used it and its working nicely .
Note 2: After sending and receiving couple of millions of messages the problem described in the following issue might occur, till the issue is fixed you can use the (@Aspect) solution described in the issue comments.
