Question: sqs how to handle duplicate messages?
Answer: There is no API level way of preventing duplicate messages to be posted to a SQS queue. You would need to handle this at application level I am afraid. You can use a DynamoDB table to store your Domain Names waiting to be crawled and only add them to the queue if they are not in DynamoDB for example.