Azure + MongoDB on EC2 + Connection reset

with No Comments

Problem

We have configured java web app with tomcat on Microsoft Azure which Store/Read data from MongoDB which is located on Amazon EC2. We have a lot of traffic between EC2 and Azure – hundreds of threads doing actions each our. Our MongoDB config was (we use java driver):

autoConnectRetry = true
connectTimeout = 10000
connectionsPerHost = 100
threadsAllowedToBlockForConnectionMultiplier = 50
socketKeepAlive = true
socketTimeout = 60000

From time to time on our server we got

Caused by: java.net.SocketException: Connection reset
 at java.net.SocketInputStream.read(SocketInputStream.java:168)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
 at org.bson.io.Bits.readFully(Bits.java:46)
 at org.bson.io.Bits.readFully(Bits.java:33)
 at org.bson.io.Bits.readFully(Bits.java:28)
 at com.mongodb.Response.<init>(Response.java:40)
 at com.mongodb.DBPort.go(DBPort.java:124)
 at com.mongodb.DBPort.call(DBPort.java:74)
 at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)

Reason

The reason of this is that Azure has global firewall which kill all idle connections in pool that are older then about 3 min – did not find any docs about this, just empirically got it.

Solution

Fix can be done in two ways:

  1. Via java driver we can set max idle time for connections and destroy them ourselves. This is already requested for MongoDB Java driver https://jira.mongodb.org/browse/JAVA-710 but still not done
  2. On Linux server we can configure ‘tcp_keepalive_time’, so that connections will be destroyed by system. We use 60 seconds live period for them.
sudo bash -c 'echo 60 > /proc/sys/net/ipv4/tcp_keepalive_time'

Restart HTTPD after this and problem gone. Of course, you need to understand that in any case single connection should not live more then 60 sec, and if you need this then better to move from Azure :)

Leave a Reply