Cannot connect to DB from a process inside Docker container ? This could fix it.

I had recently hit a blocker to production deployment, when my processes just won’t connect to my Cassandra DB from within their containers.

This is how I could fix it –

• Start by checking if the target port is open. For eg, if it is the Cassandra DB that the processes cannot access, make sure that its host port 9042 is not blocked by any firewall rules (or security group settings in case of AWS)
• From the host machine where your docker’ed process has a connectivity issue, use telnet or netcat to check if you can access the target machine and port.

 nc -vz dbhost 9042

• Once you verify that you can access the port of the service you want to access, from the host machine where your docker process is failing, and still the problem persists, do these steps-
• Verify the IP address of the problem container –

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' your-container-name-or-id
172.32.15.36

• If the IP address so obtained is found to be conflicting with the IP addresses of your network, provide a new sub-net for the the docker containers within the host as below-
  • vim /etc/docker/daemon.js
  • Add the following lines with the subnet address of your preference

{
  "bip": "10.0.0.1/16"
}

• Restart docker service and start your containers.
• Check your connectivity issue is solved.
• If your application still cannot connect to your target service, lets reset docker by deleting the docker inventories and restarting it-

sudo service docker stop

sudo rm -rf /var/lib/docker

sudo service docker start

#init your swarm if needed
docker swarm init

pull  your images

start your containers

• The above steps usually fixes the faulty bridge network created previously by deleting and recreating it.