For each developer is software is immune to memory lake, this is usually false and in many cases is not related to bugs introduced by the developer but in the interpreter on the libraries used. In other cases the memory drift can be related to a unexpected behaviour, people try to think to everything but the world (including the computer world) is not lacking in inventiveness.
I started to think about when one of our dockerised microservice that we considered bulletproof, because working well for years, started grow in memory without apparently reasons, arriving to consume the 2000% of normal. We discovered that the library we used to connect to mysql was the guilty for this problem. This library started to consume a huge amount of memory when mysql went down for 30 seconda after a wrong minor update from AWS (RDS db). In order to avoid similar problem we decided to limit our docker containers in memory.
Docker implements a way to limit the memory consumption of a container on the run. More information and options can be found on this page.
Is important to consider a good maximum value of memory to kill the containers. The Docker daemon is a process itself and it set it’s own OOM priority to avoid to be killed before the single containers, but this is true until kernel have containers to kill, after that Docker can be killed like other processes.
We didn’t mount any swap space on disk so we decided to simply limit the memory to 1024M using -m / –memory= option, this mean that the container can consume up to 1GB after that an out of memory error will be generated.
In order to kill the hungry container we must be sure that the oom killer is enabled for docker daemon. This option is a bit misleading, because we should set –oom-disable-kill=False.
Ok now if everything works as expected we have our container off and our system is safe but someone should wake up to restart the container. Also in this case Docker guys thought to us and they implemented a restart policy for the container, we use it setting –restart always.
Now it’s time to test, we must create a container with a script to simulate an abnormal memory consumption.
First of all we create the script crash.sh
#/bin/bash sleep 5 yes | tr \\n x | head -c $((1024*1024*1024)) | pv -L $((1024*1024)) | grep n
this simple script will use grep to use ram memory at a rate of about 1MB per second until 1GB.
We need also to create the Dockerfile:
FROM ubuntu:14.04 ADD crash.sh / CMD /bin/bash /crash.sh
With the Dockerfile defined, we can now build our custom container using the docker build command and next run it using the docker run command with the parameter we described above.
sudo docker build -t testing_restarts ./ sudo docker run -d --name testing_restarts --restart always -m 1024M --oom-disable-kill=False testing_restarts
Now using the docker stats you can see the memory consumption of container testing_restarts increase and at one point disappear and restart again. If everything work as expected you can remove the container using
docker rm -f testing_restarts
In our case we would add this behaviour to our docker-compose.yml and we did in this way:
version: "2.2" services: container1: image: dockerhubUser/image1 cpuset: "1" mem_limit: 1024M environment: LOGGING_LEVEL: INFO ENVIRONMENT: production volumes: restart: always network_mode: "bridge" container2: image: dockerhubUser/image2 ports: - 2244:2233 cpuset: "0" mem_limit: 1024M oom_kill_disable: False environment: LOGGING_LEVEL: INFO ENVIRONMENT: aws-production restart: always network_mode: "bridge"