How Many Queues Are Best For Max RabbitMQ Performance?

As always, the answer to such a question is linked to many other factors.
However this question is relevant in many implementations using Rabbitmq. You can reach 1 Million of messages per seconds as pivotal did in this article, but they didn’t have a huge amount of queues, they had multiple queues and this improve the parallelism using multiple cpus and huge amount of messages. In many real implementations is possible to have a big amount of queues and a reasonable message rate. So what happen to Rabbitmq performance?

From the RabbitMQ blog

RabbitMQ’s queues are fastest when they’re empty. When a queue is empty, and it has consumers ready to receive messages, then as soon as a message is received by the queue, it goes straight out to the consumer. In the case of a persistent message in a durable queue, yes, it will also go to disk, but that’s done in an asynchronous manner and is buffered heavily. The main point is that very little book-keeping needs to be done, very few data structures are modified, and very little additional memory needs allocating.

This small post tell some important things to us. Rabbitmq use memory to save memory structure for each queue, we will came back to this topic, however this footprint is related to the messages stored and not yet delivered and to some data structures used by the broker. this mean that until queues are kept almost empty the performance is not impacted as huge as can be the number of the queues.

This means that the queues can be infinite?

The answer is no of curse as for all computer related stuff. Queue use a file descriptor and this means that huge amount of queues use a huge amount of file descriptors.
According to the kernel documentation, /proc/sys/fs/file-max is the maximum, total, global number of file descriptors the kernel will allocate before choking. This is the kernel’s limit, not your current user’s. So you can open 812158, provided you’re alone on an idle system (single-user mode, no daemons running). we can say that the maximum number or queues manageable by a single linux machine is less than this number and with the right amount of ram and disk it should keep good performance.

So how many queues i should have for best performance?

Even if thousands of queues can be managed by Rabbitmq maintaining good performance, queues are single-threaded, and one queue can handle up to about 50k messages/s. You will achieve better throughput on a multi-core system if you have multiple queues and consumers. You will achieve optimal throughput if you have as many queues as cores on the underlying node.

Leave a Reply

Your email address will not be published. Required fields are marked *