Micro service benchmark: Chroniclequeue is 750 times faster than Kafka?

Home > Sci-Tech

Micro service benchmark: Chroniclequeue is 750 times faster than Kafka?

2022-01-28 00:03:03 36 ℃

Comparison of Chronicle Queue and Apache Kafka, please note: Kafka may not be the best suitable tool, Kafka is suitable for high throughput and large data scalable applications.

Apache Kafka is a common choice for service communication. Kafka's parallel processing of the message is a nice choice for log aggregation. Kafka claims to be low delay, high throughput. However, whether many micro service applications in the clouds are fast enough?

When I write Chronicle Queue Open Source, my goal is to develop a message delivery frame with a microsecond delay, and banks around the world are used to delay sensitive trading systems.

In this article, I will describe how Kafka is not extended as Chronicle Queue as a micro-service application in terms of throughput. Even if the throughput is low, the speed of Chronicle Queue has also increased by about 750 times.

Micro service delay using Chronicle: When 500 kmsg / s, the 99% ILE of Chronicle Queue Enterprise is delayed for 3.69 microseconds.

Using Kafka's micro service delay: using Kafka, the same test is performed, but at a lower throughput of 100 kmsg / s, 99% ILE single-stage micro-server end delay is approximately 2633 microseconds (starting from 150kmsg / s, The delay increased sharply).

Kafka was originally designed for log aggregation. It has many connectors, which is well done for this use case. I measured good results, indicating that using Kafka instead of log file writing in a typical system can improve performance and significantly improve manageability.

testing scenarios

In each case, the same test hardness is used. Everything is deployed on Ryzen 9 5950x running Ubuntu 21.04. All tests were used using the same MP600 Pro XT 2TB M.2 NVME drive. Sources of reference testing can be obtained here. https://github.com/openhft/microservice-benchmark

Note: Each message generated will create a second message as a response, so the actual number of messages has doubled compared to single-hop messaging benchmarks.

.....

in conclusion:

Although Kafka is a nice choice for log aggregation, due to its relatively high end-to-end delay, it may not be low enough for many use cases involving micro-services.

Chronicle Queue is open source to achieve a consistent delay of less than 100 microseconds in more than 99.99%, while Kafka has an exception value of 7 milliseconds even when the throughput is 1/5.

Chronicle Queue Enterprise has other functions that can be consistent with delayed delays below 10 microseconds in the case of more than 99.99%.

The original link is found in the link below:

CHRONICLE QUEUE is 750 times faster than Kafka?