For the past two years, the number of connected IoT devices has been growing by over 2 billion annually, according to IoT Analytics. At the same time, performance of MQTT brokers directly affects the number of devices that IoT systems can handle while maintaining reasonable speed and reliability. With minimal latencies, brokers need to handle high throughput without dropping messages or using too much resources.
To help you get an idea of how different MQTT products perform in various scenarios, we’ve compiled a list of benchmarks published by various researchers and vendors between 2020 and 2023. We also brought together papers that evaluate public sandboxes or compare MQTT brokers to other alternatives, as well as found a few articles on how to conduct your own benchmarks. Some of the papers are truly independent, while others present the results from one of the vendors. Still, all the reports shed some light on how the tools differ and may behave under various conditions.
In September 2021, a group of researchers from the University of Szeged (Hungary) and Wind River (a subsidiary of Aptiv) benchmarked the performance of six MQTT brokers. The tests were done from a machine-to-machine (M2M) communication perspective, since Aptiv/Wind River’s business is focused on embedded systems. Stress load was evaluated in both local and distributed/multi-core (cloud) environments—revealing latencies and throughput at 100% CPU use and when resource usage is unrestricted.
For the purposes of the comparison, the team developed the MQTT Blaster benchmarking tool based on Eclipse Paho. “The findings of this research highlight the significance of the relationship between MQTT broker system design and its performance under stress testing,” the authors noted in their analysis.
“A Comparison of MQTT Brokers for Distributed IoT Edge Computing” is, indeed, a fundamental MQTT broker comparison (see the document). Delivered by ABB in 2020, it aims to evaluate the messaging tools from a point of view of industrial engineer/software architect. Focusing on distributed MQTT brokers suitable fot the edge workloads, the authors used a scientific approach to calculate throughput, scalability, resilience, etc. The results were obtained by utilizing MZBench, vmq_mzbench, and MQTT Explorer.
In April 2023, MQTT vendor EMQX conducted a series of performance benchmarks. The team compared EMQX’s distributed broker against its own lightweight MQTT tool for IoT edge workloads—NanoMQ—as well as explored VerneMQ and Mosquitto. The benchmarks were implemented with EMQX’s XMeter (based on JMeter) and—in addition to latencies and throughput—also revealed CPU and memory usage, message rate limits, etc. The scenarios included both small-scale performance evaluation and enterprise-level tests. The benchmarking was done with a 16-bytes payload and included point-to-point, fan-out, and fan-in scenarios at QoS 1 and publish:subscribe ratios of 50,000:50,000, 5:250,000, and 50,000:5 respectively.
Here’s a consolidated report, as well as more detailed separate comparisons of EMQX to NanoMQ, VerneMQ, and Mosquitto. One more benchmark evaluating NanoMQ vs. Mosquitto can be found here. This GitHub repository describes the comparison’s methodology and scenarios in detail.
In 2021–2022, MQTT startup FlashMQ conducted several benchmarks—some of them were recorded as live demos. For instance, this video demonstrates the maximum number of clients and messages achieved for VerneMQ, EMQX, NanoMQ, and FlashMQ. In a separate recording, the team shows how FlashMQ reaches 4,000,000 connections per second, competing with HiveMQ, Mosquitto, and VerneMQ. Another benchmark from this vendor aimed to evaluate CPU use against performance (throughput) of VerneMQ, EMQX, and FlashMQ.
An insightful benchmark was conducted in October 2022 by Dr. Gough Lui, a postdoctoral researcher at Western Sydney University. The goal was to explore public and free MQTT services that can be used for very small projects or proof-of-concepts. The test were implemented by using the paho-mqtt library (the code is provided in the report). Quite expected, the author experienced disconnections, spikes in latencies, and dropped messages from time to time.
“The performance of the MQTT servers seem to be dominated mostly by network latency…While public and free MQTT servers can be useful for testing and hobby projects, it is important to remember that these services may not be around forever and may go down from time to time. As a result, any serious application should consider hosting their own server.”
Earlier report from EMQX—released in November 2021—also evaluated a number of free public brokers. It looked into accessibility (using MQTTX), international network delays (the time required for DNS resolution and to establish a TCP connection), average message transmission delays (from Pub to Sub), etc. To evaluate Pub/Sub rates, the team utilized EMQX’s lightweight eMQTT-Bench tool written in Erlang. Here’s the code for message transmission delay tests.
This benchmark discovered unstable network delays, connection failures, and a small amount of packet loss when testing one of the services, too. The authors also suspected some servers of having a scheduled restart mechanism.
In February 2023, HiveMQ delivered a performance benchmark of its platform to demonstrate the maximum number of connections the tool can support. The test was run on AWS EC2 and a HiveMQ Swarm instance, and the ramp up to 200 million connections took 37 minutes.
EMQX shares additional performance data for NanoMQ in the product’s documentation. The team ran tests on three different server configurations, using EMQX’s XMeter. Scenarios included 1:1 send:receive and broadcast message testing.
In March 2023, RabbitMQ released a performance benchmark demonstrating the improvements of its latest v3.12 compared to v3.11. The team evaluated memory usage when connecting 1 million MQTT clients and during the point-to-point scenario with the 100,000:100,000 ratio, as well as measured latencies and throughput.
BifroMQ is a brand-new MQTT broker open-sourced by Baidu in July 2023. Its documentation contains a comprehensive performance evaluation report, which was conducted—according to the screenshots—in May and June 2023. While the resulting table says the tool can handle 200,000+ messages per second for high-frequency scenarios, the main product page also notes that the tool supports up to 180 million connected clients.
In March 2023, researchers from the National Taiwan University tested four popular messaging solutions (see the blog post and the paper). The experiment encompassed two scenarios: one was run on a single machine, and the other used multiple servers connected via Ethernet. For the throughput benchmark, payload sizes varied from 8 bytes to 512 MiB, while latency was tested with the payload fixed at 64 bytes.
Streamnative, a company behind a messaging and streaming platform, conducted another benchmark in Q1 2023. The report encompasses four scenarios—node failure, topic counts, subscription counts, and backlog draining—with 1 KB messages.
In April 2023, 84codes—the team behind LavinMQ, CloudAMQP, and CloudMQTT—delivered a couple of own AMQP benchmarks. Evaluating RabbitMQ and LavinMQ, the performance was measured with different message sizes (16 and 1,024 bytes), emulating 3 producers and 5 consumers. The tests were done with the lavinmqperf tool, utilizing RabbitMQ’s PerfTest in some of the scenarios.
Researchers from the University of Antwerp ran this comparison in September 2022. The benchmark used a fixed RSU, which communicated with a mobile OBU over 4G with a payload size of 256 bytes.
In June 2023, computer science researchers at Baylor University measured the throughput and latency of these four dissimilar yet popular products. The study utilized the OpenMessaging Benchmark Framework.
Confluent—a company behind a cloud-native data streaming platform—conducted this benchmark in August 2020. The tests used three Amazon EC2 i3en.2xlarge instances for the producer, broker, and consumer.
Researchers at St. Petersburg Electrotechnical University published another paper in August 2022. The tests ran on a single server, using message sizes of 50 and 60,000 bytes.
If you want to conduct your own MQTT broker comparison, there are numerous articles that provide additional guidance on specific tools and approaches. E.g., EMQX explains how they benchmarked NanoMQ using their Open MQTT Benchmark Suite. In addition, the vendor has a tutorial on benchmarking their distributed broker with eMQTT-Bench.
In this paper, a group of researchers used Gatling to test the performance of MQTT brokers. Gatling goes into more detail about using their tool for MQTT performance testing in a recent tutorial.
This study provides a detailed guide on using MQTTBrokerBench to evaluate various MQTT brokers. Another research paper demonstrates how to test MQTT brokers with MQTTLoader. The post from JoinBase shows how to measure MQTT broker performance with the Open IoT Database Benchmark Suite (OIDBS). An article by VerneMQ covers the basics of load testing with Machinezone’s MZBench framework. Another blog post from BlazeMeter explains how to set up Apache JMeter for testing MQTT brokers.
Founder of VOLSOFT Volkan Alkılıç showcased how to benchmark MQTT brokers in his blog post. The article explains how to do this with ThingsOn MQTT Bench—a performance testing tool that Volkan developed for the purpose.
“As an IoT and messaging systems developer, I struggled to find a benchmark tool that met my needs when testing the performance of MQTT brokers for my IIoT Platform. Most tools I found were either too complicated or too basic, and lacked the level of customization and control I desired. This motivated me to create my own benchmark tool that is simple, lightweight, and flexible, allowing for easy adaptation to various scenarios and requirements.”
—Volkan Alkılıç, VOLSOFT
Benchmarks indicate that the performance of MQTT brokers depends heavily on the scenario and configuration. Still, technology benchmarking can reveal unexpected behavior—including unstable latency, dropped messages, and limits on the number of connections. In addition to other details about latency and throughput, these insights are also vital to understanding how the evaluated technology fits the specific requirements of your project.
If you know any other MQTT performance benchmarks we missed—feel free to add them in the comments.
Explore evaluations of Kafka’s performance on various cloud platforms, as well as benchmarks against other…
Governed by LF and designed to shift data analytics closer to devices, this lightweight IoT…
Aiming for operational improvements, manufacturers are struggling with system complexities. Is there a way out?
Supported by Dell, IOTA, Intel, etc., this open-source project assigns trust scores to sensor data,…
Google will sunset its IoT Core on August 16, 2023. Some of the edge devices…
For 53% of institutions, breaches in connected systems affect patients—e.g., lead to delayed surgeries or…
