One of Camunda’s founding values is our commitment to being open. We live that value daily by providing access to Camunda’s codebase on GitHub, supporting a vibrant community ecosystem, and being transparent about our technology’s capabilities.

Falko Menge, Camunda’s senior principal solution architect and open standards ambassador, shared the results of recent platform benchmarking tests with our CamundaCon audience in this spirit of openness and community.

Why benchmarking Camunda’s workflow engine is important

As developers know, benchmarking provides actionable insights into product quality and helps refine performance. It’s an important part of our continual development processes, now more than ever, for two main reasons.

First, orchestrated, end-to-end automation of disparate processes, systems, devices, data, and AI is critical to organizational growth and operational efficiency.

Gartner defines this class of integrated software technologies as business orchestration and automation technologies because they are important for driving consolidated business orchestration and automation. A workflow engine is an essential part of any process orchestration and automation framework, so we want to ensure Camunda’s Zeebe advanced workflow engine has the availability and scalability necessary to support the evolution of complex, enterprise-scale, intelligent automation systems.

Second, we want to confirm that we are developing the best technology to support our customers.

Increasingly, our customers are modernizing their architectures to empower use cases with numerous process instances that require high-throughput, high-performance workflow processing. Example use cases include real-time payments, stock trade matching and settlement, compliance checks, end-of-day asset balancing, bulk ordering, and merchant payment batch clearing.

We want to demonstrate that Camunda’s advanced workflow engine can support high-volume workflows with low latency, can scale as needed, and is a resilient system for handling errors and failovers in real time. Through benchmarking, we can also help our customers find the right configurations to optimize their automations.

How we built a benchmark generator

Camunda’s team faced a challenge: no common testing software measured the criteria we wanted to benchmark. Traditional load testing frameworks did not provide a way to test and measure the balance of process instance starts and service task completions. Camunda cofounder Bernd Ruecker explained this problem in a previous blog on the team’s journey to build a benchmark tool.

As a solution, our team built a benchmark generator in Java that runs on Kubernetes and simulates the workload that real Zeebe clients might generate against the engine. It combines a “starter” application that pushes API requests to the Zeebe cluster and a “worker” application for completing service tasks.

The tool includes a balancing algorithm that tries to find the optimal rate of process instance starts by starting projects at a given rate and adjusting the rate based on backpressure. Metrics are exposed in Prometheus and displayed in prebuilt Grafana dashboards.

To reduce the need for a lot of hardware resources, the team used asynchronous/reactive programming, which allows the generator to use a small machine to utilize even large Zeebe clusters.

Our testing methodology

The design of the benchmark generator allows us to test different modes. For example, we can test a solid workload and then ramp up the load to overload the system until backpressure prevents the system from completing any work. It’s an approach that allows us to find a breaking point and test potential reconfigurations to optimize performance.

In his presentation, Falko explains, “When we did the testing, we found ourselves starting to record different configurations and playing with different variations of Zeebe settings to find the right performance.” One test might be configured for 2,000 process instances per second. The next test might include adjustments to the size of the Zeebe cluster or other parameters.

To track the different configurations and variations, the team created a massive spreadsheet that captures all the configuration parameters and test results.

By adding automations to the sheet, each row becomes a configuration that can be executed. The sheet essentially starts the benchmarking tests, connecting with the workflow engine through an app called Zeebe Tuner, which generates values for the Camunda Helm Chart and drives the load generator to run the test.

This framework allows us to efficiently iterate test configurations and change one parameter at a time to evaluate and optimize performance. It also allows us to run tests unattended.

Benchmarking results

To give you an idea of the type of output we can generate through our benchmarking generator, the series of images below shows results from a hack week where the team focused on low latency requirements. The images illustrate the result of improvements made over the entire week by reconfiguring parameters in the spreadsheet to tune the engine.

Benchmarking lessons learned

“If you have a high-performance workload and you’re wondering if Camunda can handle it, then we have proven more than once that we are horizontally scalable.”
– Falko Menge, Senior Principal Solution Architect at Camunda

The benchmark generator gives engineers the ability to play with different variables and find the optimal configuration for each client’s use case and needs. It also helps us identify and prioritize potential areas where performance can be improved. For example, tests focused on process instance duration helped identify ways to reduce the duration through job streaming.

Key takeaway: The importance of job streaming

Job streaming is how the jobs arrive at the client. Through benchmarking, we could visualize the relationship between larger numbers of partitions and diminishing returns on job activation and saw how a mode called job streaming reduces process instance duration.

Job streaming mode allows brokers to autonomously assign jobs to clients in real time by enabling brokers to recognize which clients are connected to the system. In our tests, we also found that job streaming reduced CPU usage on the Zeebe cluster by 50%, which is significant. Based on benchmarking, our recommendation is organizations that require high performance use job streaming mode as a default.

Dive deeper into Camunda’s benchmarking data

Falko also shared with the CamundaCon audience his findings related to predictable scalability, price performance for Zeebe brokers, and engine tuning best practices. Download his slides and watch the conference replay to see the breakdown of our latest benchmarking test data in detail.

For more details on how to optimize Camunda for your organization, check out Jothi Viswanathan’s blog on performance tuning end-to-end solutions in Camunda 8, or connect with a Camunda consultant.