Azure Load Testing

We at cVation are curious about what the feature offers today, so we used a hackathon to experiment with the service. This blog describes what we learned and what's on our wish list for future expansions.

Motivation - Ongoing load test, as part of DevOps

The obvious motivation for testing performance is, of course, to ensure that a solution performs under a realistic user load. Most people have probably tried to performance test a solution before going live, only to experience that continuous expansions and changes after going live impair the performance. Azure Load testing helps avoid this, as it is built not onlyto run load testing once, but as a natural and automatic part of a pipeline in Azure DevOps and GitHub Actions. This ensures that you are able todetect which code commits degradinge performance, making it much easier to identify and correct the problem.

In the pipeline, you define different acceptance criteria, so that a pipeline fails and possibly blocks a release if the criteria are exceeded. Below is an example of a pipeline that fails because the response times from a function exceed 5 seconds.

Design of load test

Azure Load Testing is built on the Apache JMeter, which is an existing tool for load testing. It has the advantage that it has several existing features and if you already have load testing in JMeter, then you can easily run Azure Load Testing. Unfortunately, JMeter's client program is a bit of a hassle to get started with and one feels relegated to the 90s when using it. But fortunately, you can also write the xml directly, so you avoid the JMeter UI.

JMeter's xml also opens the possibility of autogenerating a load test. This can be done by writing tooling that analyses endpoints and builds a load test based on it.

Test results

Azure Load Testing also includes reporting and comparison of test runs. You can choose which Azure resources you want to see metrics for and if you choose to view metrics for a cosmos resources, then RU consumption and similar relevant data are displayed. The reporting is thus context-dependent on which Azure resources are part of the solution. However, we find that reporting, and especially comparing results, is not always accurate. Therefore, there may be some bugs in the preview version in this area.

Conclusion

After working with Azure Load testing on our hackathon, we see a service from Microsoft with potential, but also a service that is not finished either. It's easy to integrate Azure Load Testing into existing pipelines in both Azure DevOps and GitHub and you get great reporting in Azure. But it also seems that there are a few bugs in the display of test results.

What is still lacking? You have the option to define a precise limit which will cause a pipeline to fail, but we miss features that automatically send alerts about poor performance over time. We know that Microsoft already have AI built into Alerts to tell when something is happening that is out of the ordinary, so we hope it also gets well integrated into Azure Load Testing.

We look forward to seeing the final version and hopefully a lot of features and improvements over time.