HN is a great resource to gather different points of views from experts. Here is some folk wisdom I compiled on load and performance testing for a presentation:

Selenium/Web Driver handles the UI tests. Postman and a handful of others can perform the API tests. JMeter is the de facto for load testing at every place I've been to. And you can write any of these into your CI.

There are three separate reasons to do load testing:

1) Performance testing. Confirm the system does not degrade under a specified load and find out what performance can be expected under these circumstances. This is basically ensuring your system can handle X amount of traffic without issues and knowing your baseline performance. You should get the same kind of response times as you are getting from live server telemetry.

2) Stress testing. finding out what happens when the system is stressed beyond its specs. How does it degrade & where.

3) Reliability testing. Find out how your system breaks and when. The goal here is to try to break the system and test things like failover and making sure you don't lose or corrupt data. Better to die gracefully then abruptly.

I've used Jmeter quite a bit over the last year, and what seems to be the largest issue is that it doesn't "break" apps because it works at the protocol level and isn't a full "browser". As you increase load and response times start to increase, since it runs sequentially through the test plan the time between the requests also increases. But, when actual humans with browsers use apps there are loads of AJAX requests being fired which don't necessarily go in order or wait for others to complete first.

Most managers only want to see how traffic will perform in what-if scenarios (can we handle Black Friday, what will happen if our traffic goes up 10x during a special event, etc). For these, JMeter and whatever service you're using for analytics/metrics work perfectly fine (New Relic, is the current favorite). Then we can compare average latency, Error Rates, etc. with what we get during JMeter.

The issue is that most managers or other PHB want more than just a one off for load testing. And this article greatly covers that.

A lot of time people want a Swiss Army Knife of tools for "automation", where load testing falls under that category for them.

This magical tool should be able to:

* Test your API calls from a functional, integration, and a performance (not load testing; just making sure we're under specific latency)

* Test your Web-sites from the same testing perspectives, as well as being modular (ie: Selenium's POM)

* Integrate with CI so specific tests, test suites/flows can be tested for every new build, and we can run specific workflows by clicking a button

* Be used during load testing so we can measure latency and run tests while recreating customer experience from the Web side of things.

Last time I had to do load testing in a professional context, we required 10+ million long-lived TCP and websocket connections transacting multiple times a second. There weren't any off-the-shelf solutions at that time that could come within an order of magnitude of that at a reasonable cost - the most viable solution we tested required a thousand EC2 instances to sustain that traffic.

Do red line testing. Gradually increase load on your service with concurrent requests until it is saturated.

Measure how the latency and errors progress with more concurrent requests and understand at what point and how your service starts to break down under heavy load.

Based on this you may have to do many things –

1. Can you optimize your service or your service's downstream dependencies or the application calling your service.

2. Can you build in graceful degradation into your service – functionality reduction to get more useful throughput out of your service – with same resources.

3. Build circuit breakers and throttlers before your downstream dependencies so that you don't overload them and cause them to fail or you don't fail totally when they do indeed fail.

4. If you do get overloaded for some reason (say your server pool suddenly became half its size), are you able to recover quickly.

5. Do you have monitoring and alerts for these load scenarios?

I tend to avoid having to do load testing as it sucks up time without telling me much of interest. I instead opt for having decent telemetry on the live system. It will tell me how it performs and where the bottlenecks are. I can set alerts and take action when things degrade (e.g. because of a bad change). Besides, there is no substitute for having real users doing real things with real data. And in any case, having telemetry is crucial to do any meaningful stress or reliability testing. Otherwise you just know it degrades without understanding why.