Repository: avajs/ava

Issue #1366: Proposal: test.resource()
Created: 2017-04-21T07:11:38Z
The problem
When integration-testing web apps (which AVA is great for since it runs tests in parallel), I sometimes run into issues with what I think is resource constraints, even on my high-end Macbook Pro. This results in the following sadness:

To fix this, I need to run AVA with --c 6 to lower the concurrency which is fine but if my MBP is using more resources for other things, I need to lower it even more.
The reason for this, is that I in each test suite run my HTTP server, set up a bunch of helpers for testing my API, connect to websockets, and then finally run my test. This is my pattern:
test('GET /todos returns all todos', withServer(async (t, api) => {
  const todos = await api.getTodos()
  t.true(todos.length > 0)
}))
Where withServer is just a factory function that sets up the server:
const axios = require('axios')
const createServer = require('../src/server')

function withServer (testFn) {
  return async function (t) {
    const server = (await createServer()).listen()
    const host = `http://127.0.0.1:${server.address().port}`
    const client = axios.create({ baseURL: host })
    const api = {
      getTodos: () => client.get('/todos').then(r => r.data)
    }
    
    await testFn(t, api)
  }
}
This makes it super easy for me to write integration tests for my API. Problem is, each test need their own connection to the server (in case of websockets), and even when I memoize the server (which only works per suite due to each suite being in a separate process), I still run into the above issue.
I think this is because I am using rethinkdbdash which manages a large connection pool. For a single server instance or 4 that might be fine, but for 10+, that might be why it starts having issues.
The proposed solution
What I would really like to do, is set up just a single instance for all my tests to work with.
One way to do it is to run npm start & npm test, but if npm start takes too long to actually start the server, then that's gonna be a problem.
Instead, what if we had a way to tell AVA to run a JS function before starting the test processes? For example, if I was able to do this:
// integration-test.resource.js
import test from 'ava'
import createServer from '../src/createServer'

// This runs before any other test processes have been spun up
test.resource('Set up API server', async (t) => {
  const server = (await createServer()).listen()
  const host = `http://127.0.0.1:${server.address().port}`
  
  // Only JSON-serializable values can be provided to test processes
  t.resource('serverHost', host)
  
  return async function dispose () {
    // When all tests are done, dispose of the resource
    server.close()
  }
})
And in my test
test('GET /todos', async (t) => {
  const client = axios.create({ baseURL: t.resource('serverHost') })
  const { data } = await client.get('/todos')
  t.true(data.length > 0)
})
The result is a single HTTP server that is ready when tests run, and a way to tear it down once all tests have run (pass or fail, does not matter). This is way more CI friendly than the npm start & npm test approach.
Proposed Configuration
To load these resources via CLI:
ava 'test/*.test.js' --resources 'test/resources/*.resource.js'
Via package.json
{
  "ava": {
    "resources": [
      "test/resources/*.resource.js"
    ]
  }
}

Duplicate issue #2048: Mutexes / external resources
Created: 2019-02-22T06:16:29Z
Sometimes we need to limit some of the tests to run serial, or having a maximum parallel jobs, or wait some external resource to be prepared. Therefore introducing such concepts would be helpful for network-oriented tests or stress tests.
Purposed API:
test('Some stress test', async t => {
    const handle = await t.resources('stress-test-capacity').acquire(1000);
    // The "stress-test-capacity" is a kind of resource, we need 1000 of them.
    //
    // All resources would be automatically released after the test ends.
    // Alternatively, use handle.release() to manually release a resource.
});

Closing comment (2019-11-20T09:07:31Z): I'm not sure how many real-world scenarios would actually benefit from quantified resource-management. ... nor do I envision many use-cases where a testing database can handle 2 concurrent connections, but not 8.

I've got a scenario that would qualify, I think.  I'm essentially doing integration testing on my front-end JS code communicating with a back-end server which I run locally for testing.   I'd like to run my tests in parallel, with the server handling an independent connection from each test.  But I may need to limit the number of simultaneous connections to the server so I don't thrash my local machine.   In this case it's not a question of 2 or 8 concurrent connections but potentially many more: I've currently got ~65 such integration tests, and of course that number keeps growing.
[Careful readers will note that in #2268 I said my backend server doesn't support simultaneous requests.  But that will hopefully be changing soon, in which case I'd like to take advantage of that to parallelize my tests.  But I'm concerned about too much parallelism!]
Being able to limit the amount of simultaneous access to a resource feels to me to be analogous to the --concurrency CLI option & default limit, which I think nobody debates as an unreasonable thing to have :)
One simple way to handle this might be to have a --concurrent-tests CLI option to would limit the total number of tests that are run simultaneously, though that would be overly restrictive if only some of the tests connect to the resource.  Even if --concurrent-tests only affect to the number of concurrent tests in a single file, I could get some overall control of.
Thinking about it more, I think that case actually is the same as  #1366, but with concurrency limits.  Will go make a comment there...