ubq 0.0.7(beta)

This page as markdown

ubq guide

The API, the rules, and the commands.

How to write and run tests with ubqtest. This text comes from the installed package, so it is correct for the installed version. ubq guide prints it. ubq guide convert tells how to move a project from a different test tool. ubq doctor checks the setup of a project: it prints one FIX line for each problem, and its exit code is 1 if there is a problem.

A test file#

// test/parse.test.ts
import { assert, describe, test } from 'ubqtest'
import { parse } from '../src/parse.ts'

describe('parse', () => {
  test('circle', () => {
    const shape = parse('c')
    assert(shape) // narrows away undefined
    assert.equal(shape.kind, 'circle')
  })

  test('unknown input', () => {
    assert.throws(() => parse('?'), /unknown shape/)
  })

  for (const [input, kind] of [['c', 'circle'], ['s', 'square']] as const)
    test(`kind of ${input}`, () => {
      assert.match(parse(input), { kind })
    })
})

Run#

ubq                      all test files of the project
ubq test/parse.test.ts   one file (or a folder)
ubq test/parse.test.ts:9 the test at that line
ubq '<test id>'          one test, or the tests of a group
ubq --failed             the tests that failed in their last run
ubq --changed            the tests that ran code that changed
node test/parse.test.ts  a test file is a plain program (also: deno run -A, bun)

Each run also runs the type checker, if the project has tsconfig.json and the typescript package. Each type error is a TYPE block, and the run fails. The tests run in any case.

Use npx ubq if ubq is not on the path. On Bun: bun node_modules/.bin/ubq.

The last line of the output is the summary: pass: 0 failed, 12 passed (run 7) or fail: 2 failed, 10 passed (run 8). Passed tests print nothing.

Exit codes: 0 pass, 1 failure, 2 no tests ran, 4 wrong command or configuration.

A failure block has the test ID, the message, the location, the source line, the difference, and the command that runs the test again:

FAIL test/parse.test.ts > parse > circle
  expected values to be equal
  at test/parse.test.ts:9:12
  > assert.equal(shape.kind, 'circle')
  actual:   'square'
  expected: 'circle'
  rerun: ubq test/parse.test.ts:6

Rules that fail a test#

Do not weaken a test to get a pass. Each of these rules catches a test that would pass for the wrong reason.

API#

Complete API reference

test(name, fn)
test(name, { skip?: string | false, timeout?: number }, fn)
describe(name, fn)                 // a name prefix for its tests. fn is sync.
describe(name, { skip?, timeout? }, fn)

assert(value, message?)            // truthy. Narrows the type.
assert.equal(actual, expected)     // Object.is
assert.notEqual(actual, expected)
assert.deepEqual(actual, expected) // strict: same structure, values, prototypes
assert.match(actual, pattern)      // RegExp for a string, or a partial object
assert.throws(fn, expected)        // expected is required. Returns the error.
await assert.rejects(promise, expected)   // resolves to the error
await assert.eventually(fn, pattern)      // as match. Calls fn again until it matches, 2000 ms at most.
assert.snapshot(value)             // compares with saved text
assert.fail(message)
isDeepEqual(a, b)                  // true or false, by the rules of deepEqual. Not an assertion.

spy(impl?)                         // records calls in spy.calls: { args, result, error }
spyOn(object, key, impl?)          // replaces a method until the test ends. A second spyOn on the key replaces the first.
await mockImport(() => import('../src/a.ts'), [
  [() => import('../src/dep.ts'), { fn: replacement }],
])                                 // a new instance of a.ts with a mocked dep.ts

testdir({ 'a.txt': 'hi', sub: { 'b.json': '{}' } })  // a new folder, returns its path
env({ NAME: 'value', OTHER: undefined })             // set or delete, until the test ends
cleanup(() => server.close())                        // runs when the test ends
const clock = fakeClock({ now: 0 })                  // await clock.tick(ms), await clock.runAll()

Names that do not exist#

Do not use Use
it test
expect(x).toBe(y), toEqual assert.equal(x, y), assert.deepEqual(x, y)
assert.ok, strictEqual, deepStrictEqual assert(value), assert.equal, assert.deepEqual
beforeEach, afterEach, beforeAll, afterAll a setup function, cleanup, top-level code
test.only, describe.only ubq <file>:<line> or ubq '<test id>'
test.skip, test.todo, skipIf test(name, { skip: 'reason' }, fn)
test.each a for loop around test()
retry none. ubq --repeat <n> counts the failures of a flaky test.
vi.mock, jest.mock mockImport
vi.fn, jest.fn spy
vi.useFakeTimers fakeClock
toMatchSnapshot assert.snapshot(value)
assert.doesNotThrow call the function, then assert on its result
a subtest in a test, t.plan, t.end describe. A test ends when its function ends.

Questions about stored runs#

Each run is stored in .ubq/runs.db (git ignores .ubq/). These commands read the store and run no test:

ubq failures [run]        the failures of a run again (default: the last run)
ubq show '<test id>'      the last result of a test, with its console output
ubq history '<test id>'   the result of a test in each stored run. Says if the test is flaky.
ubq diff [run] [run]      the tests whose status differs in two runs
ubq slow [run]            the slowest tests
ubq uncovered [file]      code that no test ran, in each file that a test loaded. Not test files, and not test helpers. Exit code 1 if it lists code.
ubq uncovered --diff      code that no test ran, in the lines that you changed (from the last git commit)
ubq merge <store>...      adds the coverage of other stores: a project folder, a .ubq folder, or a runs.db file
ubq covers <file>:<line>  the tests that ran a line
ubq sql '<query>'         a read-only SQL query, as JSON lines

Coverage is always on, for each test. Before a change to a line, ubq covers <file>:<line> names the tests that protect it.

After a change to the source, do these in order:

ubq --changed           runs only the tests that ran the code that changed
ubq uncovered --diff    lists the code that you changed and that no test runs: write a test for it
ubq mutate --diff       a mutation check of the lines that you changed (see below)

--diff is the difference from the last git commit, with files that git does not have yet. --diff=main is the difference from a different commit.

A mutation check tells if the tests of a file are strong:

ubq mutate src/cart.ts       makes small changes to the file, one at a time, and runs the tests of the changed code
ubq mutate src/cart.ts:20    only the changes on that line
ubq mutate --diff            only the lines that you changed

It changes an operator, a number, or a boolean (- to +, < to <=, 0 to 1), in the test processes only. The file does not change. Each SURVIVED block is a change that no test failed for: add a test for that case. Some changes cannot make a test fail, because the changed code does the same thing. Do not change correct code for them. Run the tests before the check (ubq).

More run commands:

ubq --repeat <n> <selection>        run the tests n times, count the failures of each
ubq --baseline <commit> <selection> show only the failures that the git commit does not have

Console output of a test is captured. It shows only in the failure block of its test, and in ubq show.

Tests in a browser#

For code that runs in a browser. The driver is Playwright, with Chromium. Set up one time:

npm install --save-dev playwright-core
npx ubq browser install     downloads the browser, if it is not there
import { assert, test } from 'ubqtest'
import { browser } from 'ubqtest/browser'

test('the button counts', async () => {
  const page = await browser()           // a Page of Playwright, in a new context
  await page.goto('/test/pages/count.html')
  await page.click('#add')
  await assert.eventually(() => page.textContent('#total'), '1')
})

Coverage: the code of the project that a page runs is coverage of the test, as the code that the test process runs. So ubq --changed, ubq uncovered, and ubq covers work for it. This needs a script whose URL path is the path of a project file. The script is the file as it is, or a server made it from that one file and put a source map at its end (Vite in development does this). A bundle of many files has no coverage.

Which page page.goto('/path') loads:

A different browser version, or a shared download:

Code for more than one runtime or system#

Some code runs only on Bun, or only on Windows. One run cannot run each line of such a project. The store keeps the coverage of each environment (a runtime and a system, such as bun-darwin), and ubq uncovered lists the code that no environment ran.

ubq uncovered tells you when you need this: it prints a note if a range is on a line that names a runtime or a system (process.platform, Bun, Deno).

On one machine, run the tests on each runtime. The second run does not remove the coverage of the first:

npx ubq                        on Node
bun node_modules/.bin/ubq      on Bun
npx ubq uncovered              code that no runtime ran

For other systems, each CI job keeps its store, and one job merges them. With GitHub Actions:

jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
      - run: npm ci
      - run: npx ubq
      - uses: actions/upload-artifact@v7
        with:
          name: store-${{ matrix.os }}
          path: .ubq/runs.db*
          include-hidden-files: true
  coverage:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
      - run: npm ci
      - uses: actions/download-artifact@v8
        with:
          pattern: store-*
          path: stores
      - run: npx ubq merge stores/*
      - run: npx ubq uncovered       # exit code 1 if no job ran some code

Tests that start a process#

A test can start the program under test as a child process: a command, or a server. The code of the project that the child runs is coverage of the test, as the code that the test process runs. So ubq --changed, ubq uncovered, and ubq covers work for code that only a child runs. No setup is necessary.

import { spawnSync } from 'node:child_process'
import { assert, test } from 'ubqtest'

test('the command prints its version', () => {
  const r = spawnSync('node', ['src/cli.ts', '--version'], { encoding: 'utf8' })
  assert.match(r.stdout, /^\d+\.\d+\.\d+\n$/)
})

Tests of a Cloudflare Worker#

The test file runs in workerd, the runtime of Workers, with the local bindings of the wrangler config (D1, R2, KV, Durable Objects, vars). The project must have wrangler. Tell ubq which test files are Workers tests, in package.json:

{
  "ubq": {
    "workers": [
      { "config": "wrangler.jsonc", "include": ["test/**/*.test.ts"], "vars": { "API_KEY": "a test value" } }
    ]
  }
}

vars is optional. Its values go on top of the vars of the wrangler config.

import { env } from 'cloudflare:workers'
import { assert, test } from 'ubqtest'
import { executionContext, migrate } from 'ubqtest/workers'
import worker from '../src/index.ts'

await migrate('DB')                      // the files of migrations_dir, for the D1 binding DB

test('lists the notes', async () => {
  const ctx = executionContext()
  const response = await worker.fetch(new Request('http://x/notes'), env, ctx)
  await ctx.settled()                    // the promises of ctx.waitUntil
  assert.deepEqual(await response.json(), [])
})

Not available in a Workers test file: testdir, mockImport, assert.snapshot, and JSX files. A test that goes over its time limit fails, but the report does not list what the test left open. There is no SELF fetcher: call the handler of the Worker with a Request.

Faster runs#

ubq server starts a background server for the project. It keeps test processes ready, so a run of a few files is about 30 ms faster, and the first run after an edit about 45 ms. --repeat is about 3 times faster. The commands, the output, and the exit codes do not change. It stops after 30 minutes with no use, or with ubq server stop. It is optional.

Configuration#

Most projects need none. The ubq key of package.json can have include and exclude (lists of glob patterns for test files), helpers (glob patterns for test helpers: see below), build (glob patterns for build output and build tools: see below), timeout (milliseconds), hangRerun (false turns off the rerun of a test that timed out), and bunInspectorCoverage (true records coverage for a plain bun <test file>, with places that can be wrong. ubq <test file> on Bun does not need it), and browser (see "Tests in a browser"), and workers (see "Tests of a Cloudflare Worker"). A folder with its own package.json is a different project: run ubq in that folder. At the root of a monorepo with no tests of its own, ubq runs each workspace.

Build output and build tools are not source of the program. ubq uncovered does not list them. With no build key, they are the files in dist/ and scripts/ of the project. The key replaces that rule: "build": ["out/**", "tools/**"]. An empty list means that each file is source. If a test loads a built file that has a source map, the coverage is coverage of the source files that the map names: the result is the same as for a test that loads the source. After a change to a source file, build again before the tests run: until then, the run has no coverage of that file.

A test helper is a file that is part of the tests and is not a test file: a setup module, or the scenario of an end-to-end test. ubq uncovered does not list its code, and ubq mutate --diff does not change it. Do not write tests for a helper. With no helpers key, a helper is each file in a folder with the name test, tests, or __tests__. The key replaces that rule: "helpers": ["src/testing/**"]. An empty list means that no file is a helper. A change to a helper still makes ubq --changed run the tests that loaded it.

Code that no test can run can have a comment with the reason: // ubq: not measured: <reason>. Examples: a fallback for a value that is never absent, or code that runs before the coverage hooks start. Put the comment on a line of its own above the code, or at the end of the first line of the code. For a block, the first line in the block also works. ubq uncovered does not list that code, and it prints how many ranges it left out. A comment with no reason does not count. Look for a test first: the comment is for code that a test cannot reach.

Requirements#

Node 24 or later, or Bun, or Deno. TypeScript runs with no build step, so the code must use only syntax that can be erased: no enum, no namespace, no parameter properties. In tsconfig.json, set "allowImportingTsExtensions": true, "noEmit": true, and "erasableSyntaxOnly": true.

Bun and Deno also run .tsx and .jsx files, with coverage. Node does not run them.

Tests of a Cloudflare Worker run in workerd, and the project must have wrangler: see "Tests of a Cloudflare Worker".