WIP

Minor:

* browsers: rename Headless Firefox => Firefox Headless.
* browsers: fix safari() failing to run multiple test files.
* client: window.qunit_config_reporters_html = false
* client: debugMode `<pre>`
* server: change clientId from "client_1" to "client_S1_C1"
  to make verbose logs more useful by making it obvious which
  clients are for the same testFile (and thus ControlServer).
* reporter: flesh out more of the default reporter,
  and refine the events we emit to ease the work of reporters.
* qtap: De-duplicate browsers and test files.

Major:

* qtap,server: Streamline error handling and signal handling to avoid
  errors going unhandled, bypassing teardown, or causing
  the process to be stuck waiting for nothing.

  In particular, "Could not open <file>" is now propagated to a
  top-level error (e.g. run throws/rejects) rather than a test-level
  "bail". This means we emit "error" instead of "result" or "finish",
  and other tests are cancelled as well.

  With this in place, other uncaught errors are now handled better
  as well, such as errors from reporters.

* qtap: handle uncaught errors from reporters, which otherwise
  cause emit() to throw. Provide a dedicated EventEmitter proxy
  to improve attribution in the error message to a given reporter.
  This is limited to "on" both because its simpler that way, and
  because it protects our EventEmitter object from reporters
  tampering with "emit" or "on".

Author: Krinkle

API.md

@@ -155,70 +155,86 @@ async function mybrowser (url, signals) {
 }
 ```
 
-## QTap basic events
+## QTap summary events
 
-### Event: `'error'`
+These are emitted at most once for the run overall.
 
-* `error <Error|string>`
+### Event: `'clients'`
 
-### Event: `'finish'`
+The `clients` event conveys which browsers are being started, and which tests will be run. It is emitted as soon as QTap has validated the parameters. Each client is a browser process that runs one test suite. For example, if you run 2 test suites in 3 different browsers, there will be 6 clients.
 
-Summary event that is ideal for when you run one test suite in one browser, or if you otherwise don't need a break down of results by client.
+* `event.clients {Object<string,Object>}` Keyed by clientId
+  * `clientId {string}` An identifier unique within the current qtap process (e.g. `client_123`).
+  * `testFile {string}` Relative file path or URL (e.g. `test/index.html` or `http://localhost/test/`).
+  * `browserName {string}` Browser name, as specified in config or CLI (e.g. `firefox`).
+  * `displayName {string}` Browser pretty name (e.g. "Headless Firefox").
+
+### Event: `'error'`
 
-* `event.ok <boolean>` Aggregate status of each client's results. If any failed, this is false.
-* `event.exitCode <number>` Suggested exit code, 0 for success, 1 for failed.
-* `event.total <number>` Aggregated from `result` events.
-* `event.passed <number>` Aggregated from `result` events.
-* `event.failed <number>` Aggregated from `result` events.
-* `event.skips <array>` Carried from the first client that failed, or empty.
-* `event.todos <array>` Carried from the first client that failed, or empty.
-* `event.failures <array>` Carried from the first client that failed, or empty.
-* `event.bailout <false|string>` Carried from the first client that failed, or false.
+* `error {Error|string}`
 
-## QTap reporter events
+### Event: `'finish'`
 
-A client will emit each of these events only once, except `consoleerror` which may be emitted any number of times.
+Summary event that is ideal for when you run one test suite in one browser, or if you don't need a break down of results by client.
 
-### Event: `'client'`
+* `event.ok {boolean}` Aggregate status of each client's results. If any failed, this is false.
+* `event.exitCode {number}` Suggested exit code, 0 for success, 1 for failed.
+* `event.total {number}` Aggregated from `result` events.
+* `event.passed {number}` Aggregated from `result` events.
+* `event.failed {number}` Aggregated from `result` events.
+* `event.skips {array}` Carried from the first client that failed, or empty.
+* `event.todos {array}` Carried from the first client that failed, or empty.
+* `event.failures {array}` Carried from the first client that failed, or empty.
+* `event.bailout {false|string}` Carried from the first client that failed, or false.
 
-The `client` event is emitted when a client is created. A client is a dedicated browser instance that runs one test suite. For example, if you run 2 test suites in 3 different browsers, there will be 6 clients.
+## QTap client events
 
-* `event.clientId <string>` An identifier unique within the current qtap process (e.g. `client_123`).
-* `event.testFile <string>` Relative file path or URL (e.g. `test/index.html` or `http://localhost/test/`).
-* `event.browserName <string>` Browser name, as specified in config or CLI (e.g. `firefox`).
-* `event.displayName <string>` Browser pretty name, (e.g. "Headless Firefox").
+These are emitted once per client, except `consoleerror` and `assert` which may be emitted many times by a client during a test run.
 
 ### Event: `'online'`
 
 The `online` event is emitted when a browser has successfully started and opened the test file. If a browser fails to connect, a `bail` event is emitted instead.
 
-* `event.clientId <string>`
+* `event.clientId {string}`
+* `event.testFile {string}`
+* `event.browserName {string}`
+* `event.displayName {string}`
 
 ### Event: `'result'`
 
 The `result` event is emitted when a browser has completed a test run. This is mutually exclusive with the `bail` event.
 
-* `event.clientId <string>`
-* `event.ok <boolean>`
-* `event.total <number>`
-* `event.passed <number>`
-* `event.failed <number>`
-* `event.skips <array>` Details about skipped tests (count as passed).
-* `event.todos <array>` Details about todo tests (count as passed).
-* `event.failures <array>` Details about failed tests.
+* `event.clientId {string}`
+* `event.ok {boolean}`
+* `event.total {number}`
+* `event.passed {number}`
+* `event.failed {number}`
+* `event.skips {array}` Details about skipped tests (count as passed).
+* `event.todos {array}` Details about todo tests (count as passed).
+* `event.failures {array}` Details about failed tests.
 
 ### Event: `'bail'`
 
-The `bail` event is emitted when a browser was unable to start or complete a test run.
+The `bail` event is emitted when a browser was unable to start or unable to complete a test run.
 
-* `event.clientId <string>`
-* `event.reason <string>`
+* `event.clientId {string}`
+* `event.reason {string}`
 
 ### Event: `'consoleerror'`
 
 The `consoleerror` event relays any warning or error messages from the browser console. These are for debug purposes only, and do not indicate that a test has failed. A complete and successful test run, may nonetheless print warnings or errors to the console.
 
 It is recommended that reporters only display console errors if a test run failed (i.e. there was a failed test result, or the cilent bailed).
 
-* `event.clientId <string>`
-* `event.message <string>`
+* `event.clientId {string}`
+* `event.message {string}`
+
+### Event: `'assert'`
+
+The `assert` event describes a single test result (whether passing or failing). This can be used by reporters to indicate activity, display the name of a test in real-time, or to convey failures early.
+
+* `event.clientId {string}`
+* `event.result {Object}`
+  * `ok {boolean}`
+  * `fullname {string}`
+  * `diag {undefined|Object}`

ARCHITECTURE.md

@@ -436,7 +436,7 @@ Safari has long resisted the temptation to offer a reasonable command-line inter
   {"value":null}
   ```
 
-  This addresses all previous concerns, and seems to be the best as of 2025. The only downside is that it requires a bit more code to setup (find available port, and perform various HTTP requests).
+  This addresses all previous concerns, and seems to work best as of 2025. The only downside is that it requires more code to set up (find available port, and perform various HTTP requests).
 
   - https://webkit.org/blog/6900/webdriver-support-in-safari-10/
   - https://developer.apple.com/documentation/webkit/macos-webdriver-commands-for-safari-12-and-later

eslint.config.js

@@ -25,7 +25,7 @@ export default [
     }
   },
   {
-    files: ['test/*.js'],
+    files: ['test/**/*.js'],
     languageOptions: {
       globals: {
         QUnit: 'readonly'

src/browsers.js

@@ -118,7 +118,7 @@ async function firefox (url, signals, logger, debugMode) {
   const profileDir = LocalBrowser.makeTempDir(signals, logger);
   const args = [url, '-profile', profileDir, '-no-remote', '-wait-for-browser'];
   if (!debugMode) {
-    firefox.displayName = 'Headless Firefox';
+    firefox.displayName = 'Firefox Headless';
     args.push('-headless');
   }
 
@@ -164,7 +164,7 @@ firefox.displayName = 'Firefox';
 function makeChromium (displayName, getPaths) {
   /** @type {Browser} - https://github.com/microsoft/TypeScript/issues/22063 */
   const chromium = async function (url, signals, logger, debugMode) {
-    chromium.displayName = debugMode ? displayName : `Headless ${displayName}`;
+    chromium.displayName = debugMode ? displayName : `${displayName} Headless`;
     // https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md
     const dataDir = LocalBrowser.makeTempDir(signals, logger);
     const args = [

src/client.cjs

@@ -4,10 +4,12 @@
 function qtapClientHead () {
   // Support QUnit 2.24+: Enable TAP reporter, declaratively.
   window.qunit_config_reporters_tap = true;
+  window.qunit_config_reporters_html = false;
 
   // Cache references to original methods, to avoid getting trapped by mocks (e.g. Sinon)
   var setTimeout = window.setTimeout;
   var XMLHttpRequest = window.XMLHttpRequest;
+  var createTextNode = document.createTextNode && document.createTextNode.bind && document.createTextNode.bind(document);
 
   // Support IE 9: console.log.apply is undefined.
   // Don't bother with Function.apply.call. Skip super call instead.
@@ -67,6 +69,7 @@ function qtapClientHead () {
   function createBufferedWrite (url) {
     var buffer = '';
     var isSending = false;
+    var debugElement = false;
     function send () {
       var body = buffer;
       buffer = '';
@@ -81,8 +84,16 @@ function qtapClientHead () {
       };
       xhr.open('POST', url, true);
       xhr.send(body);
+
+      // Optimization: Only check this once, during the first send
+      if (debugElement === false) {
+        debugElement = document.getElementById('__qtap_debug_element') || null;
+      }
+      if (debugElement) {
+        debugElement.appendChild(createTextNode(body));
+      }
     }
-    return function write (str) {
+    return function writeTap (str) {
       buffer += str + '\n';
       if (!isSending) {
         isSending = true;

src/qtap.js

@@ -8,6 +8,7 @@ import util from 'node:util';
 import browsers from './browsers.js';
 import reporters from './reporters.js';
 import { ControlServer } from './server.js';
+import { BrowserStopSignal } from './util.js';
 
 /**
  * @typedef {Object} Logger
@@ -102,14 +103,16 @@ function makeLogger (defaultChannel, printVerbose, verbose = false) {
  * @return {EventEmitter}
  */
 function run (files, browserNames = 'detect', runOptions = {}) {
-  if (typeof files === 'string') files = [files];
-  if (typeof browserNames === 'string') browserNames = [browserNames];
   if (!files || !files.length) {
     throw new Error('Must pass one or more test files to run');
   }
   if (!browserNames || !browserNames.length) {
     throw new Error('Must pass one or more browser names or omit for the default');
   }
+  // Remove duplicates by using a Set
+  // TODO: Add test to verify duplicates are ignored
+  files = Array.from(new Set(typeof files === 'string' ? [files] : files));
+  browserNames = Array.from(new Set(typeof browserNames === 'string' ? [browserNames] : browserNames));
   const options = {
     cwd: process.cwd(),
     idleTimeout: 5,
@@ -124,11 +127,38 @@ function run (files, browserNames = 'detect', runOptions = {}) {
   const logger = makeLogger('qtap_main', options.printVerbose, options.verbose);
   const eventbus = new EventEmitter();
   const globalController = new AbortController();
+  const globalSignal = globalController.signal;
 
   if (options.reporter) {
     if (options.reporter in reporters) {
       logger.debug('reporter_init', options.reporter);
-      reporters[options.reporter](eventbus);
+
+      // Create a safe event listener object, which will:
+      // - Catch any errors from reporter functions,
+      //   so that emit() is safe in our internal code.
+      // - Prevent reporter functions from tampering with the internal eventbus
+      //   object (can only call "on", not "emit"; cannot break "on" for other reporters)
+      //   to protect integrity of QTap, and other reporters.
+      // - Print detailed errors in verbose mode. Any thrown erorrs here are likely
+      //   internal to QTap and not reported in detail to users by default.
+      // - Re-throw as util.BrowserStopSignal so that ControlServer#launchBrowser
+      //   won't retry/rerun tests due to an internal error (likely deterministic).
+      //
+      // TODO: Write a test observing that reporter errors bail quickly, cleanly, and without retries.
+      const safeEventConsumer = {
+        on (event, fn) {
+          eventbus.on(event, function () {
+            try {
+              fn.apply(null, arguments);
+            } catch (e) {
+              logger.warning('reporter_caught', e);
+              // @ts-ignore - TypeScript @types/node lacks `Error(,options)`
+              throw new BrowserStopSignal(`The "${options.reporter}" reporter encountered an error in the "${event}" event handler.` + (!options.verbose ? ' Run with --verbose to output a stack trace.' : ''), { cause: e });
+            }
+          });
+        }
+      };
+      reporters[options.reporter](safeEventConsumer);
     } else {
       logger.warning('reporter_unknown', options.reporter);
     }
@@ -160,7 +190,6 @@ function run (files, browserNames = 'detect', runOptions = {}) {
         throw new Error(`Loading ${options.config} failed: ${String(e)}`, { cause: e });
       }
     }
-    const globalSignal = globalController.signal;
 
     const browerPromises = [];
     for (const browserName of browserNames) {
@@ -171,12 +200,43 @@ function run (files, browserNames = 'detect', runOptions = {}) {
       }
       browserFn.getDisplayName = () => browserFn.displayName || browserName;
       for (const server of servers) {
+        await server.proxyBasePromise;
         // Each launchBrowser() returns a Promise that settles when the browser exits.
         // Launch concurrently, and await afterwards.
         browerPromises.push(server.launchBrowser(browserFn, browserName, globalSignal));
       }
     }
 
+    // The 'clients' event must be emitted:
+    // * ... early, so that reporters can indicate the browser is starting.
+    //       Therefore, we must not await browerPromises until after this.
+    // * ... exactly once, regardless of launch retries.
+    // * ... after any custom display name is assigned to browserFn.displayName,
+    //       which may happens dynamically in the browserFn() call before any async logic.
+    //       This is used for example by the "detect" browser (to display to the selected browser),
+    //       and by the BrowserStack plugin (to expand strings like "firefox_latest").
+    //       Therefore, server.launchBrowser and server.launchBrowserAttempt must have
+    //       no async logic before it calls browserFn, as otherwise we'd emit it too soon.
+    // eventbus.emit('clients', {
+    //   clientId,
+    //   testFile: this.testFile,
+    //   browserName,
+    //   displayName: browserFn.getDisplayName(),
+    // });
+    const clients = {};
+    for (const server of servers) {
+      for (const browser of server.browsers.values()) {
+        clients[browser.clientId] = {
+          clientId: browser.clientId,
+          testFile: server.testFile + server.testFileQueryString,
+          browserName: browser.browserName,
+          displayName: browser.getDisplayName(),
+        };
+      }
+    }
+    logger.debug('clients_event', clients);
+    eventbus.emit('clients', { clients: clients });
+
     const finish = {
       ok: true,
       exitCode: 0,
@@ -209,27 +269,42 @@ function run (files, browserNames = 'detect', runOptions = {}) {
       }
     });
 
-    // Wait for all tests and browsers to finish/stop, regardless of errors thrown,
-    // to avoid dangling browser processes.
+    // Upon the first of any test server of browser error (i.e. file not found),
+    // tell other test servers to stop their browsers early.
+    let firstError;
+    try {
+      await Promise.all(browerPromises);
+    } catch (e) {
+      firstError = e;
+      for (const server of servers) {
+        // @ts-ignore - TypeScript @types/node lacks `Error(,options)`
+        server.stopBrowsers(new Error('Cancelled because another test bailed', { cause: e }));
+      }
+    }
+    // Re-await for clean shutdown, including for other failed servers
     await Promise.allSettled(browerPromises);
-
-    // Re-await, this time letting the first of any errors bubble up.
-    for (const browerPromise of browerPromises) {
-      await browerPromise;
+    // Let the first error bubble up
+    if (firstError) {
+      throw firstError;
     }
 
+    logger.debug('finish_event');
     eventbus.emit('finish', finish);
   })();
   runPromise
     .finally(() => {
+      logger.debug('runpromise_finally');
       for (const server of servers) {
         server.close();
       }
 
+      // Avoid dangling browser processes, in case we're here because
+      // one of the browerPromise bailed out early.
       logger.debug('shared_cleanup', 'Invoke global signal to clean up shared resources');
       globalController.abort();
     })
     .catch((error) => {
+      logger.debug('runpromise_catch');
       // Node.js automatically ensures users cannot forget to listen for the 'error' event.
       // For this reason, runWaitFor() is a separate method, because that converts the
       // 'error' event into a rejected Promise. If we created that Promise as part of run()