Improve test runner ergonomics

[wingo]

Add ./run-tests executable that doesn't require arguments, automatically detecting available runtimes and tests.

Simplify README.md accordingly.

Console test case reporter more concise and useful:

• Single line to indicate total results
• One copy-pastable command line per failure
• Requires refactors to record command lines, remove cwd change

Allow testing multiple engines in one run.

• Required changing TEST_RUNTIME_EXE environment variable convention to one env var per executable
• Requires refactor to collect which runtime ran a testcase

[wingo]

Rebased on top of #151.

[wingo]

So. Before, you would run this:

TEST_RUNTIME_EXE="$HOME/src/wasip3/wasmtime/target/release/wasmtime -Wcomponent-model-async=y -Sp3=y" \
  python3 test-runner/wasi_test_runner.py \
    -t tests/assemblyscript/testsuite/wasm32-wasip1 \
       tests/c/testsuite/wasm32-wasip1 \
       tests/rust/testsuite/wasm32-wasip{1,3} \
    -r adapters/wasmtime.py

And you would get output like this: before.txt

After this change, you would have to specify the Wasmtime runtime via the WASMTIME variable:

WASMTIME="$HOME/src/wasip3/wasmtime/target/release/wasmtime -Wcomponent-model-async=y -Sp3=y" \
  python3 test-runner/wasi_test_runner.py \
    -t tests/assemblyscript/testsuite/wasm32-wasip1 \
       tests/c/testsuite/wasm32-wasip1 \
       tests/rust/testsuite/wasm32-wasip{1,3} \
    -r adapters/wasmtime.py

And you get similar output, though there are section headings printed as we run tests, and the synopsis is shorter and shows the commands to re-run: after-wasi-test-runner.txt

But the neat thing is that after, you can just run this:

WASMTIME="$HOME/src/wasip3/wasmtime/target/release/wasmtime -Wcomponent-model-async=y -Sp3=y" \
  ./run-tests

That gives you similar output, though it prints some info indicating how it detects runtimes in the beginning:
after-run-tests.txt

In future I would like for run-tests to be more terse as it goes, perhaps printing just a dot per test or something, so that everything stays on a page. We can add --verbose or something to print details as we go, for CI porpoises.

WDYT @alexcrichton @pchickey @loganek ?

In future I would like to:

• Remove the fork/exec around invoking adapters: the adapter should return an argv directly, that way re-invoking a failing test can be done directly instead of popping through python
• Make run-tests more terse
• Possibly include build-tests or something (No documented way in README to edit a test and run it #127)
• Add a wasi version to manifest.json, so that the runtime adapters can choose different argv for different wasi versions
• Add a jco adapter
• Add CI to publish test status; can be a dynamic page that pulls the nightly json data directly from the repo
• Run tests against some engine as part of CI

[alexcrichton]

That looks great to me! I like having copy/pastable commands at the end and I also like the aspiration of "here's the command for running the runtime directly" to get printed eventually as that's exactly what I'd be looking for when debugging things.

[wingo]

Added on a patch so that run-tests is quiet by default:

$ WASMTIME="$HOME/src/wasip3/wasmtime/target/release/wasmtime -Wcomponent-model-async=y -Sp3=y" ./run-tests
Detecting WASI runtime availability:
  wasmtime.py: wasmtime 38.0.0
  wasm-micro-runtime.py: unavailable; pass `--runtime /home/wingo/src/wasip3/wasi-testsuite/adapters/wasm-micro-runtime.py` to debug.
  wizard.py: unavailable; pass `--runtime /home/wingo/src/wasip3/wasi-testsuite/adapters/wizard.py` to debug.

Running test suite WASI C tests [wasm32-wasip1] with wasmtime 38.0.0
...........!..

Running test suite WASI Rust tests [wasm32-wasip3] with wasmtime 38.0.0
!!.
Running test suite WASI Rust tests [wasm32-wasip1] with wasmtime 38.0.0
.................!....!........!..............
Running test suite WASI Assemblyscript  tests [wasm32-wasip1] with wasmtime 38.0.0
............
===== Test results =====
wasmtime 38.0.0: FAIL: 6/75 tests failed
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/c/testsuite/wasm32-wasip1/pwrite-with-append.wasm --dir /home/wingo/src/wasip3/wasi-testsuite/tests/c/testsuite/wasm32-wasip1/fs-tests.dir::fs-tests.dir
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip3/http-fields.wasm
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip3/test-stat-at-root.wasm
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/remove_directory_trailing_slashes.wasm --dir /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/fs-tests.dir::fs-tests.dir --arg fs-tests.dir
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/path_link.wasm --dir /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/fs-tests.dir::fs-tests.dir --arg fs-tests.dir
  /usr/bin/python3 /home/wingo/src/wasip3/wasi-testsuite/adapters/wasmtime.py --test-file /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/fd_fdstat_set_rights.wasm --dir /home/wingo/src/wasip3/wasi-testsuite/tests/rust/testsuite/wasm32-wasip1/fs-tests.dir::fs-tests.dir --arg fs-tests.dir

Also there is color on a console.