Report actual argv on errors #152
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add `./run-tests` executable that doesn't require arguments, automatically detecting available runtimes and tests. Simplify `README.md` accordingly. Console test case reporter more concise and useful: - Single line to indicate total results - One copy-pastable command line per failure - Requires refactors to record command lines, remove cwd change Allow testing multiple engines in one run. - Required changing `TEST_RUNTIME_EXE` environment variable convention to one env var per executable - Requires refactor to collect which runtime ran a testcase
Don't go beyond a page or so of output.
Instead of having the adapter files be top-level python scripts, now they are modules with exported functions. One of those functions computes the argv to execute when running a test. Also, clean up directories before and after each test, in addition to before running the suite. This makes one failure to clean up not cause other cascading failures, and makes it more reliable to be able to run test cases outside the test-runner harness.
Collaborator
Author
|
With this change, output looks like this: |
Collaborator
Author
|
Cc @mohanson for the out-of-tree pywasm adapter; wdyt? |
Collaborator
Author
|
Need similar adaptations for out-of-tree wasmedge and wazero adapters |
Contributor
|
I see that in your refactoring design, by default you need the WASI implementation to provide a binary file, and then you can call it through subprocess to run a specified test case. However, pywasm currently doesn't have binary releases and is only published as a package. Therefore, does this mean I now need to provide a binary program to adapt to this refactoring (this should be easy to implement, but I want to confirm whether this is an expected change within the scope of this refactoring)?
|
Collaborator
Author
|
Can you run pywasm through python? E.g. You don't necessarily need to ship a binary. wasi-testsuite can contain the needed trampoline. I am not trying to make your life difficult, I just want wasi-testsuite to print out command lines that can be run directly :) |
Collaborator
Author
|
To finish the other thought I had, @mohanson it may be a good idea to just put the pywasm adapter into wasi-testsuite. As you like! |
Contributor
I like this idea too! |
Contributor
|
@wingo I've updated pywasm and provided an entry script. You can download this file (or include it directly in wasi-testsuite), then treat "python wasi.py" as the binary you need, and write an adapter.py file according to your new design. This should work properly. |
Collaborator
Author
|
OK! Merging this one then, and will add an adapter for pywasm in a followup. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Instead of having the adapter files be top-level python scripts, now
they are modules with exported functions. One of those functions
computes the argv to execute when running a test.
Also, clean up directories before and after each test, in addition to
before running the suite. This makes one failure to clean up not cause
other cascading failures, and makes it more reliable to be able to run
test cases outside the test-runner harness.
Stacked on #149.