Tests are failing

[hraban]

Using latest ndebug, and loading all dependencies using latest quicklisp, I get the following errors:

CL-USER> (asdf:test-system :ndebug)
; compiling file "/Users/user/code/common-lisp/ndebug/tests/package.lisp" (written 03 JUN 2023 01:24:28 AM):

; wrote /Users/user/.cache/common-lisp/sbcl-2.3.5.nixos-macosx-arm64/Users/user/code/common-lisp/ndebug/tests/package-tmpGA31XWNO.fasl
; compilation finished in 0:00:00.007
; compiling file "/Users/user/code/common-lisp/ndebug/tests/tests.lisp" (written 03 JUN 2023 01:24:28 AM):

; wrote /Users/user/.cache/common-lisp/sbcl-2.3.5.nixos-macosx-arm64/Users/user/code/common-lisp/ndebug/tests/tests-tmp5HT23M6C.fasl
; compilation finished in 0:00:00.016

------- STARTING Testing: NDEBUG/TESTS 

Starting: NDEBUG/TESTS::DEBUGGER-HANDLER-BIND
NDEBUG/TESTS::DEBUGGER-HANDLER-BIND - ERRORS (0.00s) : 2 assertions passed
  | ERRORS (1)
  | ERROR: The file
  |        #P"/var/folders/t0/t7hdj3915fbcz6r2ldrp6w580000gn/T/tmpA60KLDKY.tmp"
  |        already exists:
  |          File exists
  | #<FILE-EXISTS {70086DA323}>
  |
  |
Starting: NDEBUG/TESTS::MULTITHREADED
NDEBUG/TESTS::MULTITHREADED - ERRORS (0.00s) : 0 assertions passed
  | ERRORS (1)
  | ERROR: The value
  |          "hello"
  |        is not of type
  |          NUMBER
  | #<TYPE-ERROR expected-type: NUMBER datum: "hello">
  |
  |
Starting: NDEBUG/TESTS::WITH-DEBUGGER-HOOK-EXPANSION
NDEBUG/TESTS::WITH-DEBUGGER-HOOK-EXPANSION - ERRORS (0.00s) : 0 assertions passed
  | ERRORS (1)
  | ERROR: The value
  |          "hello"
  |        is not of type
  |          NUMBER
  | #<TYPE-ERROR expected-type: NUMBER datum: "hello">
  |
  |
Starting: NDEBUG/TESTS::CLASS-BASED-DEBUGGING
NDEBUG/TESTS::CLASS-BASED-DEBUGGING - ERRORS (0.00s) : 0 assertions passed
  | ERRORS (1)
  | ERROR: The value
  |          "hello"
  |        is not of type
  |          NUMBER
  | #<TYPE-ERROR expected-type: NUMBER datum: "hello">
  |
  |
Starting: NDEBUG/TESTS::CLASS-BASED-DEBUGGING-OVERRIDEN
NDEBUG/TESTS::CLASS-BASED-DEBUGGING-OVERRIDEN - ERRORS (0.00s) : 0 assertions passed
  | ERRORS (1)
  | ERROR: The value
  |          "hello"
  |        is not of type
  |          NUMBER
  | #<TYPE-ERROR expected-type: NUMBER datum: "hello">
  |
  |
Starting: NDEBUG/TESTS::TRY-EVALUATE
NDEBUG/TESTS::TRY-EVALUATE - ERRORS (0.01s) : 3 assertions passed
  | ERRORS (1)
  | ERROR: The file
  |        #P"/var/folders/t0/t7hdj3915fbcz6r2ldrp6w580000gn/T/tmpD801X3GJ.tmp"
  |        already exists:
  |          File exists
  | #<FILE-EXISTS {7008D98323}>
  |
  |
Test Summary for :NDEBUG/TESTS (6 tests 0.02 sec)
  | 5 assertions total
  | 5 passed
  | 0 failed
  | 6 execution errors
  | 0 warnings
  | 0 empty
  | 0 missing tests

-------   ENDING Testing: NDEBUG/TESTS 
; 
; compilation unit aborted
;   caught 1 fatal ERROR condition
; Evaluation aborted on #<LISP-UNIT2:ALL-TESTS-COMPLETE {7008D99DF3}>.

[aartaka]

So that's a lisp-unit2 problem: it fails the tests when there's an error thrown inside the test. Even if it was handled.

Tests fail on our Github CI too, while still being useful and passing (with the same set of errors) locally.

The best course of action would be to find a way for lisp-unit2 to be less intrusive about errors. I've no idea about where to start this from :/