dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827
Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution
Pull request: #39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827
Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1200670	dotnet/sdk	Run TestBuild Tests	Log	#51458
1200643	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51419
1200488	dotnet/sdk	Run FullFramework Tests	Log	#51250
1200482	dotnet/sdk	Run TestBuild Tests	Log	#51458
1200151	dotnet/sdk	Run TestBuild Tests	Log	#51654
1200150	dotnet/sdk	Run TestBuild Tests	Log	#51653
1200055	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51223
1199717	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51419
1199521	dotnet/sdk	Run FullFramework Tests	Log	#51611
1199524	dotnet/sdk	Run TestBuild Tests	Log	#51606
1199487	dotnet/sdk	Run TestBuild Tests	Log	#51643
1199032	dotnet/sdk	Run TestBuild Tests	Log	#51535
1198942	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51307
1198509	dotnet/sdk	Run FullFramework Tests	Log	#51611
1196704	dotnet/sdk	Run FullFramework Tests	Log	#51250
1196621	dotnet/sdk	Run FullFramework Tests	Log	#51606
1196912	dotnet/sdk	Run FullFramework Tests	Log	#51611
1198283	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51565
1197665	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51624
1197534	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51624
1197365	dotnet/sdk	Run FullFramework Tests	Log	#51621
1196055	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1194776	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51462
1194360	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51545
1194313	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51376
1194101	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51566
1193887	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51509
1193461	dotnet/sdk	Run FullFramework Tests	Log	#51555
1192856	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51376
1192854	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51402
1192609	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50895
1192712	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51538
1192207	dotnet/sdk	Run FullFramework Tests	Log	#51532
1192271	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51536
1192205	dotnet/sdk	Run TestBuild Tests	Log	#51533
1191468	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51207
1190761	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51516
1190484	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1190086	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1189987	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1189903	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1189177	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
1189058	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51207
1189056	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50468
1188901	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51489
1188640	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51462
1188495	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51462
1187280	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50685
1188045	dotnet/sdk	🟣 Run FullFramework Tests	Log	#50468
1187980	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1187790	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51473
1187600	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51373
1187258	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51216
1187261	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51456
1187226	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1187158	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1187224	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51402
1187000	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51462
1187012	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51373
1187015	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50685
1186342	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1186167	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1186161	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1185327	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51440
1185317	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51450
1185205	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51414
1185150	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51445
1184460	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49409
1184141	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1184133	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51376
1184089	dotnet/sdk	🟣 Run TestBuild Tests	Log	#49409
1184095	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1184102	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51441
1184031	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51439
1183888	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51414
1183738	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51430
1183693	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50988
1183118	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1183116	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51428
1182891	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51207
1182892	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1182893	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51426
1182766	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51425
1182446	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51418
1182320	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51414
1182270	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51413
1181664	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51207
1181698	dotnet/sdk	Run TestBuild Tests	Log	#51408
1181220	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51402
1180881	dotnet/sdk	🟣 Run FullFramework Tests	Log	#51207
1180689	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51385
1180843	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51387
1180825	dotnet/sdk	Run TestBuild Tests	Log	#51394
1180676	dotnet/sdk	Run FullFramework Tests	Log	#51382
1180480	dotnet/sdk	🟣 Run TestBuild Tests	Log	#51343
2820605	dotnet-sdk	Run TestBuild Tests	Log	#54515
1180243	dotnet/sdk	Run TestBuild Tests	Log	#51369
2820476	dotnet-sdk	Run FullFramework Tests	Log	#54515
1180067	dotnet/sdk	Run TestBuild Tests	Log	#51365
1179803	dotnet/sdk	🟣 Run TestBuild Tests	Log	#50468
Displaying 100 of 149 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	23	149

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR:
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: #41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR:
#39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.