Example usage with Github Actions (or an alternative CI)

[dominykas]

• Something to kick off wiby test (triggered off a comment?)
• Something to poll for results and report them back as a comment

[dominykas]

OK, I think this issue is on the critical path - the CLI approach is all nice, but wiby needs to happen in CI.

I tentatively opened this referencing Github Actions, but actions don't make this easy, so I'll need to do some digging.

Kicking off the initial wiby test is the easy part.

The hard part is to collect the results, because you can't trigger an action off a status change in another repo. Which means some sort of polling, but there's no obvious way of implementing that using actions either, or at least no obvious way which wouldn't use up the precious actions minutes.

There could be a scheduled which basically polls all the things that are in flight, but I'm struggling to imagine a way to make it easy to set up, as GH actions don't have a way of sharing workflows.

Neither is this easy on Travis - you'd essentially have to keep a job running until it times out or the dependents finish their jobs, which can be rather wasteful as well.

I'll ponder other available tools to achieve this, but ideas are most welcome!

[dominykas]

OK, so here's the plan:

1. For the initial version, there will not be an automated results check. I don't have a good solution to save on minutes. So you will have to use a comment to wiby test, which will kick off the tests, and it will also set a status check to pending. You will then have to explicitly comment to wiby result, which will loop through the tests that were kicked off and update the status check accordingly.
   • I will open an issue on GH side to add a "sleep" step (which wouldn't count against minutes) or to add a "delayed trigger" (which would explicitly kick off X minutes after Y event or X minutes after an explicit API call).
     • https://github.community/t/feature-request-sleep-wait-delay-function-which-does-not-count-against-minutes/143618
2. There will be several GH workflows involved, so I'll add a new command to sync up the canonical workflows. Not sure about the combo of commands - an interactive wiby install would be awesome, but this also needs a silent way to just update the options. Maybe a wiby outdated and wiby update make sense. I'm thinking to do this as part of wiby, but it probably makes sense to extract as a separate package as well, if the approach proves itself viable.

[dominykas]

Things not covered in #69, to be reconsidered if we want them for MVP:

• It seems the the details_url does not do what I want it to do... Basically, the status check under the PR links to the details about the status check, but somehow does not want to link to the actual build details.
• Ideally, we should report detailed results (the markdown that we now have) as a comment. We should think wether we should keep one comment per PR and update it with details, or whether we should just post a new comment after every wiby results run. (I do hate the wiby results comment pattern... but don't have any better ideas right now, that I can also implement fast, and not waste build minutes)
• The GH API seems to indicate there is some sort of a "Require action" button functionality - maybe it is possible to use that instead of the wiby results comment?
• Can labels be better triggers than wiby test / wiby result comments?
• Right now, the branch that wiby pushes to is based on the dependency name. We should do better than that and instead use the branch name of the dependency somewhere in there. This would allow linking multiple dependencies into one, e.g. you're testing dep-one and dep-two inside user-of-one-and-two. Both dep-one and dep-two have a new feature developed on a branch named feat/new-stuff. Running wiby test in dep-one would create a wiby-feat-new-stuff branch with dep-one linked into user-of-one-and-two. Running wiby test in dep-two afterwards should see that wiby-feat-new-stuff already exists and link dep-two, while keeping dep-one linked.
  • We should probably also always rebase on top of the default branch, i.e. if the branch exists, we should fetch the package.json, detect wiby-linked packages, and then apply that on top of master as well as the new changes.
• Improve status check reporting to set the "Pending" status immediately and have a separate status for wiby results check (indicating that the results check has completed).
• Add some code to install/update the GH workflow code in the repositories where it is used.

[ljharb]

i am very interested in this “require action” functionality, and have never come across it before.

[dominykas]

https://docs.github.com/en/rest/reference/checks - look for action_required - the docs weren't super clear to me and I haven't had time to experiment with this just yet... And then there's a mention of actions which allows "a button on GitHub that can be clicked to alert your app to do additional tasks", but it is unclear to me if you have to have an app or this is achievable for the workflows.

[ljharb]

ah, that reads to me as just a way to indicate to github that the user has to click the "details" link, and do something else. visual diffing uses this, for example, to force users to go click a button confirming that yes, this visual change was intended.

[dominykas]

Yeah, quite likely, although if that can be wired up to kicking off another job - that's all that's necessary. Or a readme telling to post a comment 🤷‍♂️

[dominykas]

Looks like Github only sets the COLLABORATOR relationship if you're added to the repo directly (seems like a known issue), but it does not create it when you're added via team (or by being an org admin). This means that the permission check for the comment is probably rather inefficient...

I guess we'll need to rewrite the comment trigger to instead be a label... which can kind of work for kicking off the wiby test (the job should probably remove the label when the test ends), but there's still a need for a trigger to check wiby result and I see no trivial solution (which does not waste minutes) to achieve this.

One option is to have a label for wiby in progress. The collaborator of the repo would then remove the label to check for results. If the tests are still in progress - the job would add the label back, or something. Not a pretty flow, if you ask me... Comments are nicer :(

I guess I could rework this to check the permissions via the API, even if it would cost some actions seconds...

🤔

[dominykas]

OK, the known issue with collaborators is still there, and it is a rather annoying one to work around, so I guess I'll try to reimplement the workflow with labels. Here's what I'm thinking right now:

• Someone opens a PR
• A collaborator adds a wiby test label
• This triggers the wiby workflow:
  • it checks the wiby status on the commit - it is missing, so it sets it to pending
  • it removes the wiby test label from the PR
    • it should maybe add the wiby in progress label instead, just to make this status visible the PR list?
  • it kicks off the wiby test command
• (time passes)
• A collaborator looks at the PR and re-adds the wiby test label
• This triggers the wiby workflow again:
  • it checks the wiby status on the commit and sees it is pending
  • it removes the wiby test label from the PR
  • it kicks off the wiby results command
  • if the results are in - it sets the wiby status to red or green on the commit
  • if the results are still pending - it keeps the pending status

Gotchas:

• This has much less extensibility than the comment option, unfortunately, but at least it's simple enough?
• Suppose the wiby status is pending, and downstream dependent builds are stuck for whatever reason - how does one kick off the wiby test again? I'm trying to avoid multiple different labels (i.e. I don't want a separate wiby retest or a wiby check results label).

Next steps:

• A nightly scheduled action, which scans all open PRs, checks their wiby status and wherever it is pending - adds the wiby test label to kick off the wiby results workflow