Manages cross-distribution dependencies better

[potiuk]

This PR addresses the case where we have to manage dependencies between Airfow distributions and we have to manage the right lower bind versions for them.

Since we missed a comprehensive documentation on how our dependencies are managed in general, and we had no good place to describe it, this PR not only adds code to manage it, but also extends the documentation of the "12_airflw_dependencies_and_extras.rst" chapter of the contributing documentation to include comprehensive (but also hopefully helpful) documentation explaining to contributors how our dependencies are managed.

The changes in the doc:

• add missing documentation explaining dependency management in airflow distributions in the monorepo

• adds selective check to verify if cross-airflow-distribution dependencies have not been modified in the PR, and fails if they were, unless appropriate label is set on the PR

• adds breeze tests covering the test cases of selective check

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[potiuk]

Hey @amoghrajesh @xBis7 @eladkal @bugraoz93 @ashb -> You have recently been involved in modifying such cross-distribution dependencies - including shared distributions. I think we missed a bit comprehensive documentation on how our dependencies (especially cross-airflow-distribution) dependencies are managed in Airflow and it caused a number of back-forth. So I decided - in this one - not only to add protection against accidentally modifying the min-versions for cros-airflow-distributions by contributors, but also - in orded to properly explain it - to add more complete explanation on how those dependencies are managed - with all the nuances and caveats.

This is close to main brain-dump on all the work on dependencies and providers, done over the last year or so, down to the level that I think might be useful to explain some basic assumptions and how it works - including explaining the parts that are automated, those that shoudl be done manually and some more specific edge cases and reasoning for them.

I - of course - do not expect every contributor to read them, but this might be a good reference to send someone to when they are asking how things work.

Have fun revieing, any comments on how to make it better are most welcome.

[ashb]

Docs are longer than I'd like, but make sense.

[potiuk]

Docs are longer than I'd like, but make sense.

The thing is that AI actually likes detailed context. This documentation is also for AI agents to understand what they should do by reading the documentation. Again - i do not expect people to read them like a book, more like trying to find what they are looking for - but I do 100% expect it will be read by Gen-AI tools in it's entirety to build the right context for whatever they will be doing.

[potiuk]

Applied @ashb comments.

[potiuk]

cc: @kacpermuda -> I have not tagged you on this one -> but it basically implements 150% of what we discussed. in #58393 (comment)

[potiuk]

Docs are longer than I'd like, but make sense.

The thing is that AI actually likes detailed context. This documentation is also for AI agents to understand what they should do by reading the documentation. Again - i do not expect people to read them like a book, more like trying to find what they are looking for - but I do 100% expect it will be read by Gen-AI tools in it's entirety to build the right context for whatever they will be doing.

Speaking of which... When I realized and checked how Agents work, I kind of started to see why I added such a detailed documentation for all things in our env for years. Thanks to this I do not even have to ask Cloude how to run the tests or build breeze image and run things sometimes - it just knows by reading our docs and I do not have to explain it in my prompts.

It happened multiple times when I was using agents, that they added tests and iterated on them and created a number of them following the structure and description of our test environment without me even asking for it.

[bugraoz93]

Look good to me! I totally agree that not many contributors will read this. Since the documents are public, someone can summarise them easily. Selective checks for the providers are definitely a better place for this check rather than prek. We don't need selective checks for other distributions since they are easy to follow as long as RMs are CodeOwners on those repositories.
It would be beneficial to have a list of who can be tagged for which distribution as an RM list, perhaps. It would be easier to check and ping the correct people for Commiters, PMCs, but mostly contributors. It would also be informative in general and serve as an Onboarding piece for new members

[potiuk]

Look good to me! I totally agree that not many contributors will read this. Since the documents are public, someone can summarise them easily. Selective checks for the providers are definitely a better place for this check rather than prek. We don't need selective checks for other distributions since they are easy to follow as long as RMs are CodeOwners on those repositories. It would be beneficial to have a list of who can be tagged for which distribution as an RM list, perhaps. It would be easier to check and ping the correct people for Commiters, PMCs, but mostly contributors. It would also be informative in general and serve as an Onboarding piece for new members

I think just #releae-management is good enough (I mentioned it in the description) - I guess keeping list of RMs separately is less-than-needed if everyone is in #release-management channel. Those are rather "rare" events to handle and I guess there is little issue with whole #release-management being notified (especially also for redundancy reasons, vacations and so on).

[bugraoz93]

Look good to me! I totally agree that not many contributors will read this. Since the documents are public, someone can summarise them easily. Selective checks for the providers are definitely a better place for this check rather than prek. We don't need selective checks for other distributions since they are easy to follow as long as RMs are CodeOwners on those repositories. It would be beneficial to have a list of who can be tagged for which distribution as an RM list, perhaps. It would be easier to check and ping the correct people for Commiters, PMCs, but mostly contributors. It would also be informative in general and serve as an Onboarding piece for new members

I think just #releae-management is good enough (I mentioned it in the description) - I guess keeping list of RMs separately is less-than-needed if everyone is in #release-management channel. Those are rather "rare" events to handle and I guess there is little issue with whole #release-management being notified (especially also for redundancy reasons, vacations and so on).

True, make sense!

[github-actions]

Backport successfully created: v3-1-test

Status	Branch	Result
✅	v3-1-test