Brew updates halted by outdated Xcode version on Macos11 machines

[Haroon-Khel]

Brew update wants Xcode to be updated to >= 13.2.1 to be able to perform updates on installed packages.

test-macstadium-macos11-arm64-1

"msg": "Warning: You are using macOS 11.\nWe (and Apple) do not provide support for this old version.\n
It is expected behaviour that some formulae will fail to build in this old version.\nIt is expected behaviour that Homebrew will be buggy and slow.\nDo not create any issues about this on Homebrew's GitHub repositories.\n
Do not create any issues even if you think this message is unrelated.\nAny opened issues will be immediately closed without response.\nDo not ask for help from Homebrew or its maintainers on social media.\n
You may ask for help in Homebrew's discussions but are unlikely to receive a response.\n
Try to figure out the problem yourself and submit a fix as a pull request.\nWe will review it but may or may not accept it.\n
\nError: Your Xcode (11.7 => /Applications/Xcode-11.7.app/Xcode.app/Contents/Developer) is too outdated.\n
Please update to Xcode 13.2.1 (or delete it).\nXcode can be updated from the App Store.\n
\nError: Your Command Line Tools are too outdated.\nUpdate them from Software Update in System Preferences.\n
\nIf that doesn't show you any updates, run:\n
  sudo rm -rf /Library/Developer/CommandLineTools\n  sudo xcode-select --install\n\nAlternatively, manually download them from:\n  https://developer.apple.com/download/all/.\nYou should download the Command Line Tools for Xcode 13.2.1.",

build-macstadium-macos11-arm64-1

"msg": "Warning: You are using macOS 11.\nWe (and Apple) do not provide support for this old version.\n
It is expected behaviour that some formulae will fail to build in this old version.\n
It is expected behaviour that Homebrew will be buggy and slow.\nDo not create any issues about this on Homebrew's GitHub repositories.\n
Do not create any issues even if you think this message is unrelated.\nAny opened issues will be immediately closed without response.\nDo not ask for help from Homebrew or its maintainers on social media.\n
You may ask for help in Homebrew's discussions but are unlikely to receive a response.\n
Try to figure out the problem yourself and submit a fix as a pull request.\nWe will review it but may or may not accept it.\n\nError: Your Command Line Tools are too outdated.\n
Update them from Software Update in System Preferences.\n\nIf that doesn't show you any updates, run:\n
  sudo rm -rf /Library/Developer/CommandLineTools\n  sudo xcode-select --install\n
\nAlternatively, manually download them from:\n  https://developer.apple.com/download/all/.\n
You should download the Command Line Tools for Xcode 13.2.1.",

Two separate but similar error logs. The first suggests we need to add something to the macos part of the playbooks which switch xcode back to command line tools before running any macos ansible tasks. The second error log recommends updating command line tools' xcode to >= 13.2.1

[Haroon-Khel]

Both error logs mention that Macos11 is also outdated but at the moment it is only a warning not an error, but this should be taken into account when we get some new machines #2536

[Haroon-Khel]

Attempting to install command line tools 13.3 on test-macstadium-macos11-arm64-1

administrator@test-macstadium-macos11-arm64-1 ~ % sudo installer -pkg "/Volumes/Command Line Developer Tools/Command Line Tools.pkg" -target /
installer: Cannot install on volume / because it is disabled.
installer: macOS version 12 or later is required.

[Haroon-Khel]

According to https://xcodereleases.com/ xcode 13.2.1 is compatible with macos11.3+. Our macos machines are 11.0 and 11.0.1. But there is an update available on our macos11 machines which would upgrade them to 11.7

administrator@test-macstadium-macos11-arm64-1 ~ % softwareupdate -l
Software Update Tool

* Label: macOS Big Sur 11.7.10-20G1427
	Title: macOS Big Sur 11.7.10, Version: 11.7.10, Size: 6155086K, Recommended: YES, Action: restart

Xcode13.2.1 would then be installed on the upgraded machines (hopefully)

@sxa @andrew-m-leonard Would upgrading to 11.7 affect reproducibility?

[Haroon-Khel]

(Accidentally closed the issue)

[andrew-m-leonard]

According to https://xcodereleases.com/ xcode 13.2.1 is compatible with macos11.3+. Our macos machines are 11.0 and 11.0.1. But there is an update available on our macos11 machines which would upgrade them to 11.7

administrator@test-macstadium-macos11-arm64-1 ~ % softwareupdate -l
Software Update Tool

* Label: macOS Big Sur 11.7.10-20G1427
	Title: macOS Big Sur 11.7.10, Version: 11.7.10, Size: 6155086K, Recommended: YES, Action: restart

Xcode13.2.1 would then be installed on the upgraded machines (hopefully)

@sxa @andrew-m-leonard Would upgrading to 11.7 affect reproducibility?

@Haroon-Khel As long as we reproduce using the same Xcode we're good in theory. I think moving to osx 11.7 makes sense, thanks Haroon

[sxa]

@andrew-m-leonard Yeah, that's true for new builds but since the point of this is to upgrade the code version we'd break the ability to do a reproduce of the existing ones in the field wouldn't we?

[andrew-m-leonard]

@andrew-m-leonard Yeah, that's true for new builds but since the point of this is to upgrade the code version we'd break the ability to do a reproduce of the existing ones in the field wouldn't we?

yes, but I guess we're always going to have that when we upgrade tooling, VS, Xcode, gcc,...?
For old release reproducibility, we need to use the same tooling, hence implies we'd need to cache our old tooling?
I think this raises an interesting question, as to how long ago we go back to support re-building? as that at some point becomes difficult and requires keeping old OS's etc..?

[sxa]

yes, but I guess we're always going to have that when we upgrade tooling, VS, Xcode, gcc,...?

With gcc we install each version into a different directory. There is an issue of WHEN we do such upgrades here. If it's just "whenever brew feels like" that's quite a significant risk and feels like an invitation to break reproducibility of existing releases. All of the other platforms use a fixed level at the moment.

Related: #2966