python - build abi3 wheels in cibuildwheel

[K0lb3]

This PR makes it possible to build abi3 wheels within a cibuildwheel environment.
To not effect the performance of version specific builds the C code changes are all guarded within preprocessor conditions.

This PR also modifies the setup.py, making it:

• abi3 compatible
• python 2 compatible again
• simplifies the extension build code a bit

TODO:

• add type stubs (currently don't get included in wheel)
• fix abi3 setup.py build issue (C code works, setup.py still has some issues)

[K0lb3]

It might be worth discussing if the abi3 wheels should start at 3.6 as it's now, or 3.10, which would then allow using the Buffer interface in the abi3 wheels as well.

[eustas]

Awesome, thanks. Will review soon.

[anthrotype]

Thanks for this! I'd love to not have to worry about building brotli wheels every year that a new python minor version gets out :)

This looks good, except for the issue I raised in my previous comments about the unnamed package without its own folder, which must be addressed before merge.

[K0lb3]

Thanks for this! I'd love to not have to worry about building brotli wheels every year that a new python minor version gets out :)

The repo for building and publishing the wheels needs a PR as well to make this work in the end,
as pypi might otherwise block the upload due to already existing wheels of the same name.
Alternatively the cibuildwheel config can also be done within the pyproject.toml, and added to this PR.

e.g. like following which I use in one of my projects

[tool.cibuildwheel.linux]
archs = ["x86_64", "i686", "aarch64", "ppc64le", "s390x", "armv7l"]
build = "cp37-* pp3*"

[tool.cibuildwheel.macos]
archs = ["x86_64", "arm64"]
build = "cp37-macosx_x86_64 cp38-macosx_arm64 pp3*"

[tool.cibuildwheel.windows]
archs = ["AMD64", "x86", "ARM64"]
build = "cp37-win_amd64 cp37-win32 cp39-win_arm64 pp3*"

The archs are all written separately to ensure that cibuildwheel covers all possible ones instead of just the default ones.
pypy dosn't support abi wheels, so wheels for all version of it still have to be build.
Arm OSX earliest release python build is 3.8, therefore the version to be build for it is cp38.
Arm Windows starts at 3.9, so it's configured as cp39.

As brotli could also support 3.6 abi, the config above would have to be slightly changed, as armv7l linux support is only official for cp37...as far as I can recall.

Edit:
It's on purpose that the config has no universal(2) build, as pip will use the more specific one.
e.g. if pip finds universal2 and x86_64, then it will pick x86_64. Therefore there isn't really a point in universal builds imo.

[anthrotype]

Alternatively the cibuildwheel config can also be done within the pyproject.toml, and added to this PR.

I'd prefer to keep the wheel building in the separate repository as things are currently, though @eustas may have different views.

[K0lb3]

@eustas @anthrotype
I wish you a happy new year.
A bit late, but still better than an empty bump ping :P

[K0lb3]

@eustas @anthrotype
It's been another month......nearly 4 months, a third of a year, since your last reaction.

I had to basically "wipe" all of my commits and re-add them on a fresh branch as the master branch got multiple changes that totally broke my fork. As that took a bit of time, I don't plan to do this again.

[anthrotype]

LGTM

thank you very much for this, and apologies for the very late review (I was away on paternity leave until today).

I'll let @eustas merge this as I don't have direct commit rights on this repo.

[K0lb3]

I'm sorry for my previous comment, it wasn't appropriate one way or another.

One of the changes that happened in between added another optional parameter to the process function of the decompressor class, which I added with my last commit some days ago.

[anthrotype]

hey @K0lb3 can you please fix the merge conflicts? thanks! /cc @eustas

[K0lb3]

@eustas I guess that you might have to review it again now....
I added the addition to the stubs and also improved the limited api usage, so that it uses Python's Py_buffer if Python >= 3.11 instead of the band-aid implementation.
Basically only _brotli.c and __init__.pyi changed to before.

@anthrotype
There you go...

Edit:
I noticed that I added whitespaces at two places in _brotli.c, which I will fix tomorrow.

[K0lb3]

I fixed the whitespace issue just now, and also uniformly re-formated the pre-processor conditions to not be indented.

[eustas]

Thanks. Looking into.

[eustas]

LGTM with comments:

• fix copyright year
• resolve conflicts again (sorry)
• it would be nice on #else and #endif to have comments corresponding to condition (for better readability), e.g. #else /* Py_LIMITED_API */ (NB: two spaces before comment block)

[K0lb3]

I assume that you meant the copyright date in the __init__py(i) files?

....and while it's useful to practice some more "tricky" git stuff again....it's certainly not fun....
So....a good reminder for me to never get into a position of having to manage this stuff, to say nothing about doing so for a mix of corpo and public repo :P