[v3-1-test] Fix release process for Python Client (#58447) (#58471)

The Python Client of ours was released so far without sources, because
it was assumed it uses the same sources as Airflow - but it's not
necessarily the same and it can be released with a different cadence
and using different sources - for example to fix some client-only
issues.

This PR updates the release process to also include source tarball
snapshot as part of the release and updates breeze and release process
description to make sure sources are produced and verified.
(cherry picked from commit dda9d61)

Co-authored-by: Jarek Potiuk <[email protected]>

Author: 2 people

dev/README_RELEASE_PYTHON_CLIENT.md

@@ -120,6 +120,8 @@ git log 2.8.0..HEAD --pretty=oneline -- airflow-core/src/airflow/api_fastapi/cor
 cd ${AIRFLOW_REPO_ROOT}
 rm dist/*
 breeze release-management prepare-python-client --distribution-format both --python-client-repo "${CLIENT_REPO_ROOT}" --version-suffix ""
+breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
+
 ```
 
 - This should generate both sdist and .whl package in `dist` folder of the Airflow repository. It should
@@ -326,15 +328,34 @@ Airflow Python client supports reproducible builds, which means that the package
 sources should produce binary identical packages in reproducible way. You should check if the packages can be
 binary-reproduced when built from the sources.
 
-Checkout airflow sources and build packages in dist folder (replace X.Y.Zrc1 with the version + rc candidate)
-you are checking):
+1) Set versions of the packages to be checked:
+
+```shell script
+cd <directory where airflow is checked out>
+VERSION=X.Y.Z
+VERSION_SUFFIX=rc1
+VERSION_RC=${VERSION}${VERSION_SUFFIX}
+```
+
+2) Change directory where your airflow sources are checked out
+
+```shell
+cd "${AIRFLOW_REPO_ROOT}"
+```
+
+3) Check out the ``python-client`` tag (assume apache is the remote name of the repository):
+
+```shell
+git fetch apache --tags
+git checkout providers/${VERSION_RC}
+```
+
+4) Build the distribution and source tarball:
 
 ```shell script
-VERSION=X.Y.Zrc1
-git checkout python-client/${VERSION}
-export AIRFLOW_REPO_ROOT=$(pwd)
 rm -rf dist/*
 breeze release-management prepare-python-client --distribution-format both --version-suffix ""
+breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
 ```
 
 The last - build step - by default will use Dockerized build and building of Python client packages
@@ -343,13 +364,16 @@ will be done in a docker container.  However, if you have  `hatch` installed loc
 
 ```bash
 breeze release-management prepare-python-client --distribution-format both --use-local-hatch --version-suffix ""
+breeze release-management prepare-tarball --tarball-type apache_airflow_python_client --version "${VERSION}" --version-suffix "${VERSION_SUFFIX}"
 ```
 
 This is generally faster and requires less resources/network bandwidth.
 
-Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist folder.
+Both commands should produce reproducible `.whl`, `.tar.gz` packages in dist folder and "-source.tar.gz"
+file containing airflow sources in dist folder.
 
-Change to the directory where you have the packages from svn:
+4) Change to the directory where you have the packages from svn and check if they are identical to the ones
+you just built:
 
 ```shell script
 # First clone the repo if you do not have it
@@ -358,7 +382,7 @@ cd ..
 svn update --set-depth=infinity asf-dist/dev/airflow/clients/python
 
 # Then compare the packages
-cd asf-dist/dev/airflow/clients/python/${VERSION}
+cd asf-dist/dev/airflow/clients/python/${VERSION_RC}
 for i in ${AIRFLOW_REPO_ROOT}/dist/*
 do
   echo "Checking if $(basename $i) is the same as $i"
@@ -373,6 +397,63 @@ In case the files are different, you should see:
 Binary files apache_airflow-client-2.9.0.tar.gz and .../apache_airflow-2.9.0.tar.gz differ
 ```
 
+### Licence check
+
+This can be done with the Apache RAT tool.
+
+Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi (unpack the binary, the jar is inside)
+
+You can run this command to do it for you:
+
+```shell script
+wget -qO- https://dlcdn.apache.org//creadur/apache-rat-0.17/apache-rat-0.17-bin.tar.gz | gunzip | tar -C /tmp -xvf -
+```
+
+Unpack the release source archive (the `<package + version>-source.tar.gz` file) to a folder
+
+```shell script
+rm -rf /tmp/apache/airflow-python-client-src && mkdir -p /tmp/apache-airflow-python-client-src && tar -xzf ${PATH_TO_SVN}/providers/${RELEASE_DATE}/apache_airflow_python_client-*-source.tar.gz --strip-components 1 -C /tmp/apache-airflow-python-client-src
+```
+
+Run the check:
+
+```shell script
+java -jar /tmp/apache-rat-0.17/apache-rat-0.17.jar --input-exclude-file /tmp/apache-airflow-python-client-src/.rat-excludes /tmp/apache-airflow-python-client-src/ | grep -E "! |INFO: "
+```
+
+You should see no files reported as Unknown or with wrong licence and summary of the check similar to:
+
+```
+INFO: Apache Creadur RAT 0.17 (Apache Software Foundation)
+INFO: Excluding patterns: .git-blame-ignore-revs, .github/*, .git ...
+INFO: Excluding MISC collection.
+INFO: Excluding HIDDEN_DIR collection.
+SLF4J(W): No SLF4J providers were found.
+SLF4J(W): Defaulting to no-operation (NOP) logger implementation
+SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
+INFO: RAT summary:
+INFO:   Approved:  15615
+INFO:   Archives:  2
+INFO:   Binaries:  813
+INFO:   Document types:  5
+INFO:   Ignored:  2392
+INFO:   License categories:  2
+INFO:   License names:  2
+INFO:   Notices:  216
+INFO:   Standards:  15609
+INFO:   Unapproved:  0
+INFO:   Unknown:  0
+```
+
+There should be no files reported as Unknown or Unapproved. The files that are unknown or unapproved should be shown with a line starting with `!`.
+
+For example:
+
+```
+! Unapproved:         1    A count of unapproved licenses.
+! /CODE_OF_CONDUCT.md
+```
+
 ## Signature check
 
 Make sure you have imported into your GPG the PGP key of the person signing the release. You can find the valid keys in

dev/breeze/doc/images/output_release-management_prepare-tarball.svg

@@ -1 +1 @@
-7cb732b47aeebbb6b430cda72776a7a8
+1e525a6dd6717f0ce733a5f679c83489

dev/breeze/doc/images/output_release-management_prepare-tarball.txt

@@ -332,6 +332,11 @@ def create_tarball_release(
         if not version:
             version = date.strftime(date.today(), "%Y-%m-%d")
             console_print(f"\n[info]Using current date {version} as tarball version\n")
+    elif tarball_type == TarBallType.PYTHON_CLIENT:
+        tag = f"python-client/{version + version_suffix}" if version else "HEAD"
+        if not version:
+            version = date.strftime(date.today(), "%Y-%m-%d")
+            console_print(f"\n[info]Using current date {version} as tarball version\n")
     else:  # pragma: no cover
         console_print(f"[error]Unsupported tarball type: {tarball_type}")
         exit(1)

dev/breeze/src/airflow_breeze/commands/release_candidate_command.py

@@ -239,6 +239,7 @@ class TarBallType(Enum):
     PROVIDERS = "apache_airflow_providers"
     TASK_SDK = "apache_airflow_task_sdk"
     AIRFLOW_CTL = "apache_airflow_ctl"
+    PYTHON_CLIENT = "apache_airflow_python_client"
     HELM_CHART = "helm-chart"