Cache management

Compiled dependencies are cached at https://tmp.kiwix.org/ci. That means that, as far as possible, the cache is reused in place of recompiling the source code. These cache files are created by kiwix-build and used by the CI and CD of kiwix-build itself and by the projects/repositories with code been compiled.

The cache has two kinds of files:

• Base deps: this provides compiled versions of all external dependencies and compilation tools. This cache is only used by kiwix-build. Base deps cache files are created and uploaded by kiwix-build each time the expected cache file is not available (in a brancht too). Base deps are versioned at kiwixbuild/versions.py (look at variable called base_deps_meta_version).

• Deps: this provides compiled versions of all dependencies for a specific project. This cache is only used by the the non-kiwix-build projects (libzim, libkiwix, zim-tools, kiwix-tools, ...). Deps cache files are always uploaded by the kiwix-build CI running on main. If you are in a branch, then the deps files will still be uploaded, but in a sub-directory (removed after 30 days). Deps files cache is, to a large extend, inherited from current base deps. Non-kiwix-build project used a custom GitHub action to download the deps file, see https://github.com/kiwix/kiwix-build/tree/main/actions/dl_deps_archive.

Use GitHub dl_deps_archive action

To download the deps cache file dedicated to your project, you should insert in your workflow something like:

          - name: Install dependencies
        uses: kiwix/kiwix-build/actions/dl_deps_archive@main
        with:
          target_platform: ${{ matrix.target }}

dl_deps_archive will:

• based on the repository name it will identify which project
• the target_platform which is a string describing both system, architecture and way the compilation has been done

At this stage two scenarios are possible:

• Either the branch you are in has the same name as a branch of kiwix-build and it will pick the deps cache file in the subdirectory called the same OR
• It will take the last created deps file (so from kiwix-build main branch) for your project/target.

Change dependencies version

Changing a dependency version need (most of the time) three changes.

The declared version.

In the base_deps_versions dictionary in https://github.com/kiwix/kiwix-build/blob/master/kiwixbuild/versions.py

This version number is used to create the source and build directory.

The archive to download.

Changing a version number is not enough, you have to change the archive that will be used for the dependency. This is specific to each dependency and it is in each specific dependency file (ie, https://github.com/kiwix/kiwix-build/blob/master/kiwixbuild/dependencies/android_sdk.py for the android sdk). The archive specification is declared this way :

class Source(ReleaseDownload):
        archive = Remotefile('tools_r25.2.3-linux.zip',
                             '1b35bcb94e9a686dff6460c8bca903aa0281c6696001067f34ec00093145b560',
                             'https://dl.google.com/android/repository/tools_r25.2.3-linux.zip')

• The first argument is the name of the archive.
• The second argument is the sha256sum of the archive.
• The optional third argument is the url of the archive. If this is not specified, it will download the archive from http://download.kiwix.org/dev/ (using the archive name).

Update the base_deps_meta_version

Base dependencies are cached. And so they are not rebuild by the CI every time. To invalidate the cache, kiwix-build use the variable base_deps_meta_version (in https://github.com/kiwix/kiwix-build/blob/master/kiwixbuild/versions.py) . If you want to rebuild everything with the new version of the dependency (and you want), you also need to update the base_deps_meta_version.