From 2039e1b4f63d81707562c05e912f49a99cf73283 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Thu, 20 Nov 2025 13:56:48 -0800 Subject: [PATCH 01/16] first draft --- models/ref/python.mdx | 47 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 4 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index 25e5c959f1..cc09f8594f 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -14,13 +14,15 @@ To authenticate your machine with W&B, you must first generate an API key at htt ### Install and import packages -Install the W&B library. +Install the W&B Python SDK using pip: -``` +```bash pip install wandb ``` -### Import W&B Python SDK: +### Import W&B Python SDK + +The following code snippet demonstrates how to import the W&B Python SDK and initialize a run. Replace `` with your team entity name. ```python import wandb @@ -33,4 +35,41 @@ project = "my-awesome-project" with wandb.init(entity=entity, project=project) as run: run.log({"accuracy": 0.9, "loss": 0.1}) -```` +``` + +### Install optional dependencies + +Install compatible versions of optional dependencies for enhanced functionality. To install an optional dependency, use the following command, replacing `extra` with the desired package: + +```bash +pip install wandb[extra] +``` + +For example, to install W&B with Google Cloud Storage support, run: + +```bash +pip install wandb[gcp] +``` + +Install more than one optional dependency by separating them with commas: + +```bash +pip install wandb[gcp,aws,media] +``` + +#### Supported optional dependencies + +| Package | Purpose | +|---------|---------| +| `kubeflow` | Integrations for Kubeflow and object storage access. | +| `gcp` | Google Cloud Storage support. | +| `aws` | AWS S3 and other AWS service support. | +| `azure` | Azure Blob Storage support. | +| `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| +| `sweeps`| Manage sweeps. | +| `launch` | Installs cloud providers and orchestration platforms (includes many cloud SDKs, notebook tools, and orchestration clients). Note: tornado has a Python-version marker. | +| `perf` | High-performance JSON serializer. | +| `importers` | Data import and helper utilities. | +| `workspaces` | W&B Reports and Workspaces SDK. | + +See `optional-dependencies` in the [W&B Python SDK GitHub repository](https://github.com/wandb/wandb/blob/main/pyproject.toml#L61) for an up-to-date list of optional dependencies. \ No newline at end of file From f882f6013132d9a92a556aea9086d2fcf0462d24 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Thu, 20 Nov 2025 17:10:59 -0800 Subject: [PATCH 02/16] Added links to packages --- models/ref/python.mdx | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index cc09f8594f..f7abd6b941 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -39,7 +39,7 @@ with wandb.init(entity=entity, project=project) as run: ### Install optional dependencies -Install compatible versions of optional dependencies for enhanced functionality. To install an optional dependency, use the following command, replacing `extra` with the desired package: +Install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): ```bash pip install wandb[extra] @@ -59,17 +59,20 @@ pip install wandb[gcp,aws,media] #### Supported optional dependencies -| Package | Purpose | -|---------|---------| -| `kubeflow` | Integrations for Kubeflow and object storage access. | -| `gcp` | Google Cloud Storage support. | -| `aws` | AWS S3 and other AWS service support. | -| `azure` | Azure Blob Storage support. | -| `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| -| `sweeps`| Manage sweeps. | -| `launch` | Installs cloud providers and orchestration platforms (includes many cloud SDKs, notebook tools, and orchestration clients). Note: tornado has a Python-version marker. | -| `perf` | High-performance JSON serializer. | -| `importers` | Data import and helper utilities. | -| `workspaces` | W&B Reports and Workspaces SDK. | +The following table lists supported dependencies for the W&B Python SDK: + +| Package | Purpose | Packages included | +|---------|---------| ------------------| +| `kubeflow` | Integrations for Kubeflow and object storage access. | [kubernetes](https://pypi.org/project/kubernetes/), [minio](https://pypi.org/project/minio/), [google-cloud-storage](https://pypi.org/project/google-cloud-storage/), [sh](https://pypi.org/project/sh/) | +| `gcp` | Google Cloud Storage support. | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | +| `aws` | AWS S3 and other AWS service support. | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | +| `azure` | Azure Blob Storage support. | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | +| `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/ImageIO/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | +| `sweeps`| Manage sweeps. | [sweeps](https://pypi.org/project/sweeps/) | +| `launch` | Installs cloud providers and orchestration platforms | [awscli](https://pypi.org/project/awscli/), [azure-identity](https://pypi.org/project/azure-identity/), [azure-containerregistry](https://pypi.org/project/azure-containerregistry/), and more | +| `models`| | [cloudpickle](https://pypi.org/project/cloudpickle/) | +| `perf` | High-performance JSON serializer. | [orjson](https://pypi.org/project/orjson/) | +| `importers` | Data import and helper utilities. | [polars](https://pypi.org/project/polars/), [rich](https://pypi.org/project/rich/), [filelock](https://pypi.org/project/filelock/), [mlflow](https://pypi.org/project/mlflow/), [tenacity](https://pypi.org/project/tenacity/) | +| `workspaces` | W&B Reports and Workspaces SDK. | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | See `optional-dependencies` in the [W&B Python SDK GitHub repository](https://github.com/wandb/wandb/blob/main/pyproject.toml#L61) for an up-to-date list of optional dependencies. \ No newline at end of file From 9036a95363f24bf223d8b5a62594656ab6c6e291 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Thu, 20 Nov 2025 17:16:38 -0800 Subject: [PATCH 03/16] removed layer of section hearders --- models/ref/python.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index f7abd6b941..dac81a333f 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -6,13 +6,13 @@ The W&B Python SDK, accessible at `wandb`, enables you to train and fine-tune mo > After performing your training and fine-tuning operations with this SDK, you can use [the Public API](/models/ref/python/public-api) to query and analyze the data that was logged, and [the Reports and Workspaces API](/models/ref/wandb_workspaces) to generate a web-publishable [report](/models/reports/) summarizing your work. -## Installation and setup -### Sign up and create an API key + +## Sign up and create an API key To authenticate your machine with W&B, you must first generate an API key at https://wandb.ai/authorize. -### Install and import packages +## Install and import packages Install the W&B Python SDK using pip: @@ -20,7 +20,7 @@ Install the W&B Python SDK using pip: pip install wandb ``` -### Import W&B Python SDK +## Import W&B Python SDK The following code snippet demonstrates how to import the W&B Python SDK and initialize a run. Replace `` with your team entity name. @@ -37,7 +37,7 @@ with wandb.init(entity=entity, project=project) as run: run.log({"accuracy": 0.9, "loss": 0.1}) ``` -### Install optional dependencies +## Install optional dependencies Install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): @@ -57,7 +57,7 @@ Install more than one optional dependency by separating them with commas: pip install wandb[gcp,aws,media] ``` -#### Supported optional dependencies +### Supported optional dependencies The following table lists supported dependencies for the W&B Python SDK: From dfc084060f791b2c7c935bfc5414cd3d9985cbe5 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Fri, 21 Nov 2025 14:09:58 -0800 Subject: [PATCH 04/16] some feedback --- models/ref/python.mdx | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index dac81a333f..98605be062 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -41,19 +41,19 @@ with wandb.init(entity=entity, project=project) as run: Install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): -```bash +```shell pip install wandb[extra] ``` For example, to install W&B with Google Cloud Storage support, run: -```bash +```shell pip install wandb[gcp] ``` Install more than one optional dependency by separating them with commas: -```bash +```shell pip install wandb[gcp,aws,media] ``` @@ -70,9 +70,7 @@ The following table lists supported dependencies for the W&B Python SDK: | `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/ImageIO/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | | `sweeps`| Manage sweeps. | [sweeps](https://pypi.org/project/sweeps/) | | `launch` | Installs cloud providers and orchestration platforms | [awscli](https://pypi.org/project/awscli/), [azure-identity](https://pypi.org/project/azure-identity/), [azure-containerregistry](https://pypi.org/project/azure-containerregistry/), and more | -| `models`| | [cloudpickle](https://pypi.org/project/cloudpickle/) | | `perf` | High-performance JSON serializer. | [orjson](https://pypi.org/project/orjson/) | -| `importers` | Data import and helper utilities. | [polars](https://pypi.org/project/polars/), [rich](https://pypi.org/project/rich/), [filelock](https://pypi.org/project/filelock/), [mlflow](https://pypi.org/project/mlflow/), [tenacity](https://pypi.org/project/tenacity/) | | `workspaces` | W&B Reports and Workspaces SDK. | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | See `optional-dependencies` in the [W&B Python SDK GitHub repository](https://github.com/wandb/wandb/blob/main/pyproject.toml#L61) for an up-to-date list of optional dependencies. \ No newline at end of file From 804f106669ab684389a4d6926f31cf341a5de8f3 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Fri, 21 Nov 2025 14:10:43 -0800 Subject: [PATCH 05/16] removed launch --- models/ref/python.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index 98605be062..b4b8db03eb 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -69,7 +69,6 @@ The following table lists supported dependencies for the W&B Python SDK: | `azure` | Azure Blob Storage support. | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | | `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/ImageIO/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | | `sweeps`| Manage sweeps. | [sweeps](https://pypi.org/project/sweeps/) | -| `launch` | Installs cloud providers and orchestration platforms | [awscli](https://pypi.org/project/awscli/), [azure-identity](https://pypi.org/project/azure-identity/), [azure-containerregistry](https://pypi.org/project/azure-containerregistry/), and more | | `perf` | High-performance JSON serializer. | [orjson](https://pypi.org/project/orjson/) | | `workspaces` | W&B Reports and Workspaces SDK. | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | From 3b1911c6dd77fface871ec0e038e6968c41e9b2b Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Mon, 24 Nov 2025 10:15:31 -0800 Subject: [PATCH 06/16] minor edits --- models/ref/python.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index b4b8db03eb..da2db4ab2a 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -14,9 +14,9 @@ To authenticate your machine with W&B, you must first generate an API key at htt ## Install and import packages -Install the W&B Python SDK using pip: +Install the W&B Python SDK using `pip`: -```bash +```shell pip install wandb ``` From e88ba1f2f379c71caf91a14af2a0a358699dff87 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Mon, 24 Nov 2025 10:19:15 -0800 Subject: [PATCH 07/16] remove kubeflow --- models/ref/python.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index da2db4ab2a..a68ab9cbaa 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -63,7 +63,6 @@ The following table lists supported dependencies for the W&B Python SDK: | Package | Purpose | Packages included | |---------|---------| ------------------| -| `kubeflow` | Integrations for Kubeflow and object storage access. | [kubernetes](https://pypi.org/project/kubernetes/), [minio](https://pypi.org/project/minio/), [google-cloud-storage](https://pypi.org/project/google-cloud-storage/), [sh](https://pypi.org/project/sh/) | | `gcp` | Google Cloud Storage support. | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | | `aws` | AWS S3 and other AWS service support. | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | | `azure` | Azure Blob Storage support. | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | From 8fa94f98b67b1128f2639d3c2bc8e925e26b0a6b Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Mon, 24 Nov 2025 16:49:55 -0800 Subject: [PATCH 08/16] added scripts to generate table --- models/ref/python.mdx | 17 ++- scripts/generate_sdk_extras_table.py | 111 ++++++++++++++++++++ snippets/en/_includes/python-sdk-extras.mdx | 8 ++ 3 files changed, 125 insertions(+), 11 deletions(-) create mode 100755 scripts/generate_sdk_extras_table.py create mode 100644 snippets/en/_includes/python-sdk-extras.mdx diff --git a/models/ref/python.mdx b/models/ref/python.mdx index a68ab9cbaa..75ed7b57ff 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -2,6 +2,8 @@ title: Python SDK 0.23.0 module: --- +import PythonSDKExtras from '/snippets/en/_includes/python-sdk-extras.mdx' + The W&B Python SDK, accessible at `wandb`, enables you to train and fine-tune models, and manage models from experimentation to production. > After performing your training and fine-tuning operations with this SDK, you can use [the Public API](/models/ref/python/public-api) to query and analyze the data that was logged, and [the Reports and Workspaces API](/models/ref/wandb_workspaces) to generate a web-publishable [report](/models/reports/) summarizing your work. @@ -61,14 +63,7 @@ pip install wandb[gcp,aws,media] The following table lists supported dependencies for the W&B Python SDK: -| Package | Purpose | Packages included | -|---------|---------| ------------------| -| `gcp` | Google Cloud Storage support. | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | -| `aws` | AWS S3 and other AWS service support. | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | -| `azure` | Azure Blob Storage support. | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | -| `media` | Packages for media handling, video/image processing, plotting, and cheminformatics (RDKit).| [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/ImageIO/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | -| `sweeps`| Manage sweeps. | [sweeps](https://pypi.org/project/sweeps/) | -| `perf` | High-performance JSON serializer. | [orjson](https://pypi.org/project/orjson/) | -| `workspaces` | W&B Reports and Workspaces SDK. | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | - -See `optional-dependencies` in the [W&B Python SDK GitHub repository](https://github.com/wandb/wandb/blob/main/pyproject.toml#L61) for an up-to-date list of optional dependencies. \ No newline at end of file + + + + diff --git a/scripts/generate_sdk_extras_table.py b/scripts/generate_sdk_extras_table.py new file mode 100755 index 0000000000..952563e4a4 --- /dev/null +++ b/scripts/generate_sdk_extras_table.py @@ -0,0 +1,111 @@ +import os +import re +import tomli + +pyproject_path = "./pyproject.toml" +output_path = "../snippets/en/_includes/python-sdk-extras.mdx" +version_specifiers = [">=", "<=", "==", "~=", ">", "<", "!="] + +EXCLUDE = ["models", "kubeflow", "launch", "importers", "perf"] + +def header(): + """Return the markdown table headers.""" + return "| Extra | Packages included |\n" + +def header_row(head_topics: int = 2): + """Return the markdown table header row. + + Args: + head_topics (int): The number of header topics. + + Returns: + str: A markdown table header row. + """ + return "|---------"*head_topics + "|\n" + +def make_table_row(extra, packages): + """Generate a markdown table row for the given extra and packages. + + Args: + extra (str): The name of the extra. + packages (str): The markdown links for the packages. + + Returns: + str: A markdown table row. + """ + return f"| `{extra}` | {packages} |\n" + +def clean_deps(deps: list[str]) -> list[str]: + """Remove version specifiers from a list of dependency strings. + + Args: + deps (list[str]): A list of dependency strings. + + Returns: + list[str]: A list of dependency strings without version specifiers. + """ + pattern = r"(>=|<=|==|~=|>|<|!=).*" + return [re.sub(pattern, "", url) for url in deps] + +def make_dep_link(deps: list[str]) -> str: + """Create a markdown links for dependencies. + + Args: + deps (list[str]): A list of dependency names. + + Returns: + str: A string of markdown links for the dependencies. + """ + print(f"Creating dependency links for: {deps}") + + print("These are deps" , deps) + dep_names = [dep.split(" ")[0] for dep in deps] + return ", ".join( + f"[{name}](https://pypi.org/project/{name}/)" for name in dep_names + ) + + +def generate_table(pyproject_path: str) -> list[str]: + """Generate the markdown table for the SDK extras. + + Args: + pyproject_path (str): The path to the pyproject.toml file. + + Returns: + list[str]: A list of strings representing the markdown table. + """ + rows = [] + rows.append(header()) + rows.append(header_row()) + + with open(pyproject_path, "rb") as f: + pyproject = tomli.load(f) + extras = pyproject.get("project", {}).get("optional-dependencies", {}) + for extra, deps in extras.items(): + if extra in EXCLUDE: + continue + dep_links = make_dep_link(clean_deps(deps)) + table_row = make_table_row(extra, dep_links) + rows.append(table_row) + + return rows + + +def main(): + + # Create the output file if it doesn't exist + if not os.path.exists(output_path): + with open(output_path, 'w') as f: + f.write("Creating new file...\n") + print(f"File '{output_path}' created.") + else: + print(f"File '{output_path}' already exists.") + + # Generate the markdown table and write to the output file + table_lines = generate_table(pyproject_path) + with open(output_path, "w") as f: + f.writelines(table_lines) + print(f"Markdown table written to '{output_path}'.") + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/snippets/en/_includes/python-sdk-extras.mdx b/snippets/en/_includes/python-sdk-extras.mdx new file mode 100644 index 0000000000..634d95fdf4 --- /dev/null +++ b/snippets/en/_includes/python-sdk-extras.mdx @@ -0,0 +1,8 @@ +| Extra | Packages included | +|---------|---------| +| `gcp` | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | +| `aws` | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | +| `azure` | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | +| `media` | [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/imageio/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | +| `sweeps` | [sweeps](https://pypi.org/project/sweeps/) | +| `workspaces` | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | From a948edd5831b17f2ebaab7451801bd62a97df46b Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Tue, 25 Nov 2025 08:57:18 -0800 Subject: [PATCH 09/16] workflow in the docs repo that receives the dispatch event from wandb --- .github/workflows/update-sdk-extras.yml | 112 ++++++++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 .github/workflows/update-sdk-extras.yml diff --git a/.github/workflows/update-sdk-extras.yml b/.github/workflows/update-sdk-extras.yml new file mode 100644 index 0000000000..1965ae7c27 --- /dev/null +++ b/.github/workflows/update-sdk-extras.yml @@ -0,0 +1,112 @@ +name: Update SDK Extras Documentation + +on: + repository_dispatch: + types: [pyproject-updated] + workflow_dispatch: + +jobs: + update-sdk-extras: + runs-on: ubuntu-latest + + steps: + - name: Checkout docs repository + uses: actions/checkout@v5 + with: + fetch-depth: 0 + + - name: Checkout wandb repository + uses: actions/checkout@v5 + with: + repository: wandb/wandb + path: wandb-repo + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: '3.11' + + - name: Install dependencies + run: | + pip install tomli + + - name: Generate timestamp + id: timestamp + run: | + TIMESTAMP=$(date -u +"%Y-%m-%d %H:%M:%S UTC") + echo "timestamp=${TIMESTAMP}" >> $GITHUB_OUTPUT + echo "Generated at: ${TIMESTAMP}" + + - name: Copy pyproject.toml to scripts directory + run: | + echo "Copying wandb/pyproject.toml to scripts/" + cp wandb-repo/pyproject.toml scripts/pyproject.toml + echo "✅ File copied successfully" + + - name: Generate SDK Extras Table + run: | + echo "Generating SDK extras table from pyproject.toml..." + cd scripts + python generate_sdk_extras_table.py + echo "✅ SDK extras table generated" + + - name: Check for changes + id: check-changes + run: | + if [ -n "$(git status --porcelain)" ]; then + echo "has_changes=true" >> $GITHUB_OUTPUT + echo "Changes detected in python-sdk-extras.mdx" + git diff snippets/en/_includes/python-sdk-extras.mdx + else + echo "has_changes=false" >> $GITHUB_OUTPUT + echo "No changes detected" + fi + + - name: Create Pull Request + if: steps.check-changes.outputs.has_changes == 'true' + id: create-pr + uses: peter-evans/create-pull-request@v7 + with: + token: ${{ secrets.GITHUB_TOKEN }} + commit-message: "chore: Update SDK extras documentation" + title: "Update SDK extras documentation from pyproject.toml" + draft: false + body: | + This PR updates the SDK extras documentation based on the latest `wandb/pyproject.toml` configuration. + + **Generated on**: ${{ steps.timestamp.outputs.timestamp }} + **Source**: `wandb/wandb` repository `pyproject.toml` + **Triggered by**: Repository dispatch event from wandb/wandb + + ### Changes + - Synced latest optional dependencies from wandb package + - Updated SDK extras table in documentation + + ### Review Checklist + - [ ] Verify all extras are correctly listed + - [ ] Check that package links are accurate + - [ ] Confirm excluded extras (models, kubeflow, launch, importers, perf) are intentionally omitted + + --- + *This PR was automatically generated by the SDK extras update workflow.* + branch: update-sdk-extras-${{ github.run_number }} + delete-branch: true + labels: | + documentation + automated + sdk-extras + + - name: Display Result + run: | + if [ "${{ steps.check-changes.outputs.has_changes }}" == "true" ]; then + if [ -n "${{ steps.create-pr.outputs.pull-request-number }}" ]; then + PR_URL="https://github.com/${{ github.repository }}/pull/${{ steps.create-pr.outputs.pull-request-number }}" + echo "📝 PR created successfully!" + echo "🔗 Review the changes at: $PR_URL" + echo "::notice title=Pull Request Created::A PR has been created for SDK extras updates: $PR_URL" + fi + else + echo "✅ SDK extras documentation is up to date" + echo "::notice title=No Updates Needed::SDK extras documentation is already up to date" + fi From 3f702a2642c94731bd184bdff06387eff67086ca Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Tue, 25 Nov 2025 08:59:22 -0800 Subject: [PATCH 10/16] remove redundant line --- .github/workflows/update-sdk-extras.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/.github/workflows/update-sdk-extras.yml b/.github/workflows/update-sdk-extras.yml index 1965ae7c27..8a52e8c664 100644 --- a/.github/workflows/update-sdk-extras.yml +++ b/.github/workflows/update-sdk-extras.yml @@ -103,7 +103,6 @@ jobs: if [ -n "${{ steps.create-pr.outputs.pull-request-number }}" ]; then PR_URL="https://github.com/${{ github.repository }}/pull/${{ steps.create-pr.outputs.pull-request-number }}" echo "📝 PR created successfully!" - echo "🔗 Review the changes at: $PR_URL" echo "::notice title=Pull Request Created::A PR has been created for SDK extras updates: $PR_URL" fi else From f8e40f27271744f4c355d1a4d635a535e174ea34 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Tue, 25 Nov 2025 09:09:24 -0800 Subject: [PATCH 11/16] Cleaning up script --- scripts/generate_sdk_extras_table.py | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/scripts/generate_sdk_extras_table.py b/scripts/generate_sdk_extras_table.py index 952563e4a4..443df8f1e9 100755 --- a/scripts/generate_sdk_extras_table.py +++ b/scripts/generate_sdk_extras_table.py @@ -1,11 +1,15 @@ +""""Generate a markdown table of SDK extras from pyproject.toml. + +This script reads the `pyproject.toml` file to extract optional dependencies (extras) +and generates a markdown table listing each extra along with the packages it includes. + +Note that this script excludes certain extras defined in the EXCLUDE list. +""" import os import re import tomli -pyproject_path = "./pyproject.toml" -output_path = "../snippets/en/_includes/python-sdk-extras.mdx" -version_specifiers = [">=", "<=", "==", "~=", ">", "<", "!="] - +# W&B Python SDK Extras to exclude from the table. EXCLUDE = ["models", "kubeflow", "launch", "importers", "perf"] def header(): @@ -23,7 +27,7 @@ def header_row(head_topics: int = 2): """ return "|---------"*head_topics + "|\n" -def make_table_row(extra, packages): +def make_table_row(extra: str, packages: str) -> str: """Generate a markdown table row for the given extra and packages. Args: @@ -57,8 +61,6 @@ def make_dep_link(deps: list[str]) -> str: str: A string of markdown links for the dependencies. """ print(f"Creating dependency links for: {deps}") - - print("These are deps" , deps) dep_names = [dep.split(" ")[0] for dep in deps] return ", ".join( f"[{name}](https://pypi.org/project/{name}/)" for name in dep_names @@ -92,6 +94,11 @@ def generate_table(pyproject_path: str) -> list[str]: def main(): + # Pyproject.toml is expected to be in the same directory as this script. + pyproject_path = "./pyproject.toml" + + # Output path for the generated markdown table. + output_path = "../snippets/en/_includes/python-sdk-extras.mdx" # Create the output file if it doesn't exist if not os.path.exists(output_path): @@ -108,4 +115,5 @@ def main(): print(f"Markdown table written to '{output_path}'.") if __name__ == "__main__": + # To do: Add argument parsing for custom paths main() \ No newline at end of file From 147d73fdf7aeb1cfeebf5c114d652a02217dc958 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Wed, 26 Nov 2025 11:29:32 -0800 Subject: [PATCH 12/16] Updated scripts --- models/ref/python.mdx | 14 +++-- scripts/generate_sdk_extras_table.py | 91 ++++++++++++++++++++++------ 2 files changed, 81 insertions(+), 24 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index 75ed7b57ff..09ca6d8099 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -2,7 +2,6 @@ title: Python SDK 0.23.0 module: --- -import PythonSDKExtras from '/snippets/en/_includes/python-sdk-extras.mdx' The W&B Python SDK, accessible at `wandb`, enables you to train and fine-tune models, and manage models from experimentation to production. @@ -63,7 +62,14 @@ pip install wandb[gcp,aws,media] The following table lists supported dependencies for the W&B Python SDK: - - - +{/* python-extras-start */} +| Extra | Packages included | +|---------|---------| +| `gcp` | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | +| `aws` | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | +| `azure` | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | +| `media` | [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/imageio/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | +| `sweeps` | [sweeps](https://pypi.org/project/sweeps/) | +| `workspaces` | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | +{/* python-extras-end */} \ No newline at end of file diff --git a/scripts/generate_sdk_extras_table.py b/scripts/generate_sdk_extras_table.py index 443df8f1e9..f2161fdd39 100755 --- a/scripts/generate_sdk_extras_table.py +++ b/scripts/generate_sdk_extras_table.py @@ -67,14 +67,14 @@ def make_dep_link(deps: list[str]) -> str: ) -def generate_table(pyproject_path: str) -> list[str]: +def generate_table(pyproject_path: str) -> str: """Generate the markdown table for the SDK extras. - + Args: pyproject_path (str): The path to the pyproject.toml file. Returns: - list[str]: A list of strings representing the markdown table. + str: A string representing the markdown table. """ rows = [] rows.append(header()) @@ -90,29 +90,80 @@ def generate_table(pyproject_path: str) -> list[str]: table_row = make_table_row(extra, dep_links) rows.append(table_row) - return rows + return "".join(rows) + + +def replace_content_between_markers( + file_path: str, start_marker: str, end_marker: str, replacement: str +) -> bool: + """Replace content between start and end markers in a file. + + The markers themselves are preserved in the file. Only the content + between them is replaced with the new content. + + Args: + file_path (str): The path to the file to modify. + start_marker (str): The starting marker string. + end_marker (str): The ending marker string. + replacement (str): The text to insert between the markers. + + Returns: + bool: True if both markers were found and content was replaced, False otherwise. + """ + with open(file_path, "r") as f: + content = f.read() + + # Check if both markers exist + if start_marker not in content: + print(f"Error: Start marker '{start_marker}' not found in '{file_path}'.") + return False + + if end_marker not in content: + print(f"Error: End marker '{end_marker}' not found in '{file_path}'.") + return False + + # Find the positions of the markers + start_pos = content.find(start_marker) + end_pos = content.find(end_marker) + + # Check if markers are in the correct order + if start_pos >= end_pos: + print(f"Error: Start marker must come before end marker in '{file_path}'.") + return False + + # Split content into three parts: before, between (to replace), and after + before_start = content[:start_pos + len(start_marker)] + after_end = content[end_pos:] + + # Construct the new content with replacement between markers + updated_content = before_start + "\n" + replacement + after_end + + # Write the updated content back to the file + with open(file_path, "w") as f: + f.write(updated_content) + + return True def main(): - # Pyproject.toml is expected to be in the same directory as this script. - pyproject_path = "./pyproject.toml" + # Path to pyproject.toml in the wandb SDK repository. + pyproject_path = "../../wandb/pyproject.toml" - # Output path for the generated markdown table. - output_path = "../snippets/en/_includes/python-sdk-extras.mdx" + # MDX file path that contains the markers. + mdx_file_path = "../models/ref/python.mdx" - # Create the output file if it doesn't exist - if not os.path.exists(output_path): - with open(output_path, 'w') as f: - f.write("Creating new file...\n") - print(f"File '{output_path}' created.") - else: - print(f"File '{output_path}' already exists.") + # Start and end markers that enclose the table in the MDX file. + start_marker = "{/* python-extras-start */}" + end_marker = "{/* python-extras-end */}" - # Generate the markdown table and write to the output file - table_lines = generate_table(pyproject_path) - with open(output_path, "w") as f: - f.writelines(table_lines) - print(f"Markdown table written to '{output_path}'.") + # Generate the markdown table. + table = generate_table(pyproject_path) + + # Replace the content between markers in the MDX file with the generated table. + if replace_content_between_markers(mdx_file_path, start_marker, end_marker, table): + print(f"Successfully replaced content between markers in '{mdx_file_path}'.") + else: + print(f"Failed to replace content between markers in '{mdx_file_path}'.") if __name__ == "__main__": # To do: Add argument parsing for custom paths From d860ff55ca47c3e00bf7f46090238abea382ea5a Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Wed, 26 Nov 2025 11:30:33 -0800 Subject: [PATCH 13/16] removed snippet --- snippets/en/_includes/python-sdk-extras.mdx | 8 -------- 1 file changed, 8 deletions(-) delete mode 100644 snippets/en/_includes/python-sdk-extras.mdx diff --git a/snippets/en/_includes/python-sdk-extras.mdx b/snippets/en/_includes/python-sdk-extras.mdx deleted file mode 100644 index 634d95fdf4..0000000000 --- a/snippets/en/_includes/python-sdk-extras.mdx +++ /dev/null @@ -1,8 +0,0 @@ -| Extra | Packages included | -|---------|---------| -| `gcp` | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | -| `aws` | [boto3](https://pypi.org/project/boto3/), [botocore](https://pypi.org/project/botocore/) | -| `azure` | [azure-identity](https://pypi.org/project/azure-identity/), [azure-storage-blob](https://pypi.org/project/azure-storage-blob/) | -| `media` | [numpy](https://pypi.org/project/numpy/), [moviepy](https://pypi.org/project/moviepy/), [imageio](https://pypi.org/project/imageio/), [pillow](https://pypi.org/project/pillow/), [bokeh](https://pypi.org/project/bokeh/), [soundfile](https://pypi.org/project/soundfile/), [plotly](https://pypi.org/project/plotly/), [rdkit](https://pypi.org/project/rdkit/) | -| `sweeps` | [sweeps](https://pypi.org/project/sweeps/) | -| `workspaces` | [wandb-workspaces](https://pypi.org/project/wandb-workspaces/) | From 41875517d6e0fcf35c4ac8d3b17b0ad7368d1258 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Wed, 26 Nov 2025 11:37:56 -0800 Subject: [PATCH 14/16] Update text --- models/ref/python.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index 09ca6d8099..bec39f094b 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -40,8 +40,8 @@ with wandb.init(entity=entity, project=project) as run: ## Install optional dependencies -Install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): - +If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): +If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): ```shell pip install wandb[extra] ``` @@ -63,7 +63,7 @@ pip install wandb[gcp,aws,media] The following table lists supported dependencies for the W&B Python SDK: -{/* python-extras-start */} + | Extra | Packages included | |---------|---------| | `gcp` | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) | From ef086c60d8e65c8eca9249823f631dfa7c3010a9 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Wed, 26 Nov 2025 11:38:08 -0800 Subject: [PATCH 15/16] remove extra spacing --- models/ref/python.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index bec39f094b..6eb1d2cb5e 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -42,6 +42,7 @@ with wandb.init(entity=entity, project=project) as run: If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): + ```shell pip install wandb[extra] ``` From 7e9c8df345af14c4d84efb7ee69fe5215a60f3f7 Mon Sep 17 00:00:00 2001 From: Noah Luna <15202580+ngrayluna@users.noreply.github.com> Date: Wed, 26 Nov 2025 13:14:39 -0800 Subject: [PATCH 16/16] Added small blurb --- models/ref/python.mdx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/models/ref/python.mdx b/models/ref/python.mdx index 6eb1d2cb5e..c609caa113 100644 --- a/models/ref/python.mdx +++ b/models/ref/python.mdx @@ -40,7 +40,6 @@ with wandb.init(entity=entity, project=project) as run: ## Install optional dependencies -If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Use the following command to install an optional dependency. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): If needed, you can install optional dependencies that extend the functionality of the W&B Python SDK. Replace `extra` with the desired [optional dependency](#supported-optional-dependencies): ```shell @@ -59,12 +58,17 @@ Install more than one optional dependency by separating them with commas: pip install wandb[gcp,aws,media] ``` -### Supported optional dependencies +Use extras when you need deeper integration with specific platforms or features. For example: -The following table lists supported dependencies for the W&B Python SDK: +* Install `gcp` if you store datasets, models, or artifacts in Google Cloud Storage. +* Install `aws` if you use Amazon S3 or other AWS-backed storage in your workflows. +* Install `media` if you frequently log large or complex media (`images`, `audio`, `video`) and want better format support. +### Supported optional dependencies +The following table lists supported dependencies for the W&B Python SDK and the extras you can install: +{/* python-extras-start */} | Extra | Packages included | |---------|---------| | `gcp` | [google-cloud-storage](https://pypi.org/project/google-cloud-storage/) |