CI Job access to Object Storage

[afrittoli]

Modify PyTorch CI jobs to use the OIDC authenticated role from #6 to access S3 and ECR repos.

List of PRs:

• Add OIDC permissions to xpu workflow pytorch/pytorch#151455
• Add OIDC permissions to bazel workflow pytorch/pytorch#151456
• Add OIDC permissions to docs workflow pytorch/pytorch#151567
• Add OIDC permissions to linux-build workflow pytorch/pytorch#151581
• Add OIDC permissions to linux-test workflow pytorch/pytorch#151585
• Add OIDC perms to windows-[build|test] workflows pytorch/pytorch#151596

[zxiiro]

We'll need to update lines like these:

• https://github.com/search?q=repo%3Apytorch%2Fpytorch+aws-actions%2Fconfigure-aws-credentials&type=code&p=1
• https://github.com/pytorch/pytorch/blob/main/.github/workflows/create_release.yml#L99-L103

We can use these docs to make the necessary changes to the jobs.

https://github.com/aws-actions/configure-aws-credentials?tab=readme-ov-file#oidc

[zxiiro]

Here's a table of all the available OIDC roles in PyTorch today.

#6 (comment)

I think specifically gha_workflow_pytorch_artifacts, gha_workflow_s3_and_ecr_read_only, and gha_workflows_s3_ossci_linux_windows_read_write might be the main roles we need to rework the jobs for S3 and ECR access.

[zxiiro]

It looks like the _linux-build.yml and _linux-test.yml have some setting for configure-aws-credentials however as far as I can tell its usually skipped due to the input inputs.aws-role-to-assume not being passed to the job workflow. Maybe we need to modify this to handle OIDC and get the job working. I'll see if I can setup some tests in pytorch-canary.

[zxiiro]

I couldn't get it to work on canary system. Seems like maybe its missing some permissions that prod has. It fails with this error Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity.

However on prod I was able to get aws-configure-credentials to pass on lf.linux.2xlarge runners https://github.com/pytorch/pytorch/actions/runs/14407181584/job/40406438145#step:6:1

However I noticed it fails on the lf.ephemeral.linux.2xlarge experiment runners. I guess there's some different permissions on that one but I'm not familiar with those runners.

My in progress test changes are here: https://github.com/pytorch/pytorch/pull/151116/files

[zxiiro]

Looks like we need to switch to a different role for the sccache step at the end.

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_pytorch_artifacts/gha-linux-build is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::ossci-benchmarks/v3/pytorch/pytorch/14407181584/40406438145/sccache-stats.json" because no identity-based policy allows the s3:PutObject action

https://github.com/pytorch/pytorch/actions/runs/14407181584/job/40406438145#step:18:244

This also seems to confirm that if we use the OIDC roles the EC2 default role seems to no longer have effect.

[zxiiro]

Success! I was able to get aws-configure-credentials to switch to 2 roles to make the _linux-build.yml workflow work.

https://github.com/pytorch/pytorch/actions/runs/14411450687/job/40420316073

Had to use 2 assume role calls for artifact access and also benchmark access using these 2 roles:

• arn:aws:iam::308535385114:role/gha_workflow_pytorch_artifacts
• arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results

[zxiiro]

Below is a list of jobs that call the following actions without using OIDC to generate its id-token as potential targets for updating:

• aws-actions/configure-aws-credentials
• seemethere/upload-artifact-s3
• ./.github/actions/upload-test-artifacts

Reusable workflows:

• .github/workflows/_bazel-build-test.yml
• .github/workflows/_docs.yml
• .github/workflows/_linux-build.yml
• .github/workflows/_linux-test.yml
• .github/workflows/_win-build.yml
• .github/workflows/_win-test.yml

Workflows that call the above reusable workflows.

• .github/workflows/inductor.yml
• .github/workflows/inductor-micro-benchmark.yml
• .github/workflows/inductor-micro-benchmark-x86.yml
• .github/workflows/inductor-nightly.yml
• .github/workflows/inductor-perf-compare.yml
• .github/workflows/inductor-perf-test-nightly.yml
• .github/workflows/inductor-perf-test-nightly-aarch64.yml
• .github/workflows/inductor-perf-test-nightly-h100.yml
• .github/workflows/inductor-perf-test-nightly-x86.yml
• .github/workflows/inductor-periodic.yml
• .github/workflows/inductor-unittest.yml
• .github/workflows/linux-aarch64.yml
• .github/workflows/nightly.yml
• .github/workflows/operator_benchmark.yml
• .github/workflows/periodic.yml
• .github/workflows/pull.yml
• .github/workflows/s390x-periodic.yml
• .github/workflows/slow.yml
• .github/workflows/torchbench.yml
• .github/workflows/trunk.yml
• .github/workflows/xpu.yml

[zxiiro]

For the file .github/workflows/_xpu-test.yml id-token is not set inside the file explicitly but the calling workflows set it at the job level. Would be worth explicit declaration so that any workflows that want to use it will be flagged automatically if they forget to set id-token.