Skip to content

Releases: huggingface/accelerate

v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10

20 Oct 16:08
@SunMarc SunMarc
v1.11.0
9a81156

Choose a tag to compare

View all tags
v1.11.0: TE MXFP8, FP16/BF16 with MPS, Python 3.10 Latest
Latest

TE MXFP8 support

We've added support for MXFP8 in our TransformerEngine integration. To use that, you need to set use_mxfp8_block_scaling in fp8_config. See nvidia docs [here]. (https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#MXFP8-and-block-scaling)

  • Add support for TE MXFP8 recipe in accelerate by @pstjohn in #3688

FP16/BF16 Training for MPS devices

BF16 and FP16 support for MPS devices is finally here. You can now pass mixed_precision = "fp16" or "bf16" when training on a mac (fp16 requires torch 2.8 and bf16 requires torch 2.6)

  • Add bf16/fp16 support for amp with mps device by @SunMarc in #3373

FSDP updates

The following PRs add respectively support to ignored_params and no_sync() for FSDPv2:

  • feat: add ignored_params support for fsdp2 by @kmehant in #3731
  • fix: model.set_requires_gradient_sync(False) should be called to turn off gradient synchronization in FSDP2 by @EquationWalker in #3762

Mixed precision can now be passed as a dtype string from accelerate cli flag or fsdp_config in accelerate config file:

  • feat: allow mixed precision policy as dtype by @kmehant in #3751

Nd-parallel updates

Some minor updates concerning nd-parallelism.

Bump to Python 3.10

We've dropped support for python 3.9 as it reached EOL in October.

Lots of minor fixes:

New Contributors

Full Changelog: v1.10.1...v1.11.0

Contributors

Qubitium, pstjohn, and 12 other contributors
Assets 2
Loading

v1.10.1: Patchfix

25 Aug 13:57
@SunMarc SunMarc
v1.10.1
623cc0b

Choose a tag to compare

View all tags
v1.10.1: Patchfix

Full Changelog: v1.10.0...v1.10.1

Contributors

S1ro1 and SunMarc
Loading

v1.10.0: N-D Parallelism

07 Aug 13:39
@SunMarc SunMarc
v1.10.0
5dcd16e

Choose a tag to compare

View all tags
v1.10.0: N-D Parallelism

N-D Parallelism

Training large models across multiple GPUs can be complex, especially when combining different parallelism strategies (e.g TP, CP, DP). To simplify this process, we've collaborated with Axolotl to introduce an easy-to-use integration that allows you to apply any combination of parallelism strategies directly in your training script. Just pass a ParallelismConfig specifying the size of each parallelism type—it's that simple.
Learn more about how it works in our latest blogpost.

parallelism_config = ParallelismConfig(
    dp_shard_size=2,
    dp_replicate_size=2,
    cp_size=2,
    tp_size=2,
)
accelerator = Accelerator(
    parallelism_config=parallelism_config,
   ...
)
model = AutoModelForCausalLM.from_pretrained("your-model-name", device_mesh=accelerator.torch_device_mesh)
model = accelerator.prepare(model)
  • Parallelism config + TP + HSDP + BYODM (Bring Your Own Device Mesh) by @SalmanMohammadi in #3682
  • Feat: context parallel v2.0 by @S1ro1 in #3700
  • set default submesh_tp_size to prevent unset local variable error by @winglian in #3687
  • Add Parallelism getter property to Accelerator class by @WoosungMyung in #3703
  • Fix: prepare works even if nothing except tp specified (rare) by @S1ro1 in #3707
  • Set parallelism_config in constructor due to Trainer reset of State by @winglian in #3713
  • Fix: tp size wouldn't read from env by @S1ro1 in #3716
  • Remove ParallelismConfig from PartialState by @SunMarc in #3720

FSDP improvements

We've fixed ignored modules attribute. With this, it is now possible to train PEFT model that moe layers that contrains q_proj and v_proj parameters. This is especially important for fine-tuning gpt-oss model.

Minor improvements

  • feature: CpuOffload pre_forward don't attempt to move if already on device by @JoeGaffney in #3695
  • Fix: Ensure environment variable values are case-insensitive in Accelerate by @jp1924 in #3712
  • remove use_ipex by @SunMarc in #3721

New Contributors

Full Changelog: v1.9.0...v1.10.0

Contributors

winglian, BenjaminBossan, and 6 other contributors
Loading

v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

16 Jul 16:35
@SunMarc SunMarc
v1.9.0
5565412

Choose a tag to compare

View all tags
v1.9.0: Trackio support, Model loading speedup, Minor distributed improvements

Trackio tracker support

We've added support for a trackio, lightweight, 💯 free experiment tracking Python library built on top of 🤗 Datasets and Spaces.

Screen Recording 2025-06-11 at 5 39 32 PM

Main features are:

  • Local-first design: dashboard runs locally by default. You can also host it on Spaces by specifying a space_id.
  • Persists logs locally (or in a private Hugging Face Dataset)
  • Visualize experiments with a Gradio dashboard locally (or on Hugging Face Spaces)
  • Everything here, including hosting on Hugging Faces, is free!

To use it with accelerate, you need to set log_with and initialize the trackers

accelerator = Accelerator(log_with="trackio")
config={"learning_rate": 0.001, "batch_size": 32}
# init_kwargs in order to host the dashboard on spaces
init_kwargs = {"trackio": {"space_id": "hf_username/space_name"}
accelerator.init_trackers("example_project", config=config, init_kwargs=init_kwargs})

Thanks @pcuenca for the integration !

Model loading speedup when relying set_module_tensor_to_device

Setting tensor while clearing cache is very slow, so we added clear_device option to disable it.
Another small optimization is using non_blocking everywhere and syncing just before returning control to the user. This makes the loading slightly faster.

FDSP, Deepspeed, FP8 minor improvements

  • Add support for e5e2 and default to hybrid when launcher is used by @IlyasMoutawwakil in #3640
  • Fix FP8 tests, enable FP8 to be used without direct Accelerator() configuring by @pstjohn in #3677
  • Bunch of FSDP improvements by @S1ro1 in #3671
  • Fix: properly error when DDP + Dtensor model by @S1ro1 in #3629
  • Fix fsdp2 example typo by @shimizust in #3657
  • Added a check in no_sync() to avoid errors when using deepspeed zero2/3 by @xliu0105 in #3656

🚨🚨🚨 Breaking changes 🚨🚨🚨

find_executable_batch_size() will no longer halves the batch after every OOM. Instead, we will multiply the batch size by 0.9. This should help user not waste gpu capacity.

  • “Stop Halving My Batch!” · Default back-off 0.5 → 0.9 by @SunMarc in #3684

What's Changed

New Contributors

Full Changelog: v1.8.1...v1.9.0

Contributors

pcuenca, pstjohn, and 10 other contributors
Loading

v1.8.1: Patchfix

20 Jun 15:43
@SunMarc SunMarc
v1.8.1
f5513f1

Choose a tag to compare

View all tags
v1.8.1: Patchfix

Full Changelog: v1.8.0...v1.8.1

Contributors

SunMarc and IlyasMoutawwakil
Loading

v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

19 Jun 15:37
@SunMarc SunMarc
v1.8.0
d2ced37

Choose a tag to compare

View all tags
v1.8.0: FSDPv2 + FP8, Regional Compilation for DeepSpeed, Faster Distributed Training on Intel CPUs, ipex.optimize deprecation

FSDPv2 refactor + FP8 support

We've simplified how to prepare FSDPv2 models, as there were too many ways to compose FSDP2 with other features (e.g., FP8, torch.compile, activation checkpointing, etc.). Although the setup is now more restrictive, it leads to fewer errors and a more performant user experience. We’ve also added support for FP8. You can read about the results here. Thanks to @S1ro1 for this contribution!

Faster Distributed Training on Intel CPUs

We updated the CCL_WORKER_COUNT variable and added KMP parameters for Intel CPU users. This significantly improves distributed training performance (e.g., Tensor Parallelism), with up to a 40% speed-up on Intel 4th Gen Xeon when training transformer TP models.

Regional Compilation for DeepSpeed

We added support for regional compilation with the DeepSpeed engine. DeepSpeed’s .compile() modifies models in-place using torch.nn.Module.compile(...), rather than the out-of-place torch.compile(...), so we had to account for that. Thanks @IlyasMoutawwakil for this feature!

ipex.optimize deprecation

ipex.optimize is being deprecated. Most optimizations have been upstreamed to PyTorch, and future improvements will land there directly. For users without PyTorch 2.8, we’ll continue to rely on IPEX for now.

Better XPU Support

We've greatly expanded and stabilized support for Intel XPUs:

Trackers

We've added support for SwanLab as an experiment tracking backend. Huge thanks to @ShaohonChen for this contribution ! We also deferred all tracker initializations to prevent premature setup of distributed environments.

  • Integrate SwanLab for offline/online experiment tracking for Accelerate by @ShaohonChen in #3605
  • Fix: Defer Tracker Initialization to Prevent Premature Distributed Setup by @yuanjua in #3581

What's Changed

New Contributors

Full Changelog: v1.7.0...v1.8.0

Contributors

kashif, yao-matrix, and 12 other contributors
Loading

v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

15 May 12:33
@SunMarc SunMarc
v1.7.0
9cb1a6b

Choose a tag to compare

View all tags
v1.7.0 : Regional compilation, Layerwise casting hook, FSDPv2 + QLoRA

Regional compilation

Instead of compiling the entire model at once, regional compilation targets repeated blocks (such as decoder layers) first. This allows the compiler to cache and reuse optimized code for subsequent blocks, significantly reducing the cold start compilation time typically seen during the first inference. Thanks @IlyasMoutawwakil for the feature ! You can view the full benchmark here, and check out our updated compilation guide for more details!

compilation_time-1

To enable this feature, set use_regional_compilation=True in the TorchDynamoPlugin configuration.

# Configure the compilation backend
dynamo_plugin = TorchDynamoPlugin(
    use_regional_compilation=True,
    ... # other parameters
)
# Initialize accelerator with the plugin
accelerator = Accelerator(dynamo_plugin=dynamo_plugin)
# This will apply compile_regions to your model
model = accelerator.prepare(model)

Layerwise casting hook

We've introduced a new hook that enables per-layer upcasting and downcasting (e.g., for Linear layers) during inference. This allows users to run models with separate storage and compute dtypes, resulting in memory savings. The concept was first implemented in diffusers, where downcasting models to FP8 proved effective without major quality degradation. Contributed by @sayakpaul in #3427

model = ....
storage_dtype = torch.float8_e4m3fn
compute_dtype = torch.bfloat16
attach_layerwise_casting_hooks(
            model,
            storage_dtype=storage_dtype,
            compute_dtype=compute_dtype,
        )

Better FSDP2 support

This release includes numerous new features and bug fixes. Notably, we’ve added support for FULL_STATE_DICT, a widely used option in FSDP, now enabling .save_pretrained() in transformers to work with FSDP2 wrapped models. QLoRA training is now supported as well but more testing is needed. We have also resolved a backend issue related to parameter offloading to CPU. Additionally, a significant memory spike that occurred when cpu_ram_efficient_loading=True was enabled has been fixed. Several other minor improvements and fixes are also included—see the What’s Changed section for full details.

  • FULL_STATE_DICT have been enabled by @S1ro1 in #3527
  • QLoRA support by @winglian in #3546
  • set backend correctly for CUDA+FSDP2+cpu-offload in #3574
  • memory spike fixed when using cpu_ram_efficient_loading=True by @S1ro1 in #3482

Better HPU support:

We have added a documentation for Intel Gaudi hardware !
The support is already available since v1.5.0 through this PR.

Torch.compile breaking change for dynamic argument

We've updated the logic for setting self.dynamic to explicitly preserve None rather than defaulting to False when the USE_DYNAMIC environment variable is unset. This change aligns the behavior with the PyTorch documentation for torch.compile. Thanks to @yafshar for contributing this improvement in #3567.

What's Changed

Read more

Contributors

winglian, pstjohn, and 30 other contributors
Loading

v1.6.0: FSDPv2, DeepSpeed TP and XCCL backend support

01 Apr 13:48
@SunMarc SunMarc
v1.6.0
587bc68

Choose a tag to compare

View all tags
v1.6.0: FSDPv2, DeepSpeed TP and XCCL backend support

FSDPv2 support

This release introduces the support for FSDPv2 thanks to @S1ro1.

If you are using python code, you need to set fsdp_version=2 in FullyShardedDataParallelPlugin:

from accelerate import FullyShardedDataParallelPlugin, Accelerator

fsdp_plugin = FullyShardedDataParallelPlugin(
    fsdp_version=2
    # other options...
)
accelerator = Accelerator(fsdp_plugin=fsdp_plugin)

If want to convert a YAML config that contains the FSDPv1 config to FSDPv2 one , use our conversion tool:

accelerate to-fsdp2 --config_file config.yaml --output_file new_config.yaml`

To learn more about the difference between FSDPv1 and FSDPv2, read the following documentation.

DeepSpeed TP support

We have added initial support for DeepSpeed + TP. Not many changes were required as the DeepSpeed APIs was already compatible. We only needed to make sure that the dataloader was compatible with TP and that we were able to save the TP weights. Thanks @inkcherry for the work ! #3390.

To use TP with deepspeed, you need to update the setting in the deepspeed config file by including tensor_parallel key:

    ....
    "tensor_parallel":{
      "autotp_size": ${autotp_size}
    },
   ...

More details in this deepspeed PR.

Support for XCCL distributed backend

We've added support for XCCL which is an Intel distributed backend which can be used with XPU devices. More details in this torch PR. Thanks @dvrogozh for the integration !

What's Changed

New Contributors

Full Changelog: v1.5.2...v1.6.0

Contributors

booxter, Lothiraldan, and 13 other contributors
Loading

Patch: v1.5.2

14 Mar 14:16
@muellerzr muellerzr
v1.5.2
74e08e5

Choose a tag to compare

View all tags
Patch: v1.5.2

Bug Fixes:

  • Fixed an issue with torch.get_default_device() requiring a higher version than what we support
  • Fixed a broken pytest import in prod

Full Changelog: v1.5.0...v1.5.2

Loading

v1.5.0: HPU support

12 Mar 14:18
@muellerzr muellerzr
v1.5.0
d5ed283

Choose a tag to compare

View all tags
v1.5.0: HPU support

HPU Support

  • Adds in HPU accelerator support for 🤗 Accelerate

What's Changed

New Contributors

Full Changelog: v1.4.0...v1.5.0

Contributors

saforem2, muellerzr, and 7 other contributors
Loading