🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

Decentralized LLMs fine-tuning and inference with offloading

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

An Efficient and Versatile Inference Engine for Distributed LLM Serving

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

A distributed training framework for large language models powered by Lightning.

Fast and easy distributed model training examples.

Tensor Parallelism with JAX + Shard Map

Tencent Hy3 295B MoE (NVFP4) on 2x NVIDIA DGX Spark — TP2 over 200GbE, 256K context, 26 tok/s end-to-end. Tuned, benchmarked, agent-ready.

GPU Memory Calculator for LLM Training - Calculate GPU memory requirements for training Large Language Models with support for multiple training engines including PyTorch DDP, DeepSpeed ZeRO, Megatron-LM, and FSDP.

Communication cost modeling for tensor parallel LLM inference with TP vs PP vs hybrid comparison, VRAM analysis, pipeline bubble modeling, regime detection, and cost-efficiency. Shows TP dominates on NVLink, PP has 47% bubble at 8 GPUs, and LLaMA-70B needs 8× A100 or 2× H100 for VRAM.

MiniMax-M3 (428B MoE) running on 3× RTX PRO 6000 Blackwell at TP=3 with 240K context, FP8 KV cache, and working multimodal vision input. Includes dist_utils.py patch for non-divisible attention heads.

Multi-GPU tensor/context parallel diffusion on AMD ROCm — with the patch that makes it actually work.

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

DeepSeek-V4-Flash DSpark speculative decoding on 2x DGX Spark (GB10/sm_121), 1M context — our fp8 recipe + independent reproduction and cross-build benchmarks of the NVFP4-KV build. Honest, apples-to-apples measurements.

Optimized dual AMD Strix Halo (gfx1151) vLLM MoE inference: TP=2 over USB4, tuned int4 MoE kernel, ~8us interconnect, serialized serving

This repository outlines a comprehensive guide for training a distributed deep learning model.

Multi-GPU vLLM on consumer Radeon (RX 7900 XT/XTX, gfx1100): root cause and fix for the RCCL "operation cannot be performed in the present state" crash, plus 292 benchmark measurements across five model architectures