LLaMA Factory (now renamed LlamaFactory) is a unified fine-tuning framework for large language models. The project was initiated by Dr. Yaowei Zheng's team at Beihang University and is academically supported by an ACL 2024 paper. The framework adopts a "zero-code CLI + Web UI" design to simplify the fine-tuning process. LlamaFactory has accumulated more than 72,000 Stars on GitHub and has become a reference implementation in the field of large-model fine-tuning.
>
I. Project Overview
Dimension | Details
Full name | LlamaFactory (formerly LLaMA-Factory)
GitHub | github.com/hiyouga/LlamaFactory
Stars | 72,476 ⭐
Forks | 8,861
Primary language | Python (99.7%)
License | Apache-2.0
Created | 2023-03-21
Latest version | v0.9.5 (2026-05-30)
Total commits | 3,042 commits
Core contributor | hiyouga (Yaowei Zheng, 2,188 commits, 72% of total commits)
Academic paper | ACL 2024 System Demonstrations
Repository size | ~412.9 MB
Open Issues | 57
Founder Yaowei Zheng (hiyouga) is a doctoral student at Beihang University. As first author, he has published multiple papers at leading international conferences including ACL, CVPR, and WWW, and has served as a reviewer for conferences such as AAAI and EMNLP. His awards include the Huawei Cup, first prize in the national Blue Bridge Cup, Beihang University's Student of the Year, and the Ascend Ecosystem Open Source Excellence Contribution Award. Collaborating team members include Beihang professors Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
II. Project Positioning and Core Value
2.1 One-Sentence Positioning
A standardized large-model fine-tuning pipeline: a unified process for any model, covering the entire workflow from data preparation, model training, inference validation, and deployment.
2.2 Core Problems Addressed
Problem | LLaMA Factory's solution
Different models have different fine-tuning workflows, making switching costly | An algorithm-model-data "decoupled" design with a unified entry point for 100+ models
Fine-tuning code has a high barrier to entry and requires a deep-learning background | Zero-code CLI + LLaMA Board Web UI (Gradio)
Insufficient VRAM prevents consumer-grade GPUs from training large models | QLoRA 4-bit: 7B requires only 6GB, while 70B requires only 48GB
The open-source community lacks an all-in-one toolchain | Covers the full PT→SFT→RM→PPO→DPO lifecycle
Inference and deployment are complex | One-click export + OpenAI-style API + vLLM/SGLang
2.3 Design Philosophy: Algorithm-Model-Data "Decoupling"
In an interview, Yaowei Zheng compared the model inference process to vehicles traveling on a standardized highway: the models themselves correspond to different types of vehicles, while data corresponds to the cargo loaded onto them. As long as a model can run smoothly on this highway, fine-tuning can be completed simply by adjusting parameters.
III. In-Depth Analysis of the Technical Architecture
3.1 Parallel Dual-Architecture Design
LLaMA Factory provides both the v0 (stable) and v1 (experimental) architectures, with switching between them available through the USE_V1 environment variable.
v0 (default): api/webui → chat/eval/train → data/model → hparams → extras
v1 (experimental): trainers → core → accelerator/plugins/config → utils
The v0 architecture is the current mainstream implementation, offering complete functionality and comprehensive documentation. The v1 architecture is still experimental and adopts a more modular, layered design.
3.2 Detailed Overview of the v0 Core Modules
src/llamafactory/
├── cli.py # CLI entry point and llamafactory-cli command dispatch
├── launcher.py # Launcher; dispatches according to USE_V1
├── api/ # OpenAI-style REST API
│ ├── app.py # FastAPI application
│ ├── chat.py # Chat endpoint
│ └── protocol.py # Protocol definitions
├── chat/ # Inference engine layer
│ ├── base_engine.py # Abstract base class
│ ├── hf_engine.py # Native HuggingFace inference
│ ├── vllm_engine.py # High-performance vLLM inference
│ └── sglang_engine.py # SGLang inference
├── data/ # Data processing layer
│ ├── template.py # Prompt template registry for 100+ models
│ ├── mm_plugin.py # Multimodal (image/video/audio) processing
│ ├── loader.py # Dataset loader
│ ├── processor/ # Processors by training stage
│ └── converter.py # Format conversion (Alpaca/ShareGPT)
├── model/ # Model layer
│ ├── loader.py # Model and tokenizer loading, quantization, and LoRA
│ └── patcher.py # Model compatibility patches
├── train/ # Training layer
│ ├── sft/ # Supervised fine-tuning
│ ├── dpo/ # DPO preference optimization
│ ├── ppo/ # PPO reinforcement learning
│ ├── rm/ # Reward modeling
│ ├── pt/ # Pretraining
│ ├── kto/ # KTO training
│ └── mca/ # Megatron-Core adaptation
├── hparams/ # Parameter configuration layer
│ ├── model_args.py # Model arguments
│ ├── data_args.py # Data arguments
│ ├── finetuning_args.py # Fine-tuning arguments
│ └── training_args.py # Training arguments (inherits HF TrainingArguments)
├── extras/ # Utilities
│ ├── constants.py # Global constants/enumerations
│ └── packages.py # Dependency checks
└── webui/ # LLaMA Board web interface
├── interface.py # Gradio interface construction
├── runner.py # Training task launcher
└── components/ # UI components
3.3 Training Flow (v0 Training Flow)
llamafactory-cli train xxx.yaml
→ run_exp() [tuner.py]
→ read_args() → Parse YAML/JSON configuration
→ get_train_args() → Generate 4 typed dataclasses:
ModelArguments + DataArguments + FinetuningArguments + TrainingArguments
→ Route to the specific training function:
run_sft / run_dpo / run_ppo / run_rm / run_pt / run_kto
→ (optional) export_model() → Merge and export LoRA weights
3.4 Inference Engine Matrix
Engine | Features | Use cases
HF Engine | Native HuggingFace, best compatibility | Development and debugging, small-scale inference
vLLM Engine | 270% inference acceleration, PagedAttention | High-performance online inference
SGLang Engine | RadixAttention, structured generation | Batch inference, complex generation
KT Engine | Dedicated to knowledge transfer | Model distillation
IV. Feature System
4.1 Supported Training Paradigms (8 × 6 Fine-Tuning Methods = 48 Combinations)
Training paradigm | Full | Freeze | LoRA | QLoRA | OFT | QOFT
Pretraining (PT) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
Supervised fine-tuning (SFT) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
Reward modeling (RM) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
PPO training | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
DPO training | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
KTO training | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
ORPO training | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
SimPO training | ✅ | ✅ | ✅ | ✅ | ✅ | ✅
4.2 Supported Model Series (60+ Series, 100+ Variants)
Selected representative models:
Model series | Parameter scale | Features
Llama 3-4 | 1B–402B | Meta flagship open-source models
Qwen 2-3.6 | 0.5B–397B | Alibaba Qwen full series
DeepSeek R1 | 1.5B–671B | Distilled reasoning models
Gemma 3 | 270M–27B | Google lightweight models
InternVL 2.5-3.5 | 1B–241B | Multimodal vision-language models
MiniCPM | 0.5B–9B | Optimized for edge deployment
GLM-4/4.5 | 9B–355B | Zhipu ChatGLM series
Mistral/Mixtral | 7B–8x22B | Mixture-of-experts (MoE)
Phi-3/4 | 3.8B–14B | Microsoft's compact, high-performance models
Qwen-VL 2-3 | 2B–235B | Vision-language models
4.3 Advanced Algorithm Support
Algorithm | Function | Introduced
FlashAttention-2 | Accelerated attention computation | 2023-09
Unsloth | 170% faster training + 50% less VRAM | 2023-12
DoRA | Weight-decomposed LoRA | 2024-02
GaLore | Low-rank gradient optimization | 2024-03
BAdam | Block coordinate descent optimization | 2024-04
APOLLO | Advanced optimizer | 2025-Q1
Liger Kernel | Triton kernel acceleration | 2025
FP8 training | Transformer Engine support | 2025
4.4 LLaMA Board Web UI
Zero-code visual interface built with Gradio
Model selection → dataset configuration → fine-tuning parameters → one-click training launch
Real-time monitoring: loss curves, GPU utilization, and training progress
Supports multi-backend logging with TensorBoard, Wandb, MLflow, and SwanLab
Built-in inference Chat interface for testing immediately after training
4.5 Deployment Methods
# CLI fine-tuning (one command)
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
# Web UI
llamafactory-cli webui
# OpenAI-style API deployment
API_PORT=8000 llamafactory-cli api examples/inference/qwen3.yaml
# Docker
docker run -it --rm --gpus=all hiyouga/llamafactory:latest
V. Hardware Requirements (VRAM Estimates)
Method | Precision | 7B | 14B | 30B | 70B
Full (bf16) | 32-bit | 120GB | 240GB | 600GB | 1200GB
Full (pure_bf16) | 16-bit | 60GB | 120GB | 300GB | 600GB
Freeze/LoRA | 16-bit | 16GB | 32GB | 64GB | 160GB
QLoRA | 8-bit | 10GB | 20GB | 40GB | 80GB
QLoRA | 4-bit | 6GB | 12GB | 24GB | 48GB
QLoRA | 2-bit | 4GB | 8GB | 16GB | 24GB
4-bit QLoRA quantization can fine-tune a 14B-parameter model on a single RTX 3060 (12 GB VRAM), and a 30B-parameter model on a single RTX 4090 (24 GB VRAM).
VI. Version Evolution
Version | Date | Key Changes
v0.1.0 | 2023-03 | Initial release with LLaMA fine-tuning support
Early iterations | 2023 | SFT + LoRA/QLoRA, DPO, Web UI, FlashAttention-2, Unsloth
Early iterations | 2024 Q1-Q3 | DoRA, GaLore, BAdam, FSDP+QLoRA, vLLM, multimodal LLaVA, paper publication
v0.9.1 | 2025 Q1 | Llama 3.2 Vision, LLaVA-NeXT, Pixtral, Video-LLaVA
v0.9.2 | 2025 Q1 | MiniCPM-o, SwanLab, APOLLO optimizer
v0.9.3 | 2025 Q2 | Llama 4, Gemma 3, Qwen3, InternVL3, Qwen2.5-Omni
v0.9.4 | 2025-12 | 🔄 Repository renamed to LlamaFactory; Python 3.9-3.10 deprecated
v0.9.5 | 2026-05-30 | Qwen3.5/3.6, Gemma 4, Transformers v5 compatibility
v1.x (experimental) | 2026 | New architecture: trainers → core → accelerator; modular rewrite
Key Milestones Explained
ACL 2024 paper publication — Academic recognition drove Stars from 10,000 to over 30,000
Second half of 2024 — Multimodal support (LLaVA/Video/audio) opened new markets
2025 — Maintained a monthly release cadence, supported Qwen3/DeepSeek/Llama4, and became the preferred choice in China's open-source community
Repository rename — LLaMA-Factory → LlamaFactory, reducing dependence on the Meta brand
v1 experimental architecture — Signals a major restructuring ahead
VII. Community Ecosystem
7.1 Project Ecosystem Matrix
Project | Description | Stars
LlamaFactory | Core fine-tuning framework | 72K+
EasyR1 | VLM reinforcement learning (GRPO/DAPO/Reinforce++) | Emerging
FastEdit | Efficient model fact editing | Archived
llamafactory-community | Community hub | -
Easy Dataset | Synthetic data generation tool | Third-party
DataFlow | Data processing pipeline | Third-party
7.2 Community Popularity
Chinese AI community: CSDN, Zhihu, and Blog Garden host hundreds of tutorials and practical guides
Enterprise adoption: Officially recommended by Alibaba Cloud, Huawei Ascend, Amazon AWS, and NVIDIA
Education: Used by Fudan University and many other universities as a teaching tool for large-model courses
Chinese-language support: Native support for ModelScope (Model Studio community) and OpenMind Hub mirror downloads
LLaMA Factory Online: Previously offered an online platform service; shut down on 2026-06-20
7.3 Competitor Comparison
Framework | Stars | Core Features | Model Support | Ease of Use
LlamaFactory | 72K | All-in-one GUI/CLI, ACL paper recognition | 100+ | ⭐⭐⭐⭐⭐
HuggingFace TRL | 12K | Official HF toolchain with deep ecosystem integration | All HF models | ⭐⭐⭐
Axolotl | 8K | YAML-driven customization for advanced users | Mainstream models | ⭐⭐⭐
Unsloth | 20K+ | Extreme performance optimization (2-5x acceleration) | LLaMA/Mistral, etc. | ⭐⭐⭐⭐
Firefly | 5K | Chinese large-model optimization | Primarily Chinese models | ⭐⭐⭐
LLaMA-Adapter | 6K | Academically oriented research | LLaMA series | ⭐⭐
LlamaFactory's Core Advantages:
The only framework with an ACL paper + 70K Stars + commercial deployments + a zero-code GUI
The broadest model coverage (from GPT-2 to Llama 4, and from text-only to multimodal)
China's strongest community ecosystem (HuggingFace mirrors + ModelScope integration)
VIII. SWOT Analysis
8.1 Strengths
Strength | Details
🔬 Academic authority | ACL 2024 paper, with over 500 academic citations
🎯 Extremely low barrier to entry | One-command fine-tuning and zero-code Web UI operation
🌐 Broadest model coverage | 100+ models, 60+ series, text/vision/video/audio
🇨🇳 Chinese community ecosystem | ModelScope/OpenMind mirrors and abundant Chinese tutorials
🔧 Industrial-grade toolchain | Comprehensive Docker/distributed/API deployment support
💾 Extreme VRAM optimization | Run 70B on a consumer GPU with 4-bit QLoRA
8.2 Weaknesses
Weakness | Details
👤 Severe single-maintainer dependence | hiyouga accounts for 72% of commits (2,188/3,042), Bus Factor = 1
📚 Fragmented documentation | The README is lengthy (68KB), while information across the Wiki/ReadTheDocs/community articles is fragmented
🐛 Issue backlog | Only 57 Open Issues, but discussion volume in the Issue section is enormous
🏗️ v0/v1 dual-track split | High cost of maintaining both architectures; v1 has remained unstable for a long time
📦 Complex package dependencies | CUDA/PyTorch/FlashAttn version matrix is highly error-prone
🔒 Academic-project nature | Lacks a full-time operations team, raising questions about long-term sustainability
8.3 Opportunities
Opportunity | Details
🚀 Agent fine-tuning scenarios | Explosive demand for Function Calling / Tool Use fine-tuning
🎥 Multimodal expansion | Blue-ocean market for Video/Audio/3D model fine-tuning
☁️ Cloud-native SaaS | Although LLaMA Factory Online has shut down, demand for enterprise hosting remains
🏢 Enterprise training market | Large-model fine-tuning courses and certification programs
🔗 MCP integration | Fine-tuning backend for AI Agent infrastructure
8.4 Threats
Threat | Details
🏢 In-house frameworks from major companies | Alibaba PAI and Huawei ModelArts provide integrated solutions
🔄 Competition from official HF tools | The HuggingFace TRL/peft ecosystem is becoming increasingly mature
📉 Shrinking fine-tuning demand | Prompt Engineering + RAG partially replace fine-tuning scenarios
👨🎓 Founder graduation | Project maintenance may decline after Dr. Yaowei Zheng graduates
IX. Risk Assessment
Risk Category | Risk Level | Description
Bus Factor | 🔴 High Risk | A single core developer accounts for 72% of commits; the risk of project stagnation if they leave is extremely high
Technical Debt | 🟡 Medium Risk | Dual v0/v1 architectures, 68KB README, fragmented documentation
Dependency Chain | 🟡 Medium Risk | High maintenance cost for the CUDA/PyTorch/FlashAttn version matrix
Commercialization | 🟢 Low Risk | Apache-2.0 license, with no commercial restrictions
Community Health | 🟢 Healthy | 72K Stars, an active global community, and widespread enterprise adoption
10. Technical Highlights and Innovations
10.1 "Standardized Expressway" Architecture
In traditional fine-tuning, different models require separate adaptation. LLaMA Factory uses a template registration mechanism combined with monkey‑patch technology, significantly reducing adaptation costs.
# Adding a new model takes only 3 steps:
# 1. Register the Prompt template in the TEMPLATES dictionary in template.py
# 2. Add a model compatibility patch in patcher.py (if needed)
# 3. Add multimodal support in mm_plugin.py (if needed)
10.2 Four-Layer Parameter Configuration System
YAML/JSON configuration file
→ parsed by parser.py
→ ModelArguments (model selection, quantization)
→ DataArguments (dataset, template, preprocessing)
→ FinetuningArguments (LoRA rank/target, training method)
→ TrainingArguments (inherits from HF, distributed training/logging/checkpoints)
10.3 Data Pipeline Standardization
This tool supports the two mainstream data formats, Alpaca and ShareGPT, can automatically convert more than 50 built-in datasets, and is compatible with JSON, JSONL, CSV, Parquet, Arrow, and other file formats.
10.4 Progressive Quantization Support
It supports six precision levels, from 16‑bit full precision to 2‑bit QLoRA, and can be used across different hardware environments, including A100 server clusters and ordinary laptops.
11. Competitor Comparison Matrix
Dimension | LlamaFactory | HuggingFace TRL | Unsloth | Axolotl
Stars | 72K | 12K | 20K+ | 8K
Model Support | 100+ series | All HF | LLaMA/Qwen/Mistral | Mainstream series
GUI | ✅ LLaMA Board | ❌ | ❌ | ❌
No-Code | ✅ CLI + YAML | ❌ Coding required | ❌ Coding required | YAML only
Training Paradigms | 8 | 4 (SFT/DPO/PPO/GRPO) | 3 | 5
Quantization Depth | 2-16bit | 4-16bit | 4-16bit | 4-16bit
Multimodal | ✅ Images/Video/Audio | ❌ | ❌ | ❌
Chinese Ecosystem | ✅ ModelScope + ModelBest | ⚠️ Requires a VPN | ⚠️ Requires a VPN | ❌
Academic Paper | ACL 2024 | - | - | -
Inference Engine | vLLM/SGLang/HF | HF only | HF only | vLLM/HF
Learning Curve | Very low | Moderate | Low | Moderate to high
12. Key Deployment Practices
12.1 Quick Start (3 Steps)
# Step 1: Install
git clone --depth 1 github.com/hiyouga/LlamaFactory.git
cd LlamaFactory
pip install -e ".[torch,metrics]"
# Step 2: Launch the Web UI
llamafactory-cli webui
# Step 3: Open localhost:7860 in a browser, select a model → dataset → train!
12.2 Production Environment Recommendations
# Docker deployment (recommended)
docker run -it --rm --gpus=all \
-v ./hf_cache:/root/.cache/huggingface \
-v ./data:/app/shared_data \
-v ./output:/app/output \
-p 7860:7860 -p 8000:8000 \
hiyouga/llamafactory:latest
# API deployment (OpenAI-compatible)
API_PORT=8000 llamafactory-cli api \
examples/inference/qwen3.yaml \
infer_backend=vllm
12.3 Notes for Windows Users
Manually install the GPU version of PyTorch:
pip install torch torchvision torchaudio --index-url download.pytorch.org/whl/cu126QLoRA requires the Windows-specific precompiled version of bitsandbytes
FlashAttention-2 must be compiled manually on Windows
Set
dataloader_num_workers: 0to avoid serialization errors
13. Future Outlook
v1 Architecture Stabilization: The experimental branch is expected to become the main architecture in 2026-2027, with a higher degree of modularity
Deeper Agent Fine-Tuning: Tool Use / Function Calling fine-tuning will become a standard feature
Reasoning Model Fine-Tuning: DeepSeek-R1 distillation + GRPO fine-tuning (EasyR1 project)
Cloud-Native Hosting: Third-party hosting solutions may emerge after LLaMA Factory Online shuts down
Unified Multimodal Training: Fine-tuning text/image/video/audio within a single framework
14. Rating and Summary
Dimension | Rating | Comments
Technical Architecture | 9.5/10 | Modular design, dual-architecture evolution, and a rich engine matrix
Feature Completeness | 10/10 | 8 training paradigms × 6 fine-tuning methods × 100+ models
Ease of Use | 10/10 | One-command fine-tuning + no-code Web UI, best in the industry
Documentation Quality | 7.5/10 | Detailed but overly long README, with severe fragmentation
Community Activity | 9.5/10 | 72K Stars and phenomenon-level popularity in the Chinese community
Production Readiness | 9.0/10 | Complete Docker/API/distributed support, extensively validated by enterprises
Sustainability | 6.5/10 | Bus Factor=1, academic project characteristics, long-term prospects uncertain
Innovation | 8.5/10 | ACL paper, template registration mechanism, and four-layer parameter system
Overall Rating: 9.
One-Sentence Summary
LLaMA Factory is an open-source large language model fine-tuning framework based on the ACL 2024 paper, with approximately 72,000 Stars on GitHub. Following the design principle of decoupling algorithms, models, and data, the framework provides a no-code command-line interface (CLI) and Web UI, supports fine-tuning more than 100 models, and significantly lowers the technical barrier to LLM fine-tuning. It can serve as infrastructure for deploying LLM applications.