🧩LoRA

NEW

Also known as: Low-Rank Adaptation

A parameter-efficient fine-tuning method that inserts trainable low-rank adapters while freezing pretrained model weights.

DESCRIPTION

LoRA adapts large pretrained models by factorizing weight updates into low-rank matrices, drastically reducing trainable parameters. This preserves the original weights, enabling fast, memory-efficient training and easy rollback. LoRA modules can be selectively applied to attention and feedforward layers, balancing performance and resource use. It supports merging adapted weights into the base for deployment or keeping adapters separate for flexibility. LoRA has become a standard for customizing large language models under limited compute budgets, with libraries like PEFT streamlining use.

ALTERNATIVE DEFINITIONS

Hu et al.

We propose Low-Rank Adaptation (LoRA), which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer to reduce the number of trainable parameters.

Original definition stressing freezing base weights and low-rank matrices.

Source

Hugging Face

LoRA injects trainable low-rank matrices into existing weights and freezes the original weights, greatly reducing the number of trainable parameters.

Implementation-centric definition with emphasis on efficiency.

Source

Microsoft Research

LoRA adapts large models by expressing weight updates with low-rank factorizations, enabling efficient fine-tuning without modifying the original weights.

Research blog highlighting rationale and efficiency benefits.

Source

RELATED TERMS

Parameter-Efficient Fine-Tuning Prompt Tuning QLoRA Knowledge Distillation Mixed Precision Training Regularization Fine-Tuning

COMPETING TERMS

Full Fine-Tuning Adapter Tuning Prefix Tuning

KNOWLEDGE AREAS

AI MLOps

TARGET AUDIENCE

AI Engineer MLOps Engineer Data Scientist AI Product Manager

EXTERNAL SOURCES

• Wikipedia - Low-rank adaptation
• arXiv - Original LoRA paper
• Hugging Face - PEFT library and LoRA docs
• Lightning AI - Tutorial on LoRA for LLMs

TOP QUESTIONS

Why is LoRA popular for large language models?

LoRA reduces trainable parameters and memory, enabling fine-tuning large models on modest hardware without sacrificing much performance.

Category: Relevancy

How does LoRA differ from full fine-tuning?

Full fine-tuning updates all weights; LoRA freezes base weights and learns small low-rank adapters, saving compute and preserving the base model.

Category: Comparison

Where should LoRA be applied in a transformer?

Common choices include attention projection matrices and feedforward layers; selection impacts quality and efficiency.

Category: HowTo

Can LoRA adapters be merged into the base model?

Yes, adapters can be merged post-training for simpler deployment, or kept separate for flexibility across tasks.

Category: Deployment

What rank should I choose for LoRA?

Typical ranks are 4–64; tune based on quality vs. parameter budget for your task and model size.

Category: Guidance

How does QLoRA relate to LoRA?

QLoRA combines low-rank adaptation with low-precision quantization of base weights to further reduce memory usage.

Category: Relation

Does LoRA hurt inference latency?

Merged adapters have negligible overhead; unmerged adapters add small extra matmul cost, often acceptable.

Category: Performance

Is LoRA suitable for vision models?

Yes, LoRA applies beyond NLP, including vision transformers and multimodal models, with similar efficiency gains.

Category: Application

What are common mistakes when using LoRA?

Using too low a rank, adapting too few layers, or neglecting learning-rate tuning can underfit and degrade performance.

Category: Mistakes