🧠In-Context Learning

NEW

A large model’s ability to perform new tasks from examples given in the prompt without updating its parameters.

DESCRIPTION

In-context learning (ICL) refers to models solving tasks by conditioning on demonstrations or instructions within the prompt. It emerged prominently with large language models, which can generalize from few examples and instructions. ICL enables rapid adaptation without retraining, accelerating prototyping and reducing operational overhead. Techniques like chain-of-thought and retrieval augmentation often improve ICL reliability. However, ICL can be sensitive to prompt ordering, formatting, and spurious cues.

ALTERNATIVE DEFINITIONS

Brown et al., 2020 (GPT-3)

We demonstrate that language models can learn to perform tasks from textual instructions and examples provided in the context, without any parameter updates.

Introduces the phenomenon in large language models and no-training adaptation.

Source

Stanford AI Blog

In-context learning is the ability of a model to learn a new task simply by being prompted with examples of that task.

Plain-language definition emphasizing example-driven prompting.

Source

RELATED TERMS

Few-Shot Learning Zero-shot Learning Prompt Engineering Chain-of-Thought Retrieval-Augmented Generation Meta-Learning Instruction Tuning

COMPETING TERMS

Fine-Tuning

KNOWLEDGE AREAS

AI Data Science

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer AI Product Manager

EXTERNAL SOURCES

• Wikipedia - Large language model
• GPT-3 Paper - Seminal paper showing ICL
• ICL Survey - Survey on in-context learning
• Microsoft Learn - Prompting with examples

TOP QUESTIONS

What is In-Context Learning in large language models?

It’s the ability to learn tasks from prompt examples or instructions without gradient updates. The model adapts behavior purely from context.

Category: Overview

How does In-Context Learning differ from fine-tuning?

ICL adapts at inference time via prompts; fine-tuning changes model weights using training data and compute.

Category: Comparison

What improves In-Context Learning performance?

Careful prompt design, ordering, chain-of-thought, and retrieval-augmented examples typically improve reliability and accuracy.

Category: BestPractices

What are risks of relying on In-Context Learning?

ICL can be brittle to prompt phrasing, exhibit biases, and vary across runs. It may underperform on complex domain tasks.

Category: Risk

When should I prefer In-Context Learning over fine-tuning?

Use ICL when you need rapid iteration, low operational overhead, and tasks that respond well to examples and instructions.

Category: Guidance

Is In-Context Learning the same as few-shot learning?

They overlap: few-shot learning via prompting is a form of ICL. However, few-shot can also include parameter-updating approaches.

Category: Misunderstanding

How many examples are needed for In-Context Learning?

It varies by task and model. Many tasks work with 0–8 examples; more can help but costs token budget.

Category: Requirements

Does In-Context Learning generalize beyond text?

Yes. Multimodal models can use in-context signals across text, images, or other modalities.

Category: Scope

What is a hot topic around In-Context Learning?

Understanding the mechanisms behind ICL and improving robustness via retrieval, tool-use, and structured prompting are active research areas.

Category: Topic