DataTerminology.comNEW
Also known as: Contrastive Language-Image Pretraining
A model that learns joint image–text representations via contrastive pretraining, enabling zero-shot recognition and retrieval.
DESCRIPTION
CLIP aligns images and text in a shared embedding space using contrastive learning on large-scale web data. By pairing a visual encoder with a text encoder, it learns semantic correspondences without task-specific labels. The resulting embeddings enable zero-shot classification, retrieval, and transfer to many downstream tasks. CLIP popularized prompt engineering for vision and catalyzed multimodal research and products. It is widely available as open models and integrated in ML frameworks.
ALTERNATIVE DEFINITIONS
OpenAI
CLIP learns visual concepts from natural language supervision and can be applied to a wide range of visual classification benchmarks in a zero-shot manner.
Emphasizes natural language supervision and zero-shot capability.
arXiv
We demonstrate that models trained to predict which caption goes with which image learn state-of-the-art representations for a wide range of tasks.
Highlights contrastive caption-image training as the core mechanism.
RELATED TERMS
Vision-Language Model Contrastive Learning Zero-shot Learning OpenAI Vision Transformer ResNet Prompt Engineering Image-Text Retrieval Joint Embedding Linear Probe
COMPETING TERMS
RELATIONS
- Broader: Vision-Language Model
KNOWLEDGE AREAS
AI Data Science Real-time & Streaming Data Platform & Tooling
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - CLIP (artificial intelligence)
- OpenAI - Official CLIP research page
- arXiv - CLIP research paper
- Hugging Face - Pretrained CLIP model card
TOP QUESTIONS
How does CLIP enable zero-shot classification?
It embeds class prompts and images into the same space, then selects the class with the highest cosine similarity to the image embedding.
Category: Explanation
What architectures are used in CLIP?
CLIP typically uses a ViT or ResNet visual encoder and a Transformer text encoder trained with a contrastive loss.
Category: Architecture
What are common applications of CLIP?
Zero-shot classification, image-text retrieval, content moderation, and as a vision encoder for multimodal LLMs.
Category: Use Cases
What are limitations of CLIP?
It may reflect web data biases, struggle with fine-grained distinctions without prompt tuning, and be sensitive to prompt phrasing.
Category: Risks
How do I improve CLIP performance?
Use better prompts, prompt ensembling, linear probes, or fine-tune adapters on domain-specific data.
Category: Best Practices
How does CLIP compare to ALIGN?
Both use image-text contrastive pretraining at scale; ALIGN is a Google counterpart with large proprietary datasets.
Category: Comparison
Can CLIP be used for video tasks?
Yes, by applying frame-level encodings or using video extensions, though specialized video-text models may perform better.
Category: Capabilities
Is CLIP open source?
OpenAI released weights for select CLIP variants and code; many reimplementations and compatible models exist.
Category: Licensing
Where should I start learning CLIP?
Read the OpenAI CLIP post, the original paper, then try the Hugging Face model with zero-shot examples.
Category: Getting Started