In January 2021, OpenAI introduced CLIP (Contrastive Language–Image Pre-training), a model that learned to understand images by reading the captions that accompany them on the internet. CLIP did not just classify photos well; it showed that a vision model could recognize new categories it was never explicitly trained on, simply by being told their names in words. It became a building block of the image-generation boom that followed.

The problem with labelled datasets

Classic image classifiers are trained on fixed label sets. ImageNet, for example, has 1,000 categories. A model trained on it can only answer “which of these 1,000 things is this?” Adding a new category requires collecting and labelling new images and retraining.

Labelling is expensive, and fixed categories capture only a narrow slice of the visual world. Meanwhile, the web contains hundreds of millions of images with natural-language descriptions attached — captions, alt text, titles — which describe far richer concepts than any label list.

How CLIP learns

The OpenAI team collected a dataset of about 400 million image–text pairs from the internet. CLIP consists of two encoders:

  • an image encoder (the team tried ResNets and Vision Transformers), which turns an image into a vector;
  • a text encoder (a transformer), which turns a caption into a vector in the same space.

Training uses a contrastive objective. In each batch of image–caption pairs, the model is asked: which caption belongs to which image? It is rewarded for making the vectors of matching pairs similar and the vectors of mismatched pairs dissimilar. With large batches, each image must be matched against tens of thousands of wrong captions, which forces the model to learn fine distinctions.

Crucially, CLIP never learns a fixed list of classes. It learns a shared embedding space where images and descriptions of their content end up close together.

Zero-shot classification

Once trained, CLIP can classify images into categories it has never been trained on.

  1. Write a text prompt for each candidate class, such as “a photo of a dog” and “a photo of a cat”.
  2. Encode the prompts and the image.
  3. Pick the class whose text vector is most similar to the image vector.

Using this method with no ImageNet training examples at all, the best CLIP model matched the accuracy of the original fully supervised ResNet-50 on ImageNet. The paper evaluated CLIP across more than 30 existing datasets, including fine-grained object recognition, action recognition in video and text recognition in images.

CLIP was also notably more robust to distribution shift: on variants of ImageNet with sketches, renditions or adversarially filtered images, it degraded much less than models trained only on ImageNet.

Where CLIP falls short

  • Fine-grained and abstract tasks: counting objects, estimating distances or distinguishing car models was harder.
  • Prompt sensitivity: results depended on the wording of class prompts, which led to “prompt engineering” for vision.
  • Biases: trained on unfiltered internet data, CLIP absorbed social biases, which the authors documented — including troubling misclassifications of people into crime-related or non-human categories in certain setups.
  • Typographic attacks: because CLIP learned to read text in images, writing a word on an object could change its prediction.

Why it mattered so much

CLIP’s shared image–text space became infrastructure for generative AI:

  • Text-to-image generation: CLIP-style text encoders and CLIP-guided methods helped steer diffusion models toward images that match a prompt, and the original Stable Diffusion used a CLIP text encoder.
  • Image search: describing what you want in words and retrieving matching images.
  • Evaluation: “CLIP score” became a common way to measure how well a generated image matches its prompt.
  • Multimodal language models: many early vision-language assistants used CLIP vision encoders to turn images into features a language model could read.

Open reproductions such as OpenCLIP, trained on public datasets, extended these benefits to the wider research community.

Key takeaways

  • CLIP trained an image encoder and a text encoder on about 400 million image-caption pairs from the web.
  • A contrastive objective puts matching images and captions close together in a shared vector space.
  • It can classify images zero-shot, matching a supervised ResNet-50 on ImageNet without ImageNet training.
  • It inherited biases from internet data and could be fooled by text written on objects.
  • CLIP-style encoders became core components of image generators and multimodal assistants.

Frequently asked questions

What does “contrastive learning” mean?

Training a model to pull matching pairs together and push non-matching pairs apart in a shared vector space. CLIP learns which caption goes with which image by comparing every image in a batch with every caption.

Can CLIP generate images?

No. CLIP only encodes images and text into vectors and measures how well they match. Image generators use CLIP-like text encoders or CLIP scores to guide a separate generative model, such as a diffusion model.

What is zero-shot classification?

Classifying an image into categories the model was not explicitly trained on. With CLIP you simply describe each category in words, and the model picks the description that best matches the image.