For most of the 2010s, computer vision meant convolutional neural networks (CNNs). Language processing, meanwhile, had been transformed by the transformer architecture. In October 2020, a team at Google Research led by Alexey Dosovitskiy asked a provocative question: what if we simply treat an image like a sentence? Their paper, An Image is Worth 16x16 Words, introduced the Vision Transformer (ViT) and started the convergence of vision and language architectures that powers today’s multimodal AI.

Why CNNs dominated

Convolutional networks build in useful assumptions about images, called inductive biases:

  • Locality — nearby pixels are related, so filters look at small neighborhoods.
  • Translation equivariance — a cat is a cat wherever it appears, so the same filter slides across the whole image.

These assumptions make CNNs data-efficient: they learn well from relatively modest datasets.

Transformers have almost none of these biases. Their attention mechanism can relate any position to any other from the start. That flexibility seemed wasteful for images.

How ViT works

The architecture is intentionally close to the original transformer.

  1. Cut the image into patches. A 224×224 image is split into fixed-size patches, for example 16×16 pixels, giving 196 patches.
  2. Flatten and embed. Each patch is flattened into a vector and passed through a linear layer, becoming a “token”, just like a word embedding.
  3. Add position information. Learned position embeddings tell the model where each patch came from.
  4. Add a class token. A special learnable token is prepended; its final representation is used for classification.
  5. Run a standard transformer encoder. Layers of multi-head self-attention and feed-forward networks, with residual connections and normalization.
  6. Classify. A small head on top of the class token predicts the label.

There are no convolutions in the main body of the model.

The key result: data beats bias

ViT’s performance depended dramatically on the amount of pre-training data.

  • Trained only on ImageNet (about 1.3 million images), ViT did worse than comparable ResNets. Without built-in assumptions about images, it needed more examples to learn them.
  • Pre-trained on much larger datasets — ImageNet-21k (about 14 million images) or Google’s internal JFT-300M (about 300 million images) — and then fine-tuned, ViT matched or exceeded the best CNNs on image classification benchmarks, while requiring substantially less compute to pre-train.

The lesson echoed what was happening in language: with enough data, general-purpose architectures learn the right structure themselves, and they scale better.

What the model learned

The authors examined what ViT learned. Its early layers developed filters resembling the edge and texture detectors of CNNs. Some attention heads attended locally, like convolutions, while others integrated information across the entire image from the earliest layers — something CNNs only achieve deep in the network.

Limitations

Data hunger. Without large-scale pre-training, ViTs underperform. Follow-up work such as DeiT (2021) showed that strong data augmentation and distillation can make ViTs competitive when trained on ImageNet alone.

Cost at high resolution. Attention cost grows quadratically with the number of patches, making very high-resolution images expensive. Hierarchical variants such as the Swin Transformer reintroduced some locality to address this.

The legacy

ViT’s real impact was unification. Once images became sequences of tokens, the same architecture could process text and images together:

  • CLIP used a vision transformer to learn joint image-text representations.
  • Multimodal language models turn image patches into tokens that the language model reads alongside text.
  • Self-supervised methods such as masked autoencoders and DINO trained ViTs without labels.
  • Diffusion transformers brought the architecture into image and video generation.

Today, when a chatbot describes a photo or reads a chart, a descendant of the Vision Transformer is very likely doing the seeing.

Frequently asked questions

Why does ViT need so much more data than a CNN?

Convolutional networks start with built-in assumptions that nearby pixels are related and that patterns look the same anywhere in an image. A Vision Transformer has to learn those regularities from examples. With enough data it learns them — and more flexible relationships too — but on small datasets the missing assumptions are a handicap.

What is a “patch” in a Vision Transformer?

A small square of the image, for example 16 by 16 pixels. Each patch is flattened into a vector and treated like one word in a sentence. A 224 by 224 image therefore becomes a sequence of 196 tokens that the transformer processes with attention.

Have transformers completely replaced CNNs in vision?

No. Modern convolutional networks remain competitive, efficient and widely deployed, especially on phones and embedded devices, and hybrid designs combine both ideas. Transformers dominate where vision must be combined with language, and at very large scale.

Many multimodal models use a vision transformer as their “eyes”. The image is encoded into patch tokens, these are projected into the same space as text tokens, and the language model reads them together with your question.