Few-Shot Learning

Few-shot learning (FSL) is a subfield of machine learning focused on training models to recognize or classify new concepts using only a very small number of labeled examples—typically one to five per class. Unlike traditional deep learning approaches that require thousands of labeled samples per category, FSL algorithms are designed to generalize rapidly from minimal supervision, mimicking human-like sample efficiency.1

Background & Motivation

Traditional deep learning models excel when trained on large, diverse datasets (e.g., ImageNet with 1.2M images). However, acquiring and annotating such data is costly, time-consuming, and often impractical in specialized domains like medical imaging, rare disease diagnosis, or low-resource languages.2

Few-shot learning addresses this bottleneck by shifting the objective from learning to classify to learning how to learn. The model is pretrained on a large source domain with many classes and tasks, then adapted to a target domain with minimal examples. This paradigm has become foundational in modern AI systems, particularly in vision, natural language processing, and robotics.3

Core Concepts

FSL is often discussed alongside two related paradigms that exist on a spectrum of sample efficiency:

One-Shot Learning

A strict subset of FSL where the model must learn a new class from exactly one labeled example. Commonly used in facial recognition systems where only a single reference photo is available per person.4

Zero-Shot Learning

In zero-shot learning (ZSL), the model classifies instances of classes it has never seen during training, relying instead on semantic descriptors (e.g., attribute vectors, class embeddings, or textual descriptions). ZSL and FSL are complementary; modern systems often combine both (e.g., generalized zero/few-shot learning).5

Methodologies

FSL approaches are generally categorized into three families based on how they handle the support set S and query set Q:

1. Metric-Based Learning

These methods learn a feature embedding space where similarity between support and query samples can be measured directly using distance metrics (e.g., Euclidean, cosine). The classifier assigns a query to the nearest support class centroid.

• Siamese Networks: Compare pairs of images using shared convolutional layers and a contrastive loss.6
• Prototypical Networks: Compute class prototypes (mean embeddings of support examples) and classify queries by nearest prototype.7
• Relation Networks: Learn a explicit neural relation function to score query-support pairs.

2. Optimization-Based Learning

These methods treat adaptation as a meta-optimization problem. The model learns an efficient initialization that can be fine-tuned with a few gradient steps on the support set.

• MAML (Model-Agnostic Meta-Learning): Learns parameters such that a small number of gradient updates on new tasks yield high performance.8
• Reptile & ANIL: Variants that simplify MAML by using parameter averaging or weight initialization decoupling.

3. Transformation-Based Learning

These approaches adapt the feature extractor or input data distribution to match the target task, often using domain adaptation, data augmentation, or prompt tuning in large language models.9

Applications

• Medical Imaging: Rare pathology detection from limited annotated scans.
• Low-Resource NLP: Named entity recognition and sentiment analysis for underrepresented languages.
• Robotics: Rapid skill acquisition from few demonstration episodes.
• Industrial Inspection: Defect detection in manufacturing where failure cases are scarce.
• Large Language Models: In-context learning (ICL) in transformers, where prompts serve as the "few shots".10

Challenges & Limitations

• Task Distribution Shift: Performance degrades when target tasks differ significantly from pretraining tasks.
• Overfitting to Support Set: With extremely limited data, models may memorize rather than generalize.
• Computational Cost: Meta-learning often requires nested optimization loops, increasing training time.
• Evaluation Standardization: Benchmark datasets (e.g., MiniImageNet, tieredImageNet) vary in split strategies, complicating cross-study comparisons.

Recent Advances

As of 2024–2025, FSL research has converged on several key directions:

• Vision-Language Pretraining: CLIP-style models show strong zero/few-shot generalization by aligning visual and textual embeddings.11
• Hypernetworks & Dynamic Adaptation: Generating task-specific adapter parameters conditioned on the support set.
• Graph-Based Relational Reasoning: Modeling class relationships explicitly to improve cross-class generalization.
• Self-Supervised Foundation Models: Using MAE, DINOv2, and similar architectures as robust backbones for FSL.