Video rotoscoping is traditionally where creative enthusiasm goes to die. If you have ever tried to cut a moving subject out of a forty-second clip manually, you know the particular brand of existential dread that arrives around frame 412. When Meta dropped the original Segment Anything Model (SAM), static image annotation changed overnight. But feeding video through SAM 1 felt like running a marathon in lead boots: it treated every frame as an isolated island, completely unaware of temporal flow, racking up drift errors, and devouring compute.
Enter SAM 2 (Segment Anything Model 2) from Meta FAIR. Hosted at facebookresearch/sam2, this unified model processes both static frames and continuous video streams with promptable segmentation. It brings true temporal memory to visual tracking, running at near real-time speeds on commodity consumer GPUs.
Here is an architectural breakdown, practical setup guide, and implementation walkthrough to get SAM 2 tracking objects on your local machine.
What Is SAM 2?
Definition: SAM 2 is a foundation model developed by Meta FAIR for promptable, zero-shot visual segmentation across images and video. By employing a streaming memory architecture, SAM 2 treats images simply as single-frame videos, allowing developers to propagate clicks, bounding boxes, or masks seamlessly across temporal sequences.
┌──────────────────────────────┐
│ User Prompt (Click/Box) │
└──────────────┬───────────────┘
│
┌──────────────┐ ┌──────▼──────┐ ┌──────────────┐
│ Video Frame ├─────────►│ Mask Decoder├─────────►│ Output Mask │
└──────────────┘ └──────▲──────┘ └──────┬───────┘
│ │
┌──────┴──────┐ │
│Memory Engine│◄────────────────┘
└─────────────┘ (Embeds Spatial Context)
Architectural Key Concepts
- Streaming Memory Architecture: Unlike bidirectional video transformers that choke on longer sequences by loading whole videos into VRAM, SAM 2 processes frames sequentially. It conditions predictions on a memory bank of past frame interactions.
- Memory Encoder and Bank: When a user prompts a target on frame $N$, the memory encoder extracts spatial features and appends them to a FIFO (First-In, First-Out) memory bank, alongside explicit prompt tokens.
- Occlusion-Aware Mask Head: If an object ducks behind a tree or disappears off-screen, a dedicated pointer head predicts object presence probability, avoiding the classic "hallucinated ghost mask" problem.
SAM 1 vs SAM 2: Core Architectural Differences
| Feature | SAM 1 (2023) | SAM 2 (2024–Present) |
|---|---|---|
| Primary Domain | Static 2D Images | Unified 2D Images & Video Streams |
| Temporal Context | None (Frame-agnostic) | Streaming Memory Bank & Attention |
| Interactive Latency | ~50 ms per image (ViT-H) | ~44 FPS (Hiera-B+ / Video Inference) |
| Occlusion Handling | Fails on reappearing targets | Dedicated occlusion tracking head |
| Backbone Family | Standard Vision Transformer (ViT) | Hierarchical Vision Transformer (Hiera) |
Local Installation and Hardware Requirements
The developer community across Reddit and YouTube initially worried that video foundation models would mandate an enterprise cluster of H100s. In practice, the lightweight checkpoints (sam2_hiera_tiny and sam2_hiera_small) run smoothly on modern consumer cards with 8 GB to 12 GB of VRAM. For uncompromised mask fidelity, the larger sam2_hiera_large runs comfortably within 16 GB to 24 GB of VRAM.
Prerequisites
- Python 3.10 or newer
- PyTorch 2.3.1+ with matching CUDA runtime
- Linux or macOS (WSL2 supported on Windows)
# 1. Create a dedicated virtual environment
python3 -m venv sam2_env
source sam2_env/bin/activate
# 2. Install PyTorch with CUDA acceleration (adjust for your CUDA version)
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121
# 3. Clone and install the SAM 2 repository
git clone https://github.com/facebookresearch/sam2.git
cd sam2
pip3 install -e .
# 4. Download pre-trained model weights
cd checkpoints
./download_ckpts.sh
cd ..
Practical Code Walkthrough: Propagating Masks Across Video
Here is how to initialise the video predictor, add a single-point prompt to identify a subject on the initial frame, and propagate that mask through the entire video stream.
Ensure your video frames are extracted as an image directory (JPEG or PNG), which is the standard input format for the low-level inference state.
import os
import torch
import numpy as np
from sam2.build_sam import build_sam2_video_predictor
# Define hardware runtime and model checkpoint
device = "cuda" if torch.cuda.is_available() else "cpu"
model_cfg = "sam2_hiera_l.yaml"
checkpoint = "./checkpoints/sam2_hiera_large.pt"
# Initialise the video predictor engine
predictor = build_sam2_video_predictor(model_cfg, checkpoint, device=device)
# Path to the directory containing ordered video frames (e.g., 00000.jpg, 00001.jpg)
video_dir = "./frames/skateboard_clip"
# Initialise internal memory state for the sequence
inference_state = predictor.init_state(video_path=video_dir)
# Define positive prompt on frame 0 (x, y coordinates of the target object)
# labels: 1 indicates a foreground click; 0 indicates a background exclusion click
points = np.array([[640, 360]], dtype=np.float32)
labels = np.array([1], dtype=np.int32)
target_obj_id = 1
# Add interaction to frame index 0
_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
inference_state=inference_state,
frame_idx=0,
obj_id=target_obj_id,
points=points,
labels=labels,
)
print(f"Tracking object {target_obj_id} successfully initialised.")
# Propagate masks across all subsequent video frames
video_segments = {}
for out_frame_idx, out_obj_ids, out_mask_logits in predictor.propagate_in_video(inference_state):
# Convert binary mask logits to booleans
masks = (out_mask_logits[0] > 0.0).cpu().numpy()
video_segments[out_frame_idx] = {
out_obj_id: masks[i] for i, out_obj_id in enumerate(out_obj_ids)
}
print(f"Propagation complete. Tracked across {len(video_segments)} frames.")
Refining Tracking During Runtime
If the target turns sideways on frame 80 and the bounding boundary begins to blur, you do not need to restart inference. Call predictor.add_new_points_or_box() on frame 80 with a corrective point. SAM 2 updates its memory cache and automatically re-propagates forward and backward throughout the timeline.
Why SAM 2 Stands Out
1. True Multimodal Consistency: Most open-source vision tools strap a 2D segmenter to an external Kalman filter or optical flow tracker. SAM 2 bakes temporal awareness straight into the attention layers, cutting out plumbing code and fragile intermediate models.
2. Hiera Backbone Efficiency: By abandoning older standard Vision Transformers for the Hierarchical Vision Transformer (Hiera), SAM 2 removes redundant operations while boosting inference speeds by up to three times compared to SAM 1.
3. Robust Zero-Shot Generalisation: Trained on the SA-V dataset (containing roughly 51,000 videos and over 600,000 spatio-temporal masklets), the model generalises cleanly across noisy security footage, aerial drone sweeps, and macro lenses without bespoke fine-tuning.
If your workflows touch visual effects, sports analytics, or local video parsing for LLM vision agents, facebookresearch/sam2 is the definitive modern open-source tool for pixel-level extraction. Clone the repository, grab the checkpoint weights, and reclaim the hours you once lost to manual rotoscoping.