AI GitHub
SAM 2 for prompted image and video segmentation
Generate object masks from visual prompts and propagate them through video.
Key Takeaways
- SAM 2 uses visual prompts such as points or boxes to identify the region of interest.
- Choose a compatible model configuration and checkpoint.
- Prompt one target in a video that includes an occlusion.
- The mask follows the provided prompt and can drift to another object.
How it works
SAM 2 uses visual prompts such as points or boxes to identify the region of interest. Its video workflow carries information across frames to follow objects.
What you need
Choose a compatible model configuration and checkpoint. Prepare the visual prompts and decide how video frames and returned masks will be aligned in the application.
What to try first
Prompt one target in a video that includes an occlusion. Inspect masks before, during and after the interruption and test an additional correction prompt.
Try a small object near an edge and a larger object with a similar color. Tracking both makes it easier to see where the prompt or memory loses the target.
Where to look for failures
The mask follows the provided prompt and can drift to another object. Review ambiguous frames before treating the output as an annotation or measurement.
Related Franklin pages
Choose between text-guided boxes, prompted masks and downstream tracking. Preserve a few ambiguous images and videos so each stage can be evaluated independently.
- Find image regions that match natural-language phrases. Grounding DINO for text-guided object detection
- Estimate scene depth from a single image with published checkpoints. Depth Anything V2 for monocular depth estimation
For an overview of the types of predictions vision systems produce, see Franklin’s glossary. Computer vision
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗