SAM - Segment Anything model for promptable pixel segmentation

Описание: This video talks about SAM - an attempt to build a foundational segmentation model by Facebook researchers.
Instead of classical segmentation problems like Semantic Segmentation or Instance Segmentation, which rely on existing set of labels, SAM deals with "Promptable Segmentation" - given a bounding box or a point "prompt", return segmentation for the object to which that point belongs, no matter what that object might be.
This unusual approach allowed creating of a huge, diverse dataset for segmentation and training a good model. On the other hand, SAM model cannot solve other segmentation tasks (like instance segmentation) on its own, but it can do that with an extra model to generate prompts, like object detector for example.
The video also dives deep into the architecture of the model - prompt encoding, Decoder with query-to-image cross attention, upscaling with Transposed Convolution, maks segmentation and Dice and IoU loss.
Important links:
Original paper: https://arxiv.org/pdf/2304.02643
DETR paper: https://arxiv.org/pdf/2005.12872
MaskFormer paper: https://arxiv.org/pdf/2107.06278
Tesla Autopilot video with 3d depth estimation:    • FULL Andrej Karpathy Tesla Autonomous Driv...  
Tesla Optimus video with occupancy detection: https://www.youtube.com/live/ODSJsviD...

00:00 - Intro
01:17 - Segmentation Task
07:04 - Promptable Segmentation
08:58 - Data Engine
14:34 - Model Architecture
15:45 - Prompt Encoding
18:02 - Decoder, Cross Attention
28:04 - Loss
33:08 - Transposed Convolution
37:02 - Results