grabbou/sam3-video-CoreAI

SAM 3 video for Core AI

facebook/sam3 exported to Apple Core AI as a video segmenter with the apple/coreai-models recipe, at 672 px for iPhone:

uv run models/sam3_video/export.py --dtype float16 --image-size 672

Folder	Contents	Size	Platforms
sam3-video/	CoreAIVideoSegmentation bundle: sam3_video_672_float16.aimodel (image and text encoders, detector, tracker and memory encoder functions), the CLIP tokenizer/, metadata.json (kind: video_segmenter, tracking defaults)	1.6 GB	iOS, macOS

Give it a video and text prompts (cat, the red car). It finds the matching objects and tracks them from frame to frame with stable ids. The recipe also accepts 336 px (faster, less accurate on small objects) and 1008 px (the default, slower).

Use

With @react-native-ai/core-ai, download sam3-video/ into app storage, then:

const { output } = await CoreAI.runTask(
  'video-segmentation',
  { id: 'sam3-video', sourceType: 'file', sourceUri: pathToFolder },
  { videoUri, data: { prompts: ['cat'] } },
  { outputVideo: true }
)

In Swift, load the folder with VideoSegmenter(resourcesAt:) from the CoreAIVideoSegmentation product of coreai-models.

License

SAM 3 is licensed by Meta under the SAM License, which applies to this converted copy too: a copy of the agreement is included, as it requires. This repository only converts the model to Core AI format.