Segment Anything Model

You're Claude Code, working in my repository.

Task
Pick the Segment Anything recipe on this page that fits my project (segmenting images or video, cutting a subject out into stickers, or pairing tracked people with speech), tell me which and why, then build its first step: one streamed SAM 3.1 request for a short noun phrase on an image or an uploaded video, parsed into boxes, masks and tracks with Meta's parser.

Read this first
Meta's docs page "Segment Anything Model" as Markdown: https://dev.meta.ai/docs/cookbook/sam.md
It's the source of truth for request and response shapes. The same page for people: https://dev.meta.ai/docs/cookbook/sam

Setup
- Base URL: https://api.meta.ai/v1
- Auth: send Authorization: Bearer $MODEL_API_KEY, reading MODEL_API_KEY from the environment. Never hard-code, print or commit the key.
- Model: sam-3.1
- Python: the OpenAI SDK (pip install openai) with base_url="https://api.meta.ai/v1" and api_key=os.environ["MODEL_API_KEY"] works, or plain requests.

Facts to get right
- POST https://api.meta.ai/v1/responses with model sam-3.1 and stream true. The user message holds input_text (one short noun phrase, not a question) and input_image (a data URL or file_id) or input_video (a file_id from POST /v1/files with purpose=user_data).
- The result is output_text with box and mask tokens per frame: parse it with @meta-sam/parser (parseImageStream, parseVideoStream) or meta_sam_parser (parse_responses_stream) rather than by hand.
- A completed request with no masks is a valid empty match. Fail clearly on HTTP errors, refusals, failed or incomplete responses and malformed records.
- Images cost $2.50 per 1,000; video $0.20 per 1,000 frames with a 50-frame minimum per request (usage.video_frames_processed).

Starter code (Python)
import asyncio, base64, os

from meta_sam_parser import decode_mask_to_raster, image_segmentation_format, parse_responses_stream
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="https://api.meta.ai/v1", api_key=os.environ["MODEL_API_KEY"])


async def main() -> None:
    with open("ducks.jpg", "rb") as f:
        image = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode()
    stream = await client.responses.create(model="sam-3.1", stream=True, input=[{"role": "user", "content": [
        {"type": "input_text", "text": "duck"}, {"type": "input_image", "image_url": image}]}])
    result = await parse_responses_stream(stream, image_segmentation_format()).final_result()
    masks = [r for r in result.records if r.kind == "mask"]
    print(result.outcome.status, len(masks), "masks")  # zero masks is a valid empty match
    for m in masks:
        print(m.object_id, m.bounds, sum(decode_mask_to_raster(m.mask)), "pixels")

asyncio.run(main())

Done when
- A noun phrase on an image returns one box and one mask per match.
- An uploaded video streams frames whose object ids stay stable across frames.
- A concept that isn't there is reported as an empty match, not an error.
- The key comes from MODEL_API_KEY and never appears in code, logs or commits.
- API errors (4xx, 429) surface their message instead of failing silently.