Segmentation with Segment Anything Model

You're Claude Code, working in my repository.

Task
Make a first end-to-end SAM 3.1 call from my project: POST /v1/responses with model sam-3.1, one user message holding an input_text noun phrase and an input_image, and stream: true; then parse the streamed output with @meta-sam/parser (parseImageStream, recordsOfKind, decodeMaskToRaster) and print each object's id, box and mask size.

Read this first
Meta's docs page "Segmentation with Segment Anything Model" as Markdown: https://dev.meta.ai/docs/sam/overview.md
It's the source of truth for request and response shapes. The same page for people: https://dev.meta.ai/docs/sam/overview

Setup
- Base URL: https://api.meta.ai/v1
- Auth: send Authorization: Bearer $MODEL_API_KEY, reading MODEL_API_KEY from the environment. Never hard-code, print or commit the key.
- Model: sam-3.1
- Python: the OpenAI SDK (pip install openai) with base_url="https://api.meta.ai/v1" and api_key=os.environ["MODEL_API_KEY"] works, or plain requests.

Facts to get right
- Same base URL and Bearer key as the rest of the API. Video goes in an input_video part: MP4 or MOV with H.264, H.265 or MPEG-4.
- The result streams in the output_text lane as special-token text, one line per frame, and the stream ends on response.completed. Streaming is recommended for video, not required.
- <0f> is the frame, the leading integer the object id (stable across frames), then one box and one mask; ~ marks a lossless mask payload, ! one_bit.
- Zero matches completes with an empty output_text: branch on it. A stream can also fail after HTTP 200 (an error event, then response.failed): a failure, not zero matches. usage reports images_processed or video_frames_processed, not tokens.

Starter code (Python)
import os

from openai import OpenAI

client = OpenAI(base_url="https://api.meta.ai/v1", api_key=os.environ["MODEL_API_KEY"])

output = ""
with client.responses.stream(
    model="sam-3.1",
    input=[{"type": "message", "role": "user", "content": [
        {"type": "input_text", "text": "dog"},  # one short noun phrase
        {"type": "input_image", "image_url": "https://example.com/street.jpg"},  # or a data: URL
    ]}],
    metadata={"mask_encoding": "one_bit", "include_confidence": "true"},
) as stream:
    for event in stream:
        if event.type in ("error", "response.failed"):
            raise RuntimeError(f"SAM failed: {event}")  # a failure under HTTP 200, not zero matches
        if event.type == "response.output_text.delta":
            output += event.delta  # one line per frame
if not output:
    print("No matches: a valid result.")
for line in output.splitlines():
    print(line[:160])  # <0f>0<|box;x1=..;y1=..;x2=..;y2=..;w=..;h=..;c=..|><|mask;...|>,1<|box;...

Done when
- The call prints one line per detected object with its id, its box and its mask's H x W.
- A phrase that isn't in the image prints a no-matches message instead of failing.
- The key comes from MODEL_API_KEY and never appears in code, logs or commits.
- API errors (4xx, 429) surface their message instead of failing silently.

Segment Anything Model (SAM) is Meta's segmentation model family, originally released as open-weight research models (opens in a new tab). Its latest version, SAM 3.1, is now available on Meta Model API. Detect, segment, and follow any object in images or video. Send media and a short noun phrase such as "yellow school bus", and the model returns every match as a box and a mask. Or mark specific objects with boxes and points, and the model segments those objects and tracks them forward, backward, or both ways through a video. The box locates the object; the mask gives you the pixel region to measure or cut out.

One model handles both images and video, so you build measurement, editing, and tracking features on top of a single API. The sam-3.1 model, the latest SAM version, segments stills and clips over POST /v1/responses on https://api.meta.ai/v1, using the same Bearer key as the rest of the API.

Anatomy of a request#

Segmentation happens over the Responses API, the same OpenAI-compatible format supported by our Muse Spark model line. Send one user message: a short input_text phrase naming the object, plus one media part. Streaming is the recommended path for video: set stream to true and read the server-sent events through response.completed (see segmenting with prompts), so each frame renders as it is segmented. Streaming is not required, a non-streaming request returns the complete result in a single response.

Write the prompt as a short noun phrase that names one object, such as "yellow school bus" or "red bicycle". Concrete nouns match best. Name one concept per request; to segment several object types, send several requests. To segment objects you've already located, send box and point prompts instead of a phrase. See segmenting with prompts for prompt tuning and the full request shape.

Attach an image with an input_image part or a clip with an input_video part. Video accepts MP4 and MOV containers encoded with H.264, H.265, or MPEG-4.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.meta.ai/v1",
  apiKey: process.env.MODEL_API_KEY,
});

const stream = await client.responses.create({
  model: "sam-3.1",
  input: [
    { type: "message", role: "user", content: [
        { type: "input_text", text: "yellow school bus" },
        { type: "input_image", image_url: "https://example.com/street.png" },
    ]},
  ],
  stream: true,
});

let outputText = "";
for await (const event of stream) {
  if (event.type === "response.output_text.delta") {
    outputText += event.delta;
  }
}
console.log(outputText);

Anatomy of a response#

The result streams back in the output_text lane (the text channel of the Responses stream) as special-token text: one line for an image, one line per frame for video. Each line carries a box and a mask for every object found. The stream terminates on response.completed, which delivers the final message.

Representative response for the call above, with mask payloads truncated:

<0f>0<|box;x1=353;y1=382;x2=453;y2=427;w=537;h=561|><|mask;x=0;y=0;data=46,101,~...|>,1<|box;x1=260;y1=382;x2=340;y2=427;w=537;h=561|><|mask;x=0;y=0;data=46,81,~...|>

<0f> is the frame, the leading integer is the object ordinal, then one box then one mask; ~ marks a lossless mask payload. Full grammar in reading segmentation output.

The @meta-sam/parser library turns this text into typed records, so you don't parse the grammar by hand. Pass the stream to parseImageStream, then decode each mask into a binary raster with decodeMaskToRaster:

import OpenAI from "openai";
import { parseImageStream, recordsOfKind, decodeMaskToRaster } from "@meta-sam/parser";

const client = new OpenAI({
  baseURL: "https://api.meta.ai/v1",
  apiKey: process.env.MODEL_API_KEY,
});

const events = await client.responses.create({
  model: "sam-3.1",
  input: [
    { type: "message", role: "user", content: [
        { type: "input_text", text: "yellow school bus" },
        { type: "input_image", image_url: "https://example.com/street.png" },
    ]},
  ],
  stream: true,
  metadata: { mask_encoding: "one_bit" },
});

const result = await parseImageStream(events).finalResult;

for (const record of recordsOfKind(result.records, "mask")) {
  const raster = decodeMaskToRaster(record.mask); // Uint8Array of 0 or 1, length W * H
  console.log(raster.length);
}

Each box arrives in source pixels and each mask as a binary raster. For the grammar behind the wire format, see read segmentation output.

Each object keeps a stable identity across frames, so you follow an object through a clip. See track objects across frames for detail.

Zero matches is a valid outcome. When nothing matches the prompt, the stream completes with an empty output_text, so branch on it.

Next steps#

• Segmenting with prompts to tune prompts, upload clips once, and switch mask encodings.
• Read segmentation output to parse boxes and masks and follow objects across frames.
• Use the client libraries to parse streams, decode masks, and render overlays.

Mirrored from dev.meta.ai/docs/sam/overview (synced Oct 30, 2026). Text and examples © Meta. Live demos run on Muse Muse's key.