Sign up
Docs menu

Streaming

Set "stream": true on either endpoint. The response is text/event-stream, sent as it is generated, with no content-length and no content-encoding — a streamed response that declared a length would be buffered by intermediaries and would stall tools like Claude Code.

OpenAI wire#

data: lines carrying chat.completion.chunk objects. The last content chunk has finish_reason set; one more chunk follows with an empty choices array and the usage block; then the terminator:

data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hi"},"finish_reason":null}]}

data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"object":"chat.completion.chunk","choices":[],"usage":{"prompt_tokens":6,"completion_tokens":8,"total_tokens":14}}

data: [DONE]

usage is always sent, whether or not you set stream_options.include_usage. Tool-call deltas arrive byte-for-byte as the model produces them, which is what Cline and Roo Code parse.

Anthropic wire#

Named events, ending at message_stop: message_start (with the input token count), content_block_start, content_block_delta (text, tool-input JSON or thinking deltas), content_block_stop, message_delta (with stop_reason and the final usage), message_stop. ping events may appear in between. The full example is on the Messages page.

What is filtered#

Nothing that a client uses. The one thing removed from a stream is a nonstandard trailing frame after the terminator that some models emit and no SDK expects. Every other frame — every chunk, every named event, every ping — is forwarded in order.

Disconnects, truncation, billing#

  • If you disconnect mid-stream, the generation still finishes and is still billed. Sator keeps reading the model's output for a short grace period so the tokens it generated are counted. Cancel early only when you mean it.
  • If the stream ends without its terminator ([DONE] or message_stop), the response was cut short. The tokens up to the cut are billed from an estimate, marked as such, and corrected in a nightly reconciliation.
  • If your balance reaches $0 mid-response, the stream stops there — the Zero-Balance Cutoff described in Billing. Add credit and retry.

Errors during a stream#

An error before the first byte is a normal error response — status, envelope and x-request-id as in Errors. Once the first byte is out the status is already 200; a failure after that shows up as a stream that ends early, so treat a missing terminator as a retryable failure.