Text generation API

Text models expose an OpenAI-compatible chat completions endpoint. The server runs one selected model per process. Vision-capable text models also accept image content parts.

Health, model metadata and the error format are shared by all modalities; see the API overview.

Endpoint

POST /v1/chat/completions

Models and quick start

Browse text models

self serve qwen3.5:9b

POST /v1/chat/completions

Request body

modelstringoptional

Accepted and ignored: a server runs the model selected at startup.

messagesarrayrequired

Messages in conversation order. Each item has role and content. role is system, developer, user, or assistant; developer is treated as system. Tool and function roles are unsupported.

content is a string or an array of text parts. Vision models also accept user-only image_url parts with an HTTPS URL or image data URL.

max_tokensintegeroptional

Maximum generated tokens. Must be at least 1. max_completion_tokens takes precedence when both are present.

max_completion_tokensintegeroptional

Alias for max_tokens; must be at least 1.

temperaturenumberoptional

Sampling temperature from 0 through 2.

top_pnumberoptional

Nucleus sampling value from 0 through 1.

nintegeroptional

Only 1 is supported.

streambooleanoptional

Defaults to false. With true, the response is server-sent events ending in data: [DONE].

stream_optionsobjectoptional

Only valid with stream: true. include_usage: true adds a final usage-only chunk. include_obfuscation is accepted but has no effect.

stopstring | array of stringsoptional

One stop sequence or up to four sequences. Empty sequences are ignored.

presence_penaltynumberoptional

Sampling penalty from -2 through 2.

frequency_penaltynumberoptional

Sampling penalty from -2 through 2.

seedintegeroptional

Seed passed to the runtime when supported by the loaded engine.

response_formatobjectoptional

type is text, json_object, or json_schema. For json_schema, provide json_schema.schema as a JSON schema object. name, description, and strict are accepted inside json_schema.

reasoning_effortstringoptional

none disables reasoning. minimal, low, medium, high, and xhigh enable it.

chat_template_kwargsobjectoptional

Only enable_thinking: true|false is supported.

user, store, metadata, service_tier, prompt_cache_key, prompt_cache_retention, safety_identifier, parallel_tool_callsanyoptional

Accepted for client compatibility and ignored. Tool calling, logprobs, audio, prediction, web search, and non-text modalities return 422 unsupported_capability when non-empty. tool_choice: "auto" is accepted but does not enable tools.

Example request

curl

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "text-model",
    "messages": [
      {"role": "system", "content": "Answer briefly."},
      {"role": "user", "content": "Explain embeddings in one sentence."}
    ],
    "max_tokens": 64,
    "reasoning_effort": "none"
  }'

Python

import requests

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={
        "model": "text-model",
        "messages": [
            {"role": "user", "content": "Explain embeddings in one sentence."}
        ],
        "max_tokens": 64,
        "reasoning_effort": "none",
    },
    timeout=60,
)
response.raise_for_status()
print(response.json())

JavaScript

const response = await fetch('http://localhost:8080/v1/chat/completions', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({
    model: 'text-model',
    messages: [{ role: 'user', content: 'Explain embeddings in one sentence.' }],
    max_tokens: 64,
    reasoning_effort: 'none'
  })
})

if (!response.ok) throw new Error(await response.text())
console.log(await response.json())

Example response

{
  "id": "chatcmpl-7f3d...",
  "object": "chat.completion",
  "created": 1760000000,
  "model": "text-model",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "Embeddings are numeric vectors that represent meaning."},
    "finish_reason": "stop",
    "logprobs": null
  }],
  "usage": {"prompt_tokens": 18, "completion_tokens": 10, "total_tokens": 28}
}

Response body

idstring

Unique completion ID beginning with chatcmpl-.

objectstring

Always chat.completion for a non-streaming response.

createdinteger

Unix timestamp in seconds.

modelstring

ID of the loaded model.

choicesarray

One choice with index: 0, message, finish_reason (stop or length), and logprobs: null.

choices[0].messageobject

role is assistant, content is generated text, and reasoning-capable models may include separate reasoning_content.

usageobject

prompt_tokens, completion_tokens, and total_tokens are integer token counts. total_tokens is their sum.

Streaming request

curl

curl -N http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Count to three."}],"stream":true,"stream_options":{"include_usage":true}}'

Python

import requests

with requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={"messages": [{"role": "user", "content": "Count to three."}], "stream": True},
    stream=True,
    timeout=60,
) as response:
    response.raise_for_status()
    for line in response.iter_lines(decode_unicode=True):
        if line:
            print(line)

JavaScript

const response = await fetch('http://localhost:8080/v1/chat/completions', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({
    messages: [{ role: 'user', content: 'Count to three.' }],
    stream: true
  })
})

if (!response.ok) throw new Error(await response.text())
console.log(await response.text())

Each event is a chat.completion.chunk. Deltas contain content or reasoning_content; the final chunk contains finish_reason, followed by data: [DONE]. include_usage adds a final usage-only chunk.

Vision input

Vision models accept an image in a user message as an HTTPS URL or a supported data URL. Check /v1/model for capabilities.input and max_images.

Image content part

typestringrequired

Must be image_url.

image_url.urlstringrequired

HTTPS URL or a JPEG, PNG, or WebP data URL.

Example request

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "vision-model",
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "What color is the car?"},
      {"type": "image_url", "image_url": {"url": "https://example.com/car.jpg"}}
    ]}],
    "max_tokens": 32,
    "temperature": 0
  }'

Response

{
  "object": "chat.completion",
  "model": "vision-model",
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "Red."}, "finish_reason": "stop", "logprobs": null}],
  "usage": {"prompt_tokens": 120, "completion_tokens": 2, "total_tokens": 122}
}

Text errors

Besides the shared errors, text requests return 400 invalid_request for malformed fields, 413 request_too_large for bodies over 32 MiB, 422 unsupported_capability for unsupported features, 429 queue_full, and runtime 503 or 504 errors. A streaming error after the first event is written in-band as an error object.