Serving

mach-serve loads one Mach-1 pack and serves it over HTTP:

mach-serve SyzygyResearch/Mach-1-Additive-35B --port 8080

The checkpoint can be a HuggingFace repo id (downloaded and cached on first use) or a local pack directory. The server binds 127.0.0.1:8080 by default and is ready when it logs serving <model-id> on http://127.0.0.1:8080/v1.

Endpoints

No authentication: any API key string is accepted. The server hosts a single model; the request's model field is accepted as-is, and responses echo the served model id.

Connect a client

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="mach")

completion = client.chat.completions.create(
    model="mach-1",
    messages=[{"role": "user", "content": "Write a haiku about the sea."}],
    stream=True,
)
for chunk in completion:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://127.0.0.1:8080/v1",
  apiKey: "mach",
});

const completion = await client.chat.completions.create({
  model: "mach-1",
  messages: [{ role: "user", content: "Write a haiku about the sea." }],
});
console.log(completion.choices[0]?.message.content);

Anthropic-compatible clients work by pointing their base URL at the server (for example ANTHROPIC_BASE_URL=http://127.0.0.1:8080); requests go to /v1/messages.

Chat templates are applied automatically. For models with a thinking phase, reasoning is returned separately as reasoning_content on chat-completion messages and deltas, with the final answer in content.

Tool calling and structured output

/v1/chat/completions supports OpenAI-style tools / tool_choice, and response_format with json_schema for structured output. Tool-call grammar enforcement is on by default, which constrains generation so emitted tool calls parse and match the declared parameter schema.

Image input

With the multimodal pack (SyzygyResearch/Mach-1-Additive-35B-Multimodal), send images as standard OpenAI image_url content parts. Inline data:image/...;base64,... URLs are accepted by default; fetching remote URLs server-side is off unless explicitly enabled. The vision tower loads lazily on the first image request (~0.9 GB); text-only sessions never pay for it. Clients can check the serving.vision field of GET /v1/capabilities to see whether the loaded pack accepts images.

Speculative decoding

With the dflash extra installed and a draft checkpoint on disk, pass --draft-dir to enable speculative decoding (~163 tok/s vs ~117 tok/s single-stream on the 35B pack):

mach-serve SyzygyResearch/Mach-1-Additive-35B \
  --draft-dir ~/models/Qwen3.6-35B-A3B-DFlash

A published draft for the flagship pack is z-lab/Qwen3.6-35B-A3B-DFlash. Speculative decoding is exact: the output token stream is identical to the unassisted path. Without a usable draft the server serves the exact path on its own; pass --target-only to force that explicitly.

Continuous batching

By default, concurrent requests are served one at a time behind a generation lock. --continuous-batching coalesces concurrent chat completions into shared batched decode steps. Throughput scales to ~231 tok/s aggregate at 4 streams and ~255 tok/s at 8 on the 35B pack, and combines with speculative decoding:

mach-serve SyzygyResearch/Mach-1-Additive-35B --continuous-batching

Prefix caching

Prefix caching is on by default: repeated prompt prefixes (system prompts, chat history, agent scaffolds) skip prefill on subsequent requests. Add --disk-kv-dir to also persist KV state across restarts and multi-turn sessions:

mach-serve SyzygyResearch/Mach-1-Additive-35B --disk-kv-dir ~/.cache/mach/kv

GET /v1/cache/stats reports hits and misses. Disable with --no-prefix-cache.

Common options

Run mach-serve --help for the full list. Environment-variable tuning knobs are documented in the repo's docs/configuration.md.