Testing LLM API integrations: streaming, tool calls and failures
An LLM integration fails in ways ordinary API clients rarely meet: an answer that arrives in hundreds of server-sent events, a stream that stops halfway through a sentence, a 429 during a traffic spike, an overloaded provider, a tool call the client has to execute and answer. The Mock LLM API reproduces each of these deterministically, in the OpenAI and Anthropic request shapes, so the handling can be tested on every run instead of discovered in production.
Real endpoints from the catalog — every request above is served, and every status code shown is one that endpoint actually returns.
How to practise it
Send a streamed request to mock-echo, concatenate every delta, and assert both the final text and that the stream ended with a stop reason and the final event.
Stream from mock-stream-cut and assert your client reports an incomplete answer instead of showing half a sentence as the final result.
Call mock-429 and verify your client waits for Retry-After, retries a bounded number of times and then gives up with a clear error.
Send a request to mock-tool-call with one tool declared, execute the tool call it returns, send the result back and assert the final answer uses it.
Call mock-slow with a client timeout shorter than eight seconds and assert your fallback runs instead of the request hanging.
Pagination testing — Practise page-number and cursor pagination, and the edge cases that break clients: the last partial page, an out-of-range page, and a total count that disagrees with what you collected.
Rate limit testing — Trigger a real 429 with Retry-After and X-RateLimit headers, and verify your client backs off instead of retrying straight into the limit.
Idempotency key testing — Send the same write twice with an Idempotency-Key and confirm the operation happens once — the mechanism that stops a network retry becoming a duplicate charge.
ETag and If-Match testing — Provoke a 412 Precondition Failed by writing with a stale ETag — the check that stops two clients silently overwriting each other.
Webhook signature testing — Register a callback, trigger a delivery, verify its HMAC signature, and confirm a tampered payload is rejected.
OAuth and JWT testing — Run real client-credentials, authorization-code and device-code flows, then test what your client does when the token expires mid-session.
Negative testing — Provoke 400, 401, 403, 404, 409, 418, 422, 429, 500 and 503 deliberately, and check your client handles each one differently.
Async job testing — Start a long-running job, follow the 202 and its Location header, and poll to completion — including the failure and timeout paths.
MCP client testing — Test an MCP client or agent over Streamable HTTP — initialize and session ids, tool schemas and structured output, tool errors versus protocol errors, and authentication.