Building AI Products for Users on Bad Connections

Streaming responses, hopeful reconnects and generous payloads all assume a connection that a great many users do not have.

The default architecture for a modern AI product assumes broadband: a token-by-token stream over a held-open connection, a heavy client, and a retry when something fails. On an intermittent mobile connection, each of those assumptions produces a specific failure.

Streaming is fragile by design

Streamed responses look excellent when the connection holds. When it drops mid-response the user has half an answer, the client often has no way to resume, and the request is billed in full. A queued job with a result the client can fetch when it reconnects is less impressive to demo and far more likely to deliver an answer.

Make the expensive work resumable

Anything that costs money to compute should be recoverable. Give the request an identifier, store the result, and let the client ask again for the same identifier. Users on unreliable connections will retry; without idempotency they are charged twice and may receive two different answers to the same question.

Weigh what you send

Users paying for a bundle notice a heavy application. So does a mid-range phone with limited memory. Server-rendered pages, small payloads and doing the work server-side rather than shipping a large client are unfashionable and correct for this audience.

Voice and language are not extras here

In markets with meaningful variation in literacy and many languages in daily use, voice input and local-language support are the difference between a product being usable and being a demonstration. This is where the shortage of training data in African languages has a direct commercial cost, and where a team with access to that data has an advantage no international competitor can quickly copy.

Get the next one by email