How We Built Free-First AI Routing Without Losing Cost Control

The problem was not “find the cheapest model”

When EONAPP moved its hosted Sponsored/Free AI path toward OpenRouter, the obvious temptation was to pick the lowest advertised token price and call the problem solved. That would have been a fragile launch strategy. A free model can become unavailable, return a retryable provider error, disappear from a routing pool or behave differently under load. A paid model can quietly turn a supposedly free acquisition funnel into an inference bill. The real engineering problem was to make cost a controlled state transition rather than an optimistic assumption.

We therefore designed the route around a small number of explicit states: a qualified free primary, one approved free retry, at most one paid fallback, and a hard daily paid-fallback circuit breaker. Each upstream attempt also consumes the same launch governors, so a retry is not treated as “free” merely because it uses a zero-price model. That distinction matters because provider capacity, latency and abuse limits are operational costs even when token billing is zero.

Our routing state machine

The Production policy starts with a free model only when the request is eligible for that route. If the primary free attempt succeeds, the request ends there. If it fails in a retryable way, EONAPP may move to one approved free retry selected by request class. Simple and general requests can prefer a fast general model; harder reasoning or coding requests can prefer a different free model that is better suited to that workload. Only after the approved free routes are unavailable may the system consider one paid fallback.

Not every failure is retryable. A malformed request, authentication error or policy rejection should not trigger a chain of increasingly expensive models. The point of fallback is resilience to provider availability, not a way to hide application bugs. EONAPP records whether the final answer used the primary route, a free retry or the paid fallback so that operations can distinguish genuine model availability problems from source defects.

The governors are part of the product, not an afterthought

At launch, EONAPP uses global attempt governors and a small paid-fallback breaker. The concrete limits are intentionally conservative: the release contract caps upstream attempts per minute and per day, and the paid-fallback path has a small daily spend ceiling. These are not marketing limits. They are operational guardrails that let us observe real behavior before increasing exposure.

The important design choice is that all actual upstream attempts count. If a free primary fails and a free retry is used, that is two attempts. If the paid fallback is then used, that is a third attempt and also a billable event. Counting the whole chain prevents retries from becoming an invisible source of load. It also gives us a denominator for quality: an answer that succeeds only after repeated upstream failures is not equivalent to a first-attempt success.

Cost telemetry must follow the same cohort as acquisition

EONAPP does not treat global ad revenue or global subscription revenue as proof that a paid traffic source is profitable. The hosted AI path writes cost telemetry so acquisition cohorts can be reconciled with their own AI cost and their own sponsor evidence. That means a cheap click cannot be declared profitable merely because another group of visitors generated revenue elsewhere.

This is especially important with free models. A free-model allocation can make early experiments inexpensive, but it does not eliminate the need to record attempts, paid fallbacks and provider changes. If free capacity disappears tomorrow, the same cohort may have a different cost profile. We want the economics to reflect what actually happened rather than a static spreadsheet assumption.

Privacy constraints narrow the routing pool

For EONAPP, a model being free is not enough. Hosted routing also has privacy constraints. The server route requests provider behavior that supports zero-data-retention and denies provider-side data collection where the selected upstream supports those controls. If an endpoint cannot meet the required privacy policy, it is not considered a valid “free” success just because it returns text.

This makes the router more conservative than a generic cheapest-model selector. It also means model branding is not the final privacy authority. The effective upstream endpoint and its current policy matter. That is why we record routing behavior and keep Local AI and BYOK as separate lanes rather than silently moving private work into a hosted fallback.

Worked example: three possible outcomes

Consider a normal planning request. Outcome one: the primary free model answers successfully. The request uses one upstream attempt, no paid fallback and no paid-model cost. Outcome two: the primary returns a retryable availability failure and the approved free retry succeeds. The request consumes two attempts but still no paid-model cost. Outcome three: both approved free routes are unavailable, the paid-fallback breaker still has budget, and one paid fallback succeeds. The request consumes three attempts and the actual paid cost is attributed to the request's cohort.

If the paid breaker is exhausted, the router should not quietly buy more capacity. It should fail in a controlled way or offer another product lane. A launch governor only works if reaching it changes behavior.

What we would measure before increasing limits

• Primary-free success rate by request class.
• Free-retry rate and the reason the retry was needed.
• Paid-fallback frequency and actual paid cost.
• Latency at each state in the route.
• Provider error categories rather than one generic failure bucket.
• Attempts per successful answer.
• Same-cohort sponsor activity and reconciled revenue where applicable.
• Privacy-policy eligibility of the endpoint actually selected.

Those measurements tell us whether the free-first design is genuinely efficient. A route with a nominally free primary but a high paid-fallback rate may be more expensive than a stable low-cost model. Conversely, a strong free-primary success rate can make early product testing dramatically cheaper without compromising the ability to fail over.

What we learned

The most useful lesson is that “free-first” should be a policy, not a model name. Models change. Provider availability changes. Free quotas change. The durable part is the state machine: qualify the route, try an approved free primary, allow a bounded free retry, permit at most one controlled paid fallback, account for every attempt, and stop spending when the breaker says stop.

That policy also made the rest of the launch easier to reason about. Sponsored mediation can fail without blocking the AI answer. Paid traffic can stay off until costs and sponsor evidence reconcile. Local and BYOK paths remain private and ad-free. Most importantly, operators can explain why a particular request cost money instead of discovering the answer later in a provider invoice.