AI Gateway adds confidence-based decision fallbacks
Vercel’s AI Gateway now supports confidence-based fallback models to reroute requests when primary model responses fail preset confidence thresholds, with billing applying to both decision stages.
AI Gateway introduces a new feature allowing requests to escalate to a fallback model when the primary model’s response fails to meet a user-defined confidence threshold. Conditions can be combined, enabling escalation based on multiple signals such as error rates or confidence scores. Existing fallback mechanisms for outright errors remain unchanged, ensuring backward compatibility with current configurations.
Confidence conditions apply to Choice and Score questions, while Boolean questions use a probability range for escalation. Users can configure fallbacks by adding a conditional object to providerOptions.gateway.models, specifying thresholds such as { "confidenceBelow": 0.6 } to trigger escalation when answers fall below the set value.
Requests without a configured fallback retain their existing behavior, maintaining consistency for users who do not require this feature. The system checks conditions for every matching question type when no specific question is targeted, simplifying setup for broad confidence-based routing.
Each triggered fallback initiates a second decision stage, resulting in billing for both the primary and fallback stages. This feature is currently in beta, with detailed documentation available for implementation guidance.