checklist
AI chatbot provider outage and model failover checklist for small teams
A practical checklist for handling AI provider outages, model errors, degraded latency, fallback routes, customer communication, and safe recovery without losing control of chatbot data or actions.
Use this checklist when a model provider, gateway, retrieval service, tool, queue, or region becomes unavailable or materially degraded. It is designed for a small team that needs a clear decision path without creating a second security incident during recovery.
The short answer: define the chatbot’s safe state before an outage, detect provider and route degradation with independent signals, switch only to an approved fallback, keep data and tenant boundaries unchanged, disable high-impact tools by default, communicate a bounded service status, and verify the original route before restoring it. A fallback is not safe merely because it returns an answer; it must preserve authorization, data handling, output validation, quotas, logging, and human escalation.
Start with the AI Tool Risk Checker and attach its result to the failover record. Use the Small Team AI Security Checklist for baseline ownership, access, data, logging, and incident controls. Pair this page with the AI chatbot pause and fallback plan template, the AI chatbot abuse monitoring and rate-limit checklist, and the AI chatbot output validation checklist.
Bottom line
An AI chatbot can enter a controlled fallback only when the team can answer these questions:
- Which signal proves that the primary route is degraded, and which independent signal confirms it?
- Which users, tenants, data classes, tools, and destinations are allowed on the fallback?
- Does the fallback have its own limits for input, output, concurrency, queue depth, retries, and cost?
- Which responses must become read-only, draft-only, delayed, or human-reviewed?
- Who can switch, pause, communicate, and restore the service, including outside normal hours?
- What evidence proves that the original route and the fallback stayed within their authorization and data boundaries?
NIST AI RMF Manage calls for post-deployment monitoring, incident response, recovery, and change management. CISA and the UK NCSC also frame secure AI as a lifecycle responsibility that includes deployment and operation. Treat failover as a controlled production change with a known safe state, not as an improvised provider swap.
This checklist is operational guidance, not legal advice, a compliance certification, or a guarantee of provider availability.
When to use this checklist
| Situation or change | Use this checklist? | Minimum review |
|---|---|---|
| Primary model provider outage or elevated errors | Yes | Trigger, safe state, fallback eligibility, owner, and communication. |
| Provider rate limit or account quota pressure | Yes | Quota evidence, cost controls, retry behavior, and approved fallback. |
| Model, region, gateway, or API version change | Yes | Route comparison, data boundary, regression set, and rollback. |
| Latency or queue degradation without full outage | Yes | Degraded-service thresholds, queue shedding, and customer notice. |
| Retrieval, vector store, or source-system failure | Yes | No-source behavior, source scope, cache age, and human handoff. |
| Tool or downstream action dependency failure | Yes | Disable or draft-only action path and downstream authorization. |
| Internal prototype using synthetic data only | Maybe | Record owner and safe state before any broader use. |
| Static content with no AI route | Usually no | Use ordinary availability and deployment controls instead. |
If the team cannot name the fallback route and its data, action, and ownership limits, keep the chatbot paused or read-only until the gap is resolved.
Outage review intake form
Copy this into the incident, change, or provider review record.
| Field | Entry |
|---|---|
| Incident or change ID | |
| Detected date and timezone | |
| Chatbot and environment | |
| Primary route, model, provider, and region | |
| Affected tenants, users, and endpoints | |
| Failure signal and independent confirmation | |
| Data classes in the route | |
| Retrieval sources and cache behavior | |
| Tools and downstream actions | |
| Approved fallback route | |
| Fallback data and tenant scope | |
| Input, output, retry, queue, and cost limits | |
| Switch owner and incident owner | |
| Customer and internal communication owner | |
| Restoration criteria and next review date |
Do not paste private transcripts, API keys, provider credentials, or raw customer records into the incident channel. Record route identifiers, aggregate signals, redacted samples, and an access-controlled evidence location instead.
Failure signal inventory
Use several signals because a provider status page, a client error, or one slow request is not enough to prove the safe response.
| Signal | Useful dimensions | What it can show | Evidence to retain |
|---|---|---|---|
| Provider error rate | Provider, model, region, endpoint, and status | Dependency or route failure | Aggregate counts and time window. |
| Latency | Percentiles, queue wait, route, and tenant | Degraded service before errors rise | Percentile trend and alert decision. |
| Timeout rate | Client, gateway, provider, and route | Requests that may trigger retries | Timeout class and retry outcome. |
| Retry velocity | Client version, route, and identity | Amplification or retry storm | Retry counts and backoff behavior. |
| Queue depth and age | Route, priority, tenant, and worker | Capacity pressure and stale work | Bounded queue snapshot. |
| Provider quota | Account, model, plan, and time window | External capacity exhaustion | Quota signal and owner response. |
| Cost velocity | Model, provider, route, and tenant | Unexpected spend during fallback | Aggregate cost and budget decision. |
| Retrieval health | Source, index, freshness, and authorization | Missing or stale context | Source status and no-source behavior. |
| Tool dependency health | Tool, target, approval, and error | Unsafe or unavailable action path | Action outcome and authorization result. |
| Customer reports | Tenant, feature, region, and time | Impact not visible in metrics | Redacted report and impact scope. |
| Monitoring health | Collector, alert, dashboard, and timestamp | Whether the team can trust the view | Monitoring status and safe-state choice. |
Keep provider status pages as one input, not the only evidence. A provider can be healthy while a tenant quota, gateway, region, route, or application dependency is failing.
Incident severity matrix
| Class | Example pattern | Default response | Escalation |
|---|---|---|---|
| Advisory | Isolated slow request within the route’s service objective | Observe and review the trend. | Escalate if the signal persists or spreads. |
| Degraded | Repeated timeouts, elevated errors, or queue growth | Limit retries, reduce load, and use the approved degraded route. | Notify the incident owner and affected support team. |
| High | Provider quota, tenant isolation, cost, or shared availability is at risk | Pause expensive or nonessential work and restrict the route. | Open an incident and protect unrelated tenants. |
| Critical | Cross-tenant data risk, credential compromise, unsafe tool action, or uncontrolled spend | Fail closed, disable affected actions, revoke access if needed, and preserve minimal evidence. | Incident owner leads containment and recovery. |
Severity should consider impact and confidence. Do not make a high-impact tool available again just because the provider responds to a health check.
Safe-state matrix
Decide what the chatbot can do when the primary route or dependency is unavailable.
| Capability | Primary route unavailable | Safe state | Restore gate |
|---|---|---|---|
| Public, low-impact answer | Use approved fallback if source and policy checks pass | Read-only answer with uncertainty and source limits | Recent test and monitoring pass. |
| Customer-specific account answer | Do not substitute an unscoped model | Human handoff or no-answer response | Identity, tenant, and source scope verified. |
| Retrieval-backed answer | Do not invent a current answer | Bounded cached answer or handoff with freshness notice | Cache age, authorization, and source health verified. |
| Draft ticket or CRM update | Do not write during an unverified dependency failure | Draft-only or queue for human review | Schema, idempotency, and owner review pass. |
| Message or email send | Do not send automatically | Prepare nothing or require human approval | Recipient, content, and approval checks pass. |
| Permission, refund, deletion, or account action | Do not execute | Fail closed and route to authorized staff | Independent authorization and action test pass. |
| Expensive model route | Do not retry without bound | Cheaper bounded route or pause | Cost, quality, and rate-limit guardrails pass. |
Fail closed for high-impact actions. For ordinary availability pressure, a concise refusal, read-only route, or human handoff is usually safer than an unreviewed provider swap.
Fallback route approval matrix
Approve fallbacks before the incident. The fallback may be another provider, an earlier model, a cached source, a static answer set, or a human workflow, but every choice needs explicit boundaries.
| Fallback option | Data allowed | Actions allowed | Main risk | Approval evidence |
|---|---|---|---|---|
| Same provider, alternate model | Team-defined low-risk classes | Read-only or draft-only | Different behavior or retention terms | Route comparison and test set. |
| Alternate approved provider | Only classes contractually and technically allowed | Read-only by default | New data residency, retention, or subprocessor boundary | Vendor and data review. |
| Earlier known-good model | Same scope as the tested version | Existing allowlist only | Quality, refusal, or context changes | Regression and output review. |
| Cached or static answer set | Approved, freshness-bounded content | No state-changing actions | Stale or incomplete answer | Freshness and source review. |
| Human support handoff | Minimum context needed for the case | Human-owned actions only | Queue and disclosure pressure | Handoff capacity and notice. |
| Full pause | No model or tool processing | None | Customer availability impact | Incident decision and notice. |
Never route private data to a fallback solely because it is online. Confirm the fallback’s data terms, access scope, retention, logging, action boundaries, and owner before use.
Data and tenant boundary checks
- The fallback has an explicit approved data classification and tenant scope.
- Customer, account, ticket, billing, health, employment, and identity data are not sent to an unreviewed route.
- Retrieval filters, authorization checks, and tenant binding run independently of model output.
- Cached responses have an owner, freshness rule, invalidation path, and access check.
- Logs and alerts use correlation IDs without storing provider credentials or raw private prompts by default.
- Provider, region, subprocessor, retention, and deletion assumptions are recorded for each route.
- A fallback cannot broaden the original user’s access or change the intended data destination.
- Cross-tenant and wrong-account tests are included before restoration.
Retry, queue, and cost controls
An outage can become an availability and cost incident when every layer retries independently. Set route-specific values and document the source of each threshold.
| Control | Decision to record | Failure behavior |
|---|---|---|
| Retry eligibility | Which errors may be retried and by which layer? | Stop non-retryable and repeated failures. |
| Retry cap | What is the team-defined maximum across client, gateway, and worker? | Use a safe fallback or handoff. |
| Backoff and jitter | How is synchronized retry avoided? | Delay or shed work when the dependency is unhealthy. |
| Concurrency | What is the per-identity, tenant, route, and provider cap? | Queue within a bound, then reject or defer. |
| Queue depth and age | How much work may wait? | Shed low-priority work and communicate degradation. |
| Input and output budget | What size is accepted on the fallback? | Reject or shorten safely without changing meaning. |
| Cost guard | Which spend or token velocity requires action? | Disable expensive models or pause the feature. |
| Idempotency | Which actions must not repeat after timeout? | Require a stable operation key and verify state. |
Do not use a fallback to hide a retry storm. Stop amplification first, then decide whether a bounded degraded route is safe.
Tool and action controls
Provider failure is not a reason to grant a fallback more autonomy. Keep tool actions narrow, independently authorized, and reversible.
| Action | During outage or failover | Required evidence before restore |
|---|---|---|
| Search or retrieval | Allow only approved sources with bounded calls and data. | Source health, scope, and freshness test. |
| Ticket or CRM draft | Draft-only with field and tenant checks. | Schema, idempotency, and human review test. |
| Email or message | No automatic send unless explicitly approved. | Recipient, content, and approval record. |
| External API read | Validate endpoint, target, and parameters outside the model. | Authorization and dependency test. |
| Permission or account change | Disabled by default. | Independent authorization and human approval. |
| Delete, refund, or irreversible action | Fail closed. | Explicit approval, target check, and recovery evidence. |
Customer and internal communication plan
Communicate the user-visible impact without exposing provider credentials, internal detection logic, or another tenant’s status.
| Audience | Message should include | Avoid |
|---|---|---|
| Internal responders | Time, scope, signal, safe state, owner, next update, and evidence location | Unredacted prompts or unverified root cause. |
| Support team | User impact, approved workaround, escalation path, and status language | Promising a restoration time the team cannot support. |
| Affected customers | Feature status, safe alternative, data-handling statement when relevant, and next update | Blaming a provider or disclosing another tenant. |
| Security or privacy owner | Data scope, route change, tool status, and containment decision | Treating a fallback as pre-approved without evidence. |
| Business owner | Availability, cost, customer, and risk trade-off | Hiding degraded controls to preserve a metric. |
Test set and evidence record
Use a non-production route, synthetic traffic, approved masked fixtures, and clear stop conditions. Do not test failover with real customer data merely to create evidence.
| Test ID | Defensive scenario | Expected result | Last result |
|---|---|---|---|
| FAL-01 | Primary route returns provider errors | Detection confirms the issue and the defined safe state activates. | |
| FAL-02 | Primary route is slow but not failing | Bounded timeout and retry rules prevent amplification. | |
| FAL-03 | Alternate model is selected | Data, tenant, output, and action gates remain unchanged. | |
| FAL-04 | Fallback receives private or high-impact data | Route rejects or hands off instead of broadening scope. | |
| FAL-05 | Retrieval source is unavailable | Bot uses a freshness-bounded cache or safe no-answer path. | |
| FAL-06 | Queue depth exceeds the recovery trigger | Low-priority work is shed or deferred safely. | |
| FAL-07 | Tool action is requested during failover | Action is disabled or independently authorized. | |
| FAL-08 | Provider quota or cost alert fires | Expensive route pauses or switches to an approved bounded option. | |
| FAL-09 | Client retries after timeout | Idempotency prevents duplicate durable actions. | |
| FAL-10 | Monitoring or alerting is unavailable | Team enters the documented restricted safe state. | |
| FAL-11 | Customer-facing notice is needed | Approved message and human escalation path are ready. | |
| FAL-12 | Primary route recovers | Restoration gate verifies tests, metrics, authorization, and owner sign-off. |
For each run, record the route and version, test input class, data class, threshold source, aggregate result, tool decision, reviewer, timestamp, and evidence location.
Staged failover drill
| Stage | Scope | Entry gate | Exit signal |
|---|---|---|---|
| 1. Design | One chatbot route | Primary, fallback, safe state, owners, and data boundaries documented. | Reviewers agree the fallback is bounded. |
| 2. Offline | Synthetic requests and masked fixtures | Error, timeout, retry, queue, cost, and tool tests exist. | Critical tests pass. |
| 3. Internal | One trained team | Monitoring, handoff, and communications are ready. | No unresolved high-risk finding. |
| 4. Limited drill | One low-risk route or internal tenant | Pause and restore are observable and reversible. | Recovery gates pass without cross-tenant impact. |
| 5. Production readiness | Approved customer scope | On-call, provider, cost, and rollback owners sign off. | Next review trigger is recorded. |
Keep the first drill read-only or draft-only. Add automatic writes, sends, or account actions only after the independent authorization and idempotency tests pass.
Recovery and restoration plan
| Step | Owner | Evidence | Exit gate |
|---|---|---|---|
| Confirm scope and safe state | Incident owner | Signal timeline and affected route | Scope is bounded and communicated. |
| Stop amplification | Platform owner | Retry, queue, and concurrency decision | Pressure is within team-defined limits. |
| Restrict data and actions | Security or product owner | Fallback scope and tool state | High-impact actions fail closed. |
| Activate approved fallback or pause | Engineering owner | Route version and decision record | Fallback tests and monitoring pass. |
| Communicate impact | Support or business owner | Approved notices and timestamps | Users have a safe next step. |
| Verify primary recovery | Provider and platform owners | Health, error, latency, and dependency evidence | Recovery is stable for the defined observation window. |
| Run restoration tests | Engineering and security owners | Regression, tenant, output, and action results | Critical tests pass. |
| Restore in stages | Product owner | Canary scope and rollback trigger | No unresolved critical finding. |
| Close and learn | Incident owner | Timeline, root cause, actions, and next review | Owners and due dates are recorded. |
Do not restore all traffic at once if the route has changed. Restore in a controlled scope, keep the fallback or pause path available, and confirm that monitoring is healthy before closing the incident.
Decision table
| Finding | Decision | Required evidence |
|---|---|---|
| Primary route is healthy and gates pass | Continue normal service | Health trend, tests, and owner review. |
| Provider is degraded but fallback is approved and bounded | Activate degraded route | Data, action, cost, and monitoring checks. |
| Fallback changes data, retention, or tenant assumptions | Restrict to safe data or pause | Route comparison and approval. |
| Monitoring or quota enforcement is unavailable | Restrict or pause | Safe-state decision and recovery test. |
| High-impact action is requested during failover | Fail closed and hand off | Authorization and human review evidence. |
| Recovery is unclear or repeated failures persist | Keep restricted and investigate | Incident timeline, owner, and next decision time. |
Sign-off record
| Role | Name | Decision | Conditions | Date |
|---|---|---|---|---|
| Business owner | ||||
| Product owner | ||||
| Engineering or platform owner | ||||
| Security reviewer | ||||
| Privacy or legal reviewer, if needed | ||||
| Support or incident owner |
Approval should identify the primary and fallback routes, provider and region, data classes, tenant scope, tool state, team-defined limits, communication plan, restoration gate, unresolved risks, and next review trigger.
Action tracker
| Action | Risk | Owner | Due date | Status | Evidence |
|---|---|---|---|---|---|
| Document primary route and independent health signals | |||||
| Approve fallback routes and data boundaries | |||||
| Define safe state for each capability | |||||
| Set retry, queue, concurrency, input, output, and cost limits | |||||
| Disable or constrain high-impact tools during failover | |||||
| Test tenant, authorization, output, and idempotency controls | |||||
| Run customer communication and recovery drills |
Final provider outage and failover checklist
- Primary and fallback routes have named owners, versions, providers, regions, and review dates.
- Independent monitoring covers errors, latency, timeouts, queues, retries, provider quota, cost, and dependency health.
- A documented safe state exists for public answers, private data, retrieval, drafts, messages, and tool actions.
- Fallback data, tenant, retention, source, and destination boundaries are approved.
- IP, identity, tenant, endpoint, provider, and tool limits remain in force during failover.
- Retry caps, backoff, queue bounds, concurrency limits, and idempotency are tested.
- High-impact actions fail closed or require independent authorization and human approval.
- Expensive models and provider routes have budget alerts and a pause owner.
- Customer and internal communication uses approved, bounded language.
- Synthetic failover tests cover provider errors, timeouts, retrieval failure, quota pressure, tool actions, and restoration.
- Monitoring failure has its own restricted safe state.
- Restoration is staged, observable, reversible, and backed by critical test results.
- The AI Tool Risk Checker result is attached to the review.
- The Small Team AI Security Checklist baseline controls are reviewed.
- Residual risk, rollback trigger, owners, and next review date are signed off.
Metrics to track
Track aggregate trends and redacted summaries rather than collecting raw private prompts by default.
| Metric | Why it matters | Review cadence |
|---|---|---|
| Primary and fallback error rate | Shows whether the switch improved availability. | Near real time and daily |
| Latency and timeout percentiles | Finds degradation before a full outage. | Near real time |
| Retry and queue amplification | Detects load multiplication and stale work. | Every incident and weekly |
| Fallback traffic by tenant and route | Checks fairness and scope. | Daily and weekly |
| Data or authorization blocks | Tests boundary preservation. | Every alert and weekly |
| Tool actions blocked or approved | Shows action risk during recovery. | Every incident |
| Cost and provider quota velocity | Prevents denial of wallet during failover. | Daily and monthly |
| Handoff volume and response time | Measures whether the human fallback is usable. | Weekly |
| Restoration test pass rate | Shows readiness to return to normal. | Each drill and quarterly |
| Time to detect, restrict, communicate, and restore | Measures incident control quality. | Every incident and monthly |
Evidence checked
- NIST AI RMF Core describes Manage outcomes for post-deployment monitoring, incident response, recovery, and change management.
- NIST AI RMF Manage Playbook supports documented monitoring, risk treatment, response and recovery plans, and continual improvement for deployed AI systems.
- OWASP LLM10:2025 Unbounded Consumption identifies denial of service, economic loss, and service degradation risks and supports rate limits, quotas, timeouts, throttling, monitoring, graceful degradation, and bounded work.
- OWASP LLM06:2025 Excessive Agency supports minimizing permissions, independent downstream authorization, human approval, logging, and rate limits for tool actions.
- CISA and UK NCSC secure AI system development guidance emphasizes secure design, accountability, and operational security across AI deployment and operation.
FAQ
Should a chatbot automatically switch providers during an outage?
Only when the fallback was approved and tested for the same data, tenant, authorization, action, retention, and monitoring requirements. Otherwise use a read-only response, human handoff, or pause.
Is a provider health check enough to restore traffic?
No. Check application errors, latency, queues, retrieval, authorization, output validation, tool actions, cost, and tenant impact. Restore in stages after critical tests pass.
What is the safest fallback for private customer questions?
Usually a human handoff or a route with independently verified identity, tenant, source, and data controls. A general model or stale cache should not guess at account-specific facts.
Should cached answers be used during an outage?
Only with an explicit freshness bound, access check, source owner, invalidation path, and user-facing limitation. Cached content is not automatically current or authorized.
How do we avoid making an outage worse with retries?
Cap retries across every layer, use backoff and jitter, bound concurrency and queues, and make state-changing work idempotent. Stop amplification before activating a fallback.
What should happen to tool actions during failover?
Keep high-impact actions disabled or draft-only until independent authorization, target validation, approval, and idempotency checks pass. Fail closed for deletion, refunds, permissions, and other irreversible actions.
How much outage evidence should be stored?
Store the minimum redacted evidence needed to reconstruct the decision: route and version, aggregate signals, data scope, action state, threshold source, owner, and timestamp. Restrict raw transcripts and credentials.
Is this checklist enough for compliance?
No. It is practical operating guidance, not legal, privacy, compliance, certification, or security assurance advice. Map it to applicable contracts, laws, standards, provider terms, and risk tolerance.
Recommended next step
Run the AI Tool Risk Checker for the primary and fallback routes, attach both results to the review, and keep customer-specific data and high-impact tools restricted until the failover, authorization, cost, and restoration tests pass.