checklist
AI chatbot retirement decision checklist for small teams
A practical retirement checklist for deciding whether to keep, limit, pause, or retire a customer-facing AI chatbot capability, including trigger criteria, customer impact, data cleanup, connector shutdown, communication, and final verification.
Use this checklist when a small team needs to decide whether an AI chatbot capability should stay live, be narrowed, become human-only, be paused, or be retired completely.
Retirement is not an admission that every chatbot is a failure. It is an operating decision when the capability no longer has an acceptable balance of customer value, answer quality, human fallback, data protection, operational ownership, and reversible controls. Before deciding, run the AI Tool Risk Checker and attach the result to the retirement record.
Bottom line
Retire or keep a chatbot capability human-only when repeated wrong answers, failed handoffs, unresolved data or connector risk, unsafe tool actions, poor evidence, or support burden cannot be reduced to an acceptable level. Keep it live only when the current scope has an owner, usable evidence, tested controls, and a fast pause path. If the problem is limited to one topic, channel, source, or action, limit that component instead of taking down the whole service.
Use the Small Team AI Security Checklist for baseline ownership, access review, and incident routing. This page focuses on the keep, limit, pause, and retire decision plus the cleanup work that follows it.
When to use this checklist
| Situation | Use this checklist? | Likely first action |
|---|---|---|
| A single topic produces repeated wrong answers | Yes | Limit the topic and open a correction plan. |
| A connector exposes more data than the bot needs | Yes | Pause the connector and review permissions. |
| A high-impact tool action cannot be reliably approved or reversed | Yes | Disable the action and keep the bot informational only. |
| Support cannot handle the human fallback queue | Yes | Reduce scope or pause the affected route. |
| The vendor or product owner changes retention, access, or model behavior | Yes | Run a change review before deciding. |
| The bot has low usage but no material risk | Maybe | Compare customer value and operating cost before retiring. |
| A cosmetic copy change is needed | Usually no | Use the normal change approval path. |
Do not wait for a confirmed breach. A capability with weak evidence, unclear ownership, or no disable path is already difficult to operate safely.
Retirement trigger matrix
| Trigger | Severity | Default response | Evidence to capture |
|---|---|---|---|
| Customer data may have been exposed | High | Pause the affected route or connector; escalate to security and privacy owners. | Conversation IDs, access logs, affected data class, containment time. |
| Unauthorized account, billing, deletion, access, or outbound action | High | Disable the action immediately; review whether the whole capability should retire. | Action request, approval state, tool logs, outcome, rollback result. |
| Repeated wrong or unsafe answers after reasonable fixes | High | Make the topic human-only or retire the topic. | Sample set, correction history, source changes, pass rate. |
| Handoff fails for sensitive or disputed topics | High | Pause the route and force human handoff. | Failed handoff examples, queue status, customer impact. |
| Prompt injection or source poisoning changes behavior | High | Pause the affected source, route, or capability; retest before restart. | Attack input, retrieved content, output, controls that failed. |
| Logs cannot reconstruct high-impact events | High | Disable high-impact features until logging is fixed. | Missing fields, retention settings, affected time window. |
| No named owner for sources, connectors, or incident response | Medium | Limit scope until an owner is assigned; retire if ownership cannot be established. | Owner map, unanswered escalation requests. |
| Vendor terms, retention, or subprocessors change materially | Medium | Pause the affected data flow and complete a vendor review. | Notice, contract or documentation change, data-flow impact. |
| Low usage, low value, or high support cost | Medium | Compare keep, limit, and retire options. | Usage, support time, deflection quality, cost, customer feedback. |
The trigger is not the final decision. It starts a documented review with an explicit owner and deadline.
Keep, limit, retire decision table
| Decision | Use when | Required controls before decision closes |
|---|---|---|
| Keep | Value is clear, current scope is controlled, and evidence supports continued use. | Owner, monitoring, handoff, data rules, action limits, and pause path are verified. |
| Limit | The risk is isolated to a topic, source, channel, customer group, connector, or action. | Exact boundary, blocked scope, user notice, test set, and review date are recorded. |
| Human-only | Automation is not reliable enough, but the workflow still has customer value. | Human queue owner, response target, context transfer, and customer route are ready. |
| Pause | There may be active harm, exposure, unsafe action, or missing evidence. | Disable switch, incident owner, evidence preservation, and restart gate are assigned. |
| Retire capability | The risk or operating burden is persistent, value is low, or safe controls are not achievable. | Customer communication, data and connector cleanup, access removal, and final verification are complete. |
Do not use “keep” as the default because the team has already invested in the bot. Treat sunk implementation effort as irrelevant to the safety decision.
Retirement intake form
Copy this into the chatbot operations record.
| Field | Entry |
|---|---|
| Review date | |
| Request owner | |
| Bot and capability name | |
| Current channels | |
| Current topics and customer groups | |
| Proposed decision | Keep, limit, human-only, pause, or retire. |
| Trigger for review | |
| Customer data involved | None, public, internal, customer, account, billing, regulated, credential-like, or unknown. |
| Connectors and tools involved | |
| Human fallback owner | |
| Security/privacy owner | |
| Customer communication owner | |
| Decision deadline | |
| Evidence location |
If the data involved, connected systems, or active actions are unknown, do not close the review as “keep”.
Evidence to review
Collect enough evidence to make the decision reproducible. Prefer a small, recent sample over an impressive but stale dashboard.
- Current scope: topics, channels, users, customer groups, sources, connectors, and tool actions.
- Recent answer samples, including wrong answers, refusals, corrections, and sensitive-topic routes.
- Human handoff success, queue wait time, escalations, and customer complaints.
- Prompt injection, abuse, sensitive-information, and source-integrity test results.
- Connector permissions, tool action logs, approval records, retries, and rollback results.
- Conversation, transcript, source, and vendor retention settings.
- Open incidents, exceptions, unresolved remediation items, and vendor changes.
- Usage, support cost, customer value, and any evidence of abandoned or duplicate workflows.
- Named owners for monitoring, incidents, customer communication, and cleanup.
NIST AI RMF Core treats decommissioning as part of post-deployment monitoring and risk management. The decision should therefore use operational evidence, not only a launch-time review.
Customer impact review
Before a pause, limitation, or retirement, identify who may be affected and what route replaces the bot.
| Question | Required answer |
|---|---|
| Which customers, prospects, employees, or candidates use the capability? | |
| Which topics or journeys will change? | |
| What is the replacement route? | Human support, form, email, phone, or another controlled workflow. |
| Will response times change? | |
| Does the notice need to explain the change? | |
| Are there open conversations that need follow-up? | |
| Are any accessibility, language, or timezone needs affected? | |
| Who approves the customer-facing wording? |
Do not silently remove a route that customers rely on for account access, billing, privacy, security, or safety questions. Route those requests to a named human owner.
Data and connector cleanup
Retiring the visible chat widget does not necessarily remove stored conversations, indexes, OAuth grants, service accounts, or vendor-side copies.
| Cleanup area | Check | Completion evidence |
|---|---|---|
| Conversation history | Stop new collection where possible; apply the approved retention and deletion rule. | Setting screenshot or export plus deletion record. |
| Customer data | Identify stored data classes and unresolved requests. | Data inventory and request tracker. |
| Knowledge indexes | Remove retired private sources and derived indexes. | Source list, deletion job, verification result. |
| OAuth and API access | Revoke connector grants, tokens, service accounts, and unused scopes. | Revocation log and access review. |
| Webhooks and automations | Disable events, jobs, queues, and scheduled syncs. | Job status and test showing no new event. |
| Admin access | Remove bot-specific roles and temporary access. | Access review record. |
| Vendor account | Confirm cancellation, retention, support access, and deletion terms. | Vendor confirmation or documented limitation. |
| Backups and exports | Apply the same retention and access policy to copies. | Backup location review and deletion/hold record. |
Keep legal, security, or incident holds intact. A retirement decision does not authorize deleting evidence that must be preserved.
Source and prompt cleanup
- Remove the retired capability from system prompts, routing rules, retrieval filters, and agent instructions.
- Mark retired sources as unavailable so a future workflow cannot silently reuse them.
- Archive the final approved prompt and configuration with sensitive values redacted.
- Record why the capability was retired and which failure or value evidence supported the decision.
- Remove customer-facing links, embedded scripts, help-center articles, and internal runbook references.
- Search for duplicate bots, shadow automations, copied prompts, and old webhook routes.
- Keep a human-readable replacement instruction for support and customer success teams.
Do not paste customer transcripts, API keys, tokens, or private vendor exports into a public retirement record.
Tool action shutdown
Disable actions before removing the interface. This reduces the chance that an old client, retry queue, or direct integration can still invoke the capability.
| Action type | Shutdown order | Verification |
|---|---|---|
| CRM or ticket write | Disable bot permission, then drain or cancel queued writes. | A test request is denied and no new record appears. |
| Billing or refund action | Remove action from policy and connector, then reconcile pending requests. | Approval and denial logs show the action is unavailable. |
| Account or access change | Disable service identity or action route, then review recent activity. | Direct and indirect requests fail safely. |
| Email or outbound message | Disable send permission, templates, and retries. | Test message is not sent. |
| Deletion or export action | Stop new requests, preserve open requests, and hand off to the owner. | Request register reconciles with the replacement workflow. |
Use the AI chatbot tool action approval checklist when the capability has any write or outbound action.
Customer communication plan
Use plain, accurate wording. Do not claim that all historical data was deleted unless the team has verified that result.
Short notice template
We are changing how this support route works. Starting on [date], the AI assistant will [be limited / be unavailable / route requests to our support team]. For account, billing, privacy, security, or other sensitive questions, contact [human route]. Existing requests will be handled by [owner or process].
Communication checklist
- Confirm the effective date and timezone.
- Name the replacement route and expected response time.
- Explain what happens to open conversations and pending actions.
- Provide a human route for sensitive or disputed issues.
- Update the widget, help center, onboarding, email, and internal scripts.
- Give support and customer success teams a short answer to likely questions.
- Record the approver and published locations.
Avoid blaming customers or describing an unverified security incident as the reason. Coordinate incident communications separately when required.
Internal handoff plan
| Team | Handoff | Owner | Due date |
|---|---|---|---|
| Support | Replace bot route, scripts, macros, and queue rules. | ||
| Product | Remove UI, feature flags, routing, and analytics events. | ||
| Engineering | Disable services, queues, webhooks, connectors, and credentials. | ||
| Security/privacy | Review evidence, holds, access, and data cleanup. | ||
| Customer success | Notify affected accounts and capture feedback. | ||
| Legal or compliance | Review notice, retention, contract, or regulatory needs when relevant. | ||
| Finance or procurement | Close billing, renewal, and vendor ownership tasks. |
Assign one closure owner even when several teams perform the work. Shared responsibility without a final owner leaves partial shutdowns behind.
Final verification checklist
- The decision and scope are approved by the relevant owners.
- Customer-facing routes show the approved replacement or notice.
- New chatbot sessions cannot start in the retired scope.
- Existing queues, retries, webhooks, and scheduled jobs are drained or disabled.
- Connectors, OAuth grants, service accounts, and admin roles are removed or intentionally retained.
- Tool actions are unavailable and recent activity is reconciled.
- Sources, indexes, prompts, scripts, and automations no longer route traffic to the retired capability.
- Retention, deletion, export, legal hold, and incident evidence decisions are recorded.
- Monitoring shows no new traffic or action attempts after the effective date.
- Support, customer success, and security owners know how to handle residual requests.
- The closure record contains links to evidence without exposing secrets or customer data.
Run one safe test from every relevant channel after shutdown. A green feature flag is not proof that every route is disabled.
Retirement record
| Field | Entry |
|---|---|
| Final decision | Keep, limit, human-only, pause, or retire. |
| Capability retired | |
| Effective date | |
| Trigger and rationale | |
| Customer impact | |
| Replacement route | |
| Data cleanup status | |
| Connector and access cleanup status | |
| Tool action cleanup status | |
| Open holds or exceptions | |
| Evidence reviewed | |
| Closure owner | |
| Approvers | |
| Final verification date | |
| Next review or follow-up |
Store the record where the team keeps other AI tool decisions. Do not make the record depend on one person’s memory or a vendor dashboard that may later disappear.
Metrics to track
| Metric | Why it matters |
|---|---|
| Retired-scope traffic after effective date | Shows whether routes are actually disabled. |
| Remaining action attempts | Finds stale clients, retries, or hidden integrations. |
| Customer requests after retirement | Measures replacement-route demand. |
| Human response time after handoff | Shows whether the fallback is workable. |
| Open deletion, export, or correction requests | Shows residual data obligations. |
| Access grants removed | Shows connector and identity cleanup progress. |
| Support hours saved or added | Tests the value of the decision. |
| Reopened retirement tasks | Shows incomplete shutdown or unclear ownership. |
Do not use zero traffic alone as proof of safe retirement. Verify access, actions, queues, connectors, retention, and customer routes as well.
Evidence checked
This checklist is aligned with:
- NIST AI RMF Core, which includes post-deployment monitoring plans covering appeal and override, decommissioning, incident response, recovery, and change management.
- NIST report on challenges to monitoring deployed AI systems, which describes post-deployment monitoring as important for understanding real-world AI behavior and organizing monitoring questions around who, what, when, why, and how.
- NIST AI 800-4 publication page, which describes monitoring deployed systems for reliability, unforeseen outputs, and unexpected consequences.
- CISA joint guidance on deploying AI systems securely, which emphasizes protecting AI systems and related data and services, detecting malicious activity, and responding to incidents.
- CISA and UK NCSC secure AI system development guidance, which applies secure practices across the AI system lifecycle.
- CISA JCDC AI Cybersecurity Collaboration Playbook alert, which provides a voluntary process for sharing AI cybersecurity incidents and vulnerabilities.
- OWASP Top 10 for LLM Applications, which covers prompt injection, sensitive information disclosure, insecure plugin design, excessive agency, misinformation, and overreliance.
- FTC artificial intelligence guidance, which tracks guidance and enforcement activity related to AI claims, accuracy, privacy, confidentiality, and consumer protection.
- Cybergiz chatbot templates for launch, knowledge review, tool action approval, handoff, monitoring, escalation, pause and fallback, incident communication, and weekly review.
This page is practical operating guidance, not legal, privacy, compliance, audit, certification, customer-support, incident-response, product-management, or security assurance advice.
FAQ
Should we retire the whole chatbot after one bad answer?
Usually not. First determine whether the issue is isolated to a topic, source, connector, channel, or action. Limit or pause the affected scope when a smaller boundary controls the risk. Retire the whole capability when the problem is systemic or the team cannot operate the replacement safely.
What is the difference between pause and retire?
Pause is a temporary control while the team investigates, contains, or fixes a problem. Retire is a deliberate closure decision that includes customer routing, access and connector cleanup, data handling, and final verification.
Can we keep the chatbot for public FAQs after retiring account support?
Yes, if the public FAQ scope has separate sources, permissions, routing, monitoring, and tests. Record the blocked account-support scope explicitly so future changes do not silently re-enable it.
Do we need to delete every transcript when retiring the bot?
Follow the approved retention, deletion, export, legal hold, and incident-evidence rules. Do not delete records automatically if a customer request, contractual duty, investigation, or legal hold requires preservation.
What should happen to OAuth grants and service accounts?
Revoke grants, remove unused scopes, disable service identities, and verify that no retry queue or scheduled job can recreate access. Record any intentionally retained access and its owner.
How do we prove the bot is really retired?
Test every relevant channel, review traffic and action attempts after the effective date, inspect queues and webhooks, verify access removal, and reconcile open customer or data requests. A disabled front-end widget is not enough.
Who approves a retirement decision?
At minimum, include the product or support owner, security/privacy owner, and the owner of the customer replacement route. Add engineering, customer success, legal, procurement, or finance when their systems or obligations are affected.
What if the team cannot complete connector cleanup?
Keep the capability paused, escalate the cleanup owner, and document the residual risk. Do not call the retirement complete while active data or action access remains unexplained.