checklist
AI chatbot admin settings review checklist for small teams
A practical admin settings review checklist for customer-facing AI chatbots, covering retention, training controls, access, sources, connectors, tool actions, handoff, notices, exports, and monitoring.
Use this checklist before launching a customer-facing AI chatbot, after changing vendors, and whenever the bot gains a new source, connector, admin role, retention setting, or tool action.
The goal is simple: make the admin console match the chatbot approval record. Before changing production settings, run the AI Tool Risk Checker and attach the result to the review packet.
Bottom line
Do not approve a customer-facing AI chatbot until an owner can show:
- Which admin settings exist and who owns them.
- Whether transcripts, feedback, source traces, files, and tool calls are retained.
- Whether customer chats can be used for model training or product improvement.
- Which users, teams, vendors, and support staff can access the bot and logs.
- Which sources, connectors, and tools the bot can reach.
- Which customer-visible actions require human approval.
- How customers reach a human, request correction, or ask for deletion/export.
- Which logs, metrics, and review dates prove the settings stayed stable.
Use the Small Team AI Security Checklist for baseline owner assignment, access review, vendor review, incident response, and evidence storage. This page focuses on the chatbot admin console itself.
When to use this checklist
| Scenario | Use this checklist? | Why |
|---|---|---|
| Website support chatbot answers customer questions | Yes | Public answers, transcripts, and handoff paths need owner review. |
| Chatbot uses a help center or knowledge base | Yes | Source scope, stale content, and source access can drift. |
| Chatbot can read ticket history, CRM, billing, or account data | Yes | Customer data access needs explicit boundaries. |
| Chatbot can create tickets, change fields, issue credits, cancel accounts, or trigger workflows | Escalate | Tool actions can create excessive agency if approvals are weak. |
| Vendor releases new admin controls | Yes | Defaults can change what is stored, shared, or enabled. |
| Support team adds new admins or reviewers | Yes | Role and log access should be reviewed. |
| Bot is limited to static public FAQ answers with no accounts, logs, or actions | Maybe | Still review notice, sources, and basic retention. |
| Internal-only support bot | Maybe | Use the same checklist if employees may enter customer, account, or regulated data. |
If the bot can affect a customer account, support record, billing status, trust claim, or stored transcript, treat the settings review as high risk.
Settings inventory template
Copy this table into the chatbot review packet.
| Setting area | Current value | Owner | Evidence to save | Next review |
|---|---|---|---|---|
| Workspace or bot owner | Admin screenshot or export | |||
| Admin roles and support reviewers | Role list and access log | |||
| Transcript retention | Retention setting and policy | |||
| Feedback and quality review storage | Vendor setting or terms note | |||
| Training or product-improvement use | Exact vendor setting and approved wording | |||
| Knowledge sources | Source list and approval record | |||
| Connectors and OAuth scopes | Connector list, scopes, and owner | |||
| Tool actions | Action list and approval rules | |||
| Human handoff route | Queue, SLA, and escalation owner | |||
| Customer notice | Approved notice version | |||
| Export and deletion workflow | Admin path and owner | |||
| Audit logs and monitoring | Log location and review cadence |
Do not leave “default” as the evidence. Write the visible setting value, the date checked, and who checked it.
Minimum admin settings
| Control | Minimum acceptable setting |
|---|---|
| Named owner | One accountable owner and one backup owner. |
| Admin access | Named admins only; no shared admin accounts. |
| MFA/SSO | Required wherever the vendor supports it. |
| Role separation | Bot builders, support reviewers, and billing/admin owners are separate where practical. |
| Transcript retention | Defined retention period and deletion/export path. |
| Training/product-improvement use | Verified setting and customer-safe wording; do not guess. |
| Source scope | Only approved sources; no broad drive, mailbox, wiki, or repository access by default. |
| Connector scope | Minimum OAuth scopes and documented purpose. |
| Tool actions | Disabled unless each action has an owner, limit, test set, and approval rule. |
| Human handoff | Visible customer path and staffed internal queue. |
| Customer notice | First-message notice plus sensitive data warning. |
| Audit logs | Admin changes, transcript access, source changes, and tool calls reviewed on a schedule. |
If the admin console cannot prove a control, compensate with a written process, a narrower launch scope, or a hold decision.
Data and retention settings
| Record type | Setting to verify | Default rule |
|---|---|---|
| Raw chat transcript | Is it stored, for how long, and where? | Keep only as long as support, legal, security, or customer obligations require. |
| Bot-generated summary | Is it saved into tickets, CRM, or analytics? | Treat summaries as support records when they describe customers. |
| Feedback buttons and ratings | Are comments stored with customer identifiers? | Review as customer support data. |
| Source traces | Are retrieved snippets, document names, URLs, or file metadata retained? | Treat traces as potentially sensitive. |
| Tool call logs | Are arguments, results, errors, and approvals stored? | Keep enough for audit and incident review. |
| Attachments and uploads | Can customers or agents upload files? | Disable unless the upload path has classification and deletion rules. |
| Redaction | Can admins redact or delete sensitive entries? | Define who can do it and how evidence is preserved. |
| Customer requests | Can records be exported or deleted by customer, ticket, date, or conversation? | Route deletion/export requests to the privacy/support owner. |
Use the AI chatbot conversation log retention policy template for the retention schedule behind these settings.
Training and product-improvement settings
| Question | Required answer |
|---|---|
| Can customer chats be used to train the vendor’s models? | Record the exact setting, plan terms, and owner approval. |
| Can chats be used for vendor product improvement, quality review, abuse review, or support? | Record the exact vendor wording and whether admins can opt out. |
| Can employees send transcript examples to vendor support? | Require redaction and owner approval before sending customer data. |
| Can the bot learn from new tickets automatically? | Disable until source approval and answer review are defined. |
| Can customer feedback change production answers automatically? | Hold unless there is human review and rollback. |
| Does the public notice match the settings? | Update the notice before launch or after any setting change. |
The operational rule: if you cannot prove the setting, do not promise customers that chats are not used for training, product improvement, or review.
Access and role settings
| Role | Allowed access | Review question |
|---|---|---|
| Bot owner | Configure bot scope, sources, notices, and launch state. | Is this owner still accountable for customer impact? |
| Support lead | Review answer quality, handoff failures, and customer complaints. | Can they see only the conversations they need? |
| Security/privacy owner | Review sensitive data, incidents, deletion/export, and vendor changes. | Can they export evidence without broad admin access? |
| Content owner | Update approved knowledge sources. | Can they publish source changes without approval? |
| Developer/integration owner | Configure connectors and tool actions. | Are production credentials and scopes separated? |
| Vendor support | Access only through approved support path. | Is vendor access logged and time-bound? |
| Former employee or contractor | No access. | Was offboarding verified in the chatbot console and identity provider? |
Review this with the monthly AI tool access review and after every support role change.
Source and connector settings
| Setting | Review rule |
|---|---|
| Public help center source | Approve if owner verifies freshness and scope. |
| Internal wiki source | Limit to approved pages; avoid HR, legal, finance, security, roadmap, and customer folders unless explicitly approved. |
| Ticket history | Use only if customer data handling, retention, and redaction are defined. |
| CRM connector | Limit fields and actions; avoid broad account, billing, and private note access. |
| Email connector | Avoid unless there is a clear support workflow, minimum scope, and log review. |
| File drive connector | Prefer selected folders over whole-drive access. |
| Repository connector | Avoid for customer support bots unless there is a narrow public-docs use case. |
| Automatic source sync | Require source owner, sync cadence, stale-content review, and rollback. |
| External web browsing | Disable unless output review, source citation rules, and abuse controls are defined. |
Use the AI chatbot knowledge base review checklist before enabling or expanding sources.
Tool action settings
| Action type | Default decision | Required control |
|---|---|---|
| Create support ticket | Allow with limits | Customer-visible confirmation and support queue owner. |
| Add internal note | Allow with limits | Label as AI-generated and keep reviewer trace. |
| Change ticket priority | Human approval | Prevent automatic priority escalation loops. |
| Issue credit, refund, cancel, renew, or change plan | Hold | Billing/account owner approval required. |
| Reset password, change admin, or alter access | Hold | Use authenticated non-bot workflow. |
| Send email or public message | Human approval | Reviewer must approve final text and recipient. |
| Update CRM fields | Human approval | Field allowlist and audit log required. |
| Delete, export, or redact data | Human approval | Privacy/security owner approval required. |
| Trigger webhook or automation | Hold | Narrow allowlist, rate limit, and rollback plan required. |
Use the AI chatbot tool action approval checklist for action inventory, permission boundaries, and pre-launch test cases.
Handoff and notice settings
| Setting | Required review |
|---|---|
| First message notice | Says the customer is using AI-assisted support. |
| Sensitive data warning | Warns against passwords, payment details, private keys, recovery codes, and sensitive personal information. |
| Human support path | Customer can ask for a human without arguing with the bot. |
| Uncertainty state | Bot routes missing, conflicting, or low-confidence answers to support. |
| Privacy/security route | Bot routes deletion, export, AI data-use, security, and trust questions to approved owners. |
| Transcript notice | Customer can understand that a chat or summary may be saved if true. |
| Action confirmation | Customer sees what will happen before any account-visible action. |
| Correction path | Customer can report incorrect answers and receive human review. |
Use the AI chatbot disclosure notice template and AI chatbot human handoff policy template behind these settings.
Monitoring and export settings
| Signal | Review cadence | Action |
|---|---|---|
| Admin setting changes | Weekly during pilot, monthly after launch | Compare against the approval record. |
| Source changes | Weekly during pilot, monthly after launch | Confirm source owner and content freshness. |
| Connector scope changes | Immediately after change | Re-run approval before production use. |
| Transcript access | Monthly | Confirm only approved reviewers accessed logs. |
| Tool call failures | Weekly during pilot | Look for prompt injection, unexpected arguments, or repeated failures. |
| Customer corrections | Weekly | Feed approved corrections into source workflow. |
| Sensitive data entries | Weekly | Redact or delete according to policy and improve warnings. |
| Human handoff failures | Weekly | Fix routing, SLA, or notice wording. |
| Export/deletion requests | Monthly | Confirm owner, status, and evidence. |
| Vendor setting changes | Trigger-based | Reapprove public notice and launch record. |
If the tool cannot export settings, save screenshots with date, owner, and scope.
Red flags
Pause launch or disable the risky feature if any of these are true:
- No named owner can explain the current admin settings.
- Training or product-improvement settings are unknown.
- Transcript retention is “forever” or undocumented.
- Vendor support access is unclear.
- The bot can browse broad private sources.
- The bot can take account, billing, deletion, access, or message-sending actions without human approval.
- Source sync can publish unreviewed answers.
- Customer notice says more than the settings prove.
- Admin logs cannot show who changed sources, roles, or actions.
- Former employees, contractors, test users, or vendor trial accounts still have access.
- Sensitive data appears in chats and no redaction/deletion workflow exists.
- The bot answers privacy, security, legal, billing, or account-access questions without routing.
Use the AI chatbot prompt injection response checklist if suspicious prompts, tool calls, or source poisoning appear during review.
Approval record
Copy this into the chatbot launch or change packet.
| Field | Entry |
|---|---|
| Bot name and location | Website, app, help center, portal, or support page. |
| Review date | |
| Reviewer and owner | |
| Launch or change reason | New bot, vendor change, source change, connector change, action change, role change, or periodic review. |
| Customer data scope | Public only, support context, account data, CRM, ticket history, billing, or other. |
| Transcript and record settings | Raw chats, summaries, feedback, traces, uploads, and tool logs. |
| Training/product-improvement setting | Exact verified setting and approved customer wording. |
| Access roles | Admins, reviewers, support staff, vendor support, and offboarding status. |
| Sources and connectors | Approved sources, scopes, owners, and sync rules. |
| Tool actions | Enabled actions, limits, human approval rules, and rollback path. |
| Notices and handoff | First message, sensitive data warning, human path, and privacy/security route. |
| Monitoring | Logs, cadence, metrics, and owner. |
| Decision | Approve, approve with limits, hold, deny, or escalate. |
| Next review | Date and trigger conditions. |
Store the approval record with the vendor review packet and the chatbot launch record.
Monitoring checklist
Use this during pilot and monthly after launch.
- Compare live admin settings against the approval record.
- Check whether retention, transcript, feedback, and tool-call storage changed.
- Verify training and product-improvement controls still match approved wording.
- Confirm admins, reviewers, contractors, and vendor support access.
- Review source list, connector scopes, and sync status.
- Check tool actions for new permissions, failures, or unexpected arguments.
- Review customer complaints, corrections, and human handoff failures.
- Sample transcripts for sensitive data and prohibited topics.
- Confirm deletion/export routes still work.
- Save review evidence and next review date.
Do not wait for a formal incident to review chatbot settings. Drift is usually visible in small admin changes first.
Metrics to track
| Metric | Why it matters |
|---|---|
| Admin setting changes per month | Shows drift and change-control load. |
| New sources or connector scopes | Shows expanding data access. |
| Transcript deletion/export requests | Shows privacy operational demand. |
| Sensitive data entries | Shows whether warnings and routing are working. |
| Human handoff rate | Shows whether scope and notice are clear. |
| Tool actions attempted and approved | Shows automation risk exposure. |
| Tool action rejects or failures | Shows permission, prompt injection, or design issues. |
| Customer corrections | Shows answer quality and source health. |
| Privacy/security questions routed correctly | Shows whether the bot respects sensitive-topic boundaries. |
| Former-user access found | Shows offboarding weakness. |
Track these as operational controls, not vanity chatbot metrics.
Evidence checked
This checklist is aligned with:
- NIST AI RMF Core, which frames AI risk management around govern, map, measure, and manage functions, including roles, documentation, monitoring, third-party risks, human oversight, and lifecycle review.
- NIST AI Risk Management Framework, which provides voluntary guidance for organizations managing AI risks across design, deployment, use, and operation.
- NIST Generative AI Profile, which identifies generative AI risks and risk management actions relevant to deployed generative AI systems.
- OWASP Top 10 for LLM and Generative AI Applications 2025, which includes risks such as prompt injection, sensitive information disclosure, supply chain, vector and embedding weaknesses, misinformation, and excessive agency.
- OWASP LLM06:2025 Excessive Agency, which recommends limiting extensions, limiting permissions, requiring human approval for high-impact actions, enforcing authorization outside the LLM, and monitoring tool activity.
- FTC artificial intelligence guidance, which collects FTC guidance and enforcement activity related to AI claims, privacy, confidentiality, and consumer protection.
- FTC AI privacy and confidentiality guidance, which warns companies to honor privacy and confidentiality commitments when using customer data with AI systems.
- Cybergiz templates for chatbot launch review, source review, conversation log retention, human handoff, disclosure notices, answer correction, prompt injection response, tool action approval, customer impact assessment, and vendor review.
This page is practical operating guidance, not legal, procurement, privacy, compliance, audit, certification, or security assurance advice.
FAQ
How often should we review chatbot admin settings?
Review weekly during the pilot, monthly after launch, and immediately after any vendor, source, connector, role, retention, notice, or tool-action change.
Who should own the settings review?
Assign one accountable product or support owner, with security/privacy review for customer data, transcripts, source access, tool actions, and public trust claims.
What if the vendor does not expose a setting we need?
Narrow the launch scope, document the gap, add a compensating process, or hold the feature. Do not fill the gap with customer-facing promises you cannot verify.
Are screenshots enough evidence?
Screenshots are useful when exports are unavailable, but they should include the date, owner, workspace, setting area, and next review trigger. Prefer exported admin logs or configuration exports when the vendor supports them.
Should tool actions be disabled by default?
Yes. Enable only the actions that have a business need, minimum permissions, test cases, customer confirmation, human approval where needed, logs, and rollback.
Can we say customer chats are not used for training?
Only if the responsible owner has verified the vendor setting, plan terms, and contract wording for that exact claim. Otherwise, route to approved privacy/support wording.
What is the highest-risk setting?
Tool actions tied to account, billing, deletion, access, or outbound messages are usually the highest-risk settings because they combine AI output with real customer impact.
What should we do after a failed settings review?
Record the gap, assign an owner, pause the risky feature, fix the setting or process, and rerun the review before enabling the chatbot in production.