checklist

AI chatbot knowledge base review checklist for small teams

A practical checklist for reviewing AI chatbot knowledge bases, retrieval sources, stale content, conflicting docs, sensitive internal material, customer-safe claims, and source update workflows before customers rely on answers.

Use this checklist before connecting an AI chatbot, support copilot, helpdesk assistant, or customer portal assistant to a knowledge base, help center, product documentation set, support macro library, status page, policy snippets, or internal document collection.

The biggest chatbot failures often start in the source layer. If the knowledge base contains stale refund rules, old setup instructions, broad internal docs, conflicting pricing language, unsupported security claims, or private customer material, the AI system can turn those weaknesses into confident customer-facing answers. Before approving a new source set, run the AI Tool Risk Checker and keep the result with the review record.

Bottom line

Do not let a customer-facing AI chatbot use a knowledge base until you can show:

1. Every source has an owner, purpose, and review date.
2. Stale, duplicate, deprecated, draft, internal-only, and private customer material has been excluded or clearly scoped.
3. Product, pricing, refund, billing, security, privacy, legal, outage, and contract-related sources have an approved owner.
4. Conflicting sources have a resolution rule.
5. Sensitive topics route to a human or approved customer-safe summary.
6. Retrieval tests prove the bot can find the right source and refuse unsupported questions.
7. Source changes trigger retesting before customer launch.

The Small Team AI Security Checklist covers baseline tool approval and admin ownership. This page focuses on the source material that the chatbot uses to answer customers.

When to use this checklist

Scenario	Use this checklist?	Why
Website chatbot answers from help center articles	Yes	Public docs may be stale, conflicting, or too broad for customer-specific answers.
Support copilot retrieves internal macros	Yes	Internal shortcuts can create unsupported customer-facing promises.
Chatbot searches product docs and release notes	Yes	Old versions and beta features can produce wrong guidance.
Bot uses status page or incident notes	Yes	Incident wording must be approved and time-sensitive.
Bot answers security, privacy, or AI data use questions	Yes	Claims must match evidence and customer-safe wording.
Bot searches CRM notes, tickets, or call transcripts	Escalate	Customer records need stricter data and retention review.
Bot uses broad company file storage	Escalate	The source set is likely too broad and may contain sensitive material.
Internal employee FAQ bot with public docs only	Maybe	Use if answers can affect customer commitments.
Static non-AI help center	No	Use normal documentation review instead.

If the bot can answer customers from a source, that source is part of the customer support system.

Source inventory template

Create one row per source set, not one row for every article.

Field	Entry
Source name	Help center, product docs, support macros, status page, release notes, policy snippets, security FAQ, or internal docs.
Source owner	Person responsible for accuracy and removal.
Customer visibility	Public, authenticated customer, agent-only, internal-only, or prohibited.
Allowed bot use	Customer-facing answer, draft only, source suggestion, escalation context, or not allowed.
Data class	Public, internal, customer data, security evidence, legal wording, billing policy, incident note, or regulated data.
Update cadence	On change, weekly, monthly, quarterly, release-based, or deprecated.
Last reviewed	Date reviewed by source owner.
Sensitive topics	Refunds, pricing, cancellation, outage, security, privacy, contract, account access, or regulated workflow.
Retrieval priority	Primary, secondary, fallback, or never use.
Conflict owner	Person who resolves inconsistent answers.
Test coverage	Included in retrieval test set: yes or no.
Decision	Approve, approve for draft only, restrict, remove, hold, or escalate.

Keep this inventory with the chatbot launch record and customer impact assessment.

Source approval matrix

Source type	Default decision	Required control
Public help center articles	Approve if current	Owner, last review date, stale content removal, and source citation.
Product documentation	Approve with version control	Version mapping, release owner, and deprecated page removal.
Support macros	Draft-only first	Human review, customer-safe wording, and prohibited promise check.
Status page	Approve with strict source priority	Official status source only and incident owner approval.
Refund, billing, and cancellation policy	Approve with owner review	Billing owner approval and handoff for exceptions.
Security and privacy FAQ	Approve only from customer-safe summary	Evidence owner review and no unsupported assurance claims.
Release notes	Restrict	Prevent beta, roadmap, and unreleased feature promises.
Internal troubleshooting docs	Draft-only or restrict	Remove secrets, internal names, unsafe steps, and unsupported procedures.
Customer tickets or call transcripts	Escalate	Customer data approval, retention rule, redaction, and limited access.
Broad file storage or wiki	Deny by default	Replace with curated source set.

The narrower the source set, the easier it is to test and defend.

Stale content review

Stale content creates confident wrong answers. Review these signals before launch.

Stale signal	Action
Article has no owner or review date	Remove from bot source set until reviewed.
Product UI has changed since article was written	Update or remove.
Pricing, plan, feature, or policy wording has changed	Route to owner before use.
Article references old product names, legacy screens, or retired workflows	Remove or redirect.
Support macro was written for a one-off customer situation	Mark internal-only or remove.
Release note describes beta or limited-availability feature	Restrict unless the bot can identify eligible customers safely.
Status or incident message is no longer current	Remove from retrieval or archive outside bot scope.
Security or privacy answer references old evidence	Update customer-safe summary before use.
Article contains “coming soon” or roadmap wording	Block from customer-facing answers unless owner approves.
Conflicting articles both appear active	Escalate to conflict owner and retest after resolution.

Do not rely on the chatbot to infer which source is newer. Make the source set unambiguous.

Conflict resolution rules

Conflicting sources are common in small teams. Define precedence before launch.

Conflict	Winning source
Help center conflicts with current product UI	Product owner-reviewed documentation.
Support macro conflicts with public policy	Public policy or owner-approved customer-safe wording.
Sales deck conflicts with pricing page	Pricing owner-approved source.
Release note conflicts with current docs	Current docs unless release owner confirms exception.
Internal troubleshooting guide conflicts with safe customer steps	Safe customer-facing guide.
Security FAQ conflicts with trust page	Evidence owner-approved trust or security summary.
Status page conflicts with support note	Official status page or incident owner-approved update.
Old article conflicts with new article	New article only after owner confirms migration.
Customer-specific contract conflicts with generic help center	Human handoff; do not let bot decide.

If there is no clear winning source, the bot should say it cannot answer and route to support.

Sensitive source rules

Some sources should not be available to a customer-facing bot without extra review.

Sensitive source	Default rule
Security questionnaires	Use only approved customer-safe summary.
SOC 2 reports, pen test summaries, or audit evidence	Do not expose directly; route to approved trust process.
Contracts, order forms, and customer terms	Human handoff.
Customer tickets, call transcripts, and CRM notes	Use only after customer data approval and redaction review.
Internal incident notes	Use only approved public or customer communication.
Roadmap and sales negotiation notes	Exclude from customer-facing retrieval.
Engineering runbooks	Exclude unless converted to safe customer troubleshooting steps.
Internal Slack, email, or wiki exports	Deny by default.
Payment, account recovery, or identity verification docs	Route to authenticated workflow and human review.
Regulated data workflows	Escalate before any AI retrieval.

Pair customer-record sources with the customer data AI approval form.

Retrieval test set

Test retrieval separately from answer quality. The bot must find the right source before it can answer safely.

Test category	Expected result
Exact documented question	Retrieves the current approved source.
Question with old product name	Retrieves current source or asks for clarification.
Question about deprecated feature	Refuses outdated instruction or routes to updated guidance.
Pricing exception	Retrieves policy and hands off for exceptions.
Refund dispute	Retrieves approved policy and hands off.
Security claim	Retrieves customer-safe security summary or routes to owner.
Privacy or deletion request	Routes to approved privacy/support process.
Outage question	Retrieves official status source only.
Missing answer	Says it cannot answer from approved sources and hands off.
Prompt injection in customer text	Ignores instruction and follows bot policy.
Customer pastes private data	Minimizes, warns, and routes according to data policy.
Source conflict	Escalates or uses documented precedence.

Record the source retrieved, answer outcome, and whether the answer stayed inside approved scope.

Update workflow

Knowledge base changes should trigger chatbot review.

Trigger	Required action
New product feature	Add source owner, allowed bot use, and test cases before retrieval.
Pricing or plan change	Retest pricing, cancellation, billing, and sales questions.
Refund or support policy change	Retest disputes, exceptions, and angry-customer scenarios.
Security or privacy evidence update	Update customer-safe summary and retest trust questions.
New incident or outage message	Confirm source priority and expiration date.
Deprecated article	Remove from source set and retest related questions.
New internal macro	Keep draft-only until owner approves customer-safe wording.
Vendor or subprocessor change	Update trust/security summary before customer answers change.
Customer complaint about bot answer	Review source, answer, retrieval path, and similar questions.

Do not let documentation changes silently flow into the bot without tests when the topic affects customer commitments.

Access and retention rules

Source governance also needs access governance.

Control	Minimum expectation
Source access	Bot can retrieve only approved source sets.
Admin access	Only named owners can add or remove source collections.
Change log	Source additions, removals, and priority changes are recorded.
Customer data separation	Customer records are not mixed with public help content.
Transcript retention	Chat transcripts and feedback follow a retention rule.
Source snapshots	Keep enough version history to investigate wrong answers.
Export control	Prevent broad export of internal source sets through bot responses.
Review schedule	High-risk sources are reviewed monthly or on change.
Removal workflow	Deprecated or risky sources can be removed quickly.

If the bot vendor manages retrieval or indexing, confirm how source removal propagates and how quickly stale indexed content disappears.

Approval record

Copy this into the chatbot evidence packet.

Field	Entry
Chatbot name	Tool, vendor, workspace, and launch mode.
Source sets approved	Names and links to approved collections.
Source sets excluded	Collections that must not be used.
Sensitive sources	Security, privacy, billing, legal, customer records, incident notes, or regulated data.
Source owners	Documentation, product, support, security, privacy, billing, incident, and admin owners.
Conflict rules	Source precedence and escalation owner.
Retrieval test result	Pass rate, failed categories, remediation, and retest date.
Update triggers	Changes that require retesting.
Monitoring owner	Person who reviews wrong answers and source issues.
Decision	Approve, approve with limits, draft-only, hold, deny, or escalate.
Next review	Date and trigger conditions.

Keep this record redacted. It should prove source control without storing raw customer conversations.

Monitoring checklist

After launch, look for source-related failures.

Signal	What it means
Bot cites old article	Stale source or index cleanup failure.
Bot gives different answers to same policy question	Source conflict or retrieval instability.
Bot answers unsupported questions	Missing refusal or handoff rule.
Bot exposes internal wording	Internal source leaked into customer scope.
Bot makes unsupported security claim	Trust source or prompt rule needs owner review.
Bot promises roadmap or pricing exception	Sales or release source should be removed or restricted.
Bot misses outage context	Status source priority is wrong.
Customers correct the bot	Source accuracy or retrieval quality problem.
Support agents ignore bot drafts	Source or answer quality is too low for workflow.
Source owner cannot be found	Source should be removed until ownership is clear.

Monitoring is not just model monitoring. It is documentation quality monitoring.

Metrics to track

Metric	Why it matters
Approved source sets	Shows what the bot is allowed to use.
Sources without owner	Shows governance gaps.
Sources past review date	Shows stale content risk.
Retrieval test pass rate	Shows whether the bot finds the right source.
Wrong answer source category	Shows which source sets cause failures.
Conflict incidents	Shows documentation inconsistency.
Sensitive source attempts	Shows users asking risky questions.
Source removals after launch	Shows cleanup volume.
Time to remove stale source	Shows operational readiness.
Customer complaints tied to source	Shows real-world harm signal.

If a source set creates repeated failures, remove it before tuning prompts.

Evidence checked

This checklist is aligned with:

1. NIST AI Risk Management Framework, which frames AI risk management across design, deployment, evaluation, and operation.
2. NIST AI RMF Core, which emphasizes governance, mapping risks, measuring performance, and managing risks throughout the AI lifecycle.
3. NIST Generative AI Profile, which identifies generative AI risks and risk management actions.
4. OWASP Top 10 for Large Language Model Applications, which highlights prompt injection, insecure output handling, sensitive information disclosure, supply-chain, and overreliance risks.
5. FTC AI and data guidance, which warns businesses to be careful about collecting, retaining, and using consumer data for AI.
6. Cybergiz templates for chatbot launch review, customer impact assessment, customer data approval, trust-center wording, vendor review, evidence retention, and incident response.

This page is practical operating guidance, not legal, procurement, privacy, compliance, audit, certification, or security assurance advice.

FAQ

Is this only for RAG chatbots?

No. Use it for any chatbot or copilot that retrieves or summarizes source material, including help center search, support macros, internal docs, CRM notes, status pages, or customer-safe security summaries.

Should the chatbot be allowed to use all public help center articles?

Not automatically. Public does not mean current, accurate, or safe for every customer situation. Remove stale, conflicting, beta, roadmap, and exception-heavy content before retrieval.

What is the fastest safe source set for a pilot?

Start with a small set of current, owner-reviewed help center articles for low-risk product questions. Exclude billing exceptions, security claims, account access, incidents, and customer records until those workflows have separate review.

Should customer tickets be part of the knowledge base?

Usually not for a first pilot. Tickets can contain private data, one-off exceptions, wrong assumptions, and customer-specific commitments. Use curated, redacted, owner-approved examples instead.

What if the bot gives the right answer but cites the wrong source?

Treat it as a failure. Wrong citations reduce trust and can hide stale or conflicting source problems. Fix retrieval and source priority before expanding.

How often should sources be reviewed?

Review high-risk sources on change and at least monthly during pilot. Review low-risk public docs on the documentation owner’s normal cadence, but remove anything without an owner or review date.

Who owns the knowledge base review?

The documentation or support owner owns source accuracy. The chatbot owner owns retrieval behavior. Security, privacy, billing, product, incident, and customer-success owners approve their sensitive source areas.