checklist
AI chatbot feedback and complaint triage checklist for small teams
A practical checklist for classifying chatbot feedback and complaints, protecting submitted data, investigating unsafe answers, routing high-risk cases, and recording correction and release decisions.
Editorial note: Published on September 8, 2026, to complete the September 3, 2026 editorial slot. Sources were reviewed on September 8, 2026.
Use this checklist when a customer reports a wrong, unsafe, private, biased, misleading, inaccessible, or unexpectedly expensive chatbot result. It also applies to thumbs-down feedback, correction requests, human handoffs, support tickets, abuse reports, and repeated complaints about one route, source, model, or action.
The short answer: collect the minimum information needed, classify the concern before promising a fix, protect the complaint as potentially sensitive input, preserve a redacted evidence record, reproduce the issue with a synthetic or approved masked fixture, route high-risk cases to a human owner, and make a release or restriction decision based on the affected scope. Do not close a complaint because the model produced a different answer on a second try.
Start with the AI Tool Risk Checker to record the route, data, actions, and owner. Use the Small Team AI Security Checklist for identity, data, access, logging, incident, and vendor foundations. Pair this page with the AI chatbot human review sampling and answer quality scorecard, the AI chatbot answer correction workflow template, and the AI chatbot incident communication template.
Bottom line
Feedback is an operational signal, not a complete investigation. A complaint may identify a quality problem, a privacy or authorization problem, an unsafe downstream action, a misleading notice, an outage, or a product expectation that was never defined. The first response should protect the user and preserve options while the team determines scope.
A defensible triage process can show:
- The report was assigned an owner, category, severity, scope, and next decision time.
- The team captured a redacted or synthetic reproduction without spreading private conversation content.
- The route, prompt or policy version, model, source, connector, tool, and output destination were recorded.
- High-risk reports were contained before the team attempted a broad fix or resumed an action.
- Claims and outputs were checked against approved sources and the correct tenant and role boundary.
- A correction, notice, handoff, rollback, or pause decision has an owner and evidence.
- Trends are reviewed so repeated low-severity reports do not hide a systemic failure.
NIST AI RMF describes continuous risk management through govern, map, measure, and manage functions, including documentation, testing before deployment, monitoring during operation, and response to impacts. OWASP LLM05:2025 warns that unvalidated model output can create downstream security issues, while OWASP LLM09:2025 describes credible-looking false or misleading output and recommends validation, human oversight, risk communication, and clear limitations. This checklist turns those ideas into a small-team complaint workflow; it does not determine legal notice, refund, regulatory, or liability obligations.
This is operational guidance, not legal advice, a compliance certification, or a guarantee that a complaint will be resolved within a particular time.
When to use this checklist
| Signal or trigger | Use this checklist? | Minimum review |
|---|---|---|
| User says an answer is wrong or incomplete | Yes | Source, route, version, correction path, and severity. |
| User reports private or cross-tenant information | Yes | Containment, affected scope, access review, and security owner. |
| User reports unsafe advice or a high-impact recommendation | Yes | Human review, output validation, communication, and route restriction. |
| Chatbot performed or attempted an unexpected action | Yes | Action authorization, target, evidence, and pause decision. |
| Repeated complaints about one intent or source | Yes | Trend grouping, source review, regression case, and owner. |
| Complaint includes personal, financial, health, legal, or account data | Yes | Minimum collection, restricted access, retention, and privacy routing. |
| Service outage or quota behavior causes customer impact | Yes | Incident or outage workflow plus customer communication. |
| Internal prototype with synthetic data only | Maybe | Record owner, impact boundary, and production transition gate. |
If the team cannot identify who owns the report or what safe behavior applies while investigating, move the affected route to read-only, draft-only, human, or paused mode.
Triage intake form
Copy this into the support, quality, security, or privacy case. Keep raw conversation content in a restricted evidence location only when necessary.
| Field | Entry |
|---|---|
| Case ID and received date | feedback-YYYY-MM-DD-NNN |
| Reporter and contact path | Customer, employee, support agent, monitoring signal, or anonymous report. |
| Product and route | Chatbot, interface, environment, and route identifier. |
| User and tenant scope | Verified scope if available; do not guess from message text. |
| Concern summary | One sentence using neutral, factual language. |
| Category | Quality, privacy, authorization, safety, action, availability, cost, accessibility, abuse, or other. |
| Severity and rationale | Low, medium, high, or critical according to the team matrix. |
| Affected output or action | Answer, source, ticket, message, record, tool, or customer workflow. |
| Data class | Public, internal, customer, sensitive, regulated, or unknown. |
| Route versions | Model, prompt, policy, source, connector, tool, and deployment references. |
| Immediate containment | No change, limit, redact, handoff, disable action, rollback, or pause. |
| Owner and next update | Named triage owner, reviewer, and time for the next decision. |
| Evidence location | Restricted record for redacted sample, aggregate signals, and test result. |
| Disposition | Explain, correct, fix, notify, restrict, rollback, close, or escalate. |
Intake checklist:
- The first record avoids blame, unsupported cause, and unnecessary raw content.
- The reporter receives a safe acknowledgment that does not promise a final conclusion.
- The case is linked to a route and owner, not only to a person or inbox.
- The team has recorded whether the complaint can expose private or cross-tenant data.
- The next decision time and escalation owner are explicit.
- The evidence is access-controlled and has a retention or deletion rule.
Complaint and feedback taxonomy matrix
Use one primary category and record secondary categories when a report crosses boundaries.
| Category | Example pattern | First control | Escalate to |
|---|---|---|---|
| Factual quality | Incorrect product, policy, or account answer | Verify approved source and reproduce | Product or quality owner. |
| Missing or stale source | Answer omits or contradicts current material | Check source version, freshness, and retrieval | Knowledge or platform owner. |
| Privacy disclosure | Output includes an unexpected personal or confidential detail | Stop sharing, preserve scope evidence, and restrict | Privacy and security owner. |
| Authorization boundary | User sees another role, tenant, or account context | Pause affected route and test boundary | Security and service owner. |
| Unsafe advice | Output could cause physical, financial, legal, or security harm | Human review and safe response | Domain owner and security owner. |
| Unexpected action | Bot sends, edits, deletes, purchases, or changes a record | Disable or reverse action path where safe | Engineering and incident owner. |
| Misinformation | Confident but unsupported or fabricated claim | Mark uncertainty, verify, and correct | Quality and communication owner. |
| Availability or latency | Timeouts, loops, quota, or inaccessible fallback | Apply safe state and inspect capacity | Platform and incident owner. |
| Cost impact | Unexpected charge, usage, or plan behavior | Limit route and verify aggregate usage | Finance and platform owner. |
| Accessibility or inclusion | User cannot understand, access, or correct result | Human path and interface review | Product and support owner. |
| Abuse or manipulation | Prompt injection, harassment, or repeated probing | Preserve redacted signal and apply abuse controls | Security and moderation owner. |
Do not downgrade a privacy, authorization, or high-impact action case to ordinary feedback because the final answer looked harmless.
Evidence and privacy boundary map
Map only the data needed to understand the issue. A complaint often contains more personal information than the team needs to reproduce the failure.
| Evidence layer | Minimum record | Privacy rule |
|---|---|---|
| Report | Category, impact, timestamp, and scope | Avoid copying the full conversation into the general ticket. |
| Request context | Route, user or tenant class, intent, and safe fixture ID | Use a verified identifier or synthetic fixture, not guesswork. |
| Version state | Model, prompt, policy, source, connector, tool, and deployment | Record references and hashes or version labels, not private data. |
| Retrieval | Source ID, permission decision, freshness, and result class | Redact content and verify tenant boundary. |
| Output | Redacted excerpt or structured error class | Store only the material needed for review. |
| Action | Target class, authorization decision, and outcome | Preserve side-effect evidence without secrets. |
| Communication | Acknowledgment, correction, notice, or handoff | Use approved language and a named reviewer. |
| Closure | Fix, test, scope, owner, and residual risk | Do not claim resolution without a verification result. |
Boundary checklist:
- Complaint records use the least amount of raw content needed.
- Sensitive attachments and transcripts are held in a restricted location.
- Support, quality, security, privacy, and engineering access is role-appropriate.
- Test fixtures are synthetic or approved masked data.
- Exports, screenshots, logs, backups, and analytics copies have an owner and retention rule.
- The team can delete or restrict the complaint evidence when its purpose ends.
Severity and routing matrix
Severity should reflect plausible impact and scope, not just the reporter’s tone or the number of words in the complaint.
| Severity | Example pattern | Default response | Decision owner |
|---|---|---|---|
| Low | Minor wording or non-material answer issue | Acknowledge, verify source, correct, and trend | Product or support owner. |
| Medium | Repeated wrong answer, stale source, or confusing disclosure | Investigate, add a regression case, and set a review date | Product and quality owner. |
| High | Sensitive output, unsafe recommendation, unauthorized context, or unexpected action | Contain, restrict the route, preserve evidence, and escalate | Security, privacy, and service owner. |
| Critical | Broad disclosure, high-impact action, active abuse, or material customer harm | Pause affected behavior, open incident, coordinate communication, and verify recovery | Incident owner with executive or legal routing as required. |
Routing questions:
- Does the report involve a person, tenant, account, or sensitive record outside the expected scope?
- Could a similar request affect more users or trigger a downstream action?
- Is the issue reproducible, version-specific, source-specific, or still unknown?
- Is a customer or employee relying on the output for a high-impact decision?
- Does a provider, connector, tool, or identity boundary need to be restricted?
First-response checklist
Use a safe acknowledgment while the team investigates. Do not confirm private details or promise a legal or regulatory outcome.
- Confirm receipt and provide a case reference.
- State that the team is reviewing the route and available evidence.
- Ask only for the minimum additional information needed, preferably through an approved channel.
- Tell the reporter not to send passwords, private keys, payment details, or unrelated customer records.
- Explain any immediate safe action, such as using a human support path or pausing an affected feature.
- Assign an owner and next update time.
- Route privacy, authorization, safety, and action concerns to the correct reviewer immediately.
- Preserve a redacted fixture before changing production behavior where safe.
Suggested acknowledgment:
We received your report and assigned case ID feedback-YYYY-MM-DD-NNN. We are reviewing the chatbot route, the relevant version and source context, and the scope of the behavior. Please do not send passwords, credentials, payment details, or unrelated personal information. We will provide the next update through the approved support channel after the initial review.
Adapt the wording to the team’s support, privacy, incident, and legal process. This is a starting template, not a required notice.
Investigation and correction workflow
| Step | Action | Evidence |
|---|---|---|
| 1. Contain | Restrict the affected answer, source, action, tenant, or route when risk warrants it | Safe-state decision and timestamp. |
| 2. Scope | Identify users, tenants, versions, data classes, destinations, and time window | Aggregate scope map. |
| 3. Reproduce | Use a synthetic or approved masked fixture with the same route and policy state | Fixture ID, test input class, and result. |
| 4. Verify source | Check source authority, version, freshness, retrieval, and permissions | Source and retrieval record. |
| 5. Verify output | Check validation, encoding, citations, uncertainty, and destination handling | Output class and reviewer decision. |
| 6. Verify action | Check actor, target, authorization, confirmation, and side effect | Action evidence and rollback result. |
| 7. Correct | Fix source, prompt, rule, parser, UI, route, or human process | Change reference and owner. |
| 8. Retest | Re-run the original case and nearby negative cases | Pass, fail, or unavailable verification. |
| 9. Communicate | Send a reviewed correction or scope update when appropriate | Approved message and recipient scope. |
| 10. Decide | Close, monitor, restrict, rollback, or pause | Decision record and next trigger. |
Correction rules:
- A correction does not silently delete evidence needed for the review.
- A source fix is accompanied by a retrieval and citation test.
- An output fix is accompanied by context-appropriate validation and encoding tests.
- An action fix is accompanied by authorization, confirmation, and replay tests.
- A privacy or authorization fix is accompanied by cross-scope tests.
- A customer-facing correction has a reviewer and avoids unsupported certainty.
Human review and customer communication
Route cases to a person when the output may affect safety, rights, money, access, privacy, employment, health, legal matters, or a material customer decision.
| Case type | Human review minimum | Customer-facing action |
|---|---|---|
| Ordinary factual correction | Source check and response review | Correct or explain the limitation. |
| Sensitive or private disclosure | Security and privacy scope review | Limit access and use approved notification guidance. |
| High-impact recommendation | Domain reviewer and uncertainty review | Provide a human path and avoid automated final decision. |
| Unexpected tool action | Engineering and service-owner review | Contain, reverse where safe, and explain approved next steps. |
| Repeated misinformation | Quality owner and regression review | Correct the source or route and communicate material impact. |
| Active incident | Incident owner and required specialist routing | Use the approved internal and external communication plan. |
Communication checklist:
- The message states what is known and what is still being reviewed.
- The message does not expose another user, tenant, support case, or internal system.
- The message does not overstate model accuracy or promise a result the team cannot verify.
- The message tells the recipient how to reach a human reviewer when needed.
- A reviewer approved the scope, audience, and wording.
- The final message is linked to the case evidence and decision.
Negative test set
Run these tests with synthetic complaints and fixtures. Verify the route, evidence, scope, and response rather than checking only the final text.
| Test ID | Scenario | Expected result |
|---|---|---|
| FEED-01 | User reports a wrong answer with an approved source reference | Case is created, source is checked, and a correction path is assigned. |
| FEED-02 | Complaint includes a synthetic cross-tenant disclosure | Route is contained and no other tenant content is copied into the case. |
| FEED-03 | Output gives confident advice outside the approved domain | Human review or safe limitation is used before a final customer response. |
| FEED-04 | Output reaches an HTML, SQL, file, or tool destination | Context-specific validation and encoding block unsafe handling. |
| FEED-05 | User asks for a correction and includes unrelated sensitive data | Intake requests minimization and restricts the evidence. |
| FEED-06 | Same complaint repeats across several routes | Trend grouping identifies scope and assigns a systemic owner. |
| FEED-07 | A complaint arrives while the provider or source is degraded | Triage separates service failure from model quality and uses the safe state. |
| FEED-08 | Reporter claims an action occurred but audit evidence is absent | Case stays open; the team does not assert completion or denial without verification. |
| FEED-09 | The output is corrected but the old route remains active | Release gate blocks closure until the affected route is updated or restricted. |
| FEED-10 | Complaint asks the chatbot to reveal another case | Authorization boundary rejects the request and keeps cases separate. |
| FEED-11 | A support agent requests broad access for convenience | Case-scoped access or reviewer approval is required. |
| FEED-12 | A high-severity case reaches its response deadline | Escalation fires and the owner records the next safe action. |
Do not mark a test passed because the complaint was answered politely. Verify the source, policy, output, action, tenant, and evidence boundaries.
Monitoring and trend review
| Signal | Useful dimensions | Review question |
|---|---|---|
| Complaint count | Route, intent, tenant, language, version, and period | Is a pattern emerging outside ordinary noise? |
| Correction rate | Category, source, model, and route | Which behavior needs a durable fix? |
| Privacy or authorization cases | Scope, resource, role, tenant, and version | Is the boundary failing or drifting? |
| Human handoff rate | Intent, severity, route, and outcome | Is automation being used outside its safe scope? |
| Action reversal or block rate | Tool, target, actor, and policy | Are high-impact actions over-permissioned or unclear? |
| Time to triage | Severity, owner, and queue | Are high-risk reports reaching the right person quickly? |
| Repeat report rate | Category, release, source, and customer segment | Did the fix actually reduce recurrence? |
| Evidence completeness | Case, route, version, fixture, and decision | Can the team reproduce and defend the disposition? |
| Customer communication | Severity, audience, reviewer, and outcome | Is the explanation accurate and appropriately scoped? |
Review trends at a cadence that matches the route risk. A small number of high-impact reports can matter more than a large number of minor wording suggestions.
Release gate
Approve a correction or restore a route only when every applicable answer is yes.
| Gate | Pass condition |
|---|---|
| Intake | The case has a category, severity, scope, owner, and next decision time. |
| Privacy | Evidence is minimized, restricted, and retained only for the review purpose. |
| Reproduction | The issue was reproduced or the inability to reproduce is documented. |
| Source and output | Source, retrieval, output validation, uncertainty, and destination handling were reviewed. |
| Authorization | Subject, tenant, role, and action boundaries were checked where relevant. |
| Containment | A safe state protected users while the finding was open. |
| Regression | The original and nearby negative tests pass after a fix. |
| Communication | Customer or internal messages have an owner and reviewer. |
| Decision | The residual risk, owner, and next trigger are recorded. |
If a gate fails, keep the route limited, human-reviewed, draft-only, read-only, or paused.
Staged improvement plan
| Stage | Scope | Exit evidence |
|---|---|---|
| 1. Intake baseline | Create categories, severity rules, owners, and safe acknowledgment | Synthetic cases route correctly. |
| 2. Evidence baseline | Add redacted fixture and version records | Reviewers can reproduce without raw private conversations. |
| 3. Containment | Connect high-risk categories to pause, handoff, or restriction | Safe state is observable and reversible. |
| 4. Correction | Add source, output, authorization, and action regression cases | Original and neighboring cases pass. |
| 5. Trend review | Group cases by route, version, source, tenant, and intent | Repeated issues have owners and durable actions. |
Pause or restrict the affected route for a confirmed cross-tenant disclosure, unsafe high-impact action, repeated unverified misinformation in a sensitive workflow, missing evidence for a material complaint, or an active abuse pattern.
Findings and remediation
| Finding | Severity | Owner | Fix or restriction | Evidence and due date |
|---|---|---|---|---|
Do not close a finding because the reporter stopped responding. Record the evidence, scope, decision, and remaining uncertainty.
Decision record
| Field | Entry |
|---|---|
| Case or trend ID | |
| Decision | Explain, correct, notify, monitor, restrict, rollback, or pause. |
| Route and versions | |
| Affected scope | |
| Data and action scope | |
| Evidence reviewed | |
| Tests run | |
| Customer communication | |
| Owner and reviewer | |
| Decision date | |
| Next trigger |
Metrics to track
- Complaint and feedback volume by route, intent, severity, tenant class, and version.
- Time to acknowledge, triage, contain, correct, review, and close.
- Correction, handoff, action block, rollback, and pause rates.
- Repeat reports after a fix and the age of unresolved high-risk cases.
- Evidence completeness and negative-test pass rate.
- Privacy, authorization, safety, cost, availability, and accessibility categories.
- Customer communication review outcomes and reopened cases.
Metrics are signals for review, not proof that output is reliable or a complaint is resolved. Keep the underlying evidence and uncertainty visible.
Evidence checked
- NIST AI RMF Core
- NIST AI RMF Playbook
- OWASP LLM05:2025 Improper Output Handling
- OWASP LLM09:2025 Misinformation
- CISA secure AI system development guidance
- AI Tool Risk Checker
- Small Team AI Security Checklist
FAQ
Should every thumbs-down report become an incident?
No. Triage the category, severity, scope, and plausible impact. A privacy, authorization, unsafe-action, or high-impact case may need incident handling even if it was reported only once.
Can the team ask the customer to resend the full conversation?
Usually not through a broad support channel. Ask only for the minimum information needed, use an approved restricted path, and offer a redacted or synthetic alternative where possible.
What if the team cannot reproduce the complaint?
Keep the case open or mark it as unavailable verification, record the versions and scope that were checked, and continue monitoring. Do not claim that the behavior did not occur.
When should the route be paused?
Pause or restrict it when there is a confirmed cross-scope disclosure, unsafe high-impact action, active abuse, repeated material misinformation in a sensitive workflow, missing evidence for a material case, or no reliable safe state.
Is a correction message enough to close a privacy complaint?
Not by itself. The team should review affected scope, access, copies, retention, notification, and the relevant regression tests. Apply the organization’s privacy and incident process.
Is this a legal complaint-handling policy?
No. It is an operational triage checklist. Legal, privacy, support, security, and product owners should decide obligations and customer commitments for the service.
Recommended next step
Run the AI Tool Risk Checker for one customer-facing route. Create synthetic cases for a wrong answer, a private-data boundary, an unsafe action, a source conflict, and a repeated complaint. Route them through the checklist, record the safe state, and add the resulting cases to the Small Team AI Security Checklist.