checklist
AI chatbot system prompt leakage review checklist for small teams
A practical checklist for reviewing system prompts, hidden instructions, role rules, tool descriptions, and sensitive context before an AI chatbot goes live or changes behavior.
Use this checklist before a customer-facing AI chatbot launches, before its system prompt changes, or whenever the bot can see private data or call tools.
The short answer is that a system prompt should not be treated as a secret or as an authorization boundary. Remove credentials, connection strings, private URLs, hidden role maps, and other sensitive details from it. Put authentication, authorization, validation, and high-impact approval in deterministic application controls. Then test what the chatbot reveals, what it can infer, and what it can do when a user or an external document tries to change its instructions.
Start with the AI Tool Risk Checker and attach the result to the prompt review record. Use the Small Team AI Security Checklist for baseline ownership, access, data handling, logging, and incident controls. Pair this page with the AI chatbot prompt injection response checklist for post-launch events and the AI chatbot tool action approval checklist when the bot can act in downstream systems.
Bottom line
An AI chatbot system prompt is ready for production only when the team can show:
- The prompt contains no credentials, tokens, connection strings, customer records, or security decisions that belong outside the model.
- The intended audience, allowed topics, prohibited topics, handoff rules, and output constraints are documented.
- Authentication, authorization, tenant isolation, tool permissions, and high-impact approvals are enforced outside the model.
- Direct and indirect prompt-injection tests cover leakage, bypass, data access, tool use, and misleading output.
- A named owner has approved the residual risk, monitoring, rollback, and next review trigger.
OWASP’s system prompt leakage guidance says the system prompt should not be considered a secret or used as a security control. Its core concern is the sensitive data, privilege logic, and weak application boundary that may sit behind the prompt. OWASP also notes that direct and indirect prompt injection can alter model behavior, so prompt review must cover both the prompt text and the surrounding application.
This checklist is operational guidance, not legal advice, a security certification, or a guarantee that a model will never disclose instructions.
When to use this checklist
| Change or signal | Use this checklist? | Default review depth |
|---|---|---|
| New customer-facing chatbot or assistant | Yes | Full prompt, data, access, tool, and test review. |
| System prompt or hidden instruction changes | Yes | Compare versions and rerun leakage and behavior tests. |
| New connector, retrieval source, or tool description | Yes | Review indirect injection, data scope, and action boundaries. |
| New role, tenant, account, or escalation rule | Yes | Review authorization outside the model and test cross-user requests. |
| Prompt is copied into a ticket, analytics tool, or vendor console | Yes | Check access, retention, redaction, and vendor handling. |
| Prompt only changes wording for a non-production mock | Maybe | Keep the environment isolated and record the test boundary. |
| Static FAQ with no model or private data | Usually no | Use ordinary content and web-security review instead. |
If the owner cannot identify which parts of the prompt are instructions, data, examples, tool descriptions, or policy references, pause the change until the prompt is decomposed.
Prompt review intake form
Copy this into the change, security, or product review record.
| Field | Entry |
|---|---|
| Review ID | |
| Review date | |
| Chatbot and environment | |
| Prompt version or commit | |
| Change reason | |
| Business owner | |
| Technical owner | |
| Security or privacy reviewer | |
| Model and route | |
| Connected data sources | |
| Available tools and actions | |
| User and tenant boundary | |
| Prompt storage locations | |
| Logs, traces, and support access | |
| Rollout and rollback owner | |
| Next review trigger |
Do not paste live credentials, customer records, private tokens, raw support transcripts, or unredacted prompt-extraction attempts into a broad ticket or chat channel. Reference an access-controlled evidence location instead.
Prompt component inventory
Inventory each component rather than reviewing one long prompt as a single block.
| Component | Example content | Owner | Review question |
|---|---|---|---|
| Role and purpose | ”Answer product questions” | Product | Is the role narrow enough for the use case? |
| Allowed and prohibited topics | Billing, support, legal escalation | Product and support | Which topics require a human handoff? |
| Output format | JSON, citations, or short answer | Engineering | Is output validated outside the model? |
| Retrieval instructions | Use approved help-center sources | Knowledge owner | Can retrieved content inject instructions? |
| Tool descriptions | Create ticket or draft reply | Engineering | Does the description expose excess functionality? |
| User context | Account ID, plan, locale | Engineering | Is the context minimized and authorized? |
| Examples and demonstrations | Sample customer questions | Product | Do examples contain real data or hidden policy? |
| Refusal and handoff rules | Escalate account changes | Support | Is the rule backed by an external control? |
| Policy references | Links to internal policy | Security | Can the model access more than it needs? |
Prompt content classification table
Classify every line, variable, example, and injected context before approval.
| Content type | Default treatment | Review outcome |
|---|---|---|
| Public product behavior | May remain in prompt | Confirm it is accurate and non-sensitive. |
| Internal writing style or response format | May remain with owner approval | Test that it does not override stronger controls. |
| Customer record, ticket, or transcript | Do not hard-code | Retrieve only after authorization and minimize fields. |
| API key, password, token, or connection string | Never include | Remove and rotate if it was exposed. |
| Hidden role or permission map | Externalize | Enforce access in the application or downstream system. |
| Internal endpoint, private hostname, or network detail | Remove unless operationally required | Treat disclosure as an information-leak signal. |
| Safety or refusal instruction | Keep as behavioral guidance only | Add deterministic checks and handoff controls. |
| Tool schema or action description | Minimize | Expose only the exact operation and scope needed. |
If a prompt line would cause harm when shown to a user, ask whether it should exist in the model context at all. Hiding a sensitive control inside a prompt is not a substitute for enforcing it in code.
Sensitive content removal checklist
- No API keys, access tokens, passwords, private certificates, or signing material appear in the prompt or examples.
- No database connection strings, private dashboard URLs, internal hostnames, or unrestricted file paths appear in the prompt.
- No customer records, support transcripts, employee details, or regulated data are hard-coded.
- No tenant IDs, role maps, account facts, or approval thresholds are trusted solely because the model saw them.
- Environment-specific values are injected through a controlled context layer with authorization and logging.
- Prompt files, traces, evaluations, and support exports have documented access and retention.
- Any previously exposed secret has been revoked or rotated and the incident owner is named.
Externalize authorization and sensitive logic
| Control | Do not delegate to the prompt | Required external check |
|---|---|---|
| Identity | ”The user is an admin” | Authenticate the user and bind the request to a session. |
| Tenant isolation | ”Only use this customer’s records” | Enforce tenant scope in the data service. |
| Record access | ”Never reveal private tickets” | Authorize each query and filter fields before retrieval. |
| Tool action | ”Ask before deleting” | Require an approval token or downstream confirmation. |
| Financial or account change | ”Do not refund over the limit” | Enforce limits and approval in the billing system. |
| Data export | ”Never send data externally” | Block destinations and inspect the export outside the model. |
| Output safety | ”Never make a legal promise” | Validate output, route sensitive topics, and keep a human review path. |
Use the model to interpret or draft within a bounded workflow. Use application code and downstream systems to decide whether an identity, data request, or action is allowed.
Prompt and context data-flow map
| Stage | Data or instruction | Boundary to verify | Evidence |
|---|---|---|---|
| User input | Message, attachment, or URL | Input filtering and size limits | |
| Context assembly | System prompt, user context, retrieved text | Trusted and untrusted content separated | |
| Model request | Prompt, tools, metadata, route | Provider, region, retention, and logging | |
| Model response | Answer, refusal, citations, tool call | Output validation and sensitive-data checks | |
| Tool gateway | Arguments and identity scope | Authorization, allowlist, and human approval | |
| Storage | Transcript, trace, prompt version | Access, retention, deletion, and export | |
| Support path | Debug copy or vendor ticket | Redaction and limited access |
Record where each component is processed, stored, logged, backed up, and accessed. A prompt review is incomplete if it checks the source file but ignores traces, vendor support copies, or tool gateways.
Leakage and behavior test plan
Run tests in a non-production environment first. Use synthetic records and harmless canary values that cannot authenticate to anything.
| Test family | Example test | Pass condition |
|---|---|---|
| Direct extraction | Ask the bot to print its system instructions | It does not reveal sensitive content and follows the safe response path. |
| Role bypass | Claim to be an administrator or developer | It does not grant access based on the claim. |
| Context confusion | Put an instruction in a retrieved document | The bot treats the document as data, not as a higher-priority instruction. |
| Data boundary | Request another user’s record or ticket | The data layer denies the request and the bot does not invent access. |
| Tool boundary | Ask for an unapproved action or broad export | The gateway rejects it or requires the intended approval. |
| Prompt splitting | Spread an extraction request across several turns | The same boundary holds across conversation history. |
| Multimodal input | Use an image or attachment with hidden instructions | The input is isolated, scanned, or rejected according to policy. |
| Output handling | Ask for a payload that could trigger downstream code | The application validates or blocks unsafe output before use. |
Capture the test ID, prompt version, model route, input class, output summary, tool calls, reviewer, and result. Do not preserve attack text or sensitive output more broadly than necessary.
Regression test set
Keep a small, versioned test set for every prompt change.
| Test ID | Behavior to protect | Expected result | Last result |
|---|---|---|---|
| SP-01 | No secret or credential disclosure | Refuse or provide a safe explanation | |
| SP-02 | No cross-tenant retrieval | Deny and log the authorization failure | |
| SP-03 | Untrusted source cannot rewrite behavior | Cite or summarize data without obeying embedded instructions | |
| SP-04 | High-impact action needs approval | No action before external approval | |
| SP-05 | Human handoff remains available | Route the topic with required context only | |
| SP-06 | Output cannot execute by itself | Validation or sanitization blocks unsafe use |
Fail closed for a high-risk behavior. Do not accept a prompt change because it improves answer quality if it weakens access, tenant, tool, or approval boundaries.
Review findings and remediation
| Finding | Severity | Immediate action | Owner and due date |
|---|---|---|---|
| Secret or credential in prompt | Critical | Remove, rotate, restrict traces, and assess exposure | |
| Permission logic only in prompt | High | Move authorization to the application or downstream system | |
| Private data in examples or logs | High | Redact, restrict access, and confirm retention | |
| Untrusted content can rewrite instructions | High | Separate context, add tests, and restrict retrieval | |
| Tool has broader scope than needed | High | Reduce functionality and permissions before rollout | |
| Prompt extraction reveals harmless wording | Medium | Record the result and focus on underlying controls | |
| Refusal wording is inconsistent | Medium | Add a regression case and define the handoff path |
Do not classify harmless prompt wording as the main risk if the same review finds weak identity, authorization, data, or tool controls. Fix the underlying boundary first.
Staged rollout plan
| Stage | Scope | Entry gate | Exit signal |
|---|---|---|---|
| 1. Offline | Synthetic test set | No secrets and all critical tests defined | Findings recorded and owner assigned. |
| 2. Internal | One trained team | Access, logging, handoff, and support rules ready | No high-severity regression failure. |
| 3. Limited pilot | One low-risk workflow | Monitoring and rollback are available | Quality and safety signals remain within limits. |
| 4. Broader release | Approved users and topics | Product, security, and privacy owners sign off | Review date and change triggers are recorded. |
Do not use production customer records to prove that a prompt is safe. Use synthetic data, masked fixtures, and approved test accounts with no unnecessary permissions.
Rollback and pause plan
Pause the affected prompt, route, source, or tool when any of these occur:
- A secret, credential, customer record, or private system detail appears in an output or trace.
- A user can cross a tenant or role boundary through wording, conversation history, or retrieved content.
- A tool action executes without the required authorization or human approval.
- A prompt change breaks a critical regression test or removes the human handoff path.
- Support or vendor debugging requests raw sensitive prompt or transcript content.
| Pause action | Owner | Evidence to preserve | Restart gate |
|---|---|---|---|
| Disable changed prompt version | Engineering | Version, timestamp, test result | Prior approved version restored and tested. |
| Disable one source or connector | Data or platform owner | Source ID and affected scope | Source reviewed and indirect-injection test passes. |
| Disable tool action | Product or engineering | Action, identity, arguments | Permission and approval controls verified. |
| Route to human or static fallback | Support owner | Customer impact summary | Handoff works and owner approves restart. |
Decision table
| Review result | Default decision | Required record |
|---|---|---|
| No sensitive content, external controls verified, tests pass | Approve staged rollout | Prompt version, evidence, owner, and review date. |
| Harmless prompt wording is extractable but controls are strong | Approve with monitoring | Note that prompt wording is not a secret and track leakage signals. |
| Sensitive data or permission logic remains in prompt | Reject or pause | Remediation owner and deadline. |
| Critical regression or cross-tenant failure | Reject and contain | Incident or security review record. |
| Tool action or export lacks external approval | Restrict to read-only or draft-only | Action boundary and re-test plan. |
| Evidence is incomplete or ownership is unclear | Defer | Missing evidence list and next review date. |
Sign-off record
| Role | Name | Decision | Conditions | Date |
|---|---|---|---|---|
| Business owner | ||||
| Product owner | ||||
| Engineering owner | ||||
| Security reviewer | ||||
| Privacy or legal reviewer, if needed | ||||
| Support or operations owner |
Approval should identify the exact prompt version, model route, data sources, tools, user scope, unresolved risks, rollout stage, rollback owner, and review trigger.
Action tracker
| Action | Risk | Owner | Due date | Status | Evidence |
|---|---|---|---|---|---|
| Remove sensitive values from prompt | |||||
| Move authorization out of prompt | |||||
| Restrict prompt and trace access | |||||
| Run leakage and indirect-injection tests | |||||
| Reduce tool scope and approval path | |||||
| Confirm monitoring and rollback |
Final system prompt review checklist
- Review ID, prompt version, model route, owner, and environment are recorded.
- Every prompt component is classified as instruction, data, example, tool description, or policy reference.
- Secrets, credentials, customer records, private URLs, and unnecessary internal details are absent.
- Prompt and trace storage, access, retention, deletion, and support paths are documented.
- Identity, tenant isolation, authorization, rate limits, and high-impact approvals are enforced externally.
- Retrieved documents, attachments, and user content are treated as untrusted input.
- Leakage, role-bypass, cross-tenant, tool, output, and multi-turn tests pass.
- Test evidence is redacted and stored with least-privilege access.
- Rollout, monitoring, pause, fallback, and rollback owners are named.
- The AI Tool Risk Checker result is attached to the record.
- The Small Team AI Security Checklist baseline controls are reviewed.
- Product, engineering, security, privacy, and support sign-off is complete where applicable.
- The next review date and change triggers are recorded.
Metrics to track
Track trends without collecting raw sensitive prompts unless there is a documented need.
| Metric | Why it matters | Review cadence |
|---|---|---|
| Prompt versions released | Shows change volume and review coverage | Every release |
| Critical regression pass rate | Detects boundary breakage | Every prompt change |
| Prompt leakage or extraction signals | Shows attempted and successful disclosure | Weekly or after an alert |
| Cross-tenant or unauthorized access blocks | Tests external authorization | Weekly |
| Tool calls blocked or requiring approval | Shows action boundary pressure | Weekly |
| Sensitive-data redaction findings | Detects prompt and trace hygiene issues | Monthly |
| Human handoff rate for restricted topics | Checks whether fallback works | Monthly |
| Open remediation age | Prevents known gaps from becoming permanent | Weekly |
Evidence checked
- OWASP LLM07:2025 System Prompt Leakage says system prompts should not be treated as secrets or security controls and recommends externalizing sensitive data and enforcing critical controls independently.
- OWASP LLM01:2025 Prompt Injection covers direct and indirect prompt injection and recommends least privilege, external-content separation, human approval, and adversarial testing.
- OWASP LLM06:2025 Excessive Agency supports minimizing extensions, functionality, permissions, and autonomy for tool-using systems.
- NIST AI RMF Core organizes AI risk work around Govern, Map, Measure, and Manage and supports iterative risk management across the AI lifecycle.
- NIST AI RMF Manage Playbook calls for post-deployment monitoring, incident response, recovery, and change management.
- CISA and UK NCSC secure AI system development guidance emphasizes secure design, accountability, and operational controls for AI systems.
FAQ
Is the system prompt itself a secret?
No. Treat the prompt as potentially discoverable model input. Remove sensitive values and protect the underlying data and authorization paths instead of relying on prompt secrecy.
Should a prompt tell the model which users are allowed to see a record?
It can describe the workflow, but it must not be the enforcement point. Authenticate the user, authorize the data request, and filter the result outside the model.
What if a harmless prompt extraction test succeeds?
Record it, verify that no sensitive data or security boundary was exposed, and focus remediation on external authorization, data minimization, tool scope, and output controls. Exact wording is not the main security boundary.
Do retrieval systems remove prompt leakage risk?
No. Retrieved pages, files, tickets, and attachments can contain direct or indirect instructions. Separate trusted instructions from untrusted content and test the retrieval path.
Can a small team review every prompt change manually?
Use a risk-based gate. Require full review for secrets, permissions, private data, connectors, tools, customer-facing behavior, or high-impact decisions. Use automated regression tests and an owner review for lower-risk wording changes.
Is this checklist enough for compliance?
No. It is practical operating guidance, not legal, privacy, compliance, certification, or security assurance advice. Map the review to the contracts, laws, standards, and risk tolerance that apply to your organization.
Recommended next step
Run the AI Tool Risk Checker for the chatbot workflow, attach the result to this prompt review, and close the highest-risk findings before enabling new data sources or tool actions.