checklist
AI chatbot capacity planning and load testing checklist for small teams
A practical checklist for modeling chatbot demand, testing model and provider capacity, protecting tenant fairness, setting queue and cost guardrails, and recording safe go-live decisions.
Use this checklist before launching a new AI chatbot route, increasing traffic, adding a model or provider, enabling retrieval or tools, or relying on a fallback during a busy period. It turns an optimistic traffic estimate into documented capacity evidence and a bounded go-live decision.
The short answer: model demand by identity, tenant, endpoint, input and output size, concurrency, retries, queue work, tool calls, provider quota, cost, and latency; test normal, burst, degraded, and recovery scenarios with synthetic or approved masked data; protect shared capacity from one tenant or route; and only expand after the measured results fit team-owned limits and a tested fallback. Capacity is a security concern when unbounded use can exhaust service, budget, provider quota, or downstream actions.
Start with the AI Tool Risk Checker and attach its result to the capacity review. Use the Small Team AI Security Checklist for identity, data, access, logging, and incident controls. Pair this page with the AI chatbot abuse monitoring and rate-limit checklist for usage controls and the AI chatbot provider outage and model failover checklist for recovery decisions.
Bottom line
Do not approve a larger chatbot launch because a single happy-path request worked. Approve it only when the team can show:
- A workload model that names users, tenants, endpoints, request shape, concurrency, burst behavior, retries, and tool actions.
- Measured capacity for the model, provider, gateway, retrieval system, queue, and downstream services.
- Limits that protect individual identities, tenants, shared capacity, provider quota, cost, and high-impact actions.
- Clear behavior when a limit, service objective, queue, cost guard, or dependency fails.
- A test set with pass criteria, evidence, owners, and a rollback or pause trigger.
- Monitoring that can detect degradation and a recovery path that has been exercised before production.
OWASP LLM10 describes unbounded consumption as a source of denial of service, economic loss, model theft, and service degradation. Its mitigations include input validation, rate limiting and quotas, resource allocation, timeouts, throttling, logging, anomaly detection, graceful degradation, and bounded queued or total actions. Capacity planning gives those controls a workload and evidence context; it does not replace authorization or normal application security.
This checklist is operational guidance, not legal advice, a compliance certification, or a guarantee of performance or availability.
When to use this checklist
| Situation or change | Use this checklist? | Minimum review |
|---|---|---|
| New public chatbot or anonymous trial | Yes | Demand model, burst limits, queue policy, cost guard, and safe fallback. |
| New customer or tenant rollout | Yes | Tenant isolation, fairness, concurrency, data scope, and service objectives. |
| New model, provider, region, or context window | Yes | Route comparison, quota, latency, cost, output, and failure tests. |
| New retrieval source, index, or attachment path | Yes | Context size, freshness, authorization, and dependency capacity. |
| New tool, agent loop, or external action | Yes | Per-action budgets, downstream authorization, idempotency, and fail-closed behavior. |
| Traffic, seat, or plan expansion | Yes | Re-run the model with updated demand and shared-capacity assumptions. |
| Recent incident, outage, or cost anomaly | Yes | Reproduce the failure defensively and update limits and recovery gates. |
| Internal prototype with synthetic data only | Maybe | Record owner, environment, safe state, and path to production evidence. |
If the team cannot state what happens when the modeled capacity is exceeded, keep the route limited, read-only, draft-only, or paused.
Capacity planning intake form
Copy this into the launch, change, or quarterly capacity review.
| Field | Entry |
|---|---|
| Review ID | |
| Review date and timezone | |
| Chatbot and environment | |
| Route, model, provider, and region | |
| Expected user and tenant scope | |
| Workload source and confidence | |
| Peak and burst assumptions | |
| Input, output, context, and attachment assumptions | |
| Concurrency, retry, queue, and tool assumptions | |
| Service objectives and team-defined limits | |
| Provider quota and cost owner | |
| Retrieval and downstream dependencies | |
| Test environment and data class | |
| Fallback, pause, and rollback owner | |
| Reviewer and next review trigger |
Do not put real customer prompts, private transcripts, credentials, or billing records into a broad planning document. Use synthetic workloads, aggregate metrics, and approved masked fixtures.
Route and dependency inventory
Map the whole request path. A model may be available while the gateway, retrieval store, queue, tool, or tenant database is at capacity.
| Component | Capacity signal | Security or fairness question | Owner |
|---|---|---|---|
| Client and session layer | Sessions, retries, connection pressure | Can a client create unlimited sessions or retries? | |
| Authentication and tenant binding | Auth latency, identity volume | Is the quota key verified outside the model? | |
| API gateway | Requests, concurrency, status, and latency | Are route and tenant limits enforced before expensive work? | |
| Prompt and context builder | Input size, tokens, attachments | Are size, source, and data-class limits explicit? | |
| Model route | Input, output, latency, errors, and quota | Are model limits and provider terms recorded? | |
| Retrieval or vector store | Query rate, context size, freshness | Can one tenant consume or cross a source boundary? | |
| Tool gateway | Calls, targets, errors, and side effects | Are actions independently authorized and bounded? | |
| Queue and workers | Depth, age, concurrency, and shed work | Is the queue bounded and priority-aware? | |
| Cache | Hit rate, age, size, and invalidation | Can cached data cross tenant or freshness boundaries? | |
| Logging and monitoring | Event rate, delay, and collector health | Can the team detect a failure without raw private prompts? | |
| Provider account | Quota, spend, limits, and status | Who can pause or change the external route? | |
| Human support path | Handoff queue, age, and staffing | Is the fallback usable when automation is limited? |
Workload model
Record assumptions instead of pretending that an average request predicts a peak. Keep the units and source of each assumption.
| Dimension | Question | Team-defined value or source |
|---|---|---|
| Active identities | How many users, service accounts, and anonymous sessions are expected? | |
| Tenant mix | How many tenants and what workload distribution is expected? | |
| Request rate | What is the normal rate per route and identity? | |
| Peak rate | What sustained peak is plausible during the review window? | |
| Burst shape | How short and concentrated can a legitimate burst be? | |
| Input size | What character, token, attachment, and context sizes occur? | |
| Output size | What response sizes and streaming durations occur? | |
| Concurrency | How many requests can be in flight per scope? | |
| Retry behavior | Which clients and layers retry, and how? | |
| Tool calls | How many calls per turn, session, task, and tenant? | |
| Queue work | What work can be queued, for how long, and at what priority? | |
| Provider quota | What external quota, account, or region limit applies? | |
| Cost | What budget or spend velocity requires action? | |
| Availability | Which service objective and degraded states are acceptable? |
Use distributions or scenario bands when appropriate. Do not make a false claim of capacity from a small test with a narrow request shape.
Capacity risk matrix
Classify findings by impact on users, tenants, data, cost, availability, and downstream actions.
| Class | Example finding | Default decision | Evidence |
|---|---|---|---|
| Low | Normal route remains inside measured budgets with healthy monitoring | Continue the current stage and review on schedule. | Test result and trend. |
| Medium | Headroom is uncertain or a dependency approaches its team-defined limit | Keep scope limited and add a test or guardrail. | Assumption gap and owner. |
| High | One tenant, route, provider quota, queue, or retry path can affect shared capacity | Throttle, shed, or restrict the affected scope before expansion. | Load result and mitigation. |
| Critical | Capacity pressure can cause cross-tenant exposure, uncontrolled spend, unsafe tool activity, or broad outage | Fail closed for affected actions, pause, contain, and investigate. | Incident or change record. |
Raise the classification when monitoring is incomplete, a route handles private data, a tool changes durable state, or recovery has not been tested.
Service objective and budget matrix
Write the decision in terms of user-visible behavior and safety, not just infrastructure utilization.
| Dimension | Normal state | Degraded state | Stop or pause trigger |
|---|---|---|---|
| Availability | Team-defined success objective | Read-only, delayed, or human handoff path | Safe state cannot be maintained. |
| Latency | Route-specific target and percentile | Bounded wait with a clear notice | Queue or timeout trigger is exceeded. |
| Error rate | Expected route baseline | Limited retry and fallback | Errors persist or amplify. |
| Input and output | Documented size and token budgets | Smaller bounded request or concise fallback | Budget enforcement fails. |
| Concurrency | Per-identity, tenant, route, and provider caps | Queue within a bound or shed low-priority work | Shared capacity is threatened. |
| Queue | Bounded depth and age | Priority shedding and delayed response | Queue cannot recover or evidence is stale. |
| Cost | Team-approved budget and velocity | Cheaper model, reduced context, or pause | Spend guard or provider quota is at risk. |
| Tool actions | Allowlist, approval, and action budget | Read-only or draft-only | Authorization or idempotency is unavailable. |
| Monitoring | Signals and alert owners are healthy | Reduced service with explicit uncertainty | Team cannot observe or bound the route. |
Load test scenario matrix
Use a non-production route and approved synthetic workloads. The goal is to verify defenses and capacity decisions, not to stress an external service without permission.
| Scenario | Variables | Pass condition | Safe response if it fails |
|---|---|---|---|
| Normal mixed traffic | Identity, tenant, endpoint, and request shape | Service objectives and budgets hold. | Keep the current rollout stage. |
| Sustained peak | Request rate, concurrency, input, and output | Shared capacity remains fair and observable. | Reduce scope or add capacity before launch. |
| Short legitimate burst | Burst window, tenant mix, and queue | Burst is absorbed or bounded without cross-tenant impact. | Apply the defined soft or hard limit. |
| Large input and context | Input size, attachments, and retrieval | Validation caps work and cost remains bounded. | Reject, reduce, or hand off safely. |
| Long output or stream | Output budget and connection duration | Output and timeout limits hold. | Stop safely or use concise fallback. |
| Client retry behavior | Timeout, provider errors, and backoff | Retries stop at the cap and do not amplify load. | Disable retry path and investigate. |
| One noisy tenant | Tenant rate, concurrency, and queue | Other tenants retain the intended service. | Throttle or isolate the tenant. |
| Provider quota pressure | Model, account, route, and time | Alert and fallback or pause operate as designed. | Restrict expensive traffic. |
| Retrieval dependency pressure | Query rate, source, context, and freshness | Source limits and no-source behavior hold. | Use bounded cache or handoff. |
| Tool-call loop | Action count, target, approval, and retries | Budget and downstream authorization prevent side effects. | Fail closed and disable the action. |
| Monitoring degradation | Collector delay or missing signal | Restricted safe state activates. | Pause or reduce service. |
| Recovery and ramp-up | Staged traffic after failure | Restore gates pass before wider traffic. | Return to the restricted state. |
Do not run destructive or high-volume tests against a production provider, customer tenant, or third-party service without explicit authorization and a documented stop condition.
Tenant fairness and queue controls
- Authenticated requests have verified identity and tenant keys outside model output.
- Anonymous, account, API key, tenant, endpoint, tool, and provider scopes have separate decisions where needed.
- IP or network data is supporting context, not the only fairness or abuse key.
- One tenant cannot consume all shared workers, queue slots, context capacity, or provider quota.
- Queue depth, age, priority, and shed-work behavior are bounded and observable.
- Low-priority work can be deferred without changing the meaning of high-priority work.
- Queue and retry decisions are idempotent for durable actions.
- Limit messages do not disclose another tenant’s existence or detection logic.
- A cache cannot serve one tenant’s private content to another tenant.
Cost and provider guardrails
| Guardrail | Decision to record | Owner or evidence |
|---|---|---|
| Model selection | Which model is allowed for each route and data class? | |
| Context budget | What context and attachment sizes are acceptable? | |
| Token or request budget | Which team-defined velocity requires action? | |
| Provider quota | What account, model, region, or contract limit applies? | |
| Spend velocity | What trend triggers a cheaper route, throttle, or pause? | |
| Retry cost | Which layer can retry and for how long? | |
| Tool cost | Which tools or external calls have separate budgets? | |
| Fallback cost | Is the fallback cheaper, equal, or more expensive under load? | |
| Alert route | Who receives cost and quota alerts and can act? | |
| Recovery check | What verifies that restrictions can be lifted? |
Do not use a larger provider quota as the only capacity plan. It can hide a retry storm, a noisy tenant, a queue defect, an unsafe tool loop, or a data-boundary problem.
Failure and degradation checks
- A provider error does not create unlimited retries across client, gateway, worker, and tool layers.
- A timeout cancels or bounds work and does not duplicate a state-changing action.
- A full queue returns a safe delayed or refused response rather than growing without limit.
- A missing retrieval source produces a freshness-bounded fallback or no-answer path.
- A tool action fails closed when authorization, approval, target validation, or idempotency is unavailable.
- A high-cost model can be restricted without changing tenant or data authorization.
- Monitoring failure moves the route into a documented restricted state.
- Customer and support notices use approved language and do not disclose private incident details.
- Recovery ramps traffic gradually and retains a pause or rollback path.
Test evidence record
| Field | Entry |
|---|---|
| Test run ID | |
| Route, model, provider, and version | |
| Environment and test authorization | |
| Workload source and data class | |
| Identity and tenant scope | |
| Scenario IDs | |
| Limits and threshold sources | |
| Peak result and service objective result | |
| Cost, quota, retry, and queue result | |
| Tool and authorization result | |
| Alerts and safe-state result | |
| Reviewer and timestamp | |
| Evidence location | |
| Follow-up owner and due date |
Record aggregate measurements and redacted summaries by default. Preserve raw prompts or transcripts only when the access-controlled incident or test need is documented.
Staged rollout plan
| Stage | Scope | Entry gate | Exit signal |
|---|---|---|---|
| 1. Model | One route and dependency map | Workload assumptions, budgets, owners, and safe states are documented. | Reviewers agree the model is complete enough to test. |
| 2. Offline test | Synthetic traffic and masked fixtures | Normal, peak, burst, retry, queue, cost, tool, and recovery cases exist. | Critical scenarios pass. |
| 3. Internal | One trained team or internal tenant | Monitoring, handoff, and incident owners are ready. | No unresolved high-risk capacity finding. |
| 4. Limited pilot | One low-risk customer segment or tenant | Tenant fairness, provider guardrails, and rollback are observable. | Results stay within team-defined limits. |
| 5. Broader release | Approved production scope | On-call and service owners sign off on residual risk. | Review trigger and capacity refresh date are recorded. |
Increase traffic, context, concurrency, automation, and tool scope separately when possible. A successful text-only test does not prove that a tool-enabled route has capacity or safe action boundaries.
Capacity decision table
| Finding | Decision | Required evidence |
|---|---|---|
| Normal and peak scenarios fit budgets with monitoring healthy | Approve the next rollout stage | Test report, owners, and review date. |
| Headroom or workload assumptions are uncertain | Keep limited and gather evidence | Open assumption, test plan, and due date. |
| One tenant or route threatens shared capacity | Throttle or isolate before expansion | Tenant metrics, limit action, and fairness result. |
| Cost or provider quota is at risk | Use an approved cheaper route, reduce scope, or pause | Budget signal and owner decision. |
| Retry, queue, or tool controls fail | Reject rollout and fix the control | Failure evidence and retest plan. |
| Monitoring is unavailable | Restrict or pause the affected route | Safe-state evidence and recovery test. |
| Recovery ramp passes critical tests | Restore in stages | Canary result and rollback trigger. |
Sign-off record
| Role | Name | Decision | Conditions | Date |
|---|---|---|---|---|
| Business owner | ||||
| Product owner | ||||
| Engineering or platform owner | ||||
| Security reviewer | ||||
| Privacy or legal reviewer, if needed | ||||
| Support or incident owner |
Approval should identify the workload model, route versions, limits, tenant scope, provider and cost controls, tool state, test evidence, fallback, rollback trigger, residual risks, and next review date.
Action tracker
| Action | Risk | Owner | Due date | Status | Evidence |
|---|---|---|---|---|---|
| Map route and dependency capacity signals | |||||
| Record identity, tenant, endpoint, provider, and tool scopes | |||||
| Define workload assumptions and service objectives | |||||
| Set input, output, concurrency, queue, retry, and cost limits | |||||
| Implement tenant fairness and noisy-neighbor controls | |||||
| Run synthetic peak, burst, degradation, and recovery tests | |||||
| Verify monitoring, fallback, pause, and restoration owners |
Final capacity and load test checklist
- The workload model names identities, tenants, routes, request shapes, bursts, retries, queues, tools, quotas, cost, and service objectives.
- Every component from client to provider and downstream action has a capacity signal and owner.
- Inputs, outputs, context, attachments, concurrency, timeouts, retries, queue depth, and tool calls are bounded.
- Limits protect identity, tenant, endpoint, provider, tool, and shared capacity scopes.
- IP or network data is not the only abuse or fairness key.
- One tenant cannot starve unrelated tenants or consume all shared work.
- Cost and provider quota alerts have owners who can throttle or pause the route.
- Normal, peak, burst, large-input, long-output, retry, noisy-tenant, provider, retrieval, tool, monitoring, and recovery tests are recorded.
- Tests use synthetic traffic or approved masked data with explicit authorization and stop conditions.
- High-impact tool actions fail closed when authorization, approval, target validation, or idempotency is unavailable.
- Queue, retry, cache, and fallback behavior is observable and does not broaden tenant or data scope.
- Monitoring failure has a documented restricted state.
- Rollout and recovery are staged, reversible, and backed by critical test results.
- The AI Tool Risk Checker result is attached to the review.
- The Small Team AI Security Checklist baseline controls are reviewed.
- Residual risk, rollback trigger, owners, and next review date are signed off.
Metrics to track
Track aggregate trends and redacted summaries rather than collecting raw private prompts by default.
| Metric | Why it matters | Review cadence |
|---|---|---|
| Requests and concurrency by identity and tenant | Shows demand, fairness, and noisy-neighbor pressure. | Daily and weekly |
| Latency and timeout percentiles | Finds degradation before broad failure. | Near real time |
| Error and retry rates | Detects dependency failure and amplification. | Every incident and weekly |
| Input, output, and context size | Connects request shape to capacity and cost. | Weekly |
| Queue depth, age, and shed work | Shows whether work is bounded and recoverable. | Near real time and weekly |
| Provider quota and cost velocity | Prevents external capacity or spend surprises. | Daily and monthly |
| Tool calls and blocked actions | Tests action budgets and authorization. | Weekly and after alerts |
| Fallback and handoff volume | Measures degraded-service usability. | Weekly |
| Load-test pass rate and headroom | Shows whether assumptions remain valid. | Each release and quarterly |
| Time to detect, restrict, and restore | Measures operational readiness. | Every incident and monthly |
Evidence checked
- OWASP LLM10:2025 Unbounded Consumption identifies denial of service, economic loss, model theft, and service degradation risks and supports input validation, rate limits, quotas, resource allocation, timeouts, throttling, monitoring, graceful degradation, and bounded actions.
- OWASP LLM06:2025 Excessive Agency supports minimizing permissions, independent downstream authorization, human approval, logging, and rate limiting for tool actions.
- NIST AI RMF Core describes the Govern, Map, Measure, and Manage functions and Manage outcomes for post-deployment monitoring, incident response, recovery, and change management.
- NIST AI RMF Manage Playbook supports documenting risk treatment, monitoring, response, recovery, and continual improvement for deployed AI systems.
- CISA and UK NCSC secure AI system development guidance emphasizes secure design, accountability, and operational security through AI deployment and operation.
FAQ
Is a successful load test proof that a chatbot is secure?
No. It is evidence about defined scenarios and limits. Authorization, data boundaries, output validation, tool controls, logging, and incident response still need separate checks.
How should a small team estimate chatbot capacity?
Start with observed or explicitly sourced assumptions for users, tenants, request rate, burst, input and output size, concurrency, retries, tools, queues, provider quota, cost, and service objectives. Mark uncertainty and test the highest-impact assumptions first.
Should we test against the production model provider?
Only within the provider’s terms, your contract, and an approved test plan. Prefer a non-production route, synthetic traffic, safe volumes, clear stop conditions, and no real customer data.
Is average latency enough for a capacity decision?
No. Review percentiles, timeouts, queue wait, error rate, retry rate, input and output shape, tenant distribution, and provider or downstream dependency health. Averages can hide a harmful tail.
How do we protect one tenant from exhausting shared capacity?
Use tenant-aware quotas and concurrency, bounded queues, endpoint and provider scopes, fair scheduling or shedding, alerts, and an owner who can restrict the noisy scope. IP address alone cannot provide tenant fairness.
What should happen when the load test exceeds a limit?
Apply the documented response: slow down, queue within a bound, reject, route to a lower-cost or read-only path, hand off, or pause. Do not silently raise the limit or retry indefinitely.
Should capacity tests include tools?
Yes, when tools are enabled. Test per-action budgets, downstream authorization, target validation, approval, idempotency, retries, and failure behavior. A text-only test does not prove that external actions are bounded.
How often should capacity planning be refreshed?
After a model, provider, route, retrieval, tool, plan, tenant mix, or major workload change, and on a team-defined recurring cadence. Refresh it after incidents and cost or queue anomalies.
How much load evidence should be retained?
Keep scenario definitions, route versions, aggregate results, thresholds, alerts, decisions, reviewers, and timestamps. Restrict raw prompts, transcripts, and customer records unless a documented access-controlled need exists.
Is this checklist enough for compliance?
No. It is practical operating guidance, not legal, privacy, compliance, certification, or security assurance advice. Map it to applicable contracts, laws, standards, provider terms, and risk tolerance.
Recommended next step
Run the AI Tool Risk Checker for the chatbot route and any fallback, attach the result to the capacity review, and keep the route limited or read-only until the tenant, queue, cost, tool, monitoring, and recovery tests pass.