checklist

Quarterly AI tool review scorecard for small teams

A practical quarterly scorecard for reviewing approved AI tools, covering usage, owners, access, connectors, data handling, incidents, vendor changes, renewal risk, and follow-up decisions.

Use this scorecard once per quarter to decide which approved AI tools should continue, be restricted, be renewed, be moved to a stronger control path, or be retired.

Small teams can publish a policy and still drift in practice. Users change, connectors expand, meeting bots sync more data, browser extensions get new permissions, vendor terms shift, support teams test new workflows, and billing renews quietly. A quarterly scorecard keeps the review practical: score the tool, record evidence, choose an outcome, and assign cleanup. If the tool’s risk has changed since the last review, rerun the AI Tool Risk Checker before finalizing the decision.

Bottom line

Every approved AI tool should have a quarterly status:

1. Keep approved.
2. Keep approved with follow-up actions.
3. Restrict use.
4. Move to exception.
5. Renew or downgrade.
6. Start decommissioning.
7. Escalate to leadership, legal, privacy, or security review.

Do not score based on opinions alone. Use access exports, connector lists, setting notes, incident records, exception records, renewal dates, vendor change notes, and the latest evidence packet.

When to use this scorecard

Situation	Use this scorecard?	Why
Quarterly AI tool review	Yes	Gives every approved tool a comparable status.
Tool list has grown quickly	Yes	Finds unmanaged tools, duplicate spend, and orphaned owners.
Team added connectors, bots, extensions, or automations	Yes	Reviews source-system access beyond the AI app itself.
Incidents or exceptions increased	Yes	Turns repeated issues into decisions.
Renewal is coming soon	Yes	Feeds the renewal decision with current evidence.
Owner changed recently	Yes	Confirms the handover actually worked.
Tool is already being shut down	Usually no	Use the AI tool decommissioning checklist instead.

Use the Small Team AI Security Checklist as the baseline control set.

Scorecard dimensions

Dimension	Score 0	Score 1	Score 2
Business value	No clear usage or value.	Some value, uneven adoption.	Clear approved workflow and current value.
Ownership	No named owner or stale owner.	Owner exists but backup or duties are unclear.	Business, admin, data, source-system, and finance owners are current.
Access	User and admin list is unknown or stale.	Users reviewed, but guests, bots, or service accounts need cleanup.	Users, admins, guests, bots, and service accounts are reviewed and current.
Connectors	Connectors are unknown or unmanaged.	Connectors are known, but scope or owner needs cleanup.	OAuth, extensions, bots, APIs, webhooks, and automations are reviewed.
Data handling	Allowed data and retention are unclear.	Data rules exist, but settings or exports need follow-up.	Data classes, retention, sharing, export, and deletion rules are confirmed.
Admin controls	Settings are unknown or not documented.	Settings exist, but evidence is incomplete.	SSO, logs, sharing, retention, training, and admin settings are documented.
Incidents and exceptions	Open incidents or expired exceptions are unmanaged.	Issues exist with owners and due dates.	No unmanaged incident, exception, or near-miss follow-up remains.
Vendor and renewal	Vendor changes or renewal timing are unknown.	Known, but decision is not made.	Renewal, vendor changes, support path, and cancellation path are current.

Maximum score is 16. A high score does not remove the need for judgment; it only shows the evidence is in better shape.

Decision thresholds

Total score	Default decision	Required action
14-16	Keep approved	Record next review date.
11-13	Keep approved with follow-up	Assign owners and close gaps within 30 days.
8-10	Restrict or move to exception	Narrow users, connectors, data classes, or workflow until gaps close.
5-7	Escalate	Leadership, legal, privacy, finance, or security owner must decide.
0-4	Start decommissioning	Remove access, close connectors, and preserve evidence.

Override the score if the tool touches high-risk data, production systems, source code, candidate interviews, customer records, or regulated workflows.

Quarterly review packet

Evidence	Source
Current tool owner record	Owner and review calendar.
Current access export	Admin console, identity provider, group assignment, or source system.
Connector list	OAuth apps, browser extensions, meeting bots, APIs, webhooks, and automations.
Data and retention setting note	AI workspace, source systems, transcript storage, file storage, or vendor admin console.
Incident and exception register	Incident tracker, exception register, tabletop results, or support notes.
Vendor change record	Vendor emails, release notes, terms notices, admin setting changes, or support response.
Renewal and billing note	Finance system, subscription page, procurement record, or renewal calendar.
Previous evidence packet	Last monthly, quarterly, renewal, vendor change, or audit packet.

Use the AI tool audit evidence packet template if the evidence is scattered.

Review agenda

Time	Step	Output
0-5 minutes	Confirm tool scope and owners.	Current owner map.
5-15 minutes	Review usage and business value.	Continue, restrict, or retire signal.
15-25 minutes	Review users, admins, guests, bots, and service accounts.	Access cleanup actions.
25-35 minutes	Review connectors, extensions, meeting bots, APIs, and automations.	Connector cleanup actions.
35-45 minutes	Review data, retention, sharing, export, deletion, and training settings.	Data-rule actions.
45-55 minutes	Review incidents, exceptions, vendor changes, and renewal timing.	Risk and finance actions.
55-60 minutes	Score and decide.	Decision, owner, due date, next review.

Keep the meeting to one hour. If the tool needs more time, escalate it rather than letting one tool consume the whole review.

Scorecard template

Copy this into your review record.

Quarterly AI tool review scorecard

Tool:
Vendor:
Review date:
Reviewer:
Business owner:
Admin owner:
Data owner:
Source-system owner:
Finance owner:
Current status:
Approved workflow:
Actual workflow:
Next renewal date:
Previous review date:

Scores:
- Business value:
- Ownership:
- Access:
- Connectors:
- Data handling:
- Admin controls:
- Incidents and exceptions:
- Vendor and renewal:

Total score:
Decision:
Restrictions:
Follow-up actions:
Next review date:
Evidence links:

Store links to controlled evidence, not raw customer data, source code, transcripts, secrets, or private billing records.

Follow-up tracker

Action	Owner	Due date	Status	Evidence
Remove unused users, guests, bots, or service accounts.
Remove or narrow unmanaged connectors.
Confirm retention, training, sharing, export, or deletion settings.
Close expired exceptions.
Close incident or near-miss corrective actions.
Update owner and backup owner records.
Update renewal, billing, or cancellation decision.
Start decommissioning if the tool is no longer justified.

If a follow-up misses its due date, move the tool to restricted or exception status.

Decision outcomes

Outcome	Use when	Next step
Keep approved	Score is strong and evidence is current.	Record next quarterly review.
Approved with follow-up	Gaps are small and owned.	Close actions within 30 days.
Restricted	Tool is useful, but scope is too broad.	Narrow users, data, connectors, or workflows.
Exception	Business need exists, but control gap remains.	Add expiry date and compensating controls.
Renew or downgrade	Value and risk are acceptable, but plan should change.	Use renewal decision checklist.
Decommission	Value is low or risk is unmanaged.	Use decommissioning checklist.
Escalate	Decision requires leadership, legal, privacy, security, or finance.	Create a tracked decision record.

For renewal decisions, use the AI tool renewal decision checklist.

Metrics to track

Metric	Why it matters
Tools reviewed this quarter	Shows review coverage.
Average score by tool category	Shows whether one cluster is drifting.
Tools with missing owners	Shows accountability gaps.
Tools with connector gaps	Shows hidden source-system exposure.
Tools with data-rule gaps	Shows privacy and retention uncertainty.
Tools moved to exception	Shows accepted risk volume.
Tools restricted or decommissioned	Shows whether reviews change behavior.
Follow-up actions closed on time	Shows whether the scorecard improves operations.

If scorecards never change decisions, the review is probably too soft.

Evidence checked

• NIST: AI Risk Management Framework
• NIST: Cybersecurity Framework 2.0
• NIST: Privacy Framework
• NIST CSRC: SP 800-53 Rev. 5, Security and Privacy Controls for Information Systems and Organizations
• Cybergiz: AI Tool Risk Checker
• Cybergiz: Small Team AI Security Checklist
• Cybergiz: Monthly AI tool access review checklist
• Cybergiz: AI tool audit evidence packet template

FAQ

Is this different from the monthly access review?

Yes. The monthly review focuses on users, admins, connectors, and immediate cleanup. The quarterly scorecard combines access, value, data handling, incidents, vendor changes, renewal timing, and ownership into one decision.

Should every AI tool get the same scorecard?

Use the full scorecard for approved medium and high-risk tools. For low-risk public-content tools, use a lighter owner and access check unless the scope changed.

What if a tool scores high but had a serious incident?

Override the score. A serious incident, customer impact, source-code exposure, or unresolved exception can require restriction, escalation, or decommissioning even when other fields look good.

Who should run the scorecard?

The business owner should own the decision. The admin owner should provide settings and access evidence. Source-system owners should verify connectors. Finance should confirm renewal and cancellation timing.

What if evidence is missing?

Score the relevant dimension low and create a follow-up action. Do not fill gaps with assumptions or private data copied into the scorecard.

How does this connect to the main checklist?

The Small Team AI Security Checklist defines the baseline controls. This scorecard checks whether approved tools still meet those controls each quarter.