# AI Project-Management Tool Evaluation Protocol

**Use:** a decision aid for a bounded AI-tool pilot. It is not a product certification, security assessment, procurement template, or legal advice.

## 1. Define the proposed workflow

| Field | Record |
| --- | --- |
| Job to be done | What a user needs to achieve, in one sentence. |
| User and context | Role, team, project/programme context, and systems involved. |
| Decision or action affected | What the output may inform, recommend, or execute. |
| Capability type | Generation, extraction, classification, prediction, recommendation, execution, or a combination. |
| Consequence of error | Cost of false positive, false negative, misleading explanation, data exposure, or unauthorised action. |
| Reversibility | What can be undone, by whom, and within what time. |
| Human authority | Who reviews, approves, overrides, and stops use. |

## 2. Establish the baseline and test set

- Measure the present workflow: time, rework, quality, error rate, handoffs, and user burden as relevant.
- Select representative, sanitised cases that include ordinary work, difficult edge cases, incomplete data, and known failure conditions.
- Freeze the cases and scoring rubric before comparing tools or pilot configurations.
- Define the acceptable result for each case and have an independent reviewer score outputs where feasible.

## 3. Score five dimensions

Use a 0-3 score for each dimension. Record evidence and unresolved risks; do not rely on a total alone.

| Dimension | 0 | 1 | 2 | 3 |
| --- | --- | --- | --- | --- |
| Task fit and output quality | Fails the intended job or creates unsafe outputs | Works only in selected demonstrations | Handles representative cases with material review | Meets defined performance with predictable, reviewable failure handling |
| Human review and workflow value | Adds work or hides uncertainty | Saves effort only for easy cases | Net value in the pilot with clear reviewer actions | Sustained value with manageable review, exception, and training load |
| Data, permissions, and security | Unacceptable or unknown data flow/access | Major unresolved controls | Controls tested with defined limits | Documented, tested controls meet organisational requirements |
| Integration, audit, and operational fit | Cannot fit the existing workflow | Manual workarounds or missing record | Bounded integration and usable record/override process | Reliable integration, traceability, support, monitoring, and ownership |
| Commercial and lifecycle fit | Unacceptable terms or exit risk | Material lock-in/support/training concerns | Costs, terms, support, and exit plan understood | Viable lifecycle, change, support, and exit arrangements are contractually or operationally addressed |

## 4. Log failures and controls

For each failure, record the test case, output, severity, root cause if known, human intervention, data exposed or action attempted, remediation, owner, and retest result. Include prompt-injection or instruction-conflict probes when the tool consumes untrusted text or can take connected actions.

## 5. Decide before the pilot starts

| Decision | Minimum pre-agreed condition |
| --- | --- |
| Pilot | Bounded job, approved data boundary, representative test set, baseline, named authority, and safe manual fallback. |
| Extend or scale | Defined performance and workflow-value thresholds met; high-severity failures controlled; required privacy/security/procurement reviews complete; monitoring, support, and exit route owned. |
| Stop or redesign | Unsafe/unacceptable failure; unresolvable data or permission concern; no material net workflow value; or ownership, audit, and fallback cannot be made operational. |

## Source frame and limits

This protocol is a Strategica synthesis informed by the [NIST AI RMF Playbook](https://airc.nist.gov/airmf-resources/playbook/), [UK Government AI procurement guidance](https://www.gov.uk/government/publications/guidelines-for-ai-procurement/guidelines-for-ai-procurement), [UK Government AI impact-evaluation guidance](https://www.gov.uk/government/publications/the-magenta-book/guidance-on-the-impact-evaluation-of-ai-interventions-html), [ICO AI third-party guidance](https://ico.org.uk/for-organisations/advice-and-services/audits/data-protection-audit-framework/toolkits/artificial-intelligence/contracts-and-third-parties/), and the [OWASP prompt-injection prevention guidance](https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html). It must be adapted and reviewed for the organisation, product, data, jurisdiction, and risk context.