Agentic AI changes the nature of a technology evaluation. When an AI system only generates an answer, a person reviews the output and decides whether to use it. An agent goes further: It gathers context, selects tools, makes decisions, and executes a sequence of actions across the software development lifecycle.
The central question for a pilot is therefore not whether the agent can produce an impressive result. It is: What evidence would be sufficient to authorize this agent to perform this workflow for this team?
A feature checklist or a collection of successful demos cannot answer that. Leaders need evidence about end-to-end task completion, reliability, control enforcement, level of human effort, operational risk, and economics. This article presents a framework for producing that evidence and turning an agentic AI pilot into a defensible decision.
Start with the decision, not the technology
A pilot should begin with the decision it is meant to support. "Evaluate agentic AI" is too broad. A usable decision names the workflow, the users, the proposed level of autonomy, the output the agent must produce, the acceptance criteria and metrics that define success, the quantified outcome the team expects, and the conditions under which it will proceed.
For example: Decide whether an AI agent can diagnose selected CI/CD pipeline failures and prepare a merge request with a proposed fix that passes the pipeline’s tests and is accepted by the developer, within defined repository permissions, while reducing total resolution effort by at least 25% against the current baseline.
This is more useful than "test an AI pipeline agent" because it fixes the output, the acceptance criteria, the control boundary, and the quantified outcome before anything runs.
Define what happens on success, too: a limited rollout with one team, expansion to more teams, and/or a higher level of autonomy. Without an agreed next decision, a pilot drifts into an open-ended experiment.
Define the proposed level of autonomy
Agentic AI is not simply "autonomous" or "not autonomous." The same agent can operate at different levels of authority. A useful model for a pilot has four levels:
- Recommend: The agent analyzes the situation and recommends an action.
- Prepare: The agent creates an artifact, such as a merge request, issue, or remediation plan, but does not apply it.
- Execute with approval: The agent acts only after an authorized person approves.
- Execute within boundaries: The agent acts on its own inside a predefined scope and escalates exceptions.
Test the level the team could realistically authorize. Evaluating only in recommendation mode says little about whether the agent can safely execute changes. Granting broad autonomy in an early evaluation adds risk without evidence.
Autonomy should be earned. A team can start with recommendations, move to prepared changes, and consider bounded execution only after the agent has shown sufficient reliability and control compliance.
Evaluate the workflow, not the response
Traditional model evaluation focuses on response quality. Agentic evaluation must focus on completing an end-to-end task.
Take an agent that helps resolve a failed pipeline. A plausible explanation is not the outcome. The workflow may require the agent to:
- Identify the relevant job and logs, and separate the root cause from secondary symptoms.
- Inspect the right configuration and code, and propose a valid correction without unrelated changes.
- Create a reviewable artifact and trigger the correct validation steps.
- Escalate when confidence or authorization is insufficient.
Define success at the workflow level: Did the agent reach the intended outcome correctly, within its authority, and with an acceptable amount of human effort? This keeps a convincing explanation from earning a pass when the agent does not reliably complete useful work.
Build an evaluation set from representative work
A decision-grade pilot needs an evaluation set that reflects the work the agent will actually meet, not scenarios chosen because they show the technology well. For each workflow, include:
- Realistic tasks: common, well-understood tasks and complex but valid ones
- Difficult inputs: ambiguous requests with missing or conflicting context, and unsafe or adversarial inputs where appropriate
- Cases the agent must not complete on its own: expected failures, requests outside its authorization boundary, and high-impact actions that require human approval
Use sanitized examples from real workflows whenever possible. Generic exercises remove the dependencies, coding standards, policies, and ownership boundaries that make enterprise work difficult.
Add explicit acceptance criteria: a known correct diagnosis, tests that must pass, an expected security policy result, or a rubric completed by a subject matter expert. Without them, reviewers score how persuasive an output looks, which makes results hard to compare and easy to overestimate.
Measure a distribution, not the best run
Agentic systems behave differently when input, context, or execution state changes, so a single successful run says very little about reliability. AI output is variable and context-dependent, which is why evaluation has to look for patterns across representative scenarios rather than individual results. For agents there is one addition: Variability can affect actions, not only answers.
Run important scenarios several times, vary the relevant conditions, and measure:
- Task outcomes: end-to-end success rate, reviewer acceptance rate, and failure rate by scenario type
- Human involvement: intervention and escalation rate, and rework introduced by the agent
- Controls and resilience: policy or authorization violations, plus latency and recovery from tool, context, or workflow failures
The goal is not to prove the agent always behaves identically. It is to understand the distribution of outcomes and decide whether it is acceptable for the proposed use. Where predictability is non-negotiable, as in release management, keep the agent out of the execution path. Let it prepare the artifact, such as a deployment script, and let the pipeline run that script, so the execution is deterministic and the agent’s variability is limited to what a reviewer can inspect beforehand. The same pattern works in operations: An agent can assess whether a rollback is needed, while the rollback itself runs as a tested script. In both cases the pilot measures the quality of what the agent prepares or recommends, which are the Prepare and Recommend levels.
Two cautions apply. First, aggregates hide risk: A 90% success rate is not strong if the other 10% includes unauthorized changes. Segment results by risk tier and failure type. Second, pilot samples are small, so report counts and uncertainty, not only percentages. Twenty successes in twenty runs does not demonstrate 99% reliability: At 95% confidence, the true failure rate could still be close to 14%. Eighteen out of twenty is consistent with a true success rate anywhere from roughly 70% to 97%. If a threshold matters, size the number of runs to it, or state plainly that the pilot cannot yet support the claim.
Set thresholds by risk, and treat controls as hard gates
Not every task needs the same reliability. A wrong explanation that a developer can ignore has a different impact from a wrong production change. Define risk tiers and apply different thresholds to each:
| Risk tier | Example action | Evaluation approach |
|---|---|---|
| Low | Explain a pipeline error | Measure usefulness and correctness; a human validates the response |
| Medium | Prepare a merge request | Require passing tests, scoped changes, and reviewer approval |
| High | Merge or deploy a change | Require strict authorization, mandatory controls, reversibility, and a substantially higher reliability threshold |
Some requirements are not thresholds at all. They are hard gates, and an agent cannot offset a failed gate with time saved elsewhere. Completing the task and respecting the control environment are inseparable: An agent that resolves a vulnerability by bypassing an approval rule has not completed the workflow. Typical must-pass gates include:
- Identity and authorization: Every action is attributable to the agent and the initiating user, the agent works with least-privilege access, cannot reach tools outside its scope, and retrieves only the permitted context.
- Policy and approval: Existing security and compliance policies still apply, and required human checkpoints cannot be bypassed.
- Auditability and control: Reviewers can reconstruct what the agent did and why, high-impact changes can be undone, the team can stop or disable the agent, and failures, policy violations, and abnormal activity are visible.
GitLab's guidance on governing agentic AI treats role-based permissions, tool approval guardrails, and human review checkpoints as distinct control points for agent actions. A pilot should verify them with negative tests, not just confirm they exist on paper. Attempt actions outside the permitted scope, remove required context, trigger an approval boundary, and confirm that the agent stops, refuses, or escalates as designed.
Measure total effort and cost per successful task
Time saved by the agent is only one side of the equation. An agent can finish a task in minutes and leave a lengthy review. It can cut implementation time while raising the effort needed to understand the change. It can automate routine work and still produce occasional failures that need a senior specialist to diagnose.
Measure the whole workflow: preparing requests and context, reviewing output, correcting or reworking changes, handling escalations and failures, and the manual steps avoided. The useful metric is not minutes generated by the agent but total human effort per successfully completed task. Also record where human judgment remains essential. The aim is an effective operating model between people and agents, not necessarily removing people.
Apply the same logic to cost. A low-cost execution that frequently fails can cost more than a higher-cost one that completes the workflow correctly. Count model and platform consumption, tool and infrastructure usage, human review, rework and failed attempts, and monitoring and support overhead. Then compare with the current baseline:
- Cost per successfully completed task
- Total human effort and median time to completion per task, including the share of tasks that need specialist intervention
- Rework or defect rate, and throughput under the same control requirements
The pilot does not need a perfect financial model. It needs to show whether the value stays credible once the costs a demonstration leaves out are included.
Use a decision-grade scorecard
A scorecard connects each piece of evidence to the decision the pilot must support. Hard gates are marked "Must pass"; everything else is judged against a target.
| Dimension | Decision question | Example evidence | Threshold type |
|---|---|---|---|
| Business outcome | Does the workflow improve the agreed outcome? | Baseline comparison, time and throughput metrics | Target |
| Task success | Does the agent complete the end-to-end task correctly? | Acceptance criteria, tests, expert review | Target |
| Reliability | Are outcomes consistent across representative scenarios? | Repeated runs, success and failure rates segmented by risk, with counts | Risk-based target |
| Human effort | Does the agent reduce total effort after review and rework? | Time study, intervention and escalation rate | Target |
| Economics | Is the cost per successful task credible at expected scale? | Consumption, operations, and support estimates | Target |
| Authorization | Does the agent stay within its assigned permissions? | Access and negative test results | Must pass |
| Governance | Are actions attributable, reviewable, and auditable? | Approval records, audit trail, change history | Must pass |
| Safety and recovery | Can failures be detected, contained, and reversed? | Failure injection, rollback, and disablement tests | Must pass |
| Adoption | Can the intended users understand and operate the workflow? | User observation, feedback, training needs | Target |
Define thresholds before running the evaluation. Thresholds set after seeing results let teams adjust the definition of success to match what they observed. Record the evidence behind each result: a score without evidence is hard to defend and gives little guidance for rollout.
Worked example: A pipeline failure agent
The numbers below are illustrative, not benchmarks. They show how thresholds, segmentation, and gates combine into a decision.
A team wants an agent to diagnose failed pipelines and prepare a merge request with a fix, for developer approval (the Prepare level). They sanitize 30 real failures across three types and run each scenario three times, for 90 runs. Before running anything, they set the following thresholds:
- Reviewer-accepted merge requests: at least 85% for configuration errors, 70% for flaky tests, and 85% for permission-related failures, where a wrong fix is costlier
- Total human effort per resolved failure, including review and rework: at least 25% below the baseline median of 40 minutes
- Hard gates: zero out-of-scope changes, zero bypassed approvals, and every merge request attributable to the agent and the initiating user
| Failure type | Runs | Accepted | Rate | Threshold | Result |
|---|---|---|---|---|---|
| Configuration errors | 30 | 27 | 90% | 85% | Pass |
| Flaky tests | 30 | 22 | 73% | 70% | Pass |
| Permission-related | 30 | 15 | 50% | 85% | Fail |
The median human effort was 29 minutes against 40, a 28% reduction. The gates held: no violations in 90 runs, and all 10 negative tests ended in a refusal or escalation. Zero violations in 90 runs supports a true violation rate below roughly 3% at 95% confidence, which is evidence but not proof.
The resulting decision is narrower than "the agent works." Authorize Prepare-level rollout for configuration errors and flaky tests. Keep permission-related failures at Recommend, fix the context the agent lacks, and repeat that tier. Do not authorize execution.
Governed agents across the software lifecycle
The gates above are platform-neutral, but GitLab Duo Agent Platform applies them across the whole software lifecycle instead of to a single tool. For a pilot, that means you can evaluate agents at several stages against one set of controls. Duo Agent Platform offers agents and flows at each stage:
- Plan: Planner Agent plans, prioritizes, and tracks work, and Data Analyst Agent draws insight from development metrics.
- Code: Developer Flow turns issues into merge requests, and Software Development Flow builds a multi-step plan before it acts.
- Review: Code Review Flow automates review tasks and enforces coding standards, and GitLab Duo can also resolve review discussions and merge conflicts.
- Build and deploy: Fix CI/CD Pipeline Flow diagnoses and fixes failing pipelines, and Convert to GitLab CI/CD Flow migrates legacy pipelines.
- Secure: SAST Vulnerability Resolution Flow generates fixes, the false positive detection flows filter static application security testing and secret detection findings, and Security Analyst Agent triages and analyzes vulnerabilities.
These capabilities sit at different points on the autonomy model. Developer Flow prepares a merge request for review, which is the Prepare level. Resolving merge conflicts or review discussions pushes commits, which is closer to execution, so a pilot should hold it to a stricter threshold.
The same guardrails apply across all of these stages:
- Identity: Composite identity is the default for agent sessions. A flow acts through a service account combined with the user who triggered it, and permissions are the more restrictive of the two. Agentic Chat in the UI and IDE is the exception.
- Approval: Agent tool governance (beta) sets each tool to Always Allow, Always Ask, or Always Deny at group or project level, and project rules can only be equal to or stricter than group rules. By default, write and delete tools ask for approval. Background flows have no user to ask, so their tools can only be allowed or denied, and a tool with no background rule is allowed. Set those rules explicitly, and confirm that enforcement for background flows, which is controlled by a feature flag, is on in your environment.
- Audit: AI audit events (beta) give a per-session record, with actions attributed to the service account and the human who started the session recorded alongside. Storage is off by default and must be enabled. The AI Governance Dashboard (beta) adds group-level monitoring of agent sessions and developer exposure.
- Data boundaries: Duo Agent Platform can run with self-hosted models on GitLab Self-Managed, for organizations that must control where model requests are processed.
Several of these controls are in beta, so treat them as part of what the pilot evaluates: confirm their status, enable them explicitly, and test them with the negative tests described earlier.
Turn the pilot into an authorization decision
A pilot should end with a decision about authority, scope, and operating conditions. Possible outcomes include:
- Authorize a limited rollout to selected teams, at the evaluated autonomy level or for lower-risk tasks only.
- Reduce the agent from execution to preparation or recommendation mode, or iterate on controls, context, or workflow design and repeat selected tests.
- Stop, because value, reliability, risk, or economics are not acceptable.
Write the decision down. The record should state:
- Scope: the workflows and task types covered and excluded, the authorized level of autonomy, and the systems, tools, and data the agent may access
- Evidence and controls: the thresholds demonstrated, with run counts, the known limitations, and the controls and approval points that must stay active
- Ownership: the workflow owner and escalation path, and the metrics monitored after rollout and the conditions that trigger suspension or reassessment
This ties pilot evidence to the authority actually delegated. It also stops a successful evaluation of one narrow workflow from being read as approval for broader autonomous use.
Start your own decision-grade pilot today
Evaluating agentic AI is an exercise in deciding how much authority a team is prepared to delegate, and under which conditions. The result of a decision-grade pilot should not be "the agent works." It should be a documented decision that states where the agent may act, what evidence supports that authority, which controls must stay in place, and how the team will know when to intervene. That is the standard for moving agentic AI from experimentation into responsible enterprise use.
