AI CLI Agents in CI: Claude Code vs Codex CLI vs Gemini CLI

AI CLI Agents in CI: Claude Code vs Codex CLI vs Gemini CLI#

Every comparison of terminal coding agents is written for someone sitting at a laptop, weighing which one writes better code. That is a reasonable question and it is not the one that matters when you put an agent in a pipeline. In CI the agent runs unattended, with credentials, against a checkout it did not choose, and the questions become how tightly it is sandboxed, how it authenticates without a human, and what it does when it cannot finish. Judged on those, the three tools separate cleanly.

The three questions that actually decide it#

How does it authenticate without a human? An interactive login flow is fine on a workstation and useless in a runner. You need either an API key or a machine credential, and you need to know what scope that credential carries, because the agent will be holding it while it executes model-suggested commands.

What can it do that you did not approve? An agent that edits files, runs tests, and opens a pull request is useful. The same agent with an unrestricted shell and network access, running on a build box that holds deploy credentials, is the attack surface described in hardening CI against npm worms, except you installed it deliberately.

What happens when it fails? Non-interactive mode needs to exit with a real status code, write something a human can read, and not hang waiting for input. This is where tools built primarily for interactive use tend to disappoint.

Where each one lands#

Codex CLI is the one built for this. It sandboxes by default, ships as an open-source client you can audit and pin, has native Windows support, and has the tightest GitHub integration of the three. In practice it is the strongest automated reviewer: pointed at a diff with a rubric, it produces useful review comments and exits cleanly. If you are adding an agent to a pipeline and have no other constraint, this is the default.

Claude Code is the strongest implementer, and needs more scaffolding. It handles long multi-file work with the least hand-holding, which makes it the right choice for jobs that actually change code: scheduled dependency migrations, mechanical refactors across a repo, converting a test suite. The cost is that you supply the guardrails, because its default posture assumes a developer is watching. In CI that means running it in a container you control, with a scoped token, and with explicit limits on what it may execute.

Gemini CLI is the awkward one right now. Its selling points were a very large context window and a generous free tier, both genuinely useful for large-repo exploration. Google ended consumer authentication for it on 18 June 2026, moving that experience elsewhere; the CLI remains available through Gemini Code Assist Standard and Enterprise and through paid API access. That is a licensing question to settle before you build a pipeline on it, not a capability question.

What a sane pipeline job looks like#

The pattern that holds up is narrow scope, short-lived credentials, and no write access to anything that matters:

# .github/workflows/agent-review.yml
jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read          # the agent reads the diff
      pull-requests: write    # and comments; nothing else
    steps:
      - uses: actions/checkout@v4
      - name: Agent review
        env:
          AGENT_API_KEY: ${{ secrets.AGENT_API_KEY }}
        run: |
          agent review --non-interactive \
            --base "${{ github.base_ref }}" \
            --output review.json || echo "agent failed, continuing"
      - name: Post findings
        run: ./scripts/post-review.sh review.json

Two details in there do most of the safety work. The job permissions are the minimum the task needs, so an agent that goes wrong cannot push to a branch or read other secrets. And the agent step tolerates failure rather than blocking the pipeline, because a review agent that breaks the build when the model API has a bad afternoon will be disabled by the first engineer it inconveniences.

The jobs worth automating, and the ones that are not#

Agents in CI earn their cost on work that is mechanical, reviewable, and bounded. Reviewing diffs against a checklist your team keeps forgetting. Drafting changelog entries. Triaging a failing test by reading the log and the recent commits. Proposing a dependency bump with the migration applied.

They do not earn it on work where the output is unbounded or the review cost approaches the work cost. An agent that opens a fifteen-file pull request nobody wants to read has produced a liability. The rule we apply is that if a human would not merge it without reading every line, the agent should not be generating it unattended. The same discipline that makes spec-driven development work applies here: constrain the task, not the model.

The decision, concretely#

• Adding an agent to review pull requests? Use Codex CLI. Sandboxed by default, best GitHub story, exits cleanly in non-interactive mode.
• Running scheduled jobs that rewrite code, like migrations or refactors? Use Claude Code, in a container you control, with a read-scoped token and an explicit command allowlist.
• Already standardised on Google's stack and covered by a Code Assist licence? Gemini CLI is workable, and confirm the licensing before building on it rather than after.
• Tempted to give an agent deploy credentials so it can fix production? Do not. Split the pipeline so the agent proposes and a separate, agent-free job applies.

The call we'd make#

Start with a review job, not a code-writing job. It is the highest-value use, it is bounded, it fails safely, and it gives you a month of observing how the agent behaves on your codebase before you let one open pull requests unattended. Whichever tool you pick, treat it as untrusted code executing with whatever your runner holds, because that is precisely what it is. The comparison that decides this is not which model writes better Python. It is which tool you can confidently hand a credential to.