# AI Best Practices for Engineering Teams: From Prompt Experiments to Platform Discipline

URL: https://www.devopsness.com/blog/ai-best-practices-2026-02-21-platform-discipline
Author: Kiril Urbonas
Published: 2026-02-21
Updated: 2026-02-21
Categories: AI
Tags: LLM, Evals, Security

Summary: A practical field manual for engineering teams who want AI features that survive real users, incidents, and budgets — not just demo day.

---
# AI Best Practices for Engineering Teams: From Prompt Experiments to Platform Discipline

Most AI initiatives don't fail because the model is bad. They fail because the surrounding system is missing: no clear contracts, no evaluation, no governance, and no path from one clever notebook to a repeatable product capability.

This guide is a field manual for engineering teams who want AI features that survive real users, real incidents, and real budgets — not just demo day.

---

## 1. Start with a clear AI contract

### What it is

An AI contract is a one-pager every feature must have before any prompt work begins. It defines:

- **Task statement** — What the AI is allowed to do (and not do).
- **Inputs** — Exact fields, types, and sources (for example, `sanitized Markdown, max 4k chars`).
- **Output schema** — JSON / markdown / text shape your code will parse.
- **Quality target** — What “good enough” means (accuracy, coverage, refusal rate).
- **Latency budget** — For example, `P95 ≤ 2.5s`.
- **Cost budget** — For example, `≤ $0.005 per request at forecast traffic`.

### Real-world example

A team shipping an “AI incident summary” feature skipped this step. They let the model “summarize any incident data” with no schema. Within weeks:

- Some outputs were bullet lists, others were paragraphs, some included tables.
- The notifications system broke whenever the model decided to “be helpful” and add Markdown headings.
- Product managers couldn’t tell whether things were improving — there was no quality target.

They rewrote the feature around a contract:

> Task: Summarize a production incident for engineers.  
> Inputs: Root cause, impact, timeline (ISO timestamps), remediation, incident tags.  
> Output: JSON with fields: `title`, `short_summary` (≤ 240 chars), `long_summary` (≤ 800 chars), `impact_level` (`LOW|MEDIUM|HIGH`).  
> Quality: 90% of summaries accepted without edit by on-call engineer.  
> Latency: P95 ≤ 2s.  
> Cost: ≤ $0.003 per incident.

Breakage stopped overnight because downstream code finally knew what to expect.

### Implementation tips

- **Make the contract a template** in your repo (for example, `docs/ai/contracts/incident-summary.md`) and require it in PRs for new AI features.
- Add a **checkbox in PR descriptions**: “AI contract attached and reviewed by product + eng.”
- Revisit contracts every quarter when usage, data, or risk changes.

---

## 2. Treat prompts and policies like code

### The usual failure mode

Prompts live in a vendor UI, people tweak them during incidents, and nobody knows why behavior changed last week.

**Real-world example**

A support team had a “reply draft” assistant. A well-meaning engineer changed the tone phrasing in the prompt from “professional and concise” to “friendly and conversational.” Conversion dipped on enterprise accounts — customers started complaining about “too casual” responses — but there was:

- No commit,
- No diff,
- No quick rollback.

### Better pattern

- Store prompts in your repo (for example, `prompts/support/reply-draft.prompt.md`).
- Reference them by **versioned ID** in code, like `SUPPORT_REPLY_V3`.
- For every prompt change, require:
  - A short **changelog note** (for example, “tightened escalation instructions for security issues”).
  - A link to **evaluation results** (see section 4).

### Practical setup

- Create a `prompts/` folder with:
  - `*.prompt.md` — raw prompt text.
  - `*.policy.md` — safety/guardrail instructions.
  - `CHANGELOG.md` — bullet list of meaningful changes.
- Enforce a **prompt diff review**: prompt changes must be reviewed by at least one engineer and one domain owner (for example, a support lead).

---

## 3. Fix retrieval before touching the model

For retrieval-augmented generation (RAG), prompt tuning on top of bad retrieval is like polishing a cracked lens.

### Real-world incident

A team shipped a RAG “runbook assistant” for on-call engineers. It hallucinated outdated mitigation steps. Root cause was not the model; it was:

- Runbooks older than 18 months were never re-embedded after updates.
- Chunking split tables across chunks, losing context.
- Ranking ignored incident severity; archived incidents were surfacing over fresh ones.

After they fixed retrieval:

- **Freshness rule:** re-embed within 5 minutes of document updates.
- **Chunking benchmarks:** tuned for whole procedures, not arbitrary token sizes.
- **Ranking:** boosted docs tagged with the active service and last 6 months of incidents.

Hallucinations dropped by more than 40%, with **no model change**.

### Retrieval checklist

- **Freshness**
  - [ ] Index updates within X minutes (not days).
  - [ ] Alerts when embeddings lag behind the source of truth.
- **Chunking**
  - [ ] Chunk on semantic sections (headings, steps), not just raw token counts.
  - [ ] Keep tables and code samples intact.
- **Ranking**
  - [ ] Boost by recency for operational content.
  - [ ] Boost by entity (service, team, tag) relevant to the request.
- **Evaluation**
  - [ ] Maintain 20–50 “golden questions” with known good docs.
  - [ ] Track hit-rate: “Did the top 3 chunks contain the answer?”

---

## 4. Test beyond unit tests: the three layers that matter

### Layer 1 — Deterministic tests (parsing and schema)

**Goal:** your system never crashes because of AI output shape.

- Test that every output parses into your schema.
- Test refusal handling (nulls, fallback messages) without a real model call.

### Layer 2 — Semantic evaluation suites

**Goal:** ensure the model is “good enough” for key tasks.

- Keep a set of labeled examples (input + ideal output).
- Score them automatically (for example, with another model or rule-based scoring).
- Track per-change: “Did this prompt or model version improve or regress?”

### Layer 3 — Scenario and workflow tests

**Goal:** check end-to-end behavior in real user flows.

Example: for an “incident summary + Slack notification” flow, tests should verify:

- The summary is generated.
- The Slack message uses the short summary.
- The incident link is correct.
- For “LOW” incidents, notifications don’t ping the exec channel.

### Real-world tip

- Start with **10–20 curated test cases per critical feature** instead of aiming for hundreds.
- Wire evaluation into CI so any prompt or model change runs the suite before merge.

---

## 5. Use scorecards that blend technical and business metrics

Accuracy alone is vanity if latency, cost, or policy risk are unacceptable.

### Real-world example

A customer-facing “architecture review assistant” had great semantic scores but:

- P95 latency was 8–10 seconds.
- Token burn was 3–4× budget for big accounts.
- Security flagged it because logs were sending full code samples to a third-party.

Once they built a **scorecard per release**, they exposed:

- **Technical metrics** — accuracy, latency, error rate.
- **Business metrics** — daily active users, completion rate, support escalations.
- **Risk metrics** — PII exposure incidents, policy violations, manual escalations.

They then explicitly traded a **slight drop in accuracy** for **2× faster responses and 50% lower cost**, which turned out to be the right business call.

### Suggested scorecard fields

- **Quality:** evaluation score, acceptance / edit-rate.
- **Reliability:** P95 latency, timeout rate, fallback activation rate.
- **Cost:** average cost per request, per active user, per month.
- **Safety:** policy violation rate, manual escalations.
- **Business:** user adoption, impact metric (for example, tickets resolved per agent).

---

## 6. Lightweight but explicit governance

You don’t need a committee; you need **clear ownership**.

### Minimal governance model

- **AI platform owner** — chooses providers, manages SDKs, enforces contracts.
- **Risk/Security contact** — signs off on data flow and vendors.
- **Product owner** — signs off on UX and user impact.

### Real-world pattern

Keep a single `ai/systems.yaml` file listing:

- Each AI workflow (for example, `incident_summary_v2`).
- Owner, data sources, model(s) used, and fallback behavior.
- Links to prompt, contract, and evaluation dashboards.

This file becomes your **source of truth** during incidents, audits, and vendor changes.

---

## 7. Policy-driven routing across models

### Why it matters

Blindly using the “best” model for everything is wasteful and risky.

### Example routing policy

- **Tier 0 (low risk, high volume)** — log classification, tag suggestions, deduping. Use small, cost-efficient models.
- **Tier 1 (medium risk)** — internal summaries, query rewriting. Use mid-tier models with guardrails.
- **Tier 2 (high risk, external)** — customer-visible decisions, security advice, compliance-sensitive output. Use premium models, stricter evaluation, and possibly human-in-the-loop.

### Real-world tip

Implement routing via a config file (for example, `ai/routing.json`) rather than conditionals scattered through code. That makes it easy to change behavior without redeploying everything.

---

## 8. Design UX for trust, not magic

### Good UX patterns

- Clearly label AI outputs (“AI-generated summary — last updated 2 min ago”).
- Show **confidence and completeness** hints (for example, “Based on 3 incidents and 2 runbooks”).
- Provide **simple controls**:
  - “Regenerate with more detail”
  - “Include logs from the last 30 minutes”
  - “Escalate to human reviewer”

### Real-world example

In an internal incident tool, adding a small “What data was used?” toggle reduced complaints about hallucinations. Engineers could see exactly which runbooks and incidents the assistant used.

---

## 9. Security hardening that actually prevents incidents

Assume:

- Any external content might be prompt injection.
- Any tool call could be abused if permissions are too broad.

### Practical controls

- **Input sanitization** — strip or sandbox user-provided instructions before passing them to the model.
- **Tool allowlists** — for example, the “architecture advisor” can read docs and schemas but cannot run CI jobs or modify configs.
- **Response guards** — never directly execute shell commands or SQL produced by the model; require explicit human confirmation for dangerous actions.

### Real-world example

A team added a “deploy with AI” helper. They blocked direct shell execution and forced:

1. AI suggests a deploy command.  
2. Human sees a diff and the command.  
3. Human clicks “Approve & run.”

This added 1–2 seconds but avoided an entire class of “the model misread the ticket” incidents.

---

## 10. Invest in team habits, not just tooling

### Habits that correlate with reliable AI systems

- **Short feedback loops** — weekly review of eval metrics and incident learnings.
- **Recorded “prompt post-mortems”** — for every major issue, capture:
  - The bad input, trace, and output.
  - Which contract or guardrail failed.
  - The new test case added to prevent recurrence.
- **Shared playbook** — lightweight `ai/runbooks/*.md` for:
  - Model provider outages.
  - Cost spikes.
  - Elevated policy violations.

Over time, this turns AI work from “clever prompts” into a **disciplined engineering practice** — the real competitive advantage in 2026.

---

## How to apply this next week

- **Day 1–2**
  - Write contracts for your top one or two AI features.
  - Move prompts into source control.
- **Day 3–4**
  - Build a 10–20 case eval suite for each feature.
  - Add at least one retrieval benchmark if you’re using RAG.
- **Day 5**
  - Create a minimal `ai/systems.yaml` and routing config.
  - Agree on a monthly AI scorecard review.

If you do only this, you’ll already be ahead of most teams shipping AI in production.

