The Missing Manual for
LLM CI/CD

Every engineering team has CI/CD for code. Almost none have it for the text that controls their AI.

You would never push a database migration without testing it. You would never deploy a microservice without running your test suite. But every day, teams push prompt changes to production with nothing more than a "looks good to me" in Slack.

Why Prompts Need Pipelines

Prompts are not static strings. They are executable configuration. A single word change can shift the entire behavior of your application:

  You are a helpful customer support agent.
- Be concise and professional.
+ Be friendly and use emojis! 🎉

  // This small change = completely different user experience

Without a pipeline, this change goes from someone's IDE directly to your users. No tests. No approval. No rollback plan.

The 4-Stage Prompt Pipeline

Stage 1: Lint & Validate

Before anything runs, validate the prompt structure:

• Are all {{variables}} defined?
• Does the system prompt exceed the model's context window?
• Are there known prompt injection patterns?

Stage 2: Unit Test (Deterministic)

Run fast, deterministic checks that don't require an LLM call:

// test/prompts/onboarding.test.ts
test('onboarding prompt includes required variables', () => {
    const prompt = loadPrompt('onboarding-email');
    expect(prompt.template).toContain('{{userName}}');
    expect(prompt.template).toContain('{{planName}}');
    expect(prompt.model).toBe('gpt-4');
});

Stage 3: Eval (LLM-Powered)

The real test: send the prompt to the model with known inputs and grade the output.

• Golden Dataset: 50 input/output pairs that define "correct" behavior.
• Semantic Similarity: Is the new output ≥90% similar to the golden output?
• Safety Check: Does the output contain PII, competitor mentions, or hallucinations?

Stage 4: Promote

If all tests pass, promote the prompt version through environments:

# GitHub Action
promptops promote onboarding-email --from staging --to production

GitHub Actions Example

name: Prompt CI/CD
on:
  push:
    paths: ['prompts/**']

jobs:
  test-prompts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate Prompts
        run: npx promptops validate ./prompts
      - name: Run Evals
        run: npx promptops eval --dataset golden.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - name: Promote to Staging
        if: github.ref == 'refs/heads/main'
        run: npx promptops promote --env staging

The ROI is Immediate

Teams that implement prompt CI/CD report:

• 80% fewer production incidents caused by prompt changes
• 5x faster iteration because non-engineers can propose changes
• Complete audit trail for every prompt that ever ran in production