From "Vibe Checks" to
Unit Tests

"It looks right to me" is not a testing strategy. It's a prayer.

The Maturity Model

Three Types of Prompt Tests

1. Deterministic Tests (No LLM Required)

Fast, cheap, run on every commit:

test('output is valid JSON', async () => {
    const result = await runPrompt('extract-entities', {
        input: "John works at Google in NYC"
    });
    const parsed = JSON.parse(result);
    expect(parsed).toHaveProperty('entities');
    expect(parsed.entities.length).toBeGreaterThan(0);
});

2. Semantic Tests (LLM-as-Judge)

Use a second LLM to grade the output:

test('response is polite and professional', async () => {
    const result = await runPrompt('customer-support', {
        input: "Your product sucks!"
    });
    
    const score = await judge(result, {
        criteria: 'Is the response polite, empathetic, and professional?',
        scale: { min: 1, max: 5 }
    });
    
    expect(score).toBeGreaterThanOrEqual(4);
});

3. Regression Tests (Golden Dataset)

Compare new outputs against a curated set of "correct" answers:

test('email subject lines match golden dataset', async () => {
    const goldenSet = loadGoldenDataset('email-subjects');
    
    for (const { input, expectedOutput } of goldenSet) {
        const result = await runPrompt('email-subject', { input });
        const similarity = cosineSimilarity(
            embed(result), embed(expectedOutput)
        );
        expect(similarity).toBeGreaterThan(0.85);
    }
});

The Testing Pyramid for AI