Prompt A/B Tester: Optimize Prompts Across Models Simultaneously

The Prompt Engineering Problem

Small changes in prompt wording can dramatically affect output quality. Adding "think step by step" or restructuring the system prompt can improve results by 20-50%. But testing variations manually is tedious. The Prompt A/B Tester automates this across multiple models simultaneously.

How It Works

import vincony

client = vincony.Client(api_key="YOUR_KEY")

result = client.tools.prompt_ab_test(
    prompts=[
        "Explain quicksort algorithm",
        "Explain quicksort algorithm step by step with examples",
        "You are a CS professor. Explain quicksort to a student",
    ],
    models=["gpt-5", "claude-opus-4.6"],
    scoring=["clarity", "accuracy", "completeness"]
)

for combo in result.rankings:
    print(f"#{combo.rank} {combo.model} + Prompt {combo.prompt_index}")
    print(f"  Score: {combo.total_score}/100")

Scoring Criteria

• • Clarity: How clear and well-structured is the response?
• • Accuracy: Are the facts correct?
• • Completeness: Does it cover all aspects of the question?
• • Relevance: Does it stay on topic?
• • Code quality: For code prompts — correctness, style, efficiency

Pricing

A/B testing costs the sum of individual model calls plus 2 credits for scoring. A 3-prompt × 2-model test typically costs 12-20 credits. Available on all paid plans.