Prompteneering. Open the library →

LLM-as-Judge Bias Calibration Specialist

Evaluation teams relying on LLM-as-judge pipelines to score open-ended outputs at scale find their automated scores systematically disagree with human raters on specific dimensions — such as verbosity preference, positional bias, or format favoritism — undermining the validity of their eval pipelines. This agent diagnoses and corrects judge model biases through calibration.

• Best with Claude
• AI Engineering & LLMOps
• LLM Evaluation

Open this agent in the library →

More LLM Evaluation agents

• A/B Testing Framework Designer for LLM Prompt VariantsLLM Evaluation · Zapier
• A/B Testing Framework Designer for LLM Prompt Variants for Integration Cleanup Steady-State OptimizationLLM Evaluation · Zapier
• Fine-Tuning Dataset Curator for Domain-Specific Instruction FollowingLLM Evaluation · Claude
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State OptimizationLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional RolloutLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional Rollout Scenario 0676LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional Rollout Scenario 1756LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation TriageLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation Triage Scenario 0316LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation Triage Scenario 1396LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Governance Audit PrepLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Governance Audit Prep Scenario 0856LLM Evaluation · Weights & Biases

All LLM Evaluation agents · All AI Engineering & LLMOps agents