Prompteneering. Open the library →

LLM Judge Calibration Specialist for Automated Evaluation Pipelines for Automation Rollout Steady-State Optimization

Calibrate LLM-as-judge scoring consistency against human rater agreement to make automated eval pipelines production-reliable.

• Best with Claude
• AI Engineering & LLMOps
• LLM Evaluation

Open this agent in the library →

More LLM Evaluation agents

• A/B Testing Framework Designer for LLM Prompt VariantsLLM Evaluation · Zapier
• A/B Testing Framework Designer for LLM Prompt Variants for Integration Cleanup Steady-State OptimizationLLM Evaluation · Zapier
• Fine-Tuning Dataset Curator for Domain-Specific Instruction FollowingLLM Evaluation · Claude
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State OptimizationLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional RolloutLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional Rollout Scenario 0676LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Cross-Functional Rollout Scenario 1756LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation TriageLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation Triage Scenario 0316LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Escalation Triage Scenario 1396LLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Governance Audit PrepLLM Evaluation · Weights & Biases
• Fine-Tuning Dataset Curator for Domain-Specific Instruction Following for Latency Reduction Steady-State Optimization for Governance Audit Prep Scenario 0856LLM Evaluation · Weights & Biases

All LLM Evaluation agents · All AI Engineering & LLMOps agents