Chapter 17: Automated evaluation for models and apps¶
Overview¶
Automated evaluation is useful for measuring quality and safety on a bigger scale. Azure AI Foundry evaluation tools also enable ongoing evaluations that periodically run to monitor for regression as the system, usage, and mitigations evolve over time.
Scaling Evaluation with Automation¶
Once you understand your quality requirements through manual evaluation, automation provides scale:
- Batch Processing: Evaluate thousands of outputs quickly
- Continuous Monitoring: Run evaluations on production traffic automatically
- Regression Detection: Alert when quality metrics degrade over time
- CI/CD Integration: Gate deployments on evaluation thresholds with GitHub Actions
- Cost Efficiency: Automated evaluation is far cheaper than human annotation at scale
Azure AI Foundry Automated Evaluation¶
Azure provides comprehensive automated evaluation capabilities:
- Cloud Evaluations: Run large-scale evaluations in Azure AI Foundry
- Scheduled Jobs: Periodic evaluations to catch regression
- GitHub Actions Integration: Automate evaluation in your development workflow
- Custom Metrics: Define domain-specific evaluators alongside built-in metrics
Automation doesn't replace human judgment—it amplifies it. Use manual evaluation to build intuition, then scale with Azure's automated evaluation tools.
Resources and Further Reading¶
Online Resources¶
Next Steps¶
Continue your learning journey:
Questions or feedback? Join the discussion on our GitHub repository or connect with the community.