About this lab
You are an MLOps engineer responsible for a Gemini powered summarization feature that is already live. A newer, more capable Gemini model just shipped, and your product manager wants to know whether switching is worth it. Eyeballing a handful of outputs is not enough to make that call with confidence. You need an objective, repeatable way to measure summary quality before promoting any model to production.
In this lab you will use the Vertex AI Gen AI Evaluation Service to score model output with model based metrics. You will assemble an evaluation dataset from real support conversations, run an evaluation task that measures summarization quality and groundedness, and read both the summary and per case results. You will then run a head to head comparison between two Gemini models, interpret the scores, and apply a clear promotion rule. By the end you will be able to treat evaluation as the backbone of a responsible LLM deployment workflow.
Lab details
- Level: Intermediate
- Cloud: GCP
- Domain: AI Engineering
- Duration: 75 minutes
What you will learn
- Explain why objective evaluation matters before promoting an LLM to production
- Assemble an evaluation dataset for a summarization task
- Run a Gen AI Evaluation Service task with summarization quality and groundedness metrics
- Interpret summary level and per case evaluation scores
- Compare two Gemini models head to head and apply a promotion rule
- Understand how evaluation fits into a production LLMOps workflow
Concepts covered
- LLM evaluation
- Model based metrics
- Summarization quality
- Groundedness
- Model comparison and promotion
Technologies used
- Vertex AI Gen AI Evaluation Service
- Gemini 2.5 Flash and Gemini 2.5 Pro
- Google Gen AI SDK (google-genai)
- Vertex AI Workbench
- pandas
Prerequisites
- Basic Python knowledge
- Familiarity with calling Gemini through the Gen AI SDK
- A Google Cloud account with billing enabled and the Vertex AI API available
Topics
- Gemini
- Vertex AI
- Evaluation
- LLMOps
- GenAI
- GCP
