About this lab
Quillmarsh Marine Works buys parts from 12 suppliers, and about one order in five arrives late. The reasons are written in PDFs that nobody reads: supply agreements with lead times and late-delivery credits, quality audits with capacity warnings, and a steady stream of incident reports. In this advanced lab you turn those documents into an AI and ML system that runs entirely in Snowflake.
You parse the PDFs with a Python table function, redact names and phone numbers with AI_REDACT, and extract contract, audit and incident fields with AI_AGG, then measure the extraction against an analyst's answer key (100% on 419 values) before trusting it. You register an open-source embedding model in the Model Registry, search by meaning in plain SQL with vectors, build a hybrid Cortex Search service on your own vectors, and measure retrieval quality on a test set.
The extracted fields feed point-in-time features for an XGBoost delay-risk model trained in a Snowflake Notebook: adding the document fields lifts ROC AUC from 0.69 to 0.79. You score the open orders from SQL, build a stored-procedure assistant that answers from both the documents and the model and cites a source for every fact, and automate it all: when new incident reports land on the stage, a stream and a two-task graph process them end to end and re-score the orders. The lab is designed and tested for Snowflake trial accounts, which block several Cortex features, and shows how to build the same capabilities with what they allow.
Lab details
- Level: Advanced
- Cloud: Snowflake
- Domain: AI Engineering
- Duration: 150 minutes
What you will learn
- Parse, redact and chunk PDFs inside Snowflake and extract typed fields with AI_AGG, measured against an answer key
- Register an open-source embedding model and search documents by meaning with SQL vectors and hybrid Cortex Search
- Evaluate retrieval quality at document and passage level on a labelled question set
- Train a delay-risk model on point-in-time features and prove what the document fields add
- Build a stored-procedure assistant that answers from documents and model predictions with verifiable citations
- Automate processing of newly arriving documents with a stream and a task graph
Concepts covered
- Document AI pipelines
- PII redaction before indexing
- LLM extraction with structured output and accuracy measurement
- Embeddings and vector search
- Hybrid search and retrieval evaluation
- Retrieval-augmented generation with citations
- Point-in-time feature engineering and label censoring
- Model Registry and SQL inference
- Event-driven pipelines with streams and task graphs
Technologies used
- Snowflake AI functions
- Python UDTF + directory tables
- Snowflake Model Registry
- Vectors and Cortex Search
- Snowflake Notebooks
- Snowflake Scripting
- Streams and task graphs
- Key-pair authentication
Prerequisites
- Comfortable with SQL (joins, window functions) and basic Python
- Basic ML concepts: train/test split, classification, ROC AUC
- A Snowflake account with the ACCOUNTADMIN role (tested on an Enterprise trial, AWS eu-west-1)
- A laptop with Python 3.11 and about 1 GB of disk space for packages
Topics
- Snowflake
- RAG
- Document AI
- Cortex Search
- AI_AGG
- AI_REDACT
- Embeddings
- Vector Search
- Model Registry
- Snowflake Notebooks
- XGBoost
- Streams
- Tasks
- Procurement
- Python
- SQL
