About this lab
Voltway runs 12 EV fast chargers across 5 stations. Every charger publishes charging-session events and status changes to Kafka, and the operations team wants energy, revenue, and charger health in Snowflake within a minute, with nobody running load jobs.
In this lab you run a Kafka broker and a Kafka Connect worker locally in Docker, create a least-privilege Snowflake service user that logs in with an RSA key pair, and configure the Snowflake Kafka connector to write two topics into two landing tables with Snowpipe Streaming. You trace every landed row back to its topic, partition, and offset, and see how channel offset tokens let the connector resume without loading anything twice.
Then you build the transform layer as four incremental dynamic tables: typed and de-duplicated events, rebuilt charging sessions priced with station tariffs, hourly station KPIs with a one-minute target lag, and the latest health of every charger. Three waves of real traffic put it to the test: retried duplicates, an overheating charger, a malformed message that lands in a dead-letter queue, and a backlog of late events that corrects hours you thought were finished. You finish by monitoring refreshes and lag, pausing the pipeline, and tearing everything down.
Lab details
- Level: Intermediate
- Cloud: Snowflake
- Domain: Data Engineering
- Duration: 100 minutes
What you will learn
- Create a least-privilege Snowflake service user that authenticates with an RSA key pair
- Run Kafka and Kafka Connect in Docker and stream keyed events into Snowflake with Snowpipe Streaming
- Trace landed rows to their topic, partition, and offset, and read channel offset tokens
- Build a chain of incremental dynamic tables that types, de-duplicates, sessionizes, and aggregates a stream
- Handle duplicates, a malformed message, and late-arriving events without breaking the pipeline
- Monitor refreshes and lag, pause the pipeline, and clean up every resource
Concepts covered
- Kafka topics, partitions, keys, and offsets
- Kafka Connect sink connectors
- Snowpipe Streaming channels and offset tokens
- Key-pair authentication and service users
- At-least-once delivery and de-duplication
- Dead-letter queues
- Dynamic tables, target lag, and incremental refresh
- Late-arriving data
Technologies used
- Apache Kafka
- Kafka Connect
- Snowflake Kafka Connector
- Snowpipe Streaming
- Key-Pair Authentication
- Dynamic Tables
Prerequisites
- Basic SQL (SELECT, GROUP BY, JOIN)
- A Snowflake trial account with the ACCOUNTADMIN role
- Docker Desktop installed and running
- A bash terminal with openssl and curl: Terminal on macOS or Linux, Git Bash on Windows
Topics
- Snowflake
- Kafka
- Kafka Connect
- Snowpipe Streaming
- Dynamic Tables
- Streaming
- Real-Time
- Docker
- Key-Pair Authentication
- Data Engineering
- SQL
