Practical Data Pipeline: Build a Plant Monitoring System with ksqlDB

Описание: https://cnfl.io/podcast-episode-216 | Apache Kafka® isn’t just for day jobs according to Danica Fine (Senior Developer Advocate, Confluent). It can be used to make life easier at home, too!

Building out a practical Apache Kafka® data pipeline is not always complicated—it can be simple and fun. For Danica, the idea of building a Kafka-based data pipeline sprouted with the need to monitor the water level of her plants at home. In this episode, she explains the architecture of her hardware-oriented project and discusses how she integrates, processes, and enriches data using ksqlDB and Kafka Connect, a Raspberry Pi running Confluent's Python client, and a Telegram bot. Apart from the script on the Raspberry Pi, the entire project was coded within Confluent Cloud.

Danica's model Kafka pipeline begins with moisture sensors in her plants streaming data that is requested by an endless for-loop in a Python script on her Raspberry Pi. The Pi in turn connects to Kafka on Confluent Cloud, where the plant data is sent serialized as Avro. She carefully modeled her data, sending an ID along with a timestamp, a temperature reading, and a moisture reading. On Confluent Cloud, Danica enriches the streaming plant data, which enters as a ksqlDB stream, with metadata such as moisture threshold levels, which is stored in a ksqlDB table.

She windows the streaming data into 12-hour segments in order to avoid constant alerts when a threshold has been crossed. Alerts are sent at the end of the 12-hour period if a threshold has been traversed for a consistent time period within it (one hour, for example). These are sent to the Telegram API using Confluent Cloud's HTTP Sink Connector, which pings her phone when a plant's moisture level is too low.

Potential future project improvement plans include visualizations, adding another Telegram bot to register metadata for new plants, adding machine learning to anticipate watering needs, and potentially closing the loop by pushing data back
to the Raspberry Pi, which could power a visual indicator on the plants themselves.

EPISODE LINKS
► GitHub: raspberrypi-houseplants: https://github.com/danicafine/raspber...
► Data Pipelines 101 course: https://cnfl.io/data-pipelines-course...
► Tips for Streaming Data Pipelines ft. Danica Fine: https://cnfl.io/streaming-data-pipeli...
► Danica Fine’s Twitter:   / thedanicafine  
► Kris Jenkins’ Twitter:   / krisajenkins  
►Streaming Audio Playlist:    • Streaming Audio Podcast | Apache Kafka®, C...  
► Join the Confluent Community: https://cnfl.io/join-community-episod...
► Learn more with Kafka tutorials, resources, and guides: https://cnfl.io/confluent-developer-e...
► Live demo: Intro to Event-Driven Microservices with Confluent: https://cnfl.io/event-driven-microser...
► Use PODCAST100 to get $100 of free Confluent Cloud usage: https://cnfl.io/try-cloud-episode-216
► Promo code details: https://cnfl.io/podcast100-details-ep...

EPISODE LINKS
0:00 - Intro
2:24 - Practical Kafka data pipeline
4:14 - The architecture
7:46 - Raspberry Pi
11:08 - Data modeling
16:38 - Avro schema
19:11 - Integrates, processes, and enriches data
21:55 - Windowing
24:27 - HTTP sink connector
28:49 - What's next
32:00 - It's a wrap

ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.

#datapipeline #streamprocessing #apachekafka #kafka #confluent