Migrating DuckLake Catalog from SQLite to PostgreSQL

Описание: Learn how to migrate your existing DuckLake catalog from SQLite to PostgreSQL, without losing any data, and see it in action in a production environment.

If you find this video useful or want to support the channel, please leave a like! And, if have you have any questions or suggestions, feel free to drop a comment below.

📺 Watch the full DataLab playlist here ►    • DataLab  

🔗 Resources:
💻 GitHub repo: https://github.com/DataLabTechTV/data...
📝 Blog post: https://datalabtechtv.com/posts/migra...
📚 DuckLake: https://ducklake.select/docs/stable/d...
📚 DuckDB: https://duckdb.org/docs/stable/core_e...
📚 SQLite: https://www.sqlite.org/datatype3.html
📚 PostgreSQL: https://www.postgresql.org/docs/curre...
📚 pgloader: https://pgloader.readthedocs.io/en/la...

🔔 Subscribe for more deep dives data science, data engineering, and all things data!
☁️ Follow the channel on Bluesky: https://bsky.app/profile/datalabtecht...
🎧 Join our community's Discord:   / discord  
👽 Join our subreddit:   / datalabtechtv  
💻 Check out the channel's GitHub: https://github.com/DataLabTechTV
🌐 Visit our website for blog posts: https://datalabtechtv.com

#dataengineering #devops #ducklake #duckdb #datastack #datacatalog #datamigration #sqlite #postgresql

00:00:00 Intro
00:00:35 Migration Strategy
00:01:23 Migration Script
00:02:58 Data Type Fixing
00:04:52 DuckLake Spec
00:05:31 SQLite Type Affinities
00:07:11 DuckLake Catalog DDL
00:07:43 Environment Variables
00:09:10 Migration Demo
00:09:51 Recreate Catalogs with SQLite
00:11:21 Running the Migration
00:12:57 Reconfiguring for PostgreSQL
00:13:35 DuckLake with PostgreSQL
00:14:07 Inspecting PostgreSQL DB
00:14:46 DataLab Code Changes
00:15:24 DuckLake Connection Configs: CLI
00:17:01 DuckLake Connection Configs: dbt
00:18:03 ML Server Redeployment
00:18:39 Docker Compose Changes
00:19:06 Updating CI/CD Variables
00:19:26 Docker Compose Changes (Cont.)
00:20:00 Testing ML Server
00:21:58 Outro