Data pipelines from zero to solid

Описание: (Recorded at Jfokus 2016. http://www.jfokus.com)

Data pipelines from zero to solid

Collect lots of data, do something mysterious, make profit! The big data buzz affects everyone. We all have data, and we all have a hunch that we may be able to get something valuable out of it, but the step in between is somewhat foggy, both in terms of engineering and science.- This presentation is an attempt do demystify the practice of building reliable data processing pipelines. We will go through the necessary pieces needed to build a stable processing platform: data ingestion, processing engines, workflow management, schemas, and pipeline development processes. The presentation will also include component choice considerations and recommendations, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes.

Lars Albertsson

Lars Albertsson has worked with data-intensive and scalable applications at Google, Spotify, Schibsted Media Group, natural language processing startup Recorded Future, and with stock exchange systems. He is now an independent consultant, helping companies build scalable data processing solutions.