Analyzing the analytics: auditing models from creation to production - Serge Smertin, Adyen

Описание: Data science within financial technology industry often deals with various sources of data, most of which are commercially sensitive. For compliance reasons, in some cases there may be a need to answer questions like where the data of certain machine learning model or performance optimization is coming from - tracking a data lineage. As we know, BigData decisions may follow through different stages of ETL, as well as interactive data exploration through tools like Python, R and SQL. In this presentation we’ll focus on tracking data lineage for interactive data exploration tools. A set of techniques would be shown to demonstrate how to audit data journey from code entered in notebook down to levels of execution planning, DataFrames, RDDs and Hadoop’s file formats back to visualizations displayed to data analyst.