sfspark.org: Evan Chan, Configuring and Deploying Apache Spark

Описание: ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Scale By the Bay 2019 is held on November 13-15 in sunny Oakland, California, on the shores of Lake Merritt: https://scale.bythebay.io. Join us!
-----

So you want to run Spark in production. You played with the EC2, looks cool. But the scripts are quite simple, what if a node goes away? Now what?

Evan Chan, the creator of Spark Job Server, will rely on his years of experience implementing Spark flows to show how Spark data pipelines are built.

Topics will include:

-- Running standalone vs Mesos (and Mesos fine grained vs regular)
-- Use of Job Server to expose Spark as a service
-- Running Spark on metal vs EC2 (tho at Ooyala we only ran it on metal)
-- Use a Spark distro? (We didn't)
-- Collocation of Spark and other things like Cassandra (now there's also Datastax DSE)
-- Thoughts about Docker and where it fits in

Evan loves to design, build, and improve bleeding edge distributed data and backend systems using the latest in open source technologies. He has led the design and implementation of multiple big data platforms based on Storm, Spark, Kafka, Cassandra, and Scala/Akka, including a columnar real-time distributed query engine. He is an active contributor to the Apache Spark project, a Datastax Cassandra MVP, and co-creator and maintainer of the open-source Spark Job Server. He is a big believer in GitHub, open source, and meetups, and have given talks at various conferences including Spark Summit, Cassandra Summit, and Scala Days. He has Bachelor's and Master's degrees in Electrical Engineering from Stanford University.