Work through an example to learn how to read and write Spark DataFrames

Описание: Working with DataFrames #apachespark
Work through an example to learn how to read and write Spark DataFrames.

We'll cover the following

Creating DataFrames
Writing DataFrames
Reading and writing data in Spark is very convenient given the high-level abstractions available to connect to a variety of external data sources, such as Kafka, RDBMSs, or NoSQL stores. Spark provides an interface, DataFrameReader, that allows us to read data into a DataFrame from various sources and in a number of formats such as JSON, CSV, Parquet, or Text.

For any meaningful data analysis, we’ll be creating DataFrames from data files. When doing so, we can either instruct Spark to infer the schema of the data itself or specify it for Spark. Let’s see examples of both below: