Apache Spark DataFrames row

Описание: Rows
Get hands-on practice performing various operations on the rows of a DataFrame.
We'll cover the following

Creating a row
Projections and filters
A row in Spark is an ordered collection of fields that can be accessed starting at index 0. The row is a generic object of type Row. Columns making up the row can be of the same or different types.

Creating a row
Spark allows us to instantiate rows in all the supported languages. For example, we can create a row and access its individual columns in Scala as follows:


Though we can create rows and generate DataFrames from them, we’ll generally read data from files for any serious data analysis problem.

Projections and filters
A projection is a set of rows that match a relational condition. In the context of Spark, a projection can be created using the select() method while a filter can be specified using the where() or filter() methods. Let’s say we want to find all the movies that had a rating of 9 or higher. We can write a query as follows: