Pushdown is not possible for which Spark component?

• A. Spark DataFrames
• B. Spark UDFs
• C. Spark SQL
• D. Spark Streaming

Answer: (B)

Pushdown is about moving as much of the work as possible to the data source or to an earlier stage in the data processing chain so that less data is read and moved into Spark. Spark DataFrames and Spark SQL benefit from this because the Catalyst optimizer and the data source interfaces can translate filters and projections into data-source operations, letting the source read only the necessary data. Spark Streaming similarly relies on the same idea when possible, pushing compatible operations to streaming sources.

User-defined functions, however, are arbitrary code defined by the user and run inside Spark’s execution engine after the data has been loaded. Since the data source has no knowledge of what the UDF does, it cannot push the UDF’s logic down to the source for early evaluation. As a result, UDFs prevent pushdown from being applied.