What are the new features of Apache Spark 2.4, which will be released in 2018?

This article is from the Apache Spark Meetup held at Adobe Systems Inc on September 19, 2018.

The upcoming Apache Spark 2.4 release is the fifth in the 2.x series. This article provides an overview of the key features and enhancements in Apache Spark 2.4.

• The new scheduling model (Barrier Scheduling) enables users to properly embed distributed deep learning training into Spark stages to simplify the distributed training workflow.
• Added 35 higher-order functions for array/map operations in Spark SQL.
• Added a new native AVRO data source based on Databricks' spark-avro module.
• PySpark also introduces eager evaluation mode for all operations for teaching and debuggability.
• Spark on K8S supports PySpark and R, and supports client-mode.
• Various enhancements to Structured Streaming. For example, stateful operators in continuous processing.
• Various performance improvements to built-in data sources. For example, Parquet nested schema pruning.
• Support for Scala 2.12.

Click on Shishuo.com to download this PPT.

Summarize

The above is what I introduced to you about the new features of Apache Spark 2.4, which will be launched in 2018. I hope it will be helpful to you. If you have any questions, please leave me a message and I will reply to you in time. I would also like to thank everyone for their support of the 123WORDPRESS.COM website!