Merge resource profiles within a stage ( SPARK-29153).Expose RDD APIs for Stage Level Scheduling ( SPARK-29150).Stage Level Resource configuration and Scheduling ( SPARK-27495).Optimize size of CreateArray/CreateMap to be the size of its children ( SPARK-33544).Prune unnecessary nested fields from window and sort ( SPARK-32059).Prune unnecessary nested fields over cosmetic variations ( SPARK-32163).Prune unnecessary nested fields from repartition-by-expression and join ( SPARK-31736).Prune unnecessary nested fields from aggregate and expand ( SPARK-27217).Prune unnecessary nested fields from generate without project ( SPARK-29721).Rule CoalesceBucketsInJoin ( SPARK-31350).Rule DisableUnnecessaryBucketedScan ( SPARK-32859).Rule UnwrapCastInBinar圜omparison ( SPARK-32858).Rule EliminateAggregateFilter ( SPARK-32540).Rule EliminateNullAwareAntiJoin ( SPARK-32573).Rule ExtractSingleColumnNullAwareAntiJoin ( SPARK-32290).Support subexpression elimination for interpreted predicate ( SPARK-33540).Support subexpression elimination for interpreted expression evaluation ( SPARK-33427).Support subexpression elimination in conditional expressions ( SPARK-33337).Support subexpression elimination in project with whole-stage-codegen ( SPARK-33092).
OUTGUESS 1.3 FULL
Support full outer join in shuffled hash join ( SPARK-32399).Add code-gen for shuffled hash join ( SPARK-32421).Coalesce bucketed tables for shuffled hash join ( SPARK-32286).Preserve hash join (BHJ and SHJ) stream side ordering ( SPARK-32383).Preserve shuffled hash join build side partitioning ( SPARK-32330).Remove shuffle by normalizing output partitioning and sortorder ( SPARK-33399).Remove shuffle by improving reordering join keys ( SPARK-32282).Remove shuffle by preserving output partitioning of broadcast hash join ( SPARK-31869).Push more possible predicates through Join via CNF conversion ( SPARK-31705).Push down filters through expand ( SPARK-33302).Partially push down predicates ( SPARK-32302, SPARK-32352).Remove redundant sorts before repartition nodes ( SPARK-32276).Host-local shuffle data reading without shuffle service ( SPARK-32077).Support ANSI nested bracketed comments ( SPARK-28880).Support column list in INSERT statement ( SPARK-32976).Unify temp view and permanent view behaviors ( SPARK-33138).Add SQL standard command SET TIME ZONE ( SPARK-32272).ANSI mode: new explicit cast syntax rules ( SPARK-33354).ANSI mode: runtime errors instead of returning null ( SPARK-33275).Support char/varchar data type ( SPARK-33480).Experimental node decommissioning for Kubernates and Standalone ( SPARK-20624).Shuffled hash join improvement ( SPARK-32461).Unify create table SQL syntax ( SPARK-31257).We have curated a list of high level changes here, grouped by major modules. You can consult JIRA for the detailed changes.
OUTGUESS 1.3 DOWNLOAD
To download Apache Spark 3.1.1, visit the downloads page. In addition, this release continues to focus on usability, stability, and polish while resolving around 1500 tickets. Other major updates include improved ANSI SQL compliance support, history server support in structured streaming, the general availability (GA) of Kubernetes and node decommissioning in Kubernetes and Standalone. This release adds Python type annotations and Python dependency management support as part of Project Zen. Apache Spark 3.1.1 is the second release of the 3.x line.
0 kommentar(er)
