What is the difference between Sort by in Spark SQL vs Sort of Dataset API vs Order by in Spark SQL?

Question

1. Sort by Spark SQL is resulting in Narrow dependency.
2. Sort of Dataset API and order by of Spark SQL is resulting in Wide Dependency.

Answer 1

There are two different things here:

• In general Spark uses sort as an alias for orderBy - What is the difference between sort and orderBy functions in Spark

• Hive has SORT BY clause, which sorts data locally per partition - such operation is called sortWithinPartitions in Spark.