3

I have a basic spark job that does a couple of joins. The 3 data frames that get joined are somewhat big, nearly 2 billion records in each of them. I have a spark infrastructure that automatically scales up nodes whenever necessary. It seems like a very simple spark SQL query whose results I write to disk. But the job always gets stuck at 99% when I look at from Spark UI.

Bunch of things I have tried are:

  • Increase the number of executors and executor memory.
  • Use repartition while writing the file.
  • Use the native spark join instead of spark SQL join etc

However, none of these things have worked. It would be great if somebody can share the experience of solving this problem. Thanks in advance.

Davide Spataro
  • 7,319
  • 1
  • 24
  • 36
Krishna Sangeeth
  • 37
  • 1
  • 3

1 Answers1

2

Because of the join operations, all records with the same key are shuffled to the same executor. If you data is skewed, which means that there is one or a few keys which are very dominant in terms of the number of rows. Then this single executor which has to process all these rows. Essentially your Spark job becomes single threaded since this single key needs to be processed by a single thread.

Repartitioning will not help since your join operation will shuffle the data again by hashing the join key. You could try to increase the number of partitions in case of an unlucky hash.

This video explains the problems, and suggests a solution: https://www.youtube.com/watch?v=6zg7NTw-kTQ

Cheers, Fokko

Fokko Driesprong
  • 2,075
  • 19
  • 31