How to access SparkContext from SparkSession instance?

Question

I am importing SparkSession as follows in PySpark:

from pyspark.sql import SparkSession

Then I create SparkSession:

spark = SparkSession.builder.appName("test").getOrCreate()

and try to access SparkContext:

spark.SparkContext.broadcast(...)

However, I get an error that SparkContext does not exist. How can I access it in order to set broadcast variables?

Set pyspark as environment variable [help] (https://stackoverflow.com/questions/23256536/importing-pyspark-in-python-shell) — Morse, Mar 12 '18 at 20:19
@Prateek: That's not the question I am asking how to access SparkContext using `spark`, which is the instance of SparkSession. — Markus, Mar 12 '18 at 20:24
`from pyspark import SparkContext as sc` if it doesnt work you have not set pyspark in environement variable OR spark server is not running — Morse, Mar 12 '18 at 20:33
@Prateek: No, it will not work, because it should be related to the created SparkSession. I solved this problem myself just by `spark.sparkContext.broadcast(...)` instead of `spark.SparkContext.broadcast(...)` — Markus, Mar 12 '18 at 20:36

score 46 · Accepted Answer · answered Mar 12 '18 at 20:38

46

You almost got it right, it's lowercase s at the beginning:

>>> spark.sparkContext
<SparkContext master=local[*] appName=PySparkShell>

answered Mar 12 '18 at 20:38

Roberto Congiu

score 5 · Answer 2 · answered Jul 27 '20 at 16:01

5

Asumming you have a spark session

spark_session = SparkSession \
    .builder \
    .enableHiveSupport() \
    .getOrCreate()

Spark Context can be inferred using

spark_context = spark_session._sc

or

spark_context = spark_session.sparkContext

answered Jul 27 '20 at 16:01

2 Answers2