How do I transpose a pyspark dataframe?

Question

How do I transpose this pyspark data frame such as:

Using python but in pyspark dataframe – emma19 Feb 18 '21 at 21:04 — emma19, Feb 18 '21 at 21:04

score 0 · Accepted Answer · answered Feb 18 '21 at 22:04

0

It is almost same as pandas dataframe

Let the dataframe is df

pivotdf= df.groupBy("Data_Type").pivot("Parameter").sum("Value")
pivotdf.show()

Here we are pivoting the column Parameter by grouping the column Data_Type

answered Feb 18 '21 at 22:04

Midhilesh Momidi

Thank you for the reply. After that I was actually trying to filter the pyspark dataframe by taking like rows from 40 to 50. The limit function will only take like first n rows. Is there a way to chunk the dataframe with a range? Example: like this df=df.limit[40:50] – emma19 Feb 19 '21 at 19:00
@emma19 As you need specific range `df.collect()[2:5]` but it will not give the output as dataframe format If you are looking for a dataframe format then try like this `df.toPandas()[2:5]` but if I am not wrong this will lose the distributed properties of spark but gives nice formatting – Midhilesh Momidi Feb 19 '21 at 21:34

1 Answers1