Having this dataframe I am getting Column is not iterable when I try to groupBy and getting max:
linesWithSparkDF
+---+-----+
| id|cycle|
+---+-----+
| 31| 26|
| 31| 28|
| 31| 29|
| 31| 97|
| 31| 98|
| 31| 100|
| 31| 101|
| 31| 111|
| 31| 112|
| 31| 113|
+---+-----+
only showing top 10 rows
ipython-input-41-373452512490> in runlgmodel2(model, data)
65 linesWithSparkDF.show(10)
66
---> 67 linesWithSparkGDF = linesWithSparkDF.groupBy(col("id")).agg(max(col("cycle")))
68 print "linesWithSparkGDF"
69
/usr/hdp/current/spark-client/python/pyspark/sql/column.py in __iter__(self)
241
242 def __iter__(self):
--> 243 raise TypeError("Column is not iterable")
244
245 # string methods
TypeError: Column is not iterable
4 Answers
It's because, you've overwritten the max definition provided by apache-spark, it was easy to spot because max was expecting an iterable.
To fix this, you can use a different syntax, and it should work.
inesWithSparkGDF = linesWithSparkDF.groupBy(col("id")).agg({"cycle": "max"})
or alternatively
from pyspark.sql.functions import max as sparkMax
linesWithSparkGDF = linesWithSparkDF.groupBy(col("id")).agg(sparkMax(col("cycle")))
The idiomatic style for avoiding this problem -- which are unfortunate namespace collisions between some Spark SQL function names and Python built-in function names -- is to import the Spark SQL functions module like this:
from pyspark.sql import functions as F
# USAGE: F.col(), F.max(), F.someFunc(), ...
Then, using the OP's example, you'd simply apply F like this:
linesWithSparkGDF = linesWithSparkDF.groupBy(F.col("id")) \
.agg(F.max(F.col("cycle")))
In practice, this is how the problem is avoided idiomatically. =:)
I know the question is old but this might help someone.
First import the following :
from pyspark.sql import functions as F
Then
linesWithSparkGDF = linesWithSparkDF.groupBy(col("id")).agg(F.max(col("cycle")))
I faced the similar issue, although error looks mischievous but we can resolve the same to check if we missed the following import-
from pyspark.sql.functions import *
this will get the required functions to aggregate the data if datatypes of columns are right. I fixed the similar issue by adding the required import, so don't forget that to check...
