I'm trying to create a new column that creates a string using 4 columns.
df.withColumn("input",
F.lit("".format(F.col('field1'),F.col('field2'),F.col('field3'),F.col('field4'))).show()
However, where I'm trying to insert the column values into the string, it's showing up as field1=Column<'field1'> instead of the actual value. I tried wraping it in F.format_string as well but I'm not getting the actual values.
So what it should be returning is something like this, assuming that each column has a string = VALUE.
2 Answers
You can use format_string function like this:
import pyspark.sql.functions as F
df = df.withColumn(
"input",
F.format_string(
"",
F.col('field1'), F.col('field2'), F.col('field3'), F.col('field4')
)
)
df.show(truncate=False)
#+------+------+------+------+--------------------------------------------------------------+
#|field1|field2|field3|field4|input |
#+------+------+------+------+--------------------------------------------------------------+
#|a |b |c |d |
#+------+------+------+------+--------------------------------------------------------------+
You should use concat, not lit, so something like F.concat(F.lit('), F.col('field1'), F.lit('/'), F.col('field2'))
