I am using Databricks and I already have loaded some DataTables.
However, I have a complex SQL query that I want to operate on these data tables, and I wonder if i could avoid translating it in pyspark.
Is that possible?
To give an example:
In SQL:
with table2 as (
select column1, column1
from database.table1
where
start_date <= DATE '2019-03-01' and
end_date >= DATE '2019-03-31' )
In pyspark I would already have table1 loaded but the following does not work because it can not find table1.
query = "(
select column1, column1
from table1
where
start_date <= DATE '2019-03-01' and
end_date >= DATE '2019-03-31' )"
table2 = spark.sql(query)
Thanks
1 Answer
Try giving databasename.tablename instead of tablename in query.
query = "(
select column1, column1
from *database_name.table_name*
where
start_date <= DATE '2019-03-01' and
end_date >= DATE '2019-03-31' )"
If you are using pyspark then it must be
pyspark.sql(query)
