I am trying to setup a bigquery data transfer configuration using terraform. I am using my personal gcp account. I have a setup of terraform on my laptop so that terraform and gcp can work together.
Trying this below code in main.tf,
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "4.18.0"
}
}
}
provider "google" {
# Configuration options
project="gcp-project-100"
region="us-central1"
zone="us-central1-a"
credentials = "keys.json"
}
data "google_project" "project" {
}
resource "google_project_iam_member" "permissions" {
project = data.google_project.project.project_id
role = "roles/iam.serviceAccountShortTermTokenMinter"
member = "serviceAccount:service-${data.google_project.project.number}@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com"
}
resource "google_bigquery_data_transfer_config" "query_config" {
depends_on = [google_project_iam_member.permissions]
display_name = "my-query"
location = "US"
data_source_id = "scheduled_query"
schedule = "every wednesday 09:30"
service_account_name = "service-${data.google_project.project.number}@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com"
destination_dataset_id = "practice"
params = {
destination_table_name_template = "test_gsod"
write_disposition = "WRITE_TRUNCATE"
query = "select station_number , year , month,day, mean_temp,mean_dew_point ,mean_visibility from `bigquery-public-data.samples.gsod`"
}
}
terraform apply is failing with below details
google_bigquery_data_transfer_config.query_config: Creating... ╷ │ Error: Error creating Config: googleapi: Error 403: The caller does not have permission │ │ with google_bigquery_data_transfer_config.query_config, │ on main.tf line 27, in resource "google_bigquery_data_transfer_config" "query_config": │ 27: resource "google_bigquery_data_transfer_config" "query_config" {
Can someone please help me , how to do this setup properly.
1 Answer
The issue is resolved now. I have used below piece of code along with bigquery admin role for my terraform service account.
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "4.18.0"
}
}
}
provider "google" {
# Configuration options
project="gcp-project-100"
region="us-central1"
zone="us-central1-a"
credentials = "keys.json"
}
resource "google_bigquery_data_transfer_config" "query_config" {
display_name = "my-query"
location = "US"
data_source_id = "scheduled_query"
schedule = "every 15 mins"
destination_dataset_id = "practice"
params = {
destination_table_name_template = "test_gsod"
write_disposition = "WRITE_TRUNCATE"
query = "select station_number , year , month,day, mean_temp,mean_dew_point ,mean_visibility from `bigquery-public-data.samples.gsod`"
}
}
Now it is working fine. Thanks.
