StorageWrite

type: "io.kestra.plugin.gcp.bigquery.StorageWrite"

Load an kestra internal storage file on bigquery using BigQuery Storage API

Examples

id: gcp_bq_storage_write
namespace: company.team

tasks:
  - id: read_data
    type: io.kestra.plugin.core.http.Download
    uri: https://dummyjson.com/products/1

  - id: storage_write
    type: io.kestra.plugin.gcp.bigquery.StorageWrite
    from: "{{ outputs.read_data.uri }}"
    destinationTable: "my_project.my_dataset.my_table"
    writeStreamType: DEFAULT

Properties

bufferSize

• Type: integer
• Dynamic: ❌
• Required: ✔️
• Default: 1000

The number of records to send on each query

destinationTable

• Type: string
• Dynamic: ✔️
• Required: ✔️
• Min length: 1

The table where to load data

The table must be created before.

writeStreamType

• Type: string
• Dynamic: ❌
• Required: ✔️
• Default: DEFAULT
• Possible Values:
  • DEFAULT
  • COMMITTED
  • PENDING

The type of write stream to use

from

• Type: string
• Dynamic: ✔️
• Required: ❌

The fully-qualified URIs that point to source data

location

• Type: string
• Dynamic: ✔️
• Required: ❌

The geographic location where the dataset should reside

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP project ID.

scopes

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

serviceAccount

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP service account key.

Outputs

commitTime

• Type: string
• Required: ❌
• Format: date-time

Commit time reported by BigQuery, only on PENDING writeStreamType

rows

• Type: integer
• Required: ❌

Rows count

rowsCount

• Type: integer
• Required: ❌

Rows count reported by BigQuery, only on PENDING writeStreamType

Definitions

Metrics

rows

• Type: counter

rows_count

• Type: counter