PluginsGoogle CloudTasksGcsDownloads

Downloads​Downloads

Downloads Downloads

yaml
type: "io.kestra.plugin.gcp.gcs.Downloads"

Download multiple files from a GCS bucket.

Examples

Download a list of files and move it to an archive folders

yaml
id: gcp_gcs_downloads
namespace: company.team

tasks:
  - id: downloads
    type: io.kestra.plugin.gcp.gcs.Downloads
    from: gs://my-bucket/kestra/files/
    action: MOVE
    moveDirectory: gs://my-bucket/kestra/archive/

Properties

action

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Possible Values:
    • MOVE
    • DELETE
    • NONE

The action to perform on the retrieved files. If using 'NONE' make sure to handle the files inside your flow to avoid infinite triggering.

from

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The directory to list

allVersions

  • Type: boolean
  • Dynamic: ✔️
  • Required: ❌

If set to true, lists all versions of a blob. The default is false.

listingType

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Default: DIRECTORY
  • Possible Values:
    • RECURSIVE
    • DIRECTORY

The listing type you want (like directory or recursive)

if DIRECTORY, will only list objects in the specified directory if RECURSIVE, will list objects in the specified directory recursively Default value is DIRECTORY When using RECURSIVE value, be careful to move your files to a location not in the from scope

moveDirectory

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The destination directory for MOVE action.

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP project ID.

regExp

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

A regexp to filter on full path

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account key.

Outputs

blobs

  • Type: array
  • SubType: Blob
  • Required: ❌

The bucket of the downloaded file

outputFiles

  • Type: object
  • SubType: string
  • Required: ❌

The downloaded files as a map of from/to URIs.

Definitions

io.kestra.plugin.gcp.gcs.models.Blob

Properties

bucket
  • Type: string
  • Dynamic: ❓
  • Required: ❓
cacheControl
  • Type: string
  • Dynamic: ❓
  • Required: ❓
componentCount
  • Type: integer
  • Dynamic: ❓
  • Required: ❓
contentDisposition
  • Type: string
  • Dynamic: ❓
  • Required: ❓
contentEncoding
  • Type: string
  • Dynamic: ❓
  • Required: ❓
contentLanguage
  • Type: string
  • Dynamic: ❓
  • Required: ❓
contentType
  • Type: string
  • Dynamic: ❓
  • Required: ❓
crc32c
  • Type: string
  • Dynamic: ❓
  • Required: ❓
createTime
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: date-time
customTime
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: date-time
deleteTime
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: date-time
etag
  • Type: string
  • Dynamic: ❓
  • Required: ❓
eventBasedHold
  • Type: boolean
  • Dynamic: ❓
  • Required: ❓
generatedId
  • Type: string
  • Dynamic: ❓
  • Required: ❓
isDirectory
  • Type: boolean
  • Dynamic: ❓
  • Required: ❓
kmsKeyName
  • Type: string
  • Dynamic: ❓
  • Required: ❓
md5
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Type: string
  • Dynamic: ❓
  • Required: ❓
metaGeneration
  • Type: integer
  • Dynamic: ❓
  • Required: ❓
metadata
  • Type: object
  • SubType: string
  • Dynamic: ❓
  • Required: ❓
name
  • Type: string
  • Dynamic: ❓
  • Required: ❓
retentionExpirationTime
  • Type: integer
  • Dynamic: ❓
  • Required: ❓
  • Type: string
  • Dynamic: ❓
  • Required: ❓
size
  • Type: integer
  • Dynamic: ❓
  • Required: ❓
temporaryHold
  • Type: boolean
  • Dynamic: ❓
  • Required: ❓
timeStorageClassUpdated
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: date-time
updateTime
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: date-time
uri
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: uri

Was this page helpful?