Environment variables

CLI environment variables

The CloudQuery CLI reads the following environment variables to control its behavior:

For details on OpenTelemetry integration, see the monitoring overview. For telemetry details, see the telemetry page.

Configuration file variable substitution

CloudQuery configuration .yml files support substitution of values from environment variables. Use this to keep sensitive data (like passwords & tokens) or variable data (that you want to change without touching CloudQuery configuration) outside the configuration file and load them from environment variables at run-time.

Environment variable substitution example

Inside postgresql.yml:

kind: "destination"
spec:
  name: "postgresql"
  spec:
    connection_string: ${PG_CONNECTION_STRING}

CloudQuery sources PG_CONNECTION_STRING from the environment and substitutes it before processing.

File variable substitution example

Inside postgresql.yml:

kind: "destination"
spec:
  name: "postgresql"
  spec:
    connection_string: ${file:./path/to/secret/file}

CloudQuery reads the local path ./path/to/secret/file and substitutes the file contents before processing.

Environment variables with multi-line JSON

Multi-line JSON, such as those required by the service account key for the GCP integration, can be imported by using pipe ’|’ operator. The substitution should be in the next line and it should be indented by a single tab before. You don’t need to escape any characters while passing the variable.

Inside gcp.yml:

kind: "source"
spec:
  name: "gcp"
  spec:
    service_account_key_json: |
      ${GCP_SERVICE_ACCOUNT_KEY_JSON}

JSON files in older versions

If the file or environment variable being substituted in contains JSON, import it as-is. If you’re using CloudQuery version 3.5.0 or prior, wrap it in single quotes and content should be escaped with newlines removed.

kind: "destination"
spec:
  name: "bigquery"
  spec:
    service_account_key_json: '${file:./path/to/secret/file.json}' # single quotes only for CLI versions 3.5.0 or prior

Time variable substitution example

CloudQuery configuration files support value substitution using both relative and fixed timestamps. Relative timestamps can be specified as now, x seconds [ago|from now], x minutes [ago|from now], x hours [ago|from now], x days [ago|from now], or until. Fixed timestamps follow the RFC3339 (e.g. 2025-03-21T08:53:19+00:00) or dateOnly formats (e.g. 2025-01-01)

Inside aws.yml:

kind: source
spec:
  # Source spec section
  name: aws
  spec:
    table_options:
      aws_cloudtrail_events:
        lookup_events:
          - start_time: ${time:5 days ago}
            end_time: ${time:now}
      aws_cloudwatch_metrics:
        - list_metrics:
            namespace: AWS/RDS
            metric_name: DatabaseConnections
          get_metric_statistics:
            - start_time: ${time:2022-01-01} # dateOnly format
              end_time: ${time:2022-04-02T15:04:05Z} # RFC3339 format

For the lookup_events option, CloudQuery replaces end_time with the current time and start_time with the time 5 days ago. For the get_metric_statistics option end_time and start_time will be replaced by default with RFC3339 formatted value of the fixed time.

You can also modify the output format by using the pipe operator | followed by your desired format. The time format should follow Go’s time layout specification: https://go.dev/src/time/format.go. For example, to display the date in YYYY-MM-DD format, use: ${time:now|2006-01-02}.

Next Steps

• Configuration Guide - Set up source and destination configurations
• Security - Best practices for managing secrets
• Deployment Options - Use environment variables in production deployments