Overview

Monitor CloudQuery using any of these methods:

Logging

CloudQuery utilizes structured logging (in plain and JSON formats) which can be analyzed by local tools such as jq, grep and remote aggregations tools like loki, datadog or any other popular log aggregation that supports structured logging.

OpenTelemetry (Preview)

Preview: OpenTelemetry support is in preview. The configuration interface (otel_endpoint, otel_endpoint_insecure) is functional but may change in future releases. We do not currently provide stability guarantees for this feature.

ELT workloads can be long running and sometimes it is necessary to better understand what calls are taking the most time, to optimize those on the integration side, ignore them or split them to a different workload. CloudQuery supports OpenTelemetry traces, metrics and logs out of the box. Configure it using otel_endpoint and otel_endpoint_insecure in your source spec (see Configuration).

To collect OpenTelemetry data you need a backend that supports the OpenTelemetry protocol. For example you can use Jaeger to visualize and analyze traces.

To start Jaeger locally you can use Docker:


docker run -d \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:1.58

and then specify in the source spec the following:


kind: source
spec:
  name: "aws"
  path: "cloudquery/aws"
  registry: "cloudquery"
  version: "v33.20.0"
  tables: ["aws_s3_buckets"]
  destinations: ["postgresql"]
  otel_endpoint: "localhost:4318"
  otel_endpoint_insecure: true # this is only in development when running local jaeger
  spec:

After that you can open http://localhost:16686 and see the traces:

jaeger

In production, it is common to use an OpenTelemetry collector that runs locally or as a gateway to batch the traces and forward it to the final backend. This helps with performance, fault-tolerance and decoupling of the backend in case the tracing backend changes.

CLI-level OpenTelemetry

In addition to per-integration otel_endpoint configuration in source specs, the CLI itself supports global OpenTelemetry via environment variables:

Variable	Description
`OTEL_ENDPOINT`	OpenTelemetry collector endpoint for CLI-level logs and traces
`OTEL_ENDPOINT_INSECURE`	Set to `true` to disable TLS when connecting to the endpoint

When OTEL_ENDPOINT is set, the CLI sends its own logs and traces to the specified collector, independent of any per-integration OpenTelemetry configuration.

Correlating CLI and integration traces

Use the --invocation-id flag to set a consistent identifier across CLI traces and integration traces:


cloudquery sync config.yaml --invocation-id <YOUR_UUID>

This is useful in distributed environments where you need to correlate CLI-level traces with integration-level traces across services. When not specified, a random UUID is generated for each invocation.

For the full list of CLI environment variables, see the environment variables reference.

Next Steps

OpenTelemetry with Datadog - Send traces and metrics to Datadog
OpenTelemetry with Grafana - Visualize metrics in Grafana
Performance Tuning - Use monitoring data to optimize syncs
Telemetry - Configure CLI telemetry settings