Skip to Content
CLICore ConceptsIntegrations

Integrations

Using CloudQuery Platform? See Platform Integrations for how integrations work in the managed platform, including the default ClickHouse destination and multi-destination support.

CloudQuery has an integration-based architecture, with integrations communicating over gRPC. An integration can be a source, destination, or transformer.

  • Source integration - Extracts and transforms configuration from cloud providers, SaaS apps, and other APIs (Available source integrations).
  • Destination integration - Writes data from source integrations to databases, message queues, and storage (Available destination integrations).
  • Transformer integration - Optional component that modifies data in flight between a source and destination, such as renaming tables, obfuscating sensitive columns, or flattening JSON (Available transformer integrations).

All integrations are split into official (maintained by CloudQuery) and community. You can also build your own integration.

  Sources                          Destinations
┌──────────┐                     ┌──────────────┐
│   AWS    │──┐               ┌──│  PostgreSQL  │
├──────────┤  │  ┌──────────┐ │  ├──────────────┤
│   GCP    │──┼──│ CLI sync │─┼──│   BigQuery   │
├──────────┤  │  └──────────┘ │  ├──────────────┤
│  Azure   │──┤               │  │  Snowflake   │
├──────────┤  │               │  ├──────────────┤
│  GitHub  │──┘               └──│   S3 / MySQL │
├──────────┤                     └──────────────┘
│   70+    │
│   more   │
└──────────┘

Source Integration

The core responsibilities of a source integration:

  • Define the schema (tables).
  • Authenticate with the supported APIs, SaaS services and/or cloud providers.
  • Extract data from the supported APIs and transform it into the defined schema.
  • Send the data via protobuf to the CLI for further processing and storage at the defined destination integrations.
IntegrationDescription
AWSEC2, S3, IAM, Lambda, RDS, and 300+ other AWS services
GCPCompute, Storage, IAM, BigQuery, GKE, and more
AzureVMs, Storage, Active Directory, AKS, and more
GitHubRepositories, teams, members, actions, and security advisories
KubernetesPods, deployments, services, nodes, and cluster configuration
CloudflareDNS, WAF, workers, and zone configuration
GitLabProjects, pipelines, merge requests, and group members
OracleCompute, networking, storage, and identity resources

Browse all source integrations on the CloudQuery Hub. For the full source configuration reference, see Source Integrations.

Destination Integration

The core responsibilities of a destination integration:

  • Authenticate with the destination (such as database, message queue, storage).
  • Auto-migrate the schemas defined by the source integrations.
  • Save each incoming object in the appropriate table.
DestinationUse case
PostgreSQLRelational queries, existing data pipelines
S3Data lake storage, archival
BigQueryLarge-scale analytics in GCP
MySQLRelational database, application backends
Neo4jGraph-based infrastructure analysis
SnowflakeCloud data warehouse analytics

Browse all destination integrations on the CloudQuery Hub. See Configuration Reference and learn more about CloudQuery configuration.

Transformer Integration

The core responsibilities of a transformer integration:

  • Receive records from the source integration via the CLI.
  • Apply configured transformations to each record (renaming, filtering, obfuscation, etc.).
  • Deliver the transformed records to the destination integration.

Transformers are optional - if none are configured, data flows directly from source to destination. See Transformer Integrations for configuration details and Transformations for an overview of all available transformations.

Next Steps

Last updated on
ArchitectureSyncs