Setting up a Sync

A CloudQuery sync fetches data from a source integration and delivers it to one or more destination integrations. For example, you can fetch data from AWS and deliver it to ClickHouse, or fetch data from GCP and deliver it to BigQuery, Kafka, and Neo4j at once. Learn more about syncs in the Core Concepts section.

Syncs are part of the Data Pipelines feature of CloudQuery Platform. They connect Integrations and Destinations and specify how often the data should be moved.

Prerequisites

A configured integration. See the Integration Guides for provider-specific setup, or the General Integration Setup Guide for any integration.
A destination to sync to. CloudQuery Platform includes a default ClickHouse database destination. Use this destination to benefit from platform features like SQL Console and Asset Inventory.
To add an additional destination, see the General Destination Setup Guide.

Creating a new sync with the default destination

Navigate to Data Pipelines → Integrations and click the Create Integration button. If you have an integration set up, choose Use existing integration. Otherwise leave the Create new integration selected and click the Next button.

Using an existing integration

Click the integration you want to set up the sync for. Then proceed to Select a destination.

Using an existing integration

Create a new integration

Follow the steps in the General Integration Setup Guide. Then proceed to Select a destination.

Select a destination

If you want to sync to the CloudQuery default destination or to another existing destination, leave Use existing integration selected. Use the checkboxes on the left to select the desired destinations and then click Sync selected destinations. Proceed to Configuring sync options.

If you do not want to sync to the CloudQuery default destination and you do not have another destination created yet, switch to Create new destination and follow the General Destination Setup Guide. Then proceed to Configuring sync options.

CloudQuery Platform destination selection screen with checkboxes to choose existing destinations for a sync

Configuring sync options

Give the sync a recognizable name. Set the schedule and allocate resources.

Schedule

You can choose to run syncs Daily, Weekly, Monthly, or set No Schedule to trigger runs manually. Preset schedules run at midnight UTC. The scheduled time is shown below the dropdown.

Sync schedule configuration with Daily, Weekly, Monthly, and No Schedule options

If you want to run syncs at a different frequency or at a different time, use the Advanced option and enter a cron expression. The actual frequency and time of the sync will be explained below the input.

Advanced sync schedule configuration using a cron expression for custom frequency and timing

Allocating resources

To limit or upgrade the sync’s CPU and memory usage, use the advanced options. Keep the defaults unless you experience issues with the sync. For syncs with slow sources or low row counts, you can lower the CPU and memory.

The underlying infrastructure may impose additional CPU and memory limits on syncs.

Running the sync

To run the sync immediately, click Save and run. This starts the sync immediately and continues on the specified schedule.

To save without running immediately, click Schedule (if a schedule is set) or Save (if no schedule is set).

Once the sync is started, you will be redirected to the Sync Runs page.

Sync runs page showing the status and history of sync executions after scheduling

Verify the sync

After a sync run completes, verify that data arrived at your destination.

Check sync status

On the Sync Runs page, verify your sync shows a Completed status. Click on a sync run to see details including:

Total rows synced
Duration
Errors or warnings

For more details on monitoring, see Monitoring Sync Status.

Query the data

If you are using the default ClickHouse destination, open the SQL Console and run a query to confirm data arrived:


-- List all tables created by the sync
SHOW TABLES


-- Count rows in a specific table (replace with your table name)
SELECT count(*) FROM <YOUR_TABLE_NAME>

You can also browse your synced resources in the Asset Inventory, which provides a unified view across all connected integrations.

Troubleshooting

Issue	Cause	Fix
Sync stuck in Running	Source is slow or has many resources	Check the sync logs for progress. Large syncs (e.g., `["*"]` across a large AWS organization) can take significant time. Consider syncing fewer tables.
Sync Failed with authentication error	Integration credentials expired or misconfigured	Go to Data Pipelines → Integrations, edit the integration, and re-run Test Connection. Update credentials if needed.
Sync Completed but no data	Tables list is empty or misconfigured	Check the integration’s YAML configuration to verify the `tables` field includes the tables you want to sync.
Sync Completed with warnings	Partial data due to permission issues	Check the sync logs for specific warnings. Some tables may require additional permissions beyond the base role (e.g., specific AWS services require opt-in).
Sync schedule not triggering	Cron expression invalid or timezone mismatch	Verify the cron expression under Advanced schedule settings. All schedules run in UTC.

Next steps

Monitor sync status for details on tracking sync runs and logs
Browse synced resources in the Asset Inventory
Run advanced queries in the SQL Console
Set up policies to evaluate synced resources against best practices
Configure alerts to get notified when queries detect issues
Use Insights to analyze synced data for security, compliance, and cost findings