CloudQuery

The landscape of data integration and movements constantly shifting. Choosing the right ELT or ETL tool depends on the requirements, needs, and resources available. This blog compares the pros and cons of CloudQuery vs Steampipe.

CloudQuery is an open-source, cross-language, high-performance ELT (Extract-Load-Transform) framework powered by Apache Arrow. It is extremely fast and easy to run both locally and in the cloud, it has a CLI-first design, is shipped as a single binary, and doesn’t need any additional services or UI to run.

Funded by Turbot, Steampipe is an open-source ETL (Extract-Transform-Load) framework for auditing cloud and network infrastructure. Steampipe’s major differentiating factor is real-time queries - where the SQL request is translated into live API calls. This has the benefit of always being up-to-date with the latest state but at the cost of significantly slower response times, and a massive increase in API calls. Steampipe is provided as both a single binary and via Turbot’s Pipes (essentially Steampipe-as-a-Service).

Both CloudQuery and Steampipe ship as a single binary with pluggable components. However, their key differences are in how they process data and queries.

CloudQuery extracts the data from the APIs and loads it into an instance of virtually any database/lake/warehouse you choose, where you can then query the data in the native language for that data store.

Where as Steampipe directly translates PostgreSQL queries into API calls, and the result is computed in real-time with the choice to optionally store the results.

Sources and destinations are the bread and butter of data integration solutions. With key differences, pros and cons, for each platform.

CloudQuery and Steampipe have a similar core sets of source connectors, with both focusing on cloud infrastructure and collecting usage, security, and configuration data.

With the Golang SDK for both frameworks being fairly comparable in developer experience, developing new connectors is as straightforward as possible. CloudQuery edges ahead here with its support for connectors developed in any language that supports Apache Arrow and gRPC.

Unlike CloudQuery, Steampipe has limited support for external databases and relies heavily on Postgres integration for storing results.

While CloudQuery has a competitive number of production-ready connectors (source plugins), it prioritizes performance first. This pays off in performance comparisons, where our users and customers have seen more than 50x speed improvements when switching to CloudQuery.

Using the Steampipe CLI is completely free with most plugins, whereas CloudQuery offers a free quota and charges if the number of syncs exceeds that quota.

The cloud version of Steampipe (aka Pipes), charges for the number of Users, Compute time (per second), and Storage capacity used. Whereas CloudQuery’s Cloud offering is still based on the number of rows with the same free quotas as for the CLI, with some additional fees based on data Egress, vCPUs used (per hour), and vRAM used (per hour), with no limits on the number of users.

CloudQuery is a sync-based solution, so as soon as the data is in your database, you won't be charged more. However, as Steampipe is a real-time solution, you're paying for it every time you query your data.

We’re obviously biased, but we think CloudQuery is the clear winner in flexibility, performance, and pricing.

Steampipe does have its advantages (real-time queries being the foremost - though CloudQuery’s event-based syncs definitely challenge that), but the increased load on Source APIs and the significantly worse performance make it difficult to recommend for production use cases beyond simple inspection and troubleshooting tasks.

Ready to get started with CloudQuery? You can try out CloudQuery locally with our quick start guide or explore the CloudQuery Platform (currently in beta) for a more scalable solution.

Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.