CloudQuery News
Announcing the CloudQuery BigQuery Destination Integration
Introduction #
Hot on the heels of the Snowflake destination integration, today we are introducing another CloudQuery destination plugin: the BigQuery destination plugin. BigQuery is a cloud-based data warehousing and analytics platform offered by Google Cloud that allows users to store and query large datasets quickly and efficiently. With the CloudQuery BigQuery destination plugin, you can now consolidate your organization's data from across cloud platforms into BigQuery.
Use Cases #
Cloud Infrastructure data lake and data warehouse #
CloudQuery brings all your cloud infrastructure configuration data to the same place, consolidating your security and infrastructure data and enabling new insights. If your organization already uses BigQuery for analysis, these in-house datasets can now also be combined with CloudQuery datasets to unlock insights that were previously impossible without several intermediate steps.
Historical data #
Maintaining historical data is a common use case for data warehouses. By using BigQuery as a destination, you can now cost-effectively store all your cloud infrastructure data in a data warehouse for long term storage, analysis and investigation use-cases.
Scheduled Queries #
By combining the CloudQuery data in BigQuery with scheduled queries, further business insights and security reports can be derived from the data as part of a completely serverless, cloud-based deployment.
Syncing data #
Syncing data to BigQuery can be done as with any other destination plugin, so check out our Quickstart and BigQuery plugin documentation for instructions on how to get started.
Once data is synced you can start querying either with the native BigQuery interface, code or with any of the many BI tools that support BigQuery. Here is the "Hello World" of cloud infrastructure queries, a query that finds all AWS S3 buckets that are permitted to be public:
And yes, the BigQuery destination is not just for GCP data! As the example above demonstrates, data from all major cloud providers can now be synced to BigQuery, allowing a holistic view of your cloud assets.
Summary #
This is the initial BigQuery destination plugin release. More improvements are coming, and I'd love to hear your use cases and ideas on how to leverage it. Feel free to open an issue on GitHub or join the CloudQuery Community to share your ideas.
Ready to dive deeper? Contact CloudQuery here or join the CloudQuery Community to connect with other users and experts. You can also try out CloudQuery locally with our quick start guide or explore the CloudQuery Platform (currently in beta) for a more scalable solution.
Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.
Written by Herman Schaaf
Herman is the Director of Engineering at CloudQuery and an Apache Arrow contributor. A polyglot with a preference for Go and Python, he has spoken at QCon London and Data Council New York.