Product News
CloudQuery Product Updates #13
This has been another exciting month at CloudQuery. There are new sources, metrics and destinations for you to try as well as a brand new feature for our cloud product.
New Beta Feature: CloudQuery Platform #
You can now automate syncs from our cloud using either the command-line or UI without deploying your own infrastructure. This feature, CloudQuery Platform, is now in Open Beta. You can try it for the duration of the beta testing. Read how to get started in our feature announcement.
Pricing Changes #
On March 1, we announced pricing changes for CloudQuery plugins. Read more about this in our announcement. To support those changes and the overall development of the product, some tables in various sources are now premium. This affects primarily the AWS, GCP and Azure plugins - starting with the latest major version of each plugin, the number of free tables is lower.
New Sources #
First, we have released some brand-new plugins enabling you to grab data from even more places than before. New sources include:
- BigQuery with support for custom queries and incremental syncs
- S3 Source - read parquet files from S3
Source Plugin Updates #
AWS #
A new major version has been released. This month, we have extended support for metrics from DynamoDB, CloudWatch, network rules (Route53) and AWS GuardDuty. This enables CloudQuery users to add more data to their Cloud Security Posture Management system (CSPM) without having to write code or transform data.
Read about how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.
New Tables
aws_dynamodb_table_resource_policiesaws_dynamodb_table_stream_resource_policies
Other changes
- Added support AWS AppConfig Profile tags
- Added support for AWS CloudWatch Metric Stream tags
- Added support for Route53 Resolver Rules and Endpoint tags
- Added support for resolving tags for AWS RDS DBSubnetGroup and OptionGroup
- Add
table_optionsupport foraws_guardduty_detectorsandaws_guardduty_detector_findings - Add incremental table support for
aws_guardduty_detector_findings
GCP #
A new major version has been released. This month, we have added support for GCP Identity Access Management (IAM) tables as well as the global address tables
New tables
gcp_iam_organizational_rolesgcp_iam_predefined_rolesgcp_compute_global_addresses
Oracle #
New environment variable is available for configuration:
private_key. This is an alternative to providing a path to the file with the private key and makes it easier to run Oracle syncs in production and using the CloudQuery Platform.Destinations #
- CrateDB can be now used with PostgreSQL destination plugin.
- PostgreSQL breaking change: The plugin now supports decimal types for column. Previously, decimal types were stored as string.
- S3 Destination now supports
SYNC_IDas a path variable.
SDK Improvements #
Sync ID is now passed to plugins during their Init phase. This means it is available for both source and destination plugins and can be used for logging, or configuring the destination parameters.
Other changes #
New Docker image with CloudQuery CLI based on Ubuntu makes it possible to run syncs with DuckDB plugin in a containerized environment.
In case you missed it… #
In Pulling Stock Market Data with CloudQuery, Tim goes deep into SDK and walks you through the process of creating a new CloudQuery plugin to pull the delayed market data from London Stock Exchange
Yevgeny explains how AWS Config pricing works and how you can get the same value 30 times cheaper with CloudQuery in Understanding AWS Config Cost.
We have launched the CloudQuery Platform in beta last month so you can now build a CloudQuery powered CSPM completely without deploying any infrastructure. Read how in Building a serverless open source CSPM powered by CloudQuery.
Yevgeny walks you through counting words in post titles on HackerNews: Building Local and serverless ELT Pipelines with CloudQuery and MotherDuck
Feedback #
We’re looking for feedback to help us improve our CloudQuery Platform. Participate in user research and get a 50 USD Amazon gift card! Book a research call here.
Ready to get started with CloudQuery? You can try out CloudQuery locally with our quick start guide or explore the CloudQuery Platform (currently in beta) for a more scalable solution.
Got feedback or suggestions? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.
Written by Michal Brutvan
Michal is CloudQuery's senior product manager and has responsibility for new features and CloudQuery's product roadmap. He has had a wealth of product ownership roles and prior to that, worked as a software engineer.
