AWS
Tutorials
How to Visualize CloudQuery Data with AWS QuickSight
In this guide, we will walk you through how to set up CloudQuery to build your cloud asset inventory in PostgreSQL and connect it to AWS QuickSight for visualization, monitoring and reporting.
General Architecture #
- ETL (Extract-Transform-Load) ingestion layer: CloudQuery
- Datastore: PostgreSQL
- Data Visualization and Exploration Platform: AWS QuickSight
What you will get #
- Raw SQL access to all your cloud asset inventory to create views or explore any questions or connection between resources.
- Multi-Cloud Asset Inventory: Ingest configuration from all your clouds to a single data store with a unified structure.
- Avoid yet-another-dashboard fatigue: Reuse your existing AWS QuickSight setup to build a cloud asset inventory. AWS QuickSight is not cheap but if you already use it won’t have additional cost (as the cost is per user) and you will re-use your current permissions and workflows.
Walkthrough #
Step 1: Install or Deploy CloudQuery #
If it’s your first time using CloudQuery we suggest you first run it locally to get familiar with the tool, take a look at our quickstart guide and AWS source plugin.
If you are already familiar with CloudQuery, take a look at how to deploy it to AWS on Amazon Aurora and EKS at here.
Step 2: Connecting AWS QuickSight to PostgreSQL #
You can connect QuickSight to a private PostgreSQL (RDS Aurora) if you are on the enterprise edition. See full walk through. If not, you can also make the Aurora publicly accessible but it is highly advised to only allow connections from QuickSight IP Addresses .
Click Create New Dataset and choose PostgresSQL (In this tutorial we will connect to publicly accessible RDS with authorized QuickSight IP Address) and fill-in the connection details:
Step 3: Visualize the Data #
Choose the table you want to visualize, in this case we will choose the
aws_resources view which you need to create. To create or update the view, run this query (part of the CloudQuery AWS Source plugin source code) on your database. This will allow you to visualize all resources across all accounts and regions.- Now visualize away with QuickSight! Here is a small example of global filterable resources we created pretty quickly
You can reuse QuickSight to export/share those dashboards as well as create alerts!
Summary #
In this post we showed you how to build an open-source cloud asset inventory with CloudQuery as the ETL (Extract-Transform-Load) / data-ingestion layer and QuickSight as the visualization and monitoring platforms. This approach eliminates the yet-another-dashboard fatigue and gives you the ability to pick the best-in-class visualization tools and/or reuse your current stack.
Ready to get started with CloudQuery? You can try out CloudQuery locally with our quick start guide or explore the CloudQuery Platform (currently in beta) for a more scalable solution.
Want help getting started? Join the CloudQuery community to connect with other users and experts, or message our team directly here if you have any questions.
Written by Yevgeny Pats
Yevgeny Pats is the Co-Founder & CEO at CloudQuery. Prior to establishing CloudQuery, he successfully founded and exited other startups. He has a background in software engineering and cybersecurity.
