Building an AWS Cloud Asset Inventory

What is a Cloud Asset Inventory? #

Why Do You Need a Cloud Asset Inventory? #

• Improved Security: By maintaining a comprehensive inventory of cloud assets, organizations can quickly identify and address security vulnerabilities. This proactive approach helps in mitigating potential threats before they escalate.
• Compliance: Ensuring that all cloud resources comply with regulatory and internal policies is crucial. A cloud asset inventory helps in tracking compliance status and generating audit-ready reports.
• Cost Management: Understanding where and how resources are being utilized allows for better cost management. Organizations can identify underutilized assets, optimize resource allocation, and reduce unnecessary expenditures.
• Operational Efficiency: Streamlining the management of cloud resources enhances operational efficiency. Automated tracking and reporting reduce the manual effort required for asset management, allowing engineers to focus on more strategic tasks.

Why Build a Cloud Asset Inventory with CloudQuery? #

Building an AWS Cloud Asset Inventory #

• CloudQuery

  • This tool helps us sync cloud asset data from AWS into any database, in this case, we are using PostgreSQL.
  • Why: CloudQuery is a super powerful and flexible data movement platform that allows you to sync from any data source into any data source.
• PostgreSQL (DB)

  • This is our database where we’ll store all the collected cloud asset data.
  • Why: PostgreSQL is a strong, open-source relational database. It’s great for handling complex queries and ensuring data integrity, which is exactly what we need for structured data storage and querying.
• dbt (Data Build Tool)

  • dbt is used to transform and model the data stored in PostgreSQL.
  • Why: With DBT, we can apply transformations directly within the data warehouse. This makes our data pipelines more efficient and maintainable, which is a huge win for keeping everything organized and running smoothly.

Prerequisites #

Setting Up Your Environment #

Getting Started With CloudQuery #

• First download the CloudQuery CLI (macOS) (Windows and Linux download instructions)

curl -L <https://github.com/cloudquery/cloudquery/releases/download/[PASTE_VERSION_HERE]/cloudquery_darwin_amd64> -o cloudquery

chmod a+x cloudquery

• Sign up for a CloudQuery Cloud account.
• Sign in with CloudQuery

./cloudquery login

Setting Up a Docker Container for PostgreSQL as the Data Store for AWS Cloud Asset Data #

docker run --name postgres_container \
--restart unless-stopped \
--env POSTGRES_USER=postgres \
--env POSTGRES_PASSWORD=postgres \
--env POSTGRES_HOST=db \
--env POSTGRES_DB=asset_inventory \
--publish 5432:5432 \
--volume pgdata:/var/lib/postgresql/data \
postgres

How to Sync AWS Cloud Assets with PostgreSQL with CloudQuery #

How to pull Cloud Asset data from AWS #

kind: source
spec:
  # Source spec section
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v26.9.0"
  tables: [
 "aws_apigateway_rest_api_stages",
 "aws_apigatewayv2_api_stages",
 "aws_apigatewayv2_api_routes",
 "aws_autoscaling_groups",
 "aws_codebuild_projects",
 "aws_config_configuration_recorders",
 "aws_cloudwatch_alarms",
 "aws_cloudtrail_trail_event_selectors",
 "aws_cloudwatchlogs_metric_filters",
 "aws_cloudfront_distributions",
 "aws_iam_accounts",
 "aws_iam_credential_reports",
 "aws_iam_password_policies",
 "aws_iam_users",
 "aws_ec2_network_acls",
 "aws_ec2_security_groups",
 "aws_efs_access_points",
 "aws_elasticbeanstalk_environments",
 "aws_elbv1_load_balancers",
 "aws_elbv2_load_balancers",
 "aws_rds_clusters",
 "aws_sns_subscriptions",
 "aws_s3_accounts"
  ]
  destinations: ["postgresql"]
  spec:

Setting AWS Environmental Variables #

export AWS_ACCESS_KEY_ID={Your AWS Access Key ID}
export AWS_SECRET_ACCESS_KEY={Your AWS secret access key}
export AWS_SESSION_TOKEN={Your AWS session token}

How to save AWS Cloud Assets in PostgreSQL #

---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v8.0.8"
  spec:
 connection_string: "postgresql://postgres:postgres}@db:5432/${POSTGRES_DB}?sslmode=disable"

cloudquery sync ./config.yml

docker exec -it xxx /bin/bash

psql -U postgres

select * from aws_ec2_instances;

How to Use dbt to Transform AWS Data into a Cloud Asset Inventory #

config:
  send_anonymous_usage_stats: False
  use_colors: True

aws_asset_inventory:
  target: postgres
  outputs:
 postgres:
   type: postgres
   host: "{{ env_var('POSTGRES_HOST') }}"
   user: "{{ env_var('POSTGRES_USER') }}"
   pass: "{{ env_var('POSTGRES_PASSWORD') }}"
   port: 5432
   dbname: "{{ env_var('POSTGRES_DB') }}"
   schema: public
   threads: 1

docker run --platform linux/amd64 --name dbt_container \
  --env POSTGRES_USER=postgres \
  --env POSTGRES_PASSWORD=postgres \
  --env POSTGRES_HOST=db \
  --env POSTGRES_DB=asset_inventory \
  --volume $(pwd)/cloudquery_transformation_aws-asset-inventory_vX.X.X:/usr/app \
  --volume $(pwd)/dbt-profiles.yml:/root/.dbt/profiles.yml \
  ghcr.io/dbt-labs/dbt-postgres:1.8.1 run

Running Grafana to visualize your AWS Cloud Asset Data #

docker run --name grafana_container \
  --restart unless-stopped \
  --platform linux/amd64 \
  --volume grafana:/var/lib/grafana \
  --publish 3000:3000 \
  grafana/grafana

Feeding your AWS Cloud Asset Inventory Data into Grafana for Visualization #

1. Select and Download Pre-Built Dashboards

   1. Visit the CloudQuery Hub and browse through the range of pre-built dashboards.
   2. Since you are working with AWS in this tutorial, select the AWS Asset Inventory visualization pack.
   3. Click “Download Now” and extract the zip file to your project folder.
2. Access Grafana

   1. The Grafana instance launched by our Docker Compose setup should be accessible at http://localhost:3000.
   2. If you haven’t done so already, you may need to set a password for the admin account before proceeding.

      1. Note: The default Grafana username is admin and the default password is admin.
3. Import the Dashboard

   1. In Grafana, open the hamburger menu (three horizontal lines) in the top left corner and select “Dashboards”.
   2. Click the blue “New” button on the top right, then select “Import”.
   3. Navigate to the extracted zip file directory and locate aws_asset_inventory/grafana/asset_inventory.json.
   4. Drag this file into the “Upload dashboard JSON file” area and click “Import”.
   5. This will load your dashboard.
   6. Note: You might need to add the data source in Grafana to connect to your PostgreSQL database.

Conclusion #

• Setting up CloudQuery: You configured CloudQuery to connect to your AWS account and gather detailed asset data.
• Storing Data in PostgreSQL: You set up a PostgreSQL database to store the collected asset data, enabling efficient querying and analysis.
• Transforming Data with DBT: You utilized DBT to apply data transformations, enhancing the quality and usability of your cloud asset inventory.
• Visualizing Data in Grafana: You imported pre-built dashboards into Grafana, allowing you to visualize and monitor your AWS assets dynamically.

Additional Resources #

• Google Cloud Asset Inventory 101
• What is a Cloud Asset Inventory?
• Cloud Asset Inventory with AWS Quicksight
• Build an AWS Cloud Asset Inventory

FAQs #

Code #

CloudQuery #

kind: source
spec:
  # Source spec section
  name: aws
  path: cloudquery/aws
  registry: cloudquery
  version: "v26.9.0"
  tables: [
 "aws_apigateway_rest_api_stages",
 "aws_apigatewayv2_api_stages",
 "aws_apigatewayv2_api_routes",
 "aws_autoscaling_groups",
 "aws_codebuild_projects",
 "aws_config_configuration_recorders",
 "aws_cloudwatch_alarms",
 "aws_cloudtrail_trail_event_selectors",
 "aws_cloudwatchlogs_metric_filters",
 "aws_cloudfront_distributions",
 "aws_iam_accounts",
 "aws_iam_credential_reports",
 "aws_iam_password_policies",
 "aws_iam_users",
 "aws_ec2_network_acls",
 "aws_ec2_security_groups",
 "aws_efs_access_points",
 "aws_elasticbeanstalk_environments",
 "aws_elbv1_load_balancers",
 "aws_elbv2_load_balancers",
 "aws_rds_clusters",
 "aws_sns_subscriptions",
 "aws_s3_accounts"
  ]
  destinations: ["postgresql"]
  spec:
---
kind: destination
spec:
  name: "postgresql"
  path: "cloudquery/postgresql"
  registry: "cloudquery"
  version: "v8.0.8"
  spec:
 connection_string: "postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:5432/${POSTGRES_DB}?sslmode=disable"

DBT #

config:
  send_anonymous_usage_stats: False
  use_colors: True

aws_asset_inventory:
  target: postgres
  outputs:
 postgres:
   type: postgres
   host: "{{ env_var('POSTGRES_HOST') }}"
   user: "{{ env_var('POSTGRES_USER') }}"
   pass: "{{ env_var('POSTGRES_PASSWORD') }}"
   port: 5432
   dbname: "{{ env_var('POSTGRES_DB') }}"
   schema: public
   threads: 1