List Lambdas Across AWS Accounts and Monitor Health Events Like Recursive Loops

• August 5, 2024

AWS Lambda Functions are a fundamental component of building serverless applications on AWS. However, managing and monitoring these functions across multiple AWS accounts can be complex, especially when dealing with potential issues like recursive loops and health events. This blog post will guide you through using the aws_lambda_functions, aws_health_events, aws_health_event_details and aws_health_affected_entities tables to monitor your AWS Lambda functions and identify potential health issues.

To get started, download the CloudQuery CLI and set up your CloudQuery configuration file by following the AWS Source Plugin documentation. Sync your AWS Lambda functions and health events data using CloudQuery’s AWS Source Plugin to any destination available in our destination plugins. Our most popular destination plugin is PostrgreSQL, so in the query examples below, we’ll use the PostgreSQL Destination Plugin. Ensure that you add aws_lambda_functions, aws_health_events, aws_health_event_details, and aws_health_affected_entities tables to the list of tables being synced. Note that the AWS Health API requires your account to have a business plan, so these tables can only be synced if you do.

Listing Lambda Functions #

The aws_lambda_functions table contains data about your Lambda functions, including configuration, tags, and code details. To get a quick view of your lambda functions, you can use the following query:

SELECT
    account_id,
    arn,
    region,
    configuration->>'FunctionName' AS function_name,
    configuration->>'State' AS function_state,
    configuration->>'role' AS function_role,
    configuration->>'Runtime' AS runtime,
    configuration->>'CodeSize' AS code_size,
    configuration->>'MemorySize' AS memory_size,
    configuration->'EphemeralStorage'->>'Size' as ephemeral_storage_size,
    tags
FROM
    aws_lambda_functions;

To understand how many Lambdas are associated with each account, you can run this query:

SELECT
    account_id,
    COUNT(DISTINCT arn) AS lambda_function_count
FROM
    aws_lambda_functions
GROUP BY
    account_id;

Similarly, you can run the following query to see how many Lambdas are in each AWS region:

SELECT
    region,
    COUNT(DISTINCT arn) AS lambda_function_count
FROM
    aws_lambda_functions
GROUP BY
    region;

Understanding the health status of your lambda functions #

In this section, you will use aws_health_events and its associated tables with the aws_lambda_functions table to identify Lambda functions with health alerts. To list all health events for the Lambda service, you can run this query:

SELECT
    account_id,
    arn,
    region,
    availability_zone,
    start_time,
    end_time,
    status_code
FROM
    aws_health_events
WHERE
    service LIKE '%Lambda%';

To understand what are these events, you need to join them with the aws_health_event_details table using _cq_id and _cq_parent_id

SELECT
    events.account_id,
    events.arn,
    events.region,
    events.availability_zone,
    events.start_time,
    events.end_time,
    events.status_code,
    event_details.event_description,
    event_details.event_metadata
FROM
    aws_health_events events
    INNER JOIN aws_health_event_details event_details ON events._cq_id = event_details._cq_parent_id
WHERE
    service LIKE '%Lambda%';

In the queries above, you listed all of your health events and their details. However, if you want to connect these events with Lambda functions, you need to join the aws_lambda_functions table using the aws_health_affected_entities table.

SELECT
    lambdas.account_id,
    lambdas.arn,
    lambdas.region,
    lambdas.configuration->>'FunctionName' AS function_name,
    lambdas.configuration->>'State' AS function_state,
    lambdas.configuration->>'role' AS function_role,
    lambdas.configuration->>'Runtime' AS runtime,
    lambdas.configuration->>'CodeSize' AS code_size,
    lambdas.configuration->>'MemorySize' AS memory_size,
    lambdas.configuration->'EphemeralStorage'->>'Size' as ephemeral_storage_size,
    lambdas.tags,
    events.arn as event_arn,
    events.start_time,
    events.end_time,
    events.status_code as event_status,
    event_details.event_description,
    event_details.event_metadata,
    affected_entities.status_code as entity_status
FROM
    aws_lambda_functions lambdas
    LEFT JOIN aws_health_affected_entities affected_entities ON lambdas.arn = affected_entities.entity_arn
    LEFT JOIN aws_health_events events ON affected_entities._cq_parent_id = events._cq_id
    LEFT JOIN aws_health_event_details event_details ON events._cq_id = event_details._cq_parent_id;

Now that you have listed all of your Lambda functions and their events, you can aggregate these to get the event count for each function:

WITH lambda_events AS (
    SELECT
        lambdas.account_id,
        lambdas.arn,
        lambdas.region,
        lambdas.configuration->>'FunctionName' AS function_name,
        lambdas.configuration->>'State' AS function_state,
        lambdas.configuration->>'role' AS function_role,
        lambdas.configuration->>'Runtime' AS runtime,
        lambdas.configuration->>'CodeSize' AS code_size,
        lambdas.configuration->>'MemorySize' AS memory_size,
        lambdas.configuration->'EphemeralStorage'->>'Size' as ephemeral_storage_size,
        lambdas.tags,
        events.arn as event_arn,
        events.start_time,
        events.end_time,
        events.status_code as event_status,
        event_details.event_description,
        event_details.event_metadata,
        affected_entities.status_code as entity_status
    FROM
        aws_lambda_functions lambdas
        LEFT JOIN aws_health_affected_entities affected_entities ON lambdas.arn = affected_entities.entity_arn
        LEFT JOIN aws_health_events events ON affected_entities._cq_parent_id = events._cq_id
        LEFT JOIN aws_health_event_details event_details ON events._cq_id = event_details._cq_parent_id
)
SELECT
    account_id,
    arn,
    region,
    COUNT(DISTINCT event_arn) as event_count
FROM
    lambda_events
GROUP BY
    account_id, arn, region;

Another option is to look for a specific event with specific keywords. For example, if you have enabled recursive loop detection in your AWS account, you can look for Lambda functions that triggered this event:

WITH lambda_events AS (
    SELECT
        lambdas.account_id,
        lambdas.arn,
        lambdas.region,
        lambdas.configuration->>'FunctionName' AS function_name,
        lambdas.configuration->>'State' AS function_state,
        lambdas.configuration->>'role' AS function_role,
        lambdas.configuration->>'Runtime' AS runtime,
        lambdas.configuration->>'CodeSize' AS code_size,
        lambdas.configuration->>'MemorySize' AS memory_size,
        lambdas.configuration->'EphemeralStorage'->>'Size' as ephemeral_storage_size,
        lambdas.tags,
        events.arn as event_arn,
        events.start_time,
        events.end_time,
        events.status_code as event_status,
        event_details.event_description,
        event_details.event_metadata,
        affected_entities.status_code as entity_status
    FROM
        aws_lambda_functions lambdas
        LEFT JOIN aws_health_affected_entities affected_entities ON lambdas.arn = affected_entities.entity_arn
        LEFT JOIN aws_health_events events ON affected_entities._cq_parent_id = events._cq_id
        LEFT JOIN aws_health_event_details event_details ON events._cq_id = event_details._cq_parent_id
)
SELECT
    *
FROM
    lambda_events
WHERE
    event_description->>'latestDescription' LIKE '%recursive loop%';

Using CloudQuery to monitor your AWS Lambda functions and health events across multiple accounts gives you a clear view of your serverless applications’ health and performance. CloudQuery helps you spot potential issues like recursive loops and health alerts. Start syncing your AWS Lambda functions with CloudQuery today and boost your monitoring capabilities.

Ready to enhance your AWS Lambda function monitoring? Try CloudQuery today and gain detailed insights into your serverless applications. Contact us or join our Community to connect with other users and our engineering team, and explore the full potential of CloudQuery.

Written by Jonathan Sarig

Jonathan is a software engineer at CloudQuery with a particular interest in machine learning. He primarily works in golang but also has experience in (and a passion for) Rust.

On this page

List Lambdas Across AWS Accounts and Monitor Health Events Like Recursive Loops

Listing Lambda Functions #

Understanding the health status of your lambda functions #

Written by Jonathan Sarig

Related posts

How to Identify and Address Vulnerabilities in Jamf Devices