<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pubudu Jayawardana on Medium]]></title>
        <description><![CDATA[Stories by Pubudu Jayawardana on Medium]]></description>
        <link>https://medium.com/@pubudusj?source=rss-35c84ac8c967------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Rh7I6YKtek3UG1Rz3JAYdA.jpeg</url>
            <title>Stories by Pubudu Jayawardana on Medium</title>
            <link>https://medium.com/@pubudusj?source=rss-35c84ac8c967------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:58:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pubudusj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Scan user uploaded files using AWS Lambda MicroVM and ClamAV]]></title>
            <link>https://medium.com/@pubudusj/scan-user-uploaded-files-using-aws-lambda-microvm-and-clamav-499588962564?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/499588962564</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[microvm]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 22:28:13 GMT</pubDate>
            <atom:updated>2026-08-02T22:28:13.853Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E9qcnlEJ1tHRUTX8j2MlsA.png" /></figure><p>Every application that accepts user uploads needs to answer one common question: is the uploaded file safe to process? In order to verify that, you may use an antivirus engine to scan the uploaded file. One of such well known, free and open source antivirus engines is <a href="https://www.clamav.net/">ClamAV</a>.</p><p>Running ClamAV in a serverless setup can be difficult because the virus signature database is over 200 MB and it is being updated regularly. So, keeping the virus signature database up to date for each scan while reducing latency can be challenging.</p><p>Also, file scanning should happen in an isolated environment, ideally a single disposable environment per scan. This way, once the file scan is done, those resources can be thrown away safely without impacting the rest of the system.</p><p>AWS Lambda MicroVMs, address both these problems nicely. In this blog post, I discuss why a MicroVM is a good fit for this kind of jobs, walk through a workflow I built that scans user uploaded S3 files with ClamAV running inside a MicroVM, and show how to keep the virus signature database up to date.</p><h3>Why MicroVM is the right fit</h3><p>Before MicroVMs, there were two most suitable ways to run ClamAV on AWS. You could squeeze it into a Lambda function, and accept the cold start penalty of fetching and loading the database, the packaging effort, and the 15 minute execution cap. Or you could run clamd as an service on ECS or EC2, and accept paying including the idle time, one shared daemon scanning everyone’s files, and another piece of infrastructure you need to manage yourself.</p><p>A MicroVM takes the good parts of both the approaches:</p><ol><li><strong>A disposable VM per file.</strong> Malware scanning means feeding untrusted input to a software component, so you have to assume the scanner itself can be compromised by a crafted file. Regular Lambda also runs on Firecracker, but execution environments are reused across invocations: a file that exploits the scanner could persist in the warm environment and quietly tamper with the scans that follow it. For example, it can report the infected files as clean. With MicroVM approach, every file gets a fresh VM behind a hardware virtualization boundary, with an execution role that can only read the upload bucket and emit result events, and the VM is terminated right after the scan. The blast radius of the worst case is limited to a single disposable VM that has already done its job.</li><li><strong>The snapshot removes the latency problem. </strong>When the MicroVM image is built, the platform runs the container, waits until the application reports ready, and snapshots its memory and disk. Here, I run freshclam during the image build, so the virus signature database files are baked into the snapshot’s disk and nothing is downloaded when a scan runs, and the VM boots with the database already in place. Each scan still spawns clamscan, which loads the database from the snapshot’s disk into memory before scanning. But that time duration is trivial comparing to the latency of downloading virus signature database each time per scan.</li><li><strong>Packaging simplicity. </strong>A MicroVM image is defined by a very simple plain Dockerfile, and here I use the official clamav/clamav as the base image. There is no runtime interface clients or Lambda layers. The only addition is a small Flask application that runs the scans.</li><li><strong>Serverless benefits.</strong> A VM is launched per upload and it start fast. Also it terminates itself when the scan is done. So you pay only while it runs. And with the options to configure vCPUs, memory and the running time up to 8 hours, it provides both power and simplicity of serverless.</li></ol><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FgYn_IcZAlhpqaWX.png" /></figure><p>This is the high level architecture of the virus scanning flow. There are two main parts of the system: file scanning flow and the functionality to keep the virus signature database up to date.</p><h3>File scanning flow</h3><ol><li>A user uploads a file to the upload bucket, which has EventBridge notifications enabled.</li><li>An Object Created rule on the default event bus triggers a small orchestrator Lambda function. This extra step exists because as at now, a MicroVM cannot be an EventBridge rule target directly. Something has to call the RunMicrovm API, and here a Lambda function is the middleman facilitates that.</li><li>The orchestrator calls RunMicrovm, passing the bucket and key as the run hook payload. As at now, default boto3 in the Lambda runtime doesn’t support MicroVM API. So you have to bundle boto3 version 1.43.0 or higher into this function.</li><li>The platform launches a MicroVM from the ClamAV image snapshot, the application downloads the object, runs clamscan, and publishes a FileScanned event with the result CLEAN, INFECTED or ERROR to a custom event bus.</li><li>A rule on the custom bus writes the results to CloudWatch Logs. In a real application, you would route them to whatever handles quarantining or further processing of that file.</li></ol><h4>Inside the MicroVM</h4><p>I use the MicroVM non-interactive way to scan the files. Which means the application logic triggers using the lifecycle hooks. Here are the MicroVM hooks I utilised in this application:</p><ol><li>The /ready hook is called during the image build. It verifies if the virus signature database is already exists within the MicroVM. It returns 503 and fail if the ClamAV database files not exist, which guarantees the snapshot always contains the virus signature database.</li><li>The /validate hook runs after the snapshot is taken, on a test MicroVM. I scan the <a href="https://www.eicar.org/">EICAR</a> test string there, which validates if the image actually detects malware before it ever scans actual files.</li><li>The /run hook is where the actual scan of the file happens. The payload passed to RunMicrovm is delivered as the body of this hook. The application kicks off the scan on a background thread and returns immediately. The scan downloads the object from S3, runs clamscan, and publishes the results to EventBridge. This also includes the matched signature name, the database version and build time read straight from the snapshot. MicroVM execution role provides all the permissions required to talk to the services in use: S3, EventBridge and CloudWatch.</li></ol><p>I have below idle policy for the MicroVM.</p><pre>{<br>  &quot;maxIdleDurationSeconds&quot;: 60,<br>  &quot;suspendedDurationSeconds&quot;: 0,<br>  &quot;autoResumeEnabled&quot;: false<br>}</pre><p>Once the run hook is completed, MicroVM will be suspended after 60 seconds (This is the minimum time allowed). After that, will be terminated immediately.</p><h3>Keeping the signatures database fresh</h3><p>Baking the database into the snapshot is what makes the scan fast, but it comes with an obvious downside. Virus signatures age with the image. An antivirus scanner running week-old signatures is not much of useful. One solution is to download the virus signature database at the file scan time, but it is not the best approach because it adds significant latency and, since each file scan use independent execution environments, there is no way to reuse the downloaded signature database.</p><p>The better solution is to rebuild the MicroVM image with the latest virus signature database. There is an EventBridge Scheduler invokes an image builder Lambda function every hour. Within that Lambda function it run UpdateMicrovmImage where it starts a new image build using the Dockerfile provided which includes freshclam. freshclam is the official cli tool and daemon used to download and update malware and virus signature databases for the ClamAV antivirus engine. So an hourly Lambda function that updates the scanner image produces a new active version of the image with the latest virus signatures. The next file scan after a rebuild is scanned within a MicroVM that use the latest image so that the latest database will be used.</p><blockquote>Rebuilding image with update_microvm_image is quite easy, because we can reuse the configurations from existing image including the S3 code artifacts. And Lambda service will simply take care of the build process and we don’t need to handle it.</blockquote><h4>Rebuilding only when virus database has new updates</h4><p>A full image build and snapshot is expensive, and ClamAV does not publish new signatures every hour. So before rebuilding the MicroVM image, the function checks whether it needs to.</p><p>ClamAV publishes its current signature record as a DNS TXT record at current.cvd.clamav.net, which is the same source freshclam itself uses. The record is a colon-separated string like 1.4.5:63:28072:1785016800:1:90:49192:339, and field index 3 is the UNIX timestamp at which the latest virus database was published.</p><p>If that timestamp falls inside the last hour, the function starts building the new image. Otherwise it ignores the update. However, if the DNS lookup fails for any reason, function will anyway build the image as a fall back.</p><h4>Removing old image versions</h4><p>Every rebuild creates an image version, and every version holds a snapshot you pay storage for whether or not anything is running on it. As at now, Lambda does not provide any functionality to automatically expire the old images. So, we have to manage it ourselves.</p><p>What I did here was fetching all the image versions available and order them by the created at value and delete one by one using API DeleteMicrovmImageVersion except two most recent image versions.</p><blockquote>Please note: When there is a image build in progress, the DeleteMicrovmImageVersion API will be rejected for every version of that image, not just the one being built. It is an image level lock.</blockquote><p>Because of this reason, I run the deletion of old versions before start building the new version.</p><h3>Try this yourself</h3><p>The full code is available at <a href="https://github.com/pubudusj/microvm-file-scan">github.com/pubudusj/microvm-file-scan</a>.</p><ol><li>Clone the repository and install the dependencies:</li></ol><pre>git clone https://github.com/pubudusj/microvm-file-scan<br>cd microvm-file-scan<br>python3 -m venv .venv &amp;&amp; source .venv/bin/activate<br>pip install -r requirements.txt</pre><p>2. Deploy both stacks:</p><pre>cdk deploy --all</pre><p>The first deploy takes a few minutes because CloudFormation builds and snapshots the MicroVM image.</p><blockquote>Please note: As at now, Lambda MicroVMs is available in a limited set of regions. Make sure you deploy the stacks to one of them.</blockquote><h3>Testing</h3><p>Once the project is deployed into your AWS account, there will be several outputs. To test the functionality, first upload a file to the bucket name shown in the stack outputs:</p><pre>echo &quot;hello&quot; &gt; hello.txt<br>aws s3 cp hello.txt s3://&lt;UploadBucketName&gt;/</pre><p>Shortly after, a FileScanned event with status: CLEAN appears in the results log group created by the stack. Sample event looks like below:</p><pre>{<br>    &quot;version&quot;: &quot;0&quot;,<br>    &quot;id&quot;: &quot;994e766e-cf1b-cba6-2bae-21f9c658620b&quot;,<br>    &quot;detail-type&quot;: &quot;FileScanned&quot;,<br>    &quot;source&quot;: &quot;file.scanner&quot;,<br>    &quot;account&quot;: &quot;xxxxxxxxxx&quot;,<br>    &quot;time&quot;: &quot;2026-07-29T12:59:33Z&quot;,<br>    &quot;region&quot;: &quot;eu-west-1&quot;,<br>    &quot;resources&quot;: [],<br>    &quot;detail&quot;: {<br>        &quot;bucket&quot;: &quot;microvmfilescanstack-uploadbucketxxxxxxxx-xxxxxxxxxxxx&quot;,<br>        &quot;key&quot;: &quot;hello.txt&quot;,<br>        &quot;status&quot;: &quot;CLEAN&quot;,<br>        &quot;signature&quot;: null,<br>        &quot;dbVersion&quot;: &quot;28076&quot;,<br>        &quot;dbTimestamp&quot;: &quot;29 Jul 2026 06:25 +0000&quot;,<br>        &quot;scannedAt&quot;: &quot;2026-07-29T12:59:33.861586+00:00&quot;<br>    }<br>}</pre><p>To see a detection, upload the harmless, industry standard <a href="https://www.eicar.org/">EICAR test file</a>:</p><pre>printf &#39;X5O!P%%@AP[4\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*&#39; &gt; eicar.com<br>aws s3 cp eicar.com s3://&lt;UploadBucketName&gt;/</pre><p>The resulting event looks like this:</p><pre>{<br>    &quot;version&quot;: &quot;0&quot;,<br>    &quot;id&quot;: &quot;b0c38807-a8e7-90e3-031c-20c97ff5e0f5&quot;,<br>    &quot;detail-type&quot;: &quot;FileScanned&quot;,<br>    &quot;source&quot;: &quot;file.scanner&quot;,<br>    &quot;account&quot;: &quot;xxxxxxxxxx&quot;,<br>    &quot;time&quot;: &quot;2026-07-29T13:01:40Z&quot;,<br>    &quot;region&quot;: &quot;eu-west-1&quot;,<br>    &quot;resources&quot;: [],<br>    &quot;detail&quot;: {<br>        &quot;bucket&quot;: &quot;microvmfilescanstack-uploadbucketxxxxxxxx-xxxxxxxxxxxx&quot;,<br>        &quot;key&quot;: &quot;eicar.com&quot;,<br>        &quot;status&quot;: &quot;INFECTED&quot;,<br>        &quot;signature&quot;: &quot;Eicar-Test-Signature&quot;,<br>        &quot;dbVersion&quot;: &quot;28076&quot;,<br>        &quot;dbTimestamp&quot;: &quot;29 Jul 2026 06:25 +0000&quot;,<br>        &quot;scannedAt&quot;: &quot;2026-07-29T13:01:40.882464+00:00&quot;<br>    }<br>}</pre><p>Notice the signature, dbVersion and dbTimestamp fields. Every result records exactly which signature set produced it, and dbTimestamp and dbVersion are a result of the hourly build process.</p><blockquote>Please note: When you create the eicar.com test file in your computer, your anti-virus program will most probably immediately detect and quarantine it (even it is not harmful). Instead of creating this locally, you can open up a cloudshell session in your AWS console and create and upload the test file from there.</blockquote><h3>Limitations</h3><ol><li>The virus database refresh is only as fine grained as the schedule. The interval between attempts to rebuilding the image can be set as per your need, because actual image building will happen only when there is a virus update available.</li><li>This design launches one MicroVM per upload, which is great for isolation but adds a small launch overhead per file. For high volume workloads, you could reuse a VM for a batch of files but you will loose per-file isolation.</li><li>Be aware that some of the API operations have lower transactions per second ex: RunMicrovm, SuspendMicrovm, TerminateMicrovm. And these rate limits are per account, per region which means if you have other MicroVMs running for other purposes in the same region, you will likely to be throttled. Refer <a href="https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#microvms-quotas">https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#microvms-quotas</a> for more information.</li></ol><h3>Summary</h3><p>For a requirement where you have to scan user uploaded files for malware, it is better to perform it in a disposable isolated environment. It is exactly the kind of job MicroVMs were built for with the vm-level isolation and serverless operational model.</p><p>In this project I use ClamAV an free and open source antivirus engines, but you can use any similar products suits your the best.</p><p>MicroVM is really fast to start, however, downloading virus signature database every time before a file scan will not be efficient. Baking the virus signature into the MicroVM image is the solution. However, it is necessary to have a workflow that keeps the MicroVM image up to date in regular intervals with the latest virus signature database.</p><h3>Resources</h3><ol><li>Lambda MicroVMs developer guide: <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html">https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html</a></li><li>Lambda MicroVMs API reference: <a href="https://docs.aws.amazon.com/lambda/latest/microvm-api/Welcome.html">https://docs.aws.amazon.com/lambda/latest/microvm-api/Welcome.html</a></li><li>AWS Lambda MicroVMs announcement blog: <a href="https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/">https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/</a></li><li>ClamAV: <a href="https://www.clamav.net/">https://www.clamav.net</a></li><li>ClamAV private mirror / signature distribution notes: <a href="https://docs.clamav.net/manual/Usage/SignatureManagement.html">https://docs.clamav.net/manual/Usage/SignatureManagement.html</a></li><li>EICAR test file: <a href="https://www.eicar.org/">https://www.eicar.org</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=499588962564" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Triggering Lambda Durable Functions from SQS]]></title>
            <link>https://medium.com/@pubudusj/triggering-lambda-durable-functions-from-sqs-e0d5e2702aef?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0d5e2702aef</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Wed, 03 Jun 2026 00:00:10 GMT</pubDate>
            <atom:updated>2026-06-03T22:12:12.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uhhWmiI6dNHlncR2Y_OumA.png" /></figure><p>Lambda durable functions, introduced in re:Invent 2025, are a great fit for long-running workflows that need to wait for external events, callbacks, or simply pause between steps. A single durable execution can run up to 1 year, which opens up a lot of use cases that previously required Step Functions or custom orchestration.</p><p>Even though the execution can span up to 1 year, it is better to be clear about what “1 year” actually refers to. A durable execution is <strong>not</strong> a single Lambda invocation that stays alive for a year. A durable execution can consist of multiple separate invocations. Between those invocations, the execution can be moved to wait state or go to sleep and you will not be charged for that. Every one of those individual invocations still <strong>has to finish within the standard 15 minute Lambda function timeout</strong>. And, the total duration of those individual invocations and the wait time must be maximum of 1 year. From the caller’s perspective the execution can span hours, days, or up to a year, even though a single invocation must run for maximum 15 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gtVEAW4nWsXxAh4F.png" /><figcaption>Image: Durable execution and its individual invocations</figcaption></figure><p>However, this is not always true. When you start invoking a durable function from an event source mapping (ESM), such as SQS, Kinesis, or DynamoDB Streams, that total duration of the execution is limited to only <strong>15 minutes</strong>. This is including intermediate invocations and the waits.</p><p>In this blog post, I discuss the solution that AWS provides to work around this 15 minute limitation and another (better?) alternative.</p><h3>Why the 15 minute limit exists</h3><p>Event source mappings poll the source (SQS, Kinesis, etc.) and invoke the target Lambda function <strong>synchronously</strong>. The poller holds the connection open and waits for the function to return before it marks records as processed.</p><p>For a standard Lambda function this is fine, because the maximum function timeout is 15 minutes anyway. But for a durable function, the whole durable execution (steps, waits, callbacks, everything from start to finish) must also fit inside that same synchronous call. So even though the underlying durable execution model supports up to 1 year, the synchronous nature of ESM caps the total execution time at 15 minutes.</p><blockquote><strong>Please note:</strong> When using an ESM based trigger, Durable Functions cannot be configured with an execution timeout longer than 15 minutes. Attempting to do so will cause ESM creation to fail.</blockquote><h3>The recommended workaround: using an intermediary function</h3><p>AWS recommends solving this limitation by adding a <strong>standard Lambda function</strong> between the event source mapping and the durable function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*79F0la11ky1OALDW.png" /><figcaption>Image: Using an intermediary function to execute durable function with SQS events.</figcaption></figure><ol><li>The ESM invokes the intermediary Lambda synchronously.</li><li>The intermediary Lambda invokes the durable function <strong>asynchronously</strong> (InvocationType=Event) using the Lambda Invoke API.</li><li>The intermediary Lambda returns success immediately, and the ESM is happy.</li><li>And, the durable function then runs on its own, independent of the ESM, for as long as it needs (up to 1 year).</li></ol><p>The intermediary function is an essential piece here, but now you own and maintain a Lambda function whose only job is to forward an event.</p><h3>A cleaner approach: EventBridge Pipes</h3><p>EventBridge Pipes can be the bridge between an event source and a Lambda target, and the target invocation can be configured as fire-and-forget (asynchronous). That is exactly the behavior we need. The source side polling is decoupled from the durable execution, and there is no intermediary Lambda to write or maintain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*0mS0x74c9RKs9CKW.png" /><figcaption>Image: Using EventBridge Pipe to execute durable function with SQS events.</figcaption></figure><p>The Pipe polls the SQS queue and invokes the durable function <strong>asynchronously</strong>. The Pipe considers the message processed as soon as the asynchronous invoke succeeds, not when the durable execution completes. The durable function is then free to run for a longer time.</p><p>The full code is available at <a href="https://github.com/pubudusj/sqs-to-lambda-durable-function-with-pipes">github.com/pubudusj/sqs-to-lambda-durable-function-with-pipes</a>.</p><p>Deploy it to your own AWS account using AWS CDK and Python:</p><ul><li>Clone the repository and create a virtual environment:</li></ul><pre>python3 -m venv .venv<br>source .venv/bin/activate<br>pip install -r requirements.txt</pre><ul><li>Deploy the stack:</li></ul><pre>cdk deploy</pre><ul><li>Send a test message to the QueueUrl output from the stack:</li></ul><pre>aws sqs send-message \<br>    --queue-url &lt;QueueUrl&gt; \<br>    --message-body &#39;{&quot;hello&quot;: &quot;durable&quot;}&#39;</pre><ul><li>Open the durable function in the Lambda console using the LambdaFunctionName stack output and watch the execution. You will see the first print_event_before_wait log entry almost immediately, then the wait continues for 30 minutes, then the print_event_after_wait log entry. The execution completes successfully even though it took twice as long as a standard ESM invoked Lambda would allow.</li></ul><blockquote><strong>Please note: </strong>In both the approaches the Pipe or intermediary function succeeds when the async invoke is accepted, not when the durable execution succeeds. If the durable function fails internally (an exception in a step or a callback times out etc), that failure is visible inside the durable execution history, but the SQS source has already considered the message processing is done. It is advised to build your error handling inside the durable function itself for such failures. Source queue level error handling will manage anything that failed to trigger the Pipe or intermediary function.</blockquote><h3>Summary</h3><p>Durable functions are a new way of building long-running, multi-step workflows on Lambda, but the 15 minute ceiling that appears the moment you put an event source mapping in front of them can be a real constraint. The workaround is to use an intermediary Lambda that asynchronously invokes the durable function, but you have to own and manage it.</p><p>EventBridge Pipes does the same job natively. By configuring the Pipe target with FIRE_AND_FORGET, you get the same decoupling between the synchronous poller of ESM and the long-running durable execution, without owning or managing a Lambda function.</p><p>If you are reaching for durable functions to handle SQS messages with steps, waits, or callbacks that exceed 15 minutes, using Pipes can be a better option than using an intermediary function.</p><h3>Resources</h3><ol><li>Lambda durable functions overview: <a href="https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html">https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html</a></li><li>Event source mappings with durable functions: <a href="https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html">https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html</a></li><li>Amazon EventBridge Pipes targets: <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html">https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0d5e2702aef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Lambda Tenant Isolation]]></title>
            <link>https://medium.com/@pubudusj/understanding-lambda-tenant-isolation-485c5680695c?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/485c5680695c</guid>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Sun, 22 Feb 2026 23:33:31 GMT</pubDate>
            <atom:updated>2026-02-22T23:33:31.394Z</atom:updated>
            <content:encoded><![CDATA[<p>Lambda tenant isolation is one of the important security features that came out of the 2025 re:Invent season.</p><p>Achieving tenant isolation in SaaS applications is not straightforward, and taking the single-tenant route to solve it introduces its own scaling challenges. This new feature is not a silver bullet, but it does offer much better support for keeping tenants isolated at scale.</p><p>In this blog post, I discuss what this feature is and the problems it addresses.</p><h3>Lambda execution environment</h3><p>When a invoke request reached AWS Lambda service initially, it starts a virtual environment in a EC2 host worker. We call this an execution environment. An execution environment will download the code and required dependencies and process the request. If required, it will return the response.</p><p>One of the key attributes of this execution environment is that it will not be removed or deleted immediately after processing a single request. It will be kept in ‘warm’ state to serve another incoming request. When the next request comes in, Lambda service will use the execution environment that is already available to process it, without creating a new environment.</p><p>Likewise, when you invoke a Lambda function, if there are execution environments available for that Lambda function, Lambda service will use them to process the requests else will create new execution environments.</p><p>However, this approach can be a concern when it comes to a multi-tenant Lambda function, because execution environments share ’left over’ stuff like:</p><ul><li>Global variables</li><li>Objects initialized outside of the handler</li><li>files saved in /tmp space</li></ul><h3>Multi-tenant Lambda function</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qWSp-yFJWSPcjfumyaw_Lg.png" /><figcaption>Image: Multi-tenant Lambda function</figcaption></figure><p>In this multi-tenant setup, a Lambda function is shared by more than one tenant. And based on how execution environments behave, irrespective of the tenant that invoke the Lambda function, Lambda service will use execution environments that are already available for that Lambda function. But, as mentioned earlier, execution environments sharing some data across executions can be a security issue.</p><blockquote>Having data shared across execution environments can be a great optimization if those data are accessible only by the intended tenant. However, when multiple tenants use the same execution environments, tenants will have access to data that they are not intended to.</blockquote><p>For example, if we take a single execution environment, tenant 1 might fetch some secrets from Secret Manager or save some files in /tmp directory in its execution. If the same execution environments used for an execution of tenant 2, tenant 2 will have access to the secrets fetched for tenant 1 or the contents of the /tmp directory saved by tenant 1.</p><h4>Solution</h4><p>One of the solution for this problem is to reset the execution environment just before processing each request. For example, unsetting any global variables or wipe the /tmp directory etc. However, this approach will not be practical at scale.</p><p>Another option is to go for tenant specific Lambda functions which is the single-tenant approach. In this case, tenants will have their own dedicated Lambda function each. This will solve the problem of unintended access to temporary data, because different execution environments belongs to different Lambada functions will not share the execution environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTrJeVETcAHLcicY7fcISA.png" /><figcaption>Image: Single-tenant Lambda functions</figcaption></figure><p>However having Lambda function per tenant is not scalable. When there are a lot of tenants available, you end up with a lot of Lambda functions that you need to manage. While this is possible when you use a IAC tool like CDK or Terraform, still in a situation where you need to update source code to introduce a functionality or a bug fix, you will need to update all these tenant specific resources which is not easy. And, most of the time, those resources are not fully utilized too.</p><p>What if we can have a single Lambda function shared by all of the tenant (the multi-tenant approach), yet have the isolation we need in a Lambda per tenant (single-tenant approach)?</p><h3>Lambda tenant isolation</h3><p>With Lambda tenant isolation, you can have a single Lambda function shared by all of the tenant, yet have the isolation you need in a Lambda per tenant. With this new feature, Lambda service will do the heavy lifting by creating execution environments that are dedicated to a specific tenant. This means that execution environments will not be shared across tenants, and tenants will not have access to data that they are not intended to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4wYp322dNE_skExgSGM8DA.png" /><figcaption>Image: Tenant isolated Lambda function</figcaption></figure><p>But, how Lambda determines the incoming request is from which tenant? For that, we need to provide a tenant-id in the request to Lambda. If you use Lambda invoke cli method, you can use the --tenant-id parameter as follows:</p><pre>aws lambda invoke \<br>    --function-name tenant-aware-lambda \<br>    --payload &#39;{ &quot;name&quot;: &quot;Bob&quot; }&#39; \<br>    --tenant-id t1 \<br>    response.json</pre><p>If you use Lambda API, you need to provide the value using X-Amz-Tenant-Id as follows:</p><pre>POST /2015-03-31/functions/tenant-aware-lambda/invocations HTTP/1.1<br>Host: lambda.eu-central-1.amazonaws.com<br>Content-Type: application/json<br>Authorization: AWS4-HMAC-SHA256 Credential=...<br>X-Amz-Tenant-Id: t1<br>{<br>    &quot;name&quot;: &quot;Bob&quot;<br>}</pre><p>Tenant id is case sensitive and it can be any alpha numeric character with maximum length 256 characters. There are few special characters such as hyphens (-), underscores (_), colon (:), equals (=), plus (+), at (@) and periods (.) allowed too.</p><p>One of the key attributes of tenant id is that we don’t need to pre-register those tenant ids. We can pass any dynamic value as the tenant id and Lambda service will take care of creating and maintaining a pool of execution environments for each value passed. Also, we can use any number of unique tenant ids, there is no limit.</p><h3>How to enable this feature in Lambda</h3><p>If you use the AWS Console, you can go to the Lambda create wizard and enable this in the additional security section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*cUoNLnyS6tIGX6P4KXLS6w.png" /><figcaption>Image: Creating Lambda function in AWS console</figcaption></figure><p>Also, options for CLI, CloudFormation and CDK also available.</p><p>CLI:</p><pre>aws lambda create-function \<br>    --function-name tenant-aware-lambda \<br>    --runtime python3.14 \<br>    --zip-file fileb://tenant-aware-lambda.zip \<br>    --handler index.handler \<br>    --role arn:aws:iam:123456789012:role/execution-role \<br>    --tenancy-config &#39;{&quot;TenantIsolationMode&quot;: &quot;PER_TENANT&quot;}&#39;</pre><p>CloudFormation:</p><pre>MyLambdaFunction:<br>    Type: AWS::Lambda::Function<br>    Properties:<br>      FunctionName: tenant-aware-lambda<br>      Runtime: python3.14<br>      Role: !GetAtt LambdaExecutionRole.Arn<br>      Handler: index.handler<br>      TenancyConfig:<br>        TenantIsolationMode: PER_TENANT<br>      Code:<br>        ZipFile: |<br>          .....<br>          .....          <br>      Timeout: 10<br>      MemorySize: 128</pre><p>CDK:</p><pre>tenant_aware_lambda = _lambda.Function(<br>            self,<br>            &quot;TenantAwareFunction&quot;,<br>            function_name=&quot;tenant-aware-lambda&quot;,<br>            runtime=_lambda.Runtime.PYTHON_3_13,<br>            handler=&quot;index.handler&quot;,<br>            code=_lambda.Code.from_asset(&quot;src/lambda/tenant_aware&quot;),<br>            timeout=Duration.seconds(10),<br>            tenancy_config=_lambda.TenancyConfig.PER_TENANT,<br>        )</pre><blockquote>Please note: This feature can be enabled <strong>ONLY</strong> when the Lambda function is created. You cannot enable this for an existing Lambda function.</blockquote><h3>Try this yourself</h3><p>I have created a sample application for you to see how this feature works. You can deploy it to your AWS environment using CDK with Python.</p><p>Clone the repository at <a href="https://github.com/pubudusj/lambda-tenant-isolation-demo">github.com/pubudusj/lambda-tenant-isolation-demo</a> and follow the steps below:</p><ul><li>Create a virtual environment and activate it:</li></ul><pre>python3 -m venv .venv<br>source .venv/bin/activate</pre><ul><li>Install the dependencies:</li></ul><pre>pip install -r requirements.txt</pre><ul><li>Deploy the stack:</li></ul><pre>cdk deploy</pre><p>This will create two Lambda functions:</p><ol><li>A generic Lambda function (without tenant isolation)</li><li>A Lambda function with tenant isolation enabled</li></ol><p>Both Lambda functions have a global variable counter which increments on each invocation. This is to simulate the shared state across executions. An API Gateway is also created with two endpoints to trigger these Lambda functions:</p><ul><li>/execute_generic_lambda?tenant_id=&lt;tenant_id&gt; - invokes the generic Lambda function</li><li>/execute_tenant_aware_lambda?tenant_id=&lt;tenant_id&gt; - invokes the tenant-aware Lambda function</li></ul><h3>Testing</h3><h4>Generic Lambda function</h4><p>First, let’s test the generic Lambda function. Call the /execute_generic_lambda endpoint with a tenant id:</p><pre>curl &quot;&lt;APIGW_BASE_URL&gt;/execute_generic_lambda?tenant_id=tenant1&quot;</pre><p>You will see a response like:</p><pre>{&quot;tenant_id&quot;: &quot;tenant1&quot;, &quot;invocation_count&quot;: 1}</pre><p>Now call the same endpoint again but with a different tenant id:</p><pre>curl &quot;&lt;APIGW_BASE_URL&gt;/execute_generic_lambda?tenant_id=tenant2&quot;</pre><p>You can see the invocation count keeps increasing regardless of the tenant id:</p><pre>{&quot;tenant_id&quot;: &quot;tenant2&quot;, &quot;invocation_count&quot;: 2}</pre><p>This is because the generic Lambda function shares execution environments across all tenants. The global variable counter retains its value across invocations regardless of which tenant made the request. This is the exact problem we discussed earlier - any global state, cached data or files in /tmp are accessible across tenants.</p><h4>Tenant-aware Lambda function</h4><p>Now let’s test the tenant-aware Lambda function. Call the /execute_tenant_aware_lambda endpoint with a tenant id:</p><pre>curl &quot;&lt;APIGW_BASE_URL&gt;/execute_tenant_aware_lambda?tenant_id=tenant1&quot;</pre><p>You will see a response like:</p><pre>{&quot;tenant&quot;: &quot;tenant1&quot;, &quot;invocation_count&quot;: 1}</pre><p>Now call the same endpoint again with a different tenant id:</p><pre>curl &quot;&lt;APIGW_BASE_URL&gt;/execute_tenant_aware_lambda?tenant_id=tenant2&quot;</pre><p>This time, the invocation count resets:</p><pre>{&quot;tenant&quot;: &quot;tenant2&quot;, &quot;invocation_count&quot;: 1}</pre><p>This is tenant isolation in action. Even though both requests are handled by the same Lambda function, Lambda service creates separate execution environments for each tenant. The global variable counter, or any other shared state in the execution environment, is isolated per tenant. Tenant 2 will never see the state left behind by Tenant 1.</p><p>Also note that in the tenant-aware Lambda function, we can access the tenant id from the Lambda context object (ex: using context.tenant_id in Python), instead of extracting it from the query parameters. Lambda service automatically makes the tenant id available in the context when tenant isolation is enabled. This is useful if you need to do some operations based on the tenant id - for example, fetch some data from other services.</p><p>In this example, I have used this tenant id available in the context object to publish a custom CloudWatch metric per tenant. This is helpful to monitor per-tenant invocation patterns, which can be useful for billing or auditing purposes.</p><h3>Effect on Lambda concurrency and cold starts</h3><p>One important thing to understand is how tenant isolation affects Lambda concurrency. Since execution environments are not shared across tenants, each tenant will need their own set of execution environments. This means that the overall number of concurrent execution environments can be higher compared to a non-isolated Lambda function where environments are freely shared.</p><p>For example, if you have 10 tenants each making concurrent requests, instead of reusing a pool of warm execution environments, Lambda needs to maintain separate pools per tenant. This can lead to more cold starts, especially for low-traffic tenants.</p><p>However, this is a trade-off worth making when tenant isolation is critical for your application.</p><blockquote>Please note: Make sure to consider the Lambda concurrency limits in your account when enabling tenant isolation, especially when dealing with a large number of tenants.</blockquote><h3>Integration with API Gateway</h3><p>As at now, only integration supports Lambda tenant isolation features is API gateway.</p><p>In the example project, I have mapped the incoming query string tenant_id to the integration request header X-Amz-Tenant-Id for the Lambda service.</p><pre>tenant_aware_integration = apigw.LambdaIntegration(<br>    tenant_aware_lambda,<br>    request_parameters={<br>        &quot;integration.request.header.X-Amz-Tenant-Id&quot;: &quot;method.request.querystring.tenant_id&quot;<br>    },<br>)</pre><p>When using Lambda integration on API Gateway, it gives us a lot of flexibility in choosing what value to map as the X-Amz-Tenant-Id. It could be the source AWS account id, a value from the request body, or a header value. If authentication and authorization are enabled, it could even be a Cognito user group or a claim from a JWT token. This makes API Gateway a convenient place to resolve and pass the tenant id to Lambda.</p><h3>What I’d like to see next</h3><p>While Lambda tenant isolation is a solid step forward, there are a few areas where I think it can be even more valuable.</p><ol><li>More integration support: Currently only API Gateway supports as a integration to Lambda with this feature. It would be great to see this extended to other integrations such as SQS to Lambda via event source mapping.</li><li>Native per-tenant metrics: Built-in CloudWatch metrics by tenant id would remove the need to publish custom metrics manually.</li><li>Per-tenant concurrency controls: The ability to set concurrency limits per tenant would help prevent a noisy tenant from consuming all the available concurrency.</li></ol><h3>Conclusion</h3><p>Tenant isolation is a fundamental requirement in many SaaS applications. While the single-tenant approach provides strong isolation, it comes with significant operational overhead. The multi-tenant approach is operationally simpler but introduces security risks with shared execution environments.</p><p>Lambda tenant isolation gives us the best of both worlds. We get a single Lambda function that is easy to manage and deploy, while Lambda service ensures that the execution environments are isolated per tenant. This eliminates the risk of data leakage across tenants without the burden of managing separate Lambda functions for each tenant.</p><p>This is a great addition to the serverless toolbox. If you are building multi-tenant SaaS applications on AWS Lambda, this feature is worth exploring.</p><h3>Resources</h3><ol><li>AWS Lambda tenant isolation documentation: <a href="https://docs.aws.amazon.com/lambda/latest/dg/tenant-isolation.html">https://docs.aws.amazon.com/lambda/latest/dg/tenant-isolation.html</a></li><li>Launch blog post: <a href="https://aws.amazon.com/blogs/aws/streamlined-multi-tenant-application-development-with-tenant-isolation-mode-in-aws-lambda">https://aws.amazon.com/blogs/aws/streamlined-multi-tenant-application-development-with-tenant-isolation-mode-in-aws-lambda</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=485c5680695c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Leave Management with AWS Lambda Durable Functions]]></title>
            <link>https://medium.com/@pubudusj/simple-leave-management-with-aws-lambda-durable-functions-855707d3164b?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/855707d3164b</guid>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Wed, 14 Jan 2026 00:16:40 GMT</pubDate>
            <atom:updated>2026-01-14T00:16:40.878Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>In AWS re:Invent 2025, Lambda introduced Durable Functions with a great set of features. One of the main features is that a single execution can span up to one year. Also with built-in checkpointing it is possible to track the steps that were already completed and when an execution retried from a resume or an interruption, completed steps will be skipped and resumes from the next step.</p><p>When it comes to orchestrating multiple AWS services into a workflow, up until now, AWS Step Functions was the best choice. However, now Durable Functions too offer similar functionality to build a workflow within the same familiar Lambda environment, which is great!</p><p>In this blog post, I explain how Durable Function’s callback feature can be used for a human-in-the-loop functionality.</p><h3>Durable Function callbacks</h3><p>When an execution starts, Durable Function will start an invocation. When there are multiple steps within the execution, it is possible to set the execution on hold (or wait) in a certain step for a given time period or until a signal is received from an external process. In case of a external signal, once the signal is received by the Lambda service, based on the success or failure nature of the signal, the execution that was on hold can be resumed or completely terminated.</p><p>This is a great feature supported natively by Durable Functions that is helpful for an example, in a business process where a human approval is required to continue with the flow.</p><h3>Simple leave management with Durable Functions</h3><p>In this example, an employee can send a leave request and the manager can approve or reject the request. Below are the steps involved and how Durable Function is being used.</p><ol><li>Employee sends a leave request. Durable function starts an execution.</li><li>A Leave is created in a DynamoDB table with pending state.</li><li>The employee receives an email with the request receipt confirmation.</li><li>The manager receives an email with a callback id to be used to approve or reject the leave.</li><li>Durable function keeps the execution on hold until manager approval or rejection is received.</li><li>Once the manager approves or rejects, the execution resumes.</li><li>If the manager didn’t process the request within a given time period, the request will be expired.</li><li>The leave status is updated in the DynamoDB table.</li><li>The employee receives an email with the manager’s decision or expiry.</li></ol><h3>How it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kLT5PgmsXog9ugy6.png" /><figcaption>Image: Architecture</figcaption></figure><ol><li>There is a proxy function which accepts and validates the incoming request to create a leave. Here I used a Lambda Function URL to submit the request.</li><li>Once validated, the proxy function will invoke the durable function.</li></ol><blockquote>Why use the proxy function?</blockquote><blockquote>Even with Durable functions, the synchronous invocations are limited by 15 minutes. Since I need the durable function to run longer, it is required to trigger the durable function asynchronously. Using the proxy function, durable function is invoked with invocation type ‘Event’, where it just fire-and-forget and will not wait for the durable function to complete.</blockquote><p>3. When the durable function starts its execution, first a leave record is created in the DynamoDB table.</p><p>4. Then, an email is sent to the employee confirming the receipt of the leave request.</p><p>5. In the next step, an email is sent to the manager asking to approve or reject the leave request. This step is a callback step. The execution waits at this step until the manager approves or rejects.</p><p>6. Since this is a callback step, there is a callback id generated. This callback id is indicated in the email to the manager.</p><p>7. To simulate the manager’s approval or rejection, there is a Function URL exposed to trigger the <em>Process Leave Lambda</em> function.</p><p>8. This function accepts <em>callback_id</em> and the <em>decision</em> (approve or reject) as input.</p><p>9. Based on the decision, this function will call the boto3 methods of Lambda <em>send_durable_execution_callback_success</em> or <em>send_durable_execution_callback_failure</em> using the given callback id.</p><blockquote>Please note: These latest SDK methods are not available with the boto3 version provided by Lambda by default as of now. So, it is required to package boto3 version 1.42.1 or newer with your Lambda source code.</blockquote><p>10. When Lambda service receives the decision, the durable execution that was on hold will re-start.</p><p>11. If the manager’s decision was not received on time (I set this to 5 minutes at the moment), the wait_for_callback step will throw an <em>CallableRuntimeError</em> exception with an error message — <em>Callback timed out</em>.</p><p>12. Based on the decision or expiry of callback, the leave record in DB is updated.</p><p>13. Then an email will be sent to the employee with the leave’s final status and the execution is completed.</p><h3>Try this yourself</h3><p>Here is a Github repository of a sample project I created with AWS CDK and Python for you to try out this scenario in your AWS account.</p><p><a href="https://github.com/pubudusj/simple-leave-management-with-durable-functions">https://github.com/pubudusj/simple-leave-management-with-durable-functions</a></p><ol><li>Clone the repository.</li><li>Copy the .env.example into .env and add the values. The value for <em>SYSTEM_FROM_EMAIL</em> should be already configured in Simple Email Service (SES) to send emails to the manager and employee email addresses.</li><li>Deploy the stack with CDK.</li><li>Once deployed, there will be two Lambda function urls in the output — one for creating the leave request and the other for process leave request (as manager).</li><li>Send a POST request to the create leaves endpoint with a payload similar to below:</li></ol><pre>{<br>  &quot;start_date&quot;:&quot;2026-01-10&quot;,<br>  &quot;end_date&quot;: &quot;2026-01-20&quot;,<br>  &quot;employee_email&quot;: &quot;employee@email.com&quot;<br>}</pre><p>6. This will send an email to the employee email address as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*MAxeEdNJeV1WymyK.png" /></figure><p>7. Also, an email to the manager with the callback id as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*CsbL5kmrPLcb9ZNA.png" /></figure><p>8. Manager can send a POST request to the leave process endpoint with the callback id in the email with his decision (approve or reject) as follows:</p><pre>{<br>  &quot;callback_id&quot;:&quot;[callback id from email]&quot;,<br>  &quot;decision&quot;: &quot;approve&quot;<br>}</pre><p>9. Based on the decision, the employee will receive an email with the status.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*rdJXDOPT0MS82Qnf.png" /></figure><p>10. If the manager didn’t process the decision within given time (for demo purposes, this was set to 5 minutes), the leave will be marked as expired and employee will receive an email as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*a2lMNH50P6uMr50H.png" /></figure><p>Please try this and let me know your thoughts!</p><h3>Resources</h3><ol><li>AWS Lambda Durable Functions documentation: <a href="https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html">https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html</a></li><li>Durable execution SDK: <a href="https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-sdk.html">https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-sdk.html</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=855707d3164b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring multiple dynamic resources using single Amazon CloudWatch alarm]]></title>
            <link>https://medium.com/@pubudusj/monitoring-multiple-dynamic-resources-using-single-amazon-cloudwatch-alarm-3bd87cb321fe?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bd87cb321fe</guid>
            <category><![CDATA[cloudwatch]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Tue, 21 Oct 2025 09:17:58 GMT</pubDate>
            <atom:updated>2025-10-21T09:17:58.175Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>When you need to monitor your resources with a CloudWatch alarm, what you normally have to do is to create an alarm with a specific matric of that resource. Although this gives a granular level of monitoring into your resources, you always have to add or remove alarms as and when you have new resources or when you remove a specific resource. Which is an operational overhead although it can be automated if you are using infrastructure as code tool.</p><p>Another option available is to use aggregated metrics in your alarms such as CPUUtilization for EC2 so you have coverage, but at high level into a group of resources you need to monitor. The downside of this is that it lacks granular visibility into your individual resources. Also, there is only a limited number of resources that support aggregated metrics.</p><p>In September 2025, Amazon CloudWatch introduced a nice feature of allowing monitor multiple individual metrics via a single alarm using CloudWatch Metric Insights. By using Metric insight SQL queries in the alarm, it automatically updates its query results with each evaluation and adjusts in real time when resources are created and deleted.</p><p>With the introduction of the multi-metric alarms, you can now get both granular level per resource monitoring as well as less maintenance where you don’t need to update alarms when resources are created or removed from your application because the alarm itself will automatically discover and monitor the resources.</p><h3>How it works</h3><p>You can define a generic SQL statement as the alarm metric.</p><p>For example, let’s assume that you would like to monitor all your SQS queues for available messages and if there are any messages available you need to be notified.</p><p>For this requirement, you can define a query as follows:</p><pre>SELECT MAX(ApproximateNumberOfMessagesVisible)<br>FROM SCHEMA(&quot;AWS/SQS&quot;, QueueName)<br>GROUP BY QueueName<br>ORDER BY COUNT() DESC</pre><p>With this query, the alarm will monitor all your queues for ApproximateNumberOfMessagesVisible count and trigger the alarm if a specific queue has at least one visible message. Also, if you add a new queue after the alarm is created, it will be taken into account when evaluating this query.</p><p>Also, you can use tags to filter the resources you need to monitor. For example…</p><pre>SELECT MAX(ApproximateNumberOfMessagesVisible)<br>FROM SCHEMA(&quot;AWS/SQS&quot;, QueueName)<br>WHERE tag.Component = &#39;consumer&#39;<br>AND tag.Environment = &#39;production&#39;<br>AND tag.IsDLQ = &#39;true&#39;<br>GROUP BY QueueName<br>ORDER BY COUNT() DESC</pre><blockquote>It would have been a great feature if this supports wildcards or regex based filtering into the resource properties, but as of now, only ways to use filters are using tags or the complete property value as equal or not equal.</blockquote><p>When you create the alarm with this Metrics Insights Query, you can see a new section on the alarm called “<em>Contributors</em>”. Here, you can see all the resources that match the conditions in the alarm query as well as the current state of each contributor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FlIduz0I_XbhCdE6RMxsaw.png" /><figcaption>Image: CloudWatch multi metric alarm contributors.</figcaption></figure><h3>Alarm states</h3><p>When a single contributor metric breached the threshold, then its state will be changed to in alarm state. And in the alarm details, it is clearly defined which contributor caused the alarm to change the state. This is super helpful to identify the exact resource that triggered the alarm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*ibobqWbjjATxqf9aB31jhA.png" /><figcaption>Image: CloudWatch multi metric alarm notification.</figcaption></figure><p>The alarm action is based on the contributor transitions. Which means even when the alarm is already in ‘In alarm’ state, if another contributor breached the threshold and its state becomes ‘In alarm’, the alarm action will be triggered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9fczToy0hVaJQdKPMrFAQ.png" /><figcaption>Image: CloudWatch multi metric example alarm history.</figcaption></figure><p>This is how the similar situation shows in the CloudWatch console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MZ0MhMKder_JDomU4cuUw.png" /><figcaption>Image: CloudWatch multi metric example alarm graph.</figcaption></figure><h3>Try this yourself</h3><p>I have created a Github repository with a SAM template that helps to deploy some AWS resources into your AWS account and try this scenario.</p><h4>Pre-requisites</h4><p>You need to enable the CloudWatch setting “<em>Resource tags for telemetry</em>” if you need to use tag based filters in these queries. Go to CloudWatch &gt; settings &gt; Enable resource tags on telemetry in your region to enable this.</p><ol><li>Clone the repository: <a href="https://github.com/pubudusj/cloudwatch-multi-metrics-alarm">https://github.com/pubudusj/cloudwatch-multi-metrics-alarm</a></li><li>Deploy the stack using AWS SAM CLI.</li><li>Once the stack is deployed, you can see in the created CloudWatch alarm, the contributors are available.</li><li>When you add some messages into one of the SQS queues, you can see the alarm state changed and you will receive the alarm notification which includes the contributor.</li><li>You can uncomment line 64–74 and deploy again to create a new resource.</li><li>This new resource will be evaluated and you can see it is listed in the contributors list.</li><li>If you add a message to the new queue, the alarm triggers.</li><li>Likewise, any resource new or old that matches the query will be considered to evaluate the alarm.</li></ol><h3>Limitations</h3><p>There are some notable limitations in this approach.</p><ol><li>Currently a single query can return no more than 500 time series and it is a hard limit. Which means as per the above query, maximum 500 resources can be monitored, which might not be not sufficient for a application with a lot of resources. When you have more than 500 resources to monitor, you will have to use multiple alarms.</li><li>There is no wildcard or regex supported in the query (comparing to some other features in CloudWatch)</li></ol><h3>Summary</h3><p>The new feature of CloudWatch is great for monitoring dynamic resources in your application without the management overhead of adding/removing alarms when the resources are created and deleted. There are some hard limits where this might not be suitable for all the situations. However, I believe there will be more improvements added to this feature in near future — like wild card or regex based resource filtering, that will give better developer experience.</p><h3>Resources</h3><ol><li>Release news: Amazon CloudWatch query alarms now support monitoring metrics individually. <a href="https://aws.amazon.com/about-aws/whats-new/2025/09/amazon-cloudwatch-alarm-multiple-metrics/">https://aws.amazon.com/about-aws/whats-new/2025/09/amazon-cloudwatch-alarm-multiple-metrics/</a></li><li>Creating a Metrics Insights CloudWatch alarm. <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-alarm-create.html">https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-alarm-create.html</a></li><li>Metrics Insights quotas <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-limits.html">https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-limits.html</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bd87cb321fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detect EventBridge target failure: Part 2]]></title>
            <link>https://medium.com/@pubudusj/detect-eventbridge-target-failure-part-2-999b44757092?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/999b44757092</guid>
            <category><![CDATA[aws-eventbridge]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 10:06:46 GMT</pubDate>
            <atom:updated>2025-09-30T10:06:46.939Z</atom:updated>
            <content:encoded><![CDATA[<h3>Detect EventBridge target failure: Part 2— using enhanced monitoring</h3><h3>Intro</h3><p>When delivering messages to different targets using EventBridge, it is important to get notified if there are any delivery failures. EventBridge doesn’t provide this out of the box, but there are several ways to achieve this.</p><p>In the <a href="https://medium.com/@pubudusj/detect-eventbridge-target-failure-part-1-9159d617d83a">1st part of this blog</a> we discussed how we can get notified when a delivery fails to a target using a dead letter queue.</p><p>In this blog post we will discuss another (better?) option to achieve the same using <strong>EventBridge enhanced logging</strong>.</p><h3>EventBridge Enhanced Logging</h3><p>On 15th July 2025 <a href="https://aws.amazon.com/about-aws/whats-new/2025/07/amazon-eventbridge-enhanced-logging-improved-observability/">AWS introduced enhanced logging for EventBridge</a>. Which means now you can enable logging and EvenBridge will send those logs into a configured log delivery location.</p><p>Available log destinations are:</p><ul><li>S3</li><li>CloudWatch logs</li><li>Amazon Data Firehose stream</li></ul><p>Also you can configure the standard log levels as required: Trace, Info and Errors.</p><p>You can select more than one log destination and for each destination, you can select the same or a different log level. This is really useful, for example, I can use S3 to log all the traces while using CloudWatch log stream to log only errors.</p><h3>How it works</h3><p>In this example, I have used EventBridge enhanced logging to log any errors occurring into a CloudWatch log stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hzB-I41ely6qk7KLXyysvQ.png" /><figcaption>Image: Architecture</figcaption></figure><p>Simply, EventBridge will log any errors into the CloudWatch log. Once the log record is available in the CloudWatch log stream, there are multiple ways to trigger a CloudWatch alarm. In this example, I use the number of incoming log events as the metric to trigger the alarm.</p><p>If you use a CloudWatch log stream with log level trace (which includes info and errors as well) and still you want to trigger an alarm based on any error occurring, you may use an option like creating a metric filter in the log group.</p><h3>Try this yourself</h3><p>I have created a Github repository with a AWS SAM template for you to test this scenario in your AWS account.</p><ol><li>Clone the Github repository: <a href="https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging">https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging</a></li><li>Deploy the stack using below command:</li></ol><pre>sam deploy \<br>  --template-file template.yaml \<br>  --stack-name eb-fail-detection-with-enhanced-logging \<br>  --capabilities CAPABILITY_IAM \<br>  --no-confirm-changeset \<br>  --parameter-overrides NotificationEmail=[YourEmailAddress]</pre><p>3. Here, add your email address as NotificationEmail, so you will get the notification into your email box when the target fails.</p><p>4. Once the stack is deployed, you will get a SNS subscription confirmation email. You need to confirm it in order to receive notifications.</p><p>5. Then, publish a message into the created event bus with the source as xyzcorp.</p><p>6. This way the message will match the rule and try to deliver the message to the target.</p><p>7. I have blocked the permission for publishing the target intentionally to simulate the failure.</p><p>8. In a moment, you should get an email with the alarms status.</p><p>9. If you go to the CloudWatch log stream created, you can see the entry with log level ERROR and message type INVOCATION_FAILURE.</p><p><strong>Please note:<br></strong>As of now, creating the enhanced logging and delivering them to a delivery destination is not a straight forward configuration. You need to create a CloudWatch log group, a delivery source, a delivery destination and a logs delivery.</p><p>Refer: <a href="https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging/blob/main/template.yaml#L31-L59">https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging/blob/main/template.yaml#L31-L59</a></p><p>Which means if you need to send these enhanced logs to multiple destinations, you need to repeat configuring those resources per destination.</p><p>However, nice thing about this approach is that you only need to configure this only once on the event bus, and it will log the whole message life cycle within EventBridge systems including ingestion as well as deliveries to all the targets.</p><h3>Summary</h3><ol><li>Overall, enhanced logging is a great improvement to EventBridge because up until now, message delivery was a black box (for customers), specially the consumer side of EventBridge. With this new addition, you can track and debug the flow of your message within EventBridge systems transparently using the logs generated in each and every step that the message is going through, from ingest to delivery (of course depends on the log level that was configured).</li><li>Also, you can configure more than one log destinations as well as different log levels.</li><li>Since creating single log delivery using CloudFormation requires several AWS resources to be configured, I hope EventBridge team will provide easy to use method to configure this, ideally as properties of EventBridge Bus.</li></ol><h3>Resources</h3><ol><li>Monitor and debug event-driven applications with new Amazon EventBridge logging<br><a href="https://aws.amazon.com/blogs/aws/monitor-and-debug-event-driven-applications-with-new-amazon-eventbridge-logging/">https://aws.amazon.com/blogs/aws/monitor-and-debug-event-driven-applications-with-new-amazon-eventbridge-logging/</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=999b44757092" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detect EventBridge target failure: Part 1]]></title>
            <link>https://medium.com/@pubudusj/detect-eventbridge-target-failure-part-1-9159d617d83a?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/9159d617d83a</guid>
            <category><![CDATA[aws-eventbridge]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <category><![CDATA[aws-sqs]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Tue, 23 Sep 2025 20:20:55 GMT</pubDate>
            <atom:updated>2025-09-23T20:20:55.573Z</atom:updated>
            <content:encoded><![CDATA[<h3>Detect EventBridge target failure: Part 1 — with dead letter queue</h3><h3>Intro</h3><p>When EventBridge delivers messages to its target, there can be many reasons that cause failing to send a message. There can be permission issues, rate limits or the unavailability of the target or even can be a glitch in the AWS itself, just to name a few.</p><p>No matter what the reason is, it is always ideal to get notified that there is an issue delivering messages and the reason for the failure. In this blog post I will discuss how a dead letter queue can be useful to get notified when the EventBridge fails to deliver messages to its target.</p><h3>Dead letter queues</h3><p>Dead letter queues are unsung heroes of the event driven architecture 😀. Those are easy to set up and manage yet greatly improve the resilience of a system. Also it is very cost effective.</p><p>Let’s see how we can capture the target delivery failures in EventBridge using a dead letter queue.</p><p>Please note that EventBridge supports DLQ in a couple of “levels”. EventBridge bus can have a DLQ itself, or you can set a DLQ per target basis. Let’s discuss the differences.</p><h3>DLQ on EventBridge bus level</h3><p>EventBridge bus can have a DLQ of its own. However, this is limited to capture any errors related to the KMS encryption. EventBridge sends events that aren’t successfully encrypted to the DLQ.</p><p>You can only see this DLQ setting in the EventBridge in AWS console only if customer managed KMS is used to encrypt messages. In fact, it is part of the Encryption settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IwLKmywF-KwvJFXrbFe_PQ.png" /><figcaption>Image: DLQ for Event bus only available when customer managed KMS is in use.</figcaption></figure><p>However, <strong>this DLQ will NOT capture any target related failures</strong>, so that we cannot use this for our purpose.</p><h3>DLQ on EventBridge target level</h3><p>When EventBridge cannot deliver a message to a target, we can set up a SQS queue to put that message in, on the target level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*GXP1vwfIXOQGxkum-zdIgg.png" /><figcaption>Image: DLQ on target.</figcaption></figure><p>Since one rule can have more than one target, each target can have different DLQs as well. You can use the same SQS queue as the DLQ for all the targets, but you have to configure it for each and every target separately. It may sound like repetitive work, but if you use an infrastructure as a code tool like CDK or CloudFormation, this is not complex.</p><h4>How it works</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8Tt7wfBfrxTES6LUxjdhMQ.png" /><figcaption>Image: High level architecture.</figcaption></figure><ol><li>EventBus tries to deliver a message to its target (here, it is a SQS queue) via EventBridge rule.</li><li>Let’s assume there is a permission issue, and the message cannot be delivered.</li><li>Then, EventBridge will put the message into the DLQ configured for this specific target.</li><li>In Cloudwatch, there is an alarm set up to be triggered whenever there is a message in the DLQ.</li><li>When the failed message is in the DLQ, Alarm triggers and there is a SNS topic configured as the alarm action.</li><li>And when the alarm action publishes a message to SNS topic, it will send the notification to all the subscribers to notify about the failure.</li></ol><h3>Try this yourself</h3><p>I have created an AWS SAM template to try this scenario in your AWS account.</p><ol><li>Clone the Github repository: <a href="https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq">https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq</a></li><li>Deploy the stack using below SAM command:</li></ol><pre>sam deploy \<br>  --template-file template.yaml \<br>  --stack-name event-bridge-target-failure-detection-with-dlq \<br>  --capabilities CAPABILITY_IAM \<br>  --no-confirm-changeset \<br>  --parameter-overrides NotificationEmail=[YourEmailAddress]</pre><p>3. Here, add your email address as NotificationEmail, so you will get the notification into your email box when the target fails.</p><p>4. Once the stack is deployed, you will get a SNS subscription confirmation email. You need to confirm it in order to receive notifications.</p><p>5. Then, publish a message into the created event bus with the source as xyzcorp.</p><p>6. This way the message will match the rule and try to deliver the message to the target.</p><p>7. I have blocked the permission for publishing the target intentionally to simulate the failure.</p><p>8. In a moment, you should get an email with the alarms status.</p><p>9. Further, if you check the messages in DLQ, you can see the failed message and in the message attributes, you may see the reason of failure (depends on the reason).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*zUbUfgr5w_Jsk7BKdSqJMQ.png" /><figcaption>Image: Message attributes of a failed message in DLQ.</figcaption></figure><p>10. You can configure the threshold, period and evaluation periods of the alarm as needed to control the frequency of the notifications in case of a failure.<br><a href="https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq/blob/main/template.yaml#L61-L63">https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq/blob/main/template.yaml#L61-L63</a></p><h3>Summary</h3><ol><li>EventBridge bus has a DLQ but it is for a different purpose and cannot capture any target failures.</li><li>You can use this dead letter queue approach, to capture any messages which cannot be delivered to the target. Based on the no of messages in the queue, you can get notified using CloudWatch metric and SNS. However, you will need to configure it for each and every EventBridge target separately. Using an IAC tool to configure this may be convenient.</li><li>I will discuss another solution to achieve the same in part 2 of this blog post.</li></ol><h3>Resources</h3><ol><li>Using dead-letter queues to process undelivered events in EventBridge <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html">https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9159d617d83a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EventBridge to SQS when cross region and cross account]]></title>
            <link>https://medium.com/@pubudusj/eventbridge-to-sqs-when-cross-region-and-cross-account-7538e2d952f6?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/7538e2d952f6</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[eventbridge]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Wed, 17 Sep 2025 07:42:35 GMT</pubDate>
            <atom:updated>2025-09-17T07:42:35.134Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>Delivering a message to a target SQS queue from EventBridge is a very common requirement in an event driven application. Sometimes, this SQS queue can be in a different region and may be in a different AWS account. Depending on where the target SQS queue lives, the approach to how you set up the solution differs.</p><p>In this blog post, we are going to see different scenarios on how EventBridge can deliver messages to a target SQS queue when the target SQS queue is in a different region or different AWS account.</p><h3>Scenario 1: Event bus, rule and target SQS queue in same AWS account, same region</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*ZA58QymYnirl0v2vzFxxEg.png" /><figcaption>Image: All resources in the same AWS account, same region</figcaption></figure><p>This is the most common and simplest scenario. Event bus has a rule which directly configured a SQS queue as a target. All the resources are in the same region in the same AWS account.</p><h3>Scenario 2: Event bus and rule in one region, target SQS queue in another region in the same AWS account</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XVrWpRdJdODgKrQ6xlNpIQ.png" /><figcaption>Image: Event bus and rule in one region, target SQS queue in another region in the same AWS account</figcaption></figure><p>In this scenario, the Event bus and the rule is in one region. The target SQS queue is in the same AWS account, but in a different region.</p><blockquote>As of today (September 2025), EventBridge supports SQS as a target only when it is in the same region.</blockquote><p>Because of that, first you have to create a new Event bus (or use the default Event bus) in the 2nd region. And configure the SQS queue as a target of that Event bus. Then, configure the event bus in the second region to be a target in the EventBridge rule of the first region.</p><h3>Scenario 3: Event bus, rule in one AWS account, target SQS queue in another AWS account, but all are in the same region</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rvr3u9G81G0RdjtDv2VuVw.png" /><figcaption>Image: Event bus and rule in one AWS account, target SQS queue in another AWS account, but all are in the same region</figcaption></figure><p>AWS introduced cross account support for EventBridge targets beginning of 2025. Previously, when delivering messages to cross account resource, you had to use a Event bus in the 2nd account to establish the connection (similar to scenario 2 above).</p><p>With this new feature, it is possible to have a cross account target which simplifies the message delivery from an event bus to a SQS queue in another account.</p><p>However, there is a limitation. Although the cross account delivery is possible, the EventBridge bus and the rule and the target in the 2nd AWS account must be in the same region.</p><h3>Scenario 4: Event bus, rule in one AWS account, target SQS queue in a different region in another AWS account</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mOinbuhQ9uJUphl6DL7FA.png" /><figcaption>Image: Event bus, rule in one AWS account, target SQS queue in a different region in another AWS account</figcaption></figure><p>As mentioned earlier, cross account delivery is supported only when the target SQS queue is in the same region. If the SQS queue is in a different region, you have to use an Event bus in the 2nd account in between.</p><h3>Summary</h3><p>The pattern is simple. SQS can be a target in an EventBridge rule when the rule and the SQS queue both are in the same region. Target SQS queue can be in another AWS account as long as the region is the same.</p><p>If the SQS queue is in a different region than the EventBridge rule, you have to use an intermediate EventBus in the 2nd region. This intermediate EventBus and the target SQS can be in another AWS account.</p><h3>Resources</h3><ol><li>Event bus targets in Amazon EventBridge — <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html">https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html</a></li><li>Introducing cross-account targets for Amazon EventBridge Event Buses — <a href="https://aws.amazon.com/blogs/compute/introducing-cross-account-targets-for-amazon-eventbridge-event-buses/">https://aws.amazon.com/blogs/compute/introducing-cross-account-targets-for-amazon-eventbridge-event-buses/</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev/">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7538e2d952f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I built a spelling game with AWS Serverless and GenAI]]></title>
            <link>https://medium.com/@pubudusj/how-i-built-a-spelling-game-with-aws-serverless-and-genai-9e9d08d79165?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e9d08d79165</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cdk]]></category>
            <category><![CDATA[amazon-q]]></category>
            <category><![CDATA[bedrock]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Wed, 02 Apr 2025 14:48:11 GMT</pubDate>
            <atom:updated>2025-04-02T14:48:11.340Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*-mV5xJ5hvQh0JM5GnCQsKQ.png" /></figure><h3>Intro</h3><p>Last December 2024, as part of the <a href="https://awsdevchallenge.devpost.com/">AWS Game Builder Challenge</a>, I built a simple spelling game. For that, I used AWS Serverless and GenAI services and CDK was used as the IAC tool. This was my first experience building something with the help of GenAI and using Bedrock in an application. This blog post explains the project and my experience building this simple spelling game.</p><p>👇 This is the final result — the game you can play.</p><blockquote><a href="https://spelling-game.pubudu.dev/"><strong>https://spelling-game.pubudu.dev/</strong></a></blockquote><h3>How to play the game?</h3><ol><li>Player can select a language to play the game. As of now, only English (US) and Dutch are the available languages.</li><li>When the player selects a language, there will be a maximum 5 words generated. This includes an audio, brief meaning of the word and the number of characters for each word.</li><li>Then the player needs to fill the word in the text box given.</li><li>There is an indicator next to the text box how many characters have been entered in the text box and how many characters are required for the word. This indicator will be red until the required number of characters are filled, then it turns into green.</li><li>There is a timer that starts as soon as the words are generated. This is based on the number of words generated.</li><li>When the remaining time is less than 30 seconds, the background of the page as well as the background of the timer turns into red.</li><li>Player can submit the answers before the timer runs out, else it will be automatically submitted.</li><li>Then, the answers are evaluated and based on the number of correct answers, there is a pop up visible.</li><li>If all the answers are correct, there will be a “Confetti” effect appearing on the page.</li><li>Clicking on the ‘show results’ button on the pop up, the player can see the correct/incorrect answers and the correct word (in case of an incorrect answer).</li></ol><h3>Implementation</h3><p>There are 2 main parts of this application.</p><ul><li>Backend — where words are generated and APIs are available.</li><li>Frontend — Vue.js application for the player to interact.</li></ul><h3>Source code</h3><p>There are two repositories available with the complete source code.</p><ul><li>Backend — <a href="https://github.com/pubudusj/spelling-game-backend">https://github.com/pubudusj/spelling-game-backend</a></li><li>Frontend — <a href="https://github.com/pubudusj/spelling-game-frontend">https://github.com/pubudusj/spelling-game-frontend</a></li></ul><h3>Deployment instructions</h3><p>Backend is implemented using AWS CDK and can be deployed as a generic CDK application. The URL of the Cloudfront distribution is required for the frontend to work.</p><p>In the frontend, add the Cloudfront API URL to the VITE_API_BASE_URL in the env file. Install necessary dependencies and then run the application using npm run dev in the dev mode. Or you can build the frontend app using npm run build.</p><p>When the backend stack is deployed, it will output the S3 hosting bucket name. You can copy the built frontend app into this bucket which will host the frontend via the Cloudfront distribution created in the backend stack.</p><h3>Backend</h3><p>There are 2 main components of the backend.</p><ol><li>Words generator component — to generate and save words in the database.</li><li>API component — to serve frontend.</li></ol><h3>Backend — words generator component</h3><p>Here is the high level overview of the words generator component and the steps within the state machine. Within Step Functions, there will be many AWS services called as explained below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-a-iZT4JGqUEqGAkAA9FQ.png" /><figcaption>Image: Words generator overview</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*Cz_gXeKZyqEdJCubOmIn0Q.png" /><figcaption>Image: Words generator state machine</figcaption></figure><ol><li>Words Generator Step Functions state machine is responsible for generating words.</li><li>This Step function execution takes the language code as an input. ex: en-US, nl-NL, etc.</li><li>As the first step, Bedrock InvokeModel is being called to generate 5 words with descriptions for each word, based on the language.</li><li>Here, the Anthropic Claude 3 Haiku model is being used which gives better balance of accuracy and the pricing in this scenario.</li><li>Here is the prompt I used to generate the words:</li></ol><blockquote>“Generate 5 unique words that have a random number of characters more than 4 and less than 10 in Dutch language. For each word, provide a brief description of its meaning in English with more than a couple of words. Produce output only in a minified JSON array with the keys word and description. Word must always be in lowercase.”</blockquote><p>6. Here, the response is a JSON string.</p><p>7. Within the step’s Result selector, the response will be converted to an array using the intrinsic function — <em>StringToJson</em>.</p><pre>{<br>  &quot;words.$&quot;: &quot;States.StringToJson($.Body.content[0].text)&quot;<br>}</pre><p>8. Then there is a map state where each word is an input to each map.</p><p>9. Within the map state, there are branches (currently two) based on the language.</p><p>10. In each branch, there is a step to synthesise the word using Polly using <em>StartSpeechSynthesisTask</em>.</p><p><em>11. StartSpeechSynthesisTask</em> is an async operation. So, there is an immediate step to check if the synthesis task is completed using Polly’s <em>GetSpeechSynthesisTask</em>.</p><p>12. Great thing about the <em>StartSpeechSynthesisTask</em> api is that it not only synthesises, but automatically saves the mp3 into the given S3 bucket.</p><p>13. If the speech synthesis task is not finished based on the status of the <em>GetSpeechSynthesisTask</em>, it waits and retries the status check.</p><p>14. Once synthesis is done, the execution continues to save word to DynamoDB step.</p><p>15. In this step, we use DynamoDB’s <em>PutItem</em> api to save the generated data to the table. One record consists of below data:</p><pre>pk - Primary key in the format of Word#{LanguageCode}. ex: Word#en-US<br>sk - MD5 hash of the word - here, intrinsic function States.Hash($.word, &#39;MD5&#39;)is in use. <br>word - The word generated by Bedrock.<br>description - The description generated by Bedrock.<br>s3file - Mp3 file location provided by Polly synthesis task.<br>charcount - Character count of the word. This is retrieved from Polly synthesis task.<br>updated_at - Update timestamp.</pre><p>16. Since this synthesis and save to db task runs on a map state, after a single execution, there will be a maximum 5 new words available in the ddb for the given language.</p><p>17. There are two EventBridge Schedules running every 5 minutes to call this State Machine with the different language codes — English and Dutch.</p><h3>Backend — API component</h3><p>API component is to create resources to interact with the frontend.</p><p>There are two APIs available in the backend.</p><ol><li>POST /questions - To generate questions using Step Functions to appear on the frontend.</li><li>POST /answers - To validate the answers submitted by the player.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wns4T-3-ikLZS8xPd5JPxQ.png" /><figcaption>Image: Frontend and API components</figcaption></figure><h4>Generate questions API</h4><ol><li>Generate questions API accept one argument. Which is the language code.</li><li>This /questions api has a proxy integration to a Lambda Function which will start an execution in Questions Generator State Machine synchronously using start_sync_execution SDK call.</li><li>This Questions Generator State Machine is in type — <em>Express</em>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*0h1wpTFczqhgiUTqIjwxNw.png" /><figcaption>Image: Questions Generator State Machine</figcaption></figure><p>4. A sample input is as follows:</p><pre>{<br>   &quot;language&quot;:&quot;en-US&quot;,<br>   &quot;iterate&quot;:[1, 2, 3, 4, 5]<br>}</pre><p>5. Here, “<em>iterate</em>” is a hard coded array to start a map execution within the state machine.</p><p>6. Within the state machine, first the map state is executed based on the “iterate” array from the input.</p><p>7. Inside the map state, first, it fetches max 50 records from DynamoDB. Here, DynamoDB scan is used. However, in order to fetch some random data, a random <em>ExclusiveStartKey</em> is in use with the help of the intrinsic function <em>UUID()</em>. Also, <em>FilterExpression</em> is used to filter the records applicable only for the given language code.</p><pre>{<br>  &quot;TableName&quot;: &quot;arn:aws:dynamodb:****&quot;,<br>  &quot;Limit&quot;: 50,<br>  &quot;ExclusiveStartKey&quot;: {<br>    &quot;pk&quot;: {<br>      &quot;S.$&quot;: &quot;States.Format(&#39;Word#{}&#39;, $$.Execution.Input.language)&quot;<br>    },<br>    &quot;sk&quot;: {<br>      &quot;S.$&quot;: &quot;States.UUID()&quot;<br>    }<br>  },<br>  &quot;FilterExpression&quot;: &quot;pk = :pk&quot;,<br>  &quot;ExpressionAttributeValues&quot;: {<br>    &quot;:pk&quot;: {<br>      &quot;S.$&quot;: &quot;States.Format(&#39;Word#{}&#39;, $$.Execution.Input.language)&quot;<br>    }<br>  },<br>  &quot;ReturnConsumedCapacity&quot;: &quot;TOTAL&quot;<br>}</pre><p>8. Next, the number of item counts returned from the previous step is checked.</p><p>9. If the count is more than 0, then in the next step, single random record is selected from them. This step is a Pass state with transformation using Parameters, which uses intrinsic functions — <em>ArrayGetItem</em>, <em>MathRandom</em> and <em>ArrayLength</em>.</p><pre>{<br>  &quot;item.$&quot;: &quot;States.ArrayGetItem($.items,States.MathRandom(0, States.ArrayLength($.items)))&quot;<br>}</pre><p>10. Then, the selected record is being sent to the Generate Pre signed URL step. This is a Lambda function, which generates a pre-signed URL for the s3file path of the record. So, from the frontend the mp3 file can be played using this pre-signed url. Also there is a transformation of data within this Lambda function. The expiry of the pre-signed url is set to minimum because it is only required within the session of the game.</p><p>11. This is the last step within the map state which outputs the record in below format.</p><pre>{<br>   &quot;id&quot;:&quot;XXXX&quot;,<br>   &quot;description&quot;:&quot;Description of the word&quot;,<br>   &quot;charcount&quot;:8,<br>   &quot;language&quot;:&quot;en-US&quot;,<br>   &quot;url&quot;:&quot;Presigned-url&quot;<br>}</pre><p>12. Once all the map steps are completed, there is a final aggregation Lambda function — <em>GetUniqueResultsLambda</em>. Since each map step is independent, there is a possibility of selecting the same record in more than one map state. This Lambda function simply removes such duplications.</p><p>13. And the response is returned to the frontend as the response to the /questions endpoint.</p><h4>Validate answers API</h4><ol><li>POST /answers API is responsible for validating the answers submitted in the frontend.</li><li>This API endpoint has a proxy Lambda function which accepts the payload in below format:</li></ol><pre>{<br>   &quot;language&quot;:&quot;en-US&quot;,<br>   &quot;answers&quot;:[<br>      {<br>         &quot;id&quot;:&quot;312e8b6583d4b65b&quot;,<br>         &quot;word&quot;:&quot;effect&quot;<br>      },<br>      {<br>         &quot;id&quot;:&quot;99b788c54c1a8265&quot;,<br>         &quot;word&quot;:&quot;perilous&quot;<br>      },<br>      ...<br>      ...<br>   ]<br>}</pre><p>3. Within the Lambda function, it does a DynamoDB’s <em>batch_get_item</em> SDK call to fetch words per ids and match with the word provided in the API.</p><p>4. Then it returns the response in below format:</p><pre>[<br>    {<br>        &quot;id&quot;: &quot;312e8b6583d4b65b&quot;,<br>        &quot;original_word&quot;: &quot;affect&quot;,<br>        &quot;correct&quot;: false<br>    },<br>    {<br>        &quot;id&quot;: &quot;99b788c54c1a8265&quot;,<br>        &quot;original_word&quot;: &quot;perilous&quot;,<br>        &quot;correct&quot;: true<br>    },<br>    ...<br>    ...<br>]</pre><p>5. Based on the value of “correct”, frontend will calculate and show the results.</p><h3>Frontend</h3><p>For the frontend, I have used a simple single page application built with Vue.js. I have very limited knowledge on frontend technologies. Because of that, I used <a href="https://aws.amazon.com/q/developer/">Amazon Q Developer</a> on VSCode to implement the frontend application.</p><p>Almost 95% of the frontend application was built by Amazon Q Developer. I have asked different questions and in most of the cases, Amazon Q was able to analyse the code and generate the code as per my requirements. This was a step by step process where I asked Amazon Q to generate specific functionality at a time.</p><p>Here are some work in progress “<em>versions</em>” of the application that was implemented and fine tuned step by step using Amazon Q.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*69EUaem_b-gkb8gE_oeyEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*CZyNZRCA7CLrJYjnZUV-xg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gZZn4LzxZKq1kd9QhZZjzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gIU_EdL4ZJfGbj07zg2gig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXTeuoYm0mHrPs2G4vUIfw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*nD8QMWv2uky-QxB2IMf72Q.png" /></figure><p>Here are some examples for the questions I asked from Amazon Q and how it analysed and generated code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*K8_YLKjrF7nWcIQyislaEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*n7NvJnkItwr03skhwmzkyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*viJL6R_o4eqmfWDI56SR4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*G2LQjy5pfMY6rIczlLk4Hw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*w9YI66E4ZF2eLvyMiIS5Xw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*VUO4iW1FsN15vgHOAFx7Sw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*ZxSOpCXtgSkoQY8RJJ52ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*A7jkZZHGCQmaX-97sf5-Jw.png" /></figure><h3>Lessons learnt</h3><p>Below are some of the lessons I learnt while I was working on this project, including some feedback on some of the services I used here.</p><ol><li>Bedrock does not always return JSON. Within the prompt I used, I stated — “Produce output only in a minified JSON array with the keys word and description”. However, once in a while, Bedrock returns data in different formats. This could have been more accurate if I add the beginning of the response in the prompt, so Bedrock can continue from there. However, this will increase the request token count for each API call. So, to avoid additional cost and also, the error rate is acceptable (since this is anyway a background job), I kept this prompt as it is.</li><li>Amazon Polly’s <em>StartSpeechSynthesisTask</em> doesn’t support S3 path. We can provide the <em>OutputS3BucketName</em> where the generated audio will be stored. However, we cannot specify a path to save the object. Instead, I have used the OutputS3KeyPrefix parameter to provide the path with the language code so, the audio is saved in s3://bucket_name/language/code/file_name.mp3.However, one minor issue with that is, Polly always adds a dot (.) between the file name and the prefix. So, all the files generated in the sub path start with a dot.</li><li>Cost of Polly. Polly has Generative and Neural text-to-speech engines apart from Standard. However, the cost of them is quite high compared to Standard. Also, those are available only for a limited number of languages. <a href="https://aws.amazon.com/polly/pricing/">https://aws.amazon.com/polly/pricing/</a></li><li>Selecting random items from the DynamoDB table is hard. There is no straightforward way to achieve this. That’s why I had to use a <em>ExclusiveStartKey</em> and fetch maximum 50 items and select one random item. This might introduce some duplications. That’s the reason I had to use the <em>GetUniqueResultsLambda</em> which removes any duplicates from the map job.</li><li>I initially used direct integrations to start the Step Functions express workflow to generate questions directly from API Gateway. However, the VTL is complex to build specially to get the response in a specific format. So I stuck to the Lambda proxy option which was more simple.</li><li>Initially I have hosted the frontend using <a href="https://docs.aws.amazon.com/amplify/latest/userguide/welcome.html">Amplify Hosting</a>. However, I wanted to restrict the API Gateway access only from Amplify project, but there is no option yet. So I switched to Cloudfront, S3 set up. This is described in detail in one of my previous blogs: <a href="https://pubudu.dev/posts/access-api-gw-rest-api-only-from-cloudfront/">Enforce CloudFront-Only Access for AWS API Gateway</a></li><li>For the frontend, I indeed did <a href="https://en.wikipedia.org/wiki/Vibe_coding">Vibe coding</a>. So, there is a high chance that any frontend specialist might find unacceptable code in it 😉.</li><li>Most of the time, Amazon Q generated code which includes the expected functionality. However, there were cases it couldn’t fix some code snippets. For example, when I asked to center a component, it couldn’t fix it even after 20th iteration. May be this is because of the complexity of the single page structure.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2HB4V35buOYgh2_o2HGdOw.png" /><figcaption>Image: Amazon Q attempts to align center a div</figcaption></figure><h3>Possible improvements</h3><ol><li>Currently there is no ‘history’ of the games a particular player played. Adding a logged-in mode to record the player status/progress can be a nice feature.</li><li>Add more languages: Currently only English(US) and Dutch are available to select. Having more languages is nice. However, those need to be supported by Polly.</li><li>Finished game’s standing against all other games in the results can be a nice feature. For this, recording the results and comparing against all the results will be required.</li></ol><p>What are the other possible improvements you see?</p><h3>Conclusion</h3><p>Overall, it was a great experience using Bedrock and Amazon Q. I didn’t win the hackathon, but as this was the first time I was using those GenAI services, I learnt a lot from this project and enjoyed a lot. Frontend code was mostly generated by Amazon Q and I am happy with the results of it although I am certain there can be many improvements in the frontend code.</p><h3>Useful Links</h3><ol><li>Getting started with Amazon Q Developer: <a href="https://aws.amazon.com/q/developer/getting-started/">https://aws.amazon.com/q/developer/getting-started/</a></li><li>Getting started with Amazon Bedrock: <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html">https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html</a></li><li>Invoke and customize Amazon Bedrock models with Step Functions: <a href="https://docs.aws.amazon.com/step-functions/latest/dg/connect-bedrock.html">https://docs.aws.amazon.com/step-functions/latest/dg/connect-bedrock.html</a></li><li>Claude Prompt engineering overview: <a href="https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview">https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview</a></li></ol><p>👋 I regularly create content on <strong>AWS </strong>and<strong> Serverless</strong>, and if you’re interested, feel free to follow/connect with me so you don’t miss out on my latest posts!</p><p>LinkedIn: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a><br>Twitter/X: <a href="https://x.com/pubudusj">https://x.com/pubudusj</a><br>Medium: <a href="https://medium.com/@pubudusj">https://medium.com/@pubudusj</a><br>Personal blog: <a href="https://pubudu.dev">https://pubudu.dev</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e9d08d79165" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enforce CloudFront-Only Access for AWS API Gateway]]></title>
            <link>https://medium.com/@pubudusj/enforce-cloudfront-only-access-for-aws-api-gateway-3b365279e814?source=rss-35c84ac8c967------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b365279e814</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[api-gateway]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloudfront]]></category>
            <dc:creator><![CDATA[Pubudu Jayawardana]]></dc:creator>
            <pubDate>Thu, 06 Mar 2025 16:04:26 GMT</pubDate>
            <atom:updated>2025-03-06T16:04:26.825Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kedRgOzlPUYfmhB1WXMJFQ.png" /></figure><p>Recently, I had a requirement to expose a REST API from API Gateway exclusively through CloudFront. Keeping API Gateway behind Cloudfront provides additional layer of security because Cloudfront comes with automatic protections of AWS Shield Standard, at no additional charge.</p><p>There are several ways to achieve this, for example, using a signed request using Lambda@Edge in Cloudfront, but I went for a simpler solution, which is using a custom header in the request to API Gateway from the Cloudfront and using Lambda authorizer at the API gateway to validate it. This blog post explains how to implement this solution.</p><h3>How it works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VuniK9-APYC4QrrD4VY6NA.png" /><figcaption>Image: Architecture</figcaption></figure><ol><li>In the Cloudfront distribution, we create an origin for API gateway endpoint.</li><li>Here, we define a custom header on this Cloudfront origin.</li><li>When a request comes to Cloudfront distribution, as per the behaviour we define, it will call the API gateway endpoint.</li><li>This request will contain the custom header.</li><li>In the API Gateway end, there is an Lambda authorizer which validates this incoming header.</li><li>There is a SSM secure parameter which holds the value of the header.</li><li>When it is required to validate the incoming header, Lambda authorizer will fetch this value from the parameter store.</li></ol><p>Communication between the Cloudfront distribution and the API gateway is secure and there is no way someone who doesn’t have access to Cloudfront settings will get to know the header value. However, keeping a static value for a validation seems not correct. So, it is better to rotate the value of the header for added security.</p><h4>Rotating the header value</h4><ol><li>There is a EventBridge scheduler which will invoke a Lambda function in a given interval.</li><li>This Lambda function will perform below actions:</li></ol><ul><li>Generate a random value for the header.</li><li>Update the SSM secure parameter.</li><li>Update the Cloudfront origin’s custom header value.</li></ul><p>3. Once this Lambda execution is successful, in both sides (Cloudfront and API Gateway) the new header value will be used.</p><h3>Try this yourself</h3><p>Here is a Github repository of the project I created to try out this solution. <a href="https://github.com/pubudusj/secure-api-with-cloudfront">https://github.com/pubudusj/secure-api-with-cloudfront</a></p><p>You can deploy this to your AWS account using CDK.</p><blockquote>Please note: First, you need to create a secure SSM parameter with any value. Then create an .env file copying the .env.example file in the project root directory and set the name/path of the parameter in the .env file.</blockquote><p>In the CDK code, you will notice that the initial header value is hardcoded.</p><pre>origin=origins.RestApiOrigin(<br>       rest_api,<br>       origin_path=&quot;/prod&quot;,<br>       custom_headers={custom_header_key: &quot;test&quot;},<br>   ),</pre><p>Also, this obviously does not match with the value you have in the SSM parameter.</p><p>This is fine, because this stack includes a <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html">Cloudformation custom resource</a>, which will be executed on create. This custom resource will start the initial token rotation as soon as the stack is created which generates a new header value and syncs both SSM secure parameter and Cloudfront distribution.</p><p>Once the stack is deployed, you can access the API using API Gateway endpoint and also with Cloudfront endpoint.</p><p>For example:</p><ul><li>Cloudfront endpoint: <a href="https://[CloudfrontPrefix].cloudfront.net/prod/hello">https://[CloudfrontPrefix].cloudfront.net/prod/hello</a></li><li>API Gateway endpoint: <a href="https://[APIGWPrefix].execute-api.[region].amazonaws.com/prod/hello">https://[APIGWPrefix].execute-api.[region].amazonaws.com/prod/hello</a></li></ul><p>You will notice that Cloudfront endpoint works fine while there is a 401 Unauthorized error from the API Gateway endpoint.</p><h3>Tips/Lesson Learned</h3><ol><li>Here I have used an SSM Parameter store with a custom built rotation mechanism. However, you can use Secret manager instead with its inbuilt rotation features. Make sure you are aware of the differences. Yan has done this comparison in detail in his blog post <a href="https://theburningmonk.com/2023/03/the-old-faithful-why-ssm-parameter-store-still-reigns-over-secrets-manager/">https://theburningmonk.com/2023/03/the-old-faithful-why-ssm-parameter-store-still-reigns-over-secrets-manager/</a></li><li>Keep in mind that when setting a new custom header value on the Cloudfront origin, it will perform a re-deployment. Cloudfront re-deployment takes time. It can be within a couple of minutes to 10–15 minutes too. But for a change like this, it should be faster.<br>Because of this reason, during a Cloudfront re-deployment, your SSM parameter and incoming header value at API Gateway can be different. One solution I used to address this is to cache the Lambda authorization result. Here I used a ttl of 5 minutes, but you can adjust as required.</li></ol><pre>authorizer = apigateway.RequestAuthorizer(<br>   self,<br>   &quot;LambdaHeaderAuthorizer&quot;,<br>   handler=custom_authorizer,<br>   identity_sources=[apigateway.IdentitySource.header(custom_header_key)],<br>   results_cache_ttl=Duration.minutes(5),<br>)</pre><h3>Useful Links</h3><ol><li>Cloudfront custom headers: <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/add-origin-custom-headers.html">https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/add-origin-custom-headers.html</a></li><li>API Gateway Lambda authorizer: <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html">https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html</a></li></ol><p>Please let me know your thoughts on this implementation.</p><p>You can find more AWS and Serverless contents at my personal blog: <a href="https://pubudu.dev/">https://pubudu.dev/</a></p><p>And don’t forget to follow me on LinkedIn too: <a href="https://www.linkedin.com/in/pubudusj/">https://www.linkedin.com/in/pubudusj</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b365279e814" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>