<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by bitsofinfo on Medium]]></title>
        <description><![CDATA[Stories by bitsofinfo on Medium]]></description>
        <link>https://medium.com/@bitsofinfo?source=rss-f1b4cea7e056------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kGtM7a-g9lp7N38Qa5BnTQ.png</url>
            <title>Stories by bitsofinfo on Medium</title>
            <link>https://medium.com/@bitsofinfo?source=rss-f1b4cea7e056------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:44:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bitsofinfo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Wrangling Serverless Development & Devops]]></title>
            <link>https://medium.com/devops-dudes/wrangling-serverless-development-devops-5824a800d04f?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/5824a800d04f</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Sat, 21 Sep 2024 04:56:56 GMT</pubDate>
            <atom:updated>2026-04-09T15:49:36.709Z</atom:updated>
            <content:encoded><![CDATA[<p>A few years back I wrote about an <a href="https://bitsofinfo.wordpress.com/2021/11/29/serverless-aws-lambda-architecture-for-large-scale-data-ingestion/">architecture for large-scale serverless data ingestion on AWS</a>. That post covered the design of a <a href="https://github.com/serverless/serverless">Serverless Framework</a> based import platform; an orchestrator service that picks up files landing in an S3 bucket and fans out processing to a set of consumer services, each handling a specific data format or third party integration. The design worked well and the project had been running in production for some time.</p><p>As time progressed the platform had grown to span several separate repositories, each one a variation of the same pattern. There was existing documentation and scripts that covered how to work with the services locally and deploy them, and deployments were infrequent enough that the lack of formal CI had not been a pressing issue. That said, the setup complexity was real; getting a devops runner environment running correctly required a non-trivial amount of tool chain configuration, and making a change that needed to land consistently across all the services meant having to touch each repo individually. The tooling largely existed but what was needed was a way to reduce the friction of using it, eliminate the proliferation of per-repo changes and get things into a state where CI was a realistic option.</p><p>This post covers the work done to address that: consolidating everything into a monorepo, standardizing the tooling behind a consistent task runner and containerized build environment, and wiring up CI so deployments were no longer dependent on a specific local or brittle remote runner setup.</p><h3>Where things started</h3><p>Each import consumer service lived in its own repository. They all followed the same fundamental pattern; an orchestrator ( data-import) and N consumer services ( data-import-[name]), each one a <a href="https://github.com/serverless/serverless">Serverless Framework </a>project with its own serverless.yml, its own package.json, its own set of configurations plus custom logic code. In theory they were all instances of the same architecture. In practice they had diverged enough that making a cross-cutting change meant touching every repository individually and manually verifying nothing had broken, usually on the machine of whichever admin happened to have the right set of tools.</p><p>This was a pretty complicated situation to try to address; with lots of moving parts; can’t we make this more manageable?</p><h3>Monorepo layout</h3><p>After much trial and error while progressing to a larger vision, the end result was a single repository managing more than a dozen services; one orchestrator and the rest consumers. Shared code, configuration templates, scripts, and operational tools all live at the top level. Each deployable service lives under services/:</p><pre>data-import/<br>├── services/               # all deployable Serverless Framework services<br>│   ├── data-import/            # orchestrator<br>│   ├── data-import-abc/        # consumer<br>│   ├── data-import-def/        # consumer<br>│   └── ...<br>├── configs/                # shared config templates<br>│   ├── serverless/         # serverless.yml component fragments<br>│   ├── nodejs/             # package.json component fragments<br>│   └── aws/                # aws policy config fragments<br>├── libs/                   # shared Node.js libraries<br>│   ├── imp-lib-core/<br>│   ├── imp-lib-custom1/<br>│   └── imp-lib-custom2/<br>├── tasks/                  # shared Taskfile definitions<br>├── scripts/                # shared shell automation scripts (backs the Taskfile tasks)<br>├── docker/                 # imp-shell dockerfile and build infrastructure<br>├── devops/                 # CICD pipeline templates<br>├── resources/              # local dev resources (s3, elasticmq, plugins)<br>├── templates/              # scaffold for new services<br>└── docs/</pre><p>The two key service types reflect the architecture from the original post. services/data-import is the orchestrator; files land in an S3 landing zone and it picks them up and kicks off the processing flow. services/data-import-[name] are the consumers; each receives raw rows or file pointers from the orchestrator, processes them in service-specific ways, and reports results back.</p><h3>The service directory pattern</h3><p>Every service under services/ follows an identical structure. The majority of that structure is symlinked back to shared monorepo locations rather than copied. This is the foundational design principle:</p><pre>services/data-import-[name]/<br>├── service.metadata.yml         # service-specific high-level config<br>├── serverless.service.yml       # service-specific serverless additions<br>├── package.service.json         # service-specific npm dependencies<br>├── service.taskfile.yml         # service-specific custom tasks<br>├── handler.js (or handler.py)   # lambda handler logic<br>├── Taskfile.yml -&gt; tasks/Taskfile.service.yml       (symlink)<br>├── scripts/    -&gt; ../../scripts/                    (symlink)<br>├── tasks/      -&gt; ../../tasks/                      (symlink)<br>├── devops/     -&gt; ../../devops/                     (symlink)<br>├── libs/imp-lib-core    -&gt; ../../libs/imp-lib-core  (symlink)<br>├── libs/imp-lib-custom1 -&gt; ../../libs/imp-lib-custom1 (symlink)<br>├── libs/imp-lib-custom2 -&gt; ../../libs/imp-lib-custom2 (symlink)<br>│<br>├── serverless.yml               # generated by tasks<br>└── package.json                 # generated by tasks</pre><p>Changes to shared scripts, task definitions, libs, templates automatically apply to every service in the repo without any sync or copy step. The only files that are truly service-specific are the function handler code (python or node), service.metadata.yml, and the service-specific config additions.</p><p>Much of what I am about to describe is automated via a utility called <a href="https://taskfile.dev/">Taskfile</a> (which I will cover more later).</p><h3>Config template system and service.metadata.yml</h3><p>A central problem with managing many similar-but-not-identical serverless framework services is configuration drift. The solution was a component-based config merge system. Shared serverless.yml configuration is broken into named fragments stored in configs/serverless/:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJmk5neEYgM4UkP-oL-f1Q.png" /></figure><p>The same pattern applies to package.json via fragments in configs/nodejs/.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*RSwDxBxTKZ111M-I" /></figure><p>The key driver of the generative process is each service’s service.metadata.yml. This file declares which config fragments the service needs, its port assignments for local development, and other service-specific values that the shared task definitions read at runtime. The final serverless.yml for any given service is a combination of the base serverless.yml template, the service&#39;s own serverless.service.yml additions, and the variables and fragment selections declared below. (All of this is consumed by <a href="https://taskfile.dev/">Taskfile</a> tasks, covered more below)</p><pre>service:<br>  name: imp-app1<br>  fullName: data-import-app1<br>  importType: app1<br>  importTypeDashed: app1<br>  taskConfigs:<br>    # @see tasks/Taskfile.dev.yml<br>    dev:<br>      COMMON_SERVERLESS_YML_COMPONENTS: &quot;common consumer consumer.nodejs dynamodb&quot;<br>      COMMON_PACKAGE_JSON_COMPONENTS: &quot;common consumer dynamodb xml&quot;<br>      COMMON_IAM_POLICY_COMPONENTS: &quot;common&quot;<br>    # @see tasks/Taskfile.shell.yml<br>    shell:<br>      IMP_SHELL_EXPOSE_LOCAL_S3_PORT: 4602<br>      IMP_SHELL_EXPOSE_LOCAL_ELASTICMQ_PORT1: 9364<br>      IMP_SHELL_EXPOSE_LOCAL_ELASTICMQ_PORT2: 9365<br>      IMP_SHELL_EXPOSE_LOCAL_LAMBDA_PORT: 3033<br>      IMP_SHELL_EXPOSE_LOCAL_DYNAMODB_PORT: 8034</pre><p>Running task dev:gen-serverless-yml reads this file via <a href="https://github.com/mikefarah/yq">yq</a>, merges the declared fragments, and produces the final serverless.yml for that service. The monorepo-wide task services.gen-serverless-yml does the same across all services in a single pass. When a shared fragment changes; a new security policy, a lambda layer update, a serverless framework configuration change; you update one file and regenerate. Every service that includes that fragment is updated in one operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/0*AasPiHs4ShkoEmY2" /></figure><h3>Taskfile</h3><p>One thing that comes up in projects like is the use of a “Makefile”. makefiles can work, but they were designed for building software artifacts; their dependency model, syntax, and behavior under edge cases make them awkward to use as a general-purpose task runner. In practice makefiles used this way tend to become hard to follow and fragile; it feels like and overloading of the tools true purpose.</p><p><a href="https://taskfile.dev/">Taskfile</a> is an alternative that covers this use case well. Some reasons it worked well here:</p><ul><li>YAML syntax — more human consumable and consistent with the rest of the project’s config</li><li>first-class namespacing via includes; tasks are organized by concern without naming collisions</li><li>variables and dynamic values read from external files at runtime (via yq in this case)</li><li>clear dependency declaration between tasks</li><li>works identically on osx, windows and *nix without platform-specific workarounds</li><li>single binary with no runtime dependencies</li></ul><p>It is worth noting that the Taskfile tasks themselves are largely thin orchestration wrappers. The actual heavy lifting; config generation, serverless package management, AWS setup, local dev initialization; is done by custom scripts in the top-level scripts/ directory. Taskfile provides the consistent interface and handles argument passing, environment variable wiring, and task dependencies; the scripts do the work. This kept the scripts independently usable and testable outside of Taskfile if needed.</p><p>The Taskfile tree is organized in layers. Shared task definitions live in tasks/ and are symlinked into every service directory. Each service&#39;s Taskfile.yml is itself a symlink to tasks/Taskfile.service.yml, meaning every service gets an identical set of tasks out of the box. Tasks are namespaced by concern:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ElG6A8jmXhzHo_iLlFlJEw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*I0sDkvAKin2yiR0s.jpg" /></figure><p>From within any service directory:</p><pre>task dev:gen-serverless-yml            # regenerate serverless.yml from fragments<br>task dev:gen-package-json              # regenerate package.json from fragments<br>task shell:imp-shell.start-local-dev   # start local dev environment in container<br>task shell:imp-shell.deploy.dev        # package and deploy to dev via container<br>task infra:aws-setup.{env}             # provision aws infrastructure for this service<br>task infra:ci-setup.{env}              # provision ci jobs for this service<br>... and a few more</pre><p>From the repository root, cross-service operations run across all services at once:</p><pre>task services.gen-serverless-yml       # regenerate serverless.yml for every service<br>task services.regen.all                # all generation tasks across all services<br>task services.aws-setup.{env}          # run aws-setup for all services against {env}<br>task services.ci-deploy.{env}          # trigger CI deploy for all services to {env}<br>... and a few more</pre><p>Variables flow into tasks from service.metadata.yml at runtime via <a href="https://github.com/mikefarah/yq">yq</a>. There is no per-service variable substitution step; the metadata file is the authoritative source and every task reads from it directly. This is what makes a single shared task definition work correctly across services with different configurations, port assignments, and dependency sets.</p><p>Individual services can define additional tasks in a service.taskfile.yml that is not symlinked or shared, invoked via the svc: prefix. For example a consumer that requires a running OpenSearch instance manages that lifecycle entirely in its own file:</p><pre>tasks:<br>  opensearch.docker.start:<br>    desc: Starts opensearch locally, required prior to starting this service<br>    cmds:<br>      - docker compose -f util/opensearch/opensearch.docker-compose.yml up<br>  test.trigger.via.orchestrator:<br>    desc: Copies a test file to trigger processing locally via the orchestrator<br>    cmds:<br>      - task: :test.trigger.via.orchestrator<br>        vars:<br>          DOCS_SOURCE_SAMPLES_TEST_FILENAME: &quot;mysvc-small.csv&quot;</pre><p>Shared behavior lives in the tasks/ tree and propagates everywhere via symlinks; service-specific behavior stays local to that service.</p><h3>The imp-shell docker image</h3><p>One of the more practical pieces of the project was the development of the imp-shell docker image and framework. Serverlesss projects of this complexity has significant dependencies: a specific version of Nodejs, Python, the Serverless Framework, the AWS CLI, Java for DynamoDB Local, yq, jq, Taskfile, and more. Getting these aligned across developer machines (+cicd) and across time is a chronic source of &quot;works on my machine&quot; failures and onboarding friction.</p><p>imp-shell was an idea I had that ended up being the answer to the latter issues: a Docker image that packages all required tooling at pinned versions, used identically for local development and CI/CD. It is not a sidecar or helper container; it is the execution environment for everything: local service startup, packaging, deploying, and infrastructure provisioning. The image is built on Amazon Linux; the same OS family used by AWS lambda&#39;s managed runtime environment; maximizing parity between the local shell and what lambda actually runs in production.</p><h3>Shell build specs</h3><p>Rather than a single Dockerfile with a hardwired dependencies/configs, imp-shell uses a build spec model. Each spec is a small .env file in docker/ that declares the exact versions of all variable components. The filename itself encodes the spec identity and drives the generated image tag:</p><pre>docker/<br>├── imp-shell.amazonlinuxXYZ.dockerfile<br>├── docker-entrypoint.sh<br>├── build-image.sh<br>├── build-all-images.sh<br>├── ci-build.pipeline.xml<br>├── Taskfile.yml<br>│<br>├── imp-shell.amazonlinux.node-18.20.3_py-3.11_sls-3.40.0.env   ← current<br>├── imp-shell.amazonlinux.node-18.20.3_py-3.11_sls-3.39.0.env<br>├── imp-shell.amazonlinux.node-18.19.1_py-3.11_sls-3.38.0.env<br>├── imp-shell.amazonlinux.node-18.19.1_py-3.11_sls-2.72.3.env<br>├── imp-shell.amazonlinux.node-16.20.1_py-3.11_sls-2.54.0.env<br>└── ...</pre><p>Each .env file declares the build variables consumed by build-image.sh:</p><pre>NODE_VERSION=18.x.x<br>PYTHON_VERSION=3.xx<br>SERVERLESS_VERSION=3.x.x<br>TASKFILE_VERSION=3.x.x<br>NVM_VERSION=0.xx.x<br>YQ_VERSION=4.x.x<br>BUILD_PLATFORMS=linux/arm64,linux/amd64</pre><p>All versions are pinned. There are no floating latest references inside the image; what was built is what runs. All images are built for both linux/amd64 and linux/arm64 via docker buildx, so the same image tag works natively on Intel Macs, Apple Silicon Macs, and Linux CI agents without emulation or separate tags.</p><p>imp-shell has its own dedicated CI pipeline separate from the service pipelines. When triggered it loops over every .env spec file and builds and pushes the corresponding image to ECR (docker registry). Each build produces two tags per spec: a stable -latest tag that services reference in their service.metadata.yml, and an immutable versioned tag with git metadata for audit and rollback.</p><p>Old specs are preserved in docker/ indefinitely. Services migrate to new image versions by updating IMP_SHELL_IMAGE_NAME_TAG in their service.metadata.yml and testing locally. There is no forced services wide cutover; a service on SLS 3.x (serverless) can stay on that image while another migrates to SLS 4.x.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*3VN1h7Ashp5yoEjv" /></figure><h3>How the shell is used</h3><p>Every dev:*, infra:*, and cicd:* task has a shell:* counterpart that runs it inside the container. The shell tasks read port assignments and the target image tag from service.metadata.yml and construct the appropriate docker run invocation automatically.</p><p><strong>Interactive shell</strong>: drop into a bash prompt inside the container with the service directory mounted:</p><pre>cd services/data-import-mysvc<br>AWS_PROFILE=sls-data-import-mysvc-agent-dev task shell:imp-shell</pre><p>On startup the container entrypoint prints a banner showing exactly what you are running against; the mounted host path, the base image, and the pinned tool versions. The shell prompt is also customized so there is never any confusion about the fact that commands are executing inside the container:</p><pre>########################################################<br>imp-shell: data-import-mysvc<br>     REMEMBER! your commands are running in a container<br>     but the files you see in /opt/imp-mysvc<br>     are mounted to your local machine at:<br>     /Users/you/data-import/services/data-import-mysvc<br>     base image: public.ecr.aws/amazonlinux/amazonlinux:2023...<br>     nodejs: 18.x.x<br>     python: 3.xx<br>     serverless: 3.x.x<br>     taskfile: 3.x.x<br>########################################################<br>imp-shell@imp-mysvc&gt;</pre><p><strong>Local dev</strong>: start serverless offline inside the container with all AWS service emulators:</p><pre>task shell:imp-shell.start-local-dev</pre><p>Lambda handlers become callable HTTP endpoints on the mapped port, SQS queues are emulated via elasticmq, s3 via a local stub, and dynamo via dynamodb local (and more). All ports are forwarded to the host machine. Each service declares its port assignments in service.metadata.yml so multiple services can run simultaneously without conflict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6-t08Di2k6bGmzcS" /></figure><p><strong>Ad-hoc commands</strong>: run any single command inside the container and exit:</p><pre>task shell:imp-shell-cmd -- npm install<br>task shell:imp-shell-cmd -- serverless info --stage dev<br>task shell:imp-shell-cmd -- aws s3 ls s3://some-bucket</pre><p><strong>Package and deploy</strong>:</p><pre>task shell:imp-shell.package.dev<br>task shell:imp-shell.deploy-s3-package.dev<br>task shell:imp-shell.deploy.dev        # both in sequence</pre><p>Packaging and deployment happen inside the container. The artifact produced locally is identical to what CI produces because they are running the same image executing the same tasks. The built artifact is uploaded to the service’s S3 admin bucket and deployed from there; not from the local filesystem. On subsequent deployments the original artifact from s3 is sourced again. It is immutable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*-aRauSfiTEp6gtT1" /></figure><h3>CI/CD shell usage</h3><p>The most important property of imp-shell is not any individual capability; it is the guarantee of consistency between local and CI. The CI service pipeline does not install any tools directly. It authenticates to the docker registry, pulls the image, and runs the same task cicd:* commands a developer would run locally:</p><pre>CI pulls imp-shell image from registry<br>→ task cicd:package.[env]<br>→ upload artifact to S3<br>→ task cicd:deploy-s3-package.[env]</pre><p>If task shell:imp-shell.start-local-dev works locally, CI will work. If CI breaks, the developer can reproduce the failure exactly by running the same task locally against the same image. This eliminates the entire category of &quot;it built fine locally but failed in CI/CD.&quot;</p><h3>Serverless Framework upgrade</h3><p>With the monorepo in place and a generated configuration system across all services, upgrading from Serverless Framework 2.x to 3.x was significantly more manageable than it would have been across disparate repositories. The shared fragments in configs/serverless/ were updated once and regenerated across all services in a single pass via task services.regen.all. Breaking changes in the framework upgrade were addressed in the shared templates rather than service by service. A new imp-shell spec file was added for the new framework version and services migrated to it incrementally.</p><h3>Init a new service</h3><p>The one other item that needed to be addressed once the fundamentals were in place was a simpler way to stub out a new consumer service. For this the monorepo provides a scaffolding init task at the root:</p><pre>task services.init.new</pre><p>This generates the full service directory structure from templates, creates all required symlinks, and produces a service.metadata.yml to fill in. A working reference implementation serves as a concrete example. The combination of the scaffold command, the symlink-based shared structure, and the fragment-based config system means a new consumer service goes from zero to a fully configured, deployable serverless project with CI/CD support in a fraction of the time it took before.</p><h3>Well…</h3><p>None of the individual pieces here are particularly unique; monorepos, containerized build environments, and task runners are all well established patterns. What made this project worthwhile was getting them working together consistently for a set of services that had grown organically without tight management behind them. The result is not perfect and there is still plenty of room to mature the CI setup, improve test coverage, and further tighten the operational model. But the baseline is now in a significantly better place than it was.</p><p>Configuration drift is harder to accumulate because shared config is generated from one source rather than maintained per service. On boarding a new developer no longer requires a half-day of setup; the imp-shell container handles it. Cross cutting changes that previously meant touching N repositories now mean touching one file and running one command. And critically, local development and CI now run the same spec, which removes a whole category of build issues.</p><p>For anyone dealing with a similar situation, a collection of related serverless services that grew without a unified operational story, the approach here is worth considering. It does require an upfront investment to get the monorepo structure, the container image, and the task definitions into shape. But that investment pays back quickly once the fleet starts growing or when you need to make changes that have to land consistently across all services at once.</p><p>I hope this overview might provide ideas for others, not sure I’d do it again honestly, it was a ton of work. Secondly I’m still not sold on “serverless” as an overall development platform (using the term generically here).</p><p><em>(I included a few additional supplemental diagrams below, covering some other CI tasks that were created for policy generation and CI configuration per service)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/0*ky2-icfVGsyO65ad" /><figcaption><em>generating IAM policies for a specific service (i.e. the identity config the actual lambda runtime services execute under)</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*A0TM9MOkd9gtwH4P" /><figcaption><em>Task that is provisioning a CI/CD job for a service</em></figcaption></figure><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2024/09/20/wrangling-serverless-development-devops/"><em>http://bitsofinfo.wordpress.com</em></a><em> on September 21, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5824a800d04f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/devops-dudes/wrangling-serverless-development-devops-5824a800d04f">Wrangling Serverless Development &amp; Devops</a> was originally published in <a href="https://medium.com/devops-dudes">DevOps Dudes</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Poor man’s distributed locking]]></title>
            <link>https://levelup.gitconnected.com/poor-mans-distributed-locking-bd99a2dde36d?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd99a2dde36d</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Thu, 30 May 2024 17:51:21 GMT</pubDate>
            <atom:updated>2026-04-10T02:18:46.897Z</atom:updated>
            <content:encoded><![CDATA[<p>One of my prior posts on the async processing service mentioned a distributed locking mechanism. This post touches on that implementation. The short version: the client had no existing distributed locking infrastructure; no Hazelcast, no Redis, no Zookeeper, nothing. What they did have was Postgres, which was already in use. That turned out to be enough.</p><h3>The problem</h3><p>With a background processing service running queue population sweeps on a schedule, and potentially multiple threads or service instances doing work concurrently, there were two categories of race conditions to guard against. The first was queue level: two threads calling populateWorkQueue() simultaneously and enqueuing the same items twice. The second was item level: two threads picking up the same work item and processing it concurrently. Neither outcome was acceptable; duplicate processing in particular could cause real problems depending on what the downstream document manager did with repeated submissions.</p><p>The obvious solutions, Hazelcast distributed maps, Redis SETNX, or purpose-built lock services, were all off the table given the client&#39;s infrastructure. The less obvious solution was sitting right there in Postgres: advisory locks.</p><h3>Postgres advisory locks</h3><p>Postgres advisory locks are a feature most people never touch. Unlike row locks or table locks which are tied to transactions and specific data, advisory locks are application level locks identified by a numeric key that you manage entirely yourself. Postgres provides two variants: blocking ( pg_advisory_lock) which waits until the lock is available, and non-blocking ( pg_try_advisory_lock) which returns immediately with a boolean indicating whether the lock was granted.</p><p>The numeric key is the interesting part. Postgres advisory locks take a bigint identifier, so any string lock name can be mapped to one via a hash. The implementation does exactly that:</p><pre>private static long toPgLockId(String lockId) {<br>    return lockId.hashCode() &amp; Long.MAX_VALUE;<br>}</pre><p>This gives a stable numeric identifier for any named lock, which is all Postgres needs.</p><h3>The LockMgr interface</h3><p>The locking abstraction was kept deliberately simple. As mentioned in the previous post, it implements the standard java.util.concurrent.Lock contract so callers do not need to know anything about the underlying mechanism:</p><pre>public interface LockMgr {<br>    public Lock getLock(String lockId, String context) throws LockUnavailableException;<br>}</pre><p>A lock has two things: an identifier (the resource being locked) and a context. The context serves a dual purpose; it describes the scope of the locking operation (i.e. what logical operation or component is acquiring the lock) as well as identifying who is asking for it, both of which are captured in log output for diagnostics and tracing. getLock() returns a standard Lock that the caller can use with the familiar lock() / unlock() pattern. If the lock cannot be obtained at all, LockUnavailableException is thrown.</p><h3>PgAdvisoryLock</h3><p>PgAdvisoryLock is the Lock implementation. It wraps the two Postgres advisory lock SQL calls behind the standard Lock interface methods:</p><pre>private final String NON_BLOCKING_LOCK_SQL = &quot;SELECT pg_try_advisory_lock(?)&quot;;<br>private static final String BLOCKING_LOCK_SQL = &quot;SELECT pg_advisory_lock(?)&quot;;</pre><p>lock() uses the blocking variant; it will wait until the lock is available. tryLock() uses the non-blocking variant; it returns immediately with true or false. This maps cleanly onto the java.util.concurrent.Lock contract without any custom logic needed in the callers.</p><p>The unlock() implementation issues pg_advisory_unlock() against the same connection the lock was acquired on, then defers to PostgresAdvisoryLockMgr for connection cleanup:</p><pre>public void unlock() {<br>    try {<br>        if (!lockGranted) { return; }<br>        try (val statement = connection.prepareStatement(&quot;SELECT pg_advisory_unlock(?)&quot;)) {<br>            statement.setLong(1, lockId);<br>            try (val result = statement.executeQuery()) {<br>                if (result.next()) {<br>                    unlockSuccess = result.getBoolean(1);<br>                }<br>            }<br>        }<br>    } catch (Throwable e) {<br>        throw new LockUnavailableException(&quot;unlock() FAILED &quot; + this + &quot; error:&quot; + e.getMessage(), e);<br>    } finally {<br>        this.lockMgr.handleUnlock(this);<br>    }<br>}</pre><h3>PostgresAdvisoryLockMgr</h3><p>PostgresAdvisoryLockMgr is the LockMgr implementation and handles the connection lifecycle. It maintains two thread-local maps: one tracking the JDBC connection in use for the current thread, and one tracking the stack of locks held by the current thread.</p><p>The stack matters because Postgres advisory locks are re-entrant at the connection level; the same connection can acquire the same lock multiple times and must release it the same number of times. Tracking locks in a stack per thread ensures that the JDBC connection is held open for as long as any lock on that thread is still active, and closed cleanly when the last lock is released:</p><pre>protected void handleUnlock(PgAdvisoryLock lock) {<br>    Long threadId = Thread.currentThread().threadId();<br>    Stack locks = thread2locks.get(threadId);<br>    locks.pop();<br>    if (locks.isEmpty()) {<br>        PgLockConnection pgConn = thread2connection.get(threadId);<br>        if (pgConn != null) { pgConn.close(); }<br>        this.thread2locks.remove(threadId);<br>        this.thread2connection.remove(threadId);<br>    }<br>}</pre><h3>Why a separate datasource</h3><p>This was the most non-obvious part of the implementation. The first instinct was to just use the existing Quarkus datasource, but that ran into a fundamental problem quickly that led to hard to diagnose issues during development: Quarkus’s CDI container manages the lifecycle of datasource connections in conjunction with the active transaction context. Advisory locks acquired on a CDI-managed connection are tied to that transaction; when the transaction commits or rolls back, the connection is returned to the pool and the advisory locks are released by Postgres automatically, regardless of whether pg_advisory_unlock was ever called. That completely undermines the purpose of the lock.</p><p>The solution was a dedicated, separately configured AgroalDataSource created outside of CDI&#39;s awareness and used exclusively for advisory locks:</p><pre>public static AgroalDataSource create(String url, String username, String password) throws Exception {<br>    AgroalDataSourceConfigurationSupplier configuration = new AgroalDataSourceConfigurationSupplier()<br>            .connectionPoolConfiguration( cp -&gt; cp<br>                    .minSize( 5 )<br>                    .maxSize( 20 )<br>                    .acquisitionTimeout( ofSeconds( 5 ) )<br>                    .connectionFactoryConfiguration( cf -&gt; cf<br>                            .jdbcUrl( url )<br>                            .autoCommit( false )<br>                            .jdbcTransactionIsolation( SERIALIZABLE )<br>                            .principal( new NamePrincipal( username ) )<br>                            .credential( new SimplePassword( password ) )<br>                    )<br>            );<br>    return AgroalDataSource.from(configuration);<br>}</pre><p>Connections from this pool are not managed by Quarkus’s CDI transaction machinery, so advisory locks held on them live and die by the explicit pg_advisory_lock and pg_advisory_unlock calls only.</p><h3>Usage</h3><p>From the caller’s perspective the API is the same as any java.util.concurrent.Lock. A lock is requested by id and context, acquired, work is done, and the lock is released in the finally block. A LockUnavailableException signals that the lock could not be obtained at all and the caller can decide how to handle that; in the context of the processing service this typically means re-queuing the work item for a later attempt:</p><pre>Lock lock = null;<br>try {<br>    lock = lockMgr.getLock(&quot;uniqueId&quot;, getClass() + &quot;.doSomething()&quot;);<br>    lock.lock();<br>    // do work<br>} catch (LockUnavailableException e) {<br>    throw new TryProcessingWorkItemLaterException(&quot;Could not acquire lock, try again later: &quot; + e.getMessage());<br>} catch (Exception e) {<br>    // handle<br>} finally {<br>    if (lock != null) { lock.unlock(); }<br>}</pre><p>tryLock() is used in cases where skipping is preferable to waiting; for example the scheduled queue population sweep uses tryLock() so that if another instance already holds the queue lock, the sweep just exits cleanly rather than blocking. lock() is used where waiting is the right behavior; for example acquiring an item level lock before processing a specific work item.</p><h3>Summary</h3><p>Postgres advisory locks are a simple, effective distributed locking primitive that is easy to overlook. If you already have Postgres in your stack and do not need the overhead of a dedicated locking service, they are worth considering. The main things to get right are keeping the lock connections off the CDI-managed datasource and managing the connection lifecycle carefully so locks are not held longer than intended. Wrapping the whole thing behind java.util.concurrent.Lock keeps the callers clean and leaves room to swap in a different backend; Hazelcast, Redis, or anything else; later without touching the code that uses the locks.</p><p>The implementation worked well enough in production that it was extracted into a standalone library for reuse across other projects.</p><pre>import lombok.Getter;<br>import lombok.val;<br>import org.jboss.logging.Logger;<br>import java.sql.Connection;<br>import java.sql.SQLException;<br>import java.util.concurrent.TimeUnit;<br>import java.util.concurrent.locks.Condition;<br>import java.util.concurrent.locks.Lock;<br>public class PgAdvisoryLock implements Lock {<br>    private static final Logger logger = Logger.getLogger(PgAdvisoryLock.class);<br>    private static final String NON_BLOCKING_LOCK_SQL = &quot;SELECT pg_try_advisory_lock(?)&quot;;<br>    private static final String BLOCKING_LOCK_SQL = &quot;SELECT pg_advisory_lock(?)&quot;;<br>    private final PostgresAdvisoryLockMgr.PgLockConnection pgConnection;<br>    @Getter<br>    private final String lockName;<br>    @Getter<br>    private long lockId;<br>    @Getter<br>    private boolean lockGranted;<br>    @Getter<br>    private final String lessee;<br>    private final PostgresAdvisoryLockMgr lockMgr;<br>    public PgAdvisoryLock(PostgresAdvisoryLockMgr lockMgr, String lockName, String lessee) throws SQLException {<br>        this.lockMgr = lockMgr;<br>        this.lockName = lockName;<br>        this.lockId = toPgLockId(lockName);<br>        this.lessee = lessee;<br>        this.lockGranted = false;<br>        this.pgConnection = lockMgr.getConnection(this);<br>    }<br>    @Override<br>    public void lock() {<br>        try {<br>            boolean result = this.blockingLock();<br>            if (!result) {<br>                throw new LockUnavailableException(&quot;lock() blockingLock() returned false: &quot; + this, null);<br>            }<br>        } catch(InterruptedException e) {<br>            if (Thread.interrupted()) {<br>                Thread.currentThread().interrupt();<br>            }<br>            throw new LockUnavailableException(&quot;lock() blockingLock() threw on: &quot; + e.getClass().getSimpleName() + this + &quot; &quot; + e.getMessage(), e);<br>        }<br>    }<br>    @Override<br>    public void lockInterruptibly() throws InterruptedException {<br>        this.blockingLock();<br>    }<br>    @Override<br>    public boolean tryLock() {<br>        try {<br>            return nonBlockingLock();<br>        } catch(Exception e) {<br>            if (Thread.interrupted()) {<br>                Thread.currentThread().interrupt();<br>            }<br>            throw new LockUnavailableException(&quot;lock() nonBlockingLock() threw on: &quot; + e.getClass().getSimpleName() + this + &quot; &quot; + e.getMessage(), e);<br>        }<br>    }<br>    @Override<br>    public boolean tryLock(long time, TimeUnit unit) throws InterruptedException {<br>        // need to impl this but w/ a custom connection timeout?<br>        throw new UnsupportedOperationException(&quot;tryLock(long time, TimeUnit unit) is not supported&quot;);<br>    }<br>    private boolean doPgLockLogic(String lockSql) throws InterruptedException {<br>        boolean blocking = lockSql.equals(BLOCKING_LOCK_SQL);<br>        Connection connection = pgConnection.connection;<br>        try (val statement = connection.prepareStatement(lockSql)) {<br>            statement.setLong(1, lockId);<br>            try (val result = statement.executeQuery()) {<br>                if (result.next()) {<br>                    if (blocking) {<br>                        lockGranted = true;<br>                    } else {<br>                        lockGranted = result.getBoolean(1);<br>                    }<br>                } else {<br>                    lockGranted = false;<br>                }<br>            }<br>            if (lockGranted) {<br>                logger.debug(&quot;doPgLockLogic(blocking:&quot;+blocking+&quot;) GRANTED &quot; + this);<br>            } else {<br>                logger.debug(&quot;doPgLockLogic(blocking:&quot;+blocking+&quot;) NOT GRANTED &quot; + this);<br>            }<br>        } catch(Throwable e) {<br>            String msg = &quot;doPgLockLogic(blocking:&quot;+blocking+&quot;) ERROR &quot; + this +<br>                    &quot; error: &quot; + e.getMessage();<br>            logger.error(msg,e);<br>            if (Thread.interrupted()) {<br>                throw new InterruptedException();<br>            }<br>        }<br>        return lockGranted;<br>    }<br>    private boolean nonBlockingLock() throws InterruptedException {<br>        return doPgLockLogic(NON_BLOCKING_LOCK_SQL);<br>    }<br>    private boolean blockingLock() throws InterruptedException  {<br>        return doPgLockLogic(BLOCKING_LOCK_SQL);<br>    }<br>    public void unlock() {<br>        try {<br>            if (!lockGranted) {<br>                return;<br>            }<br>            var unlockSuccess = false;<br>            Connection connection = pgConnection.connection;<br>            try (val statement = connection.prepareStatement(&quot;SELECT pg_advisory_unlock(?)&quot;)) {<br>                statement.setLong(1, lockId);<br>                try (val result = statement.executeQuery()) {<br>                    if (result.next()) {<br>                        unlockSuccess = result.getBoolean(1);<br>                    }<br>                }<br>            }<br>            if (unlockSuccess) {<br>                logger.debug(&quot;unlock() RELEASED &quot; + this);<br>            } else {<br>                logger.debug(&quot;unlock() NOT RELEASED &quot; + this);<br>            }<br>        } catch (Throwable e) {<br>            String msg = &quot;unlock() FAILED &quot; + this + &quot; error:&quot; +e.getMessage();<br>            logger.error(msg,e);<br>            throw new LockUnavailableException(msg, e);<br>        } finally {<br>            // defer to mgr for unlock connection handling<br>            this.lockMgr.handleUnlock(this);<br>        }<br>    }<br>    @Override<br>    public Condition newCondition() {<br>        throw new UnsupportedOperationException(&quot;newCondition() is not supported&quot;);<br>    }<br>    private static long toPgLockId(String lockId) {<br>        return lockId.hashCode() &amp; Long.MAX_VALUE;<br>    }<br>    @Override<br>    public String toString() {<br>        return &quot;[&quot;+getClass().getSimpleName()+&quot;]{ lessee: &quot; + lessee +<br>        &quot;, lockName: &quot; + lockName + &quot;, lockId :&quot; + lockId + &quot; &quot; + this.pgConnection + &quot; }&quot;;<br>    }<br>}</pre><pre>import io.agroal.api.AgroalDataSource;<br>import jakarta.inject.Singleton;<br>import org.eclipse.microprofile.config.inject.ConfigProperty;<br>import org.jboss.logging.Logger;<br>import java.sql.Connection;<br>import java.sql.PreparedStatement;<br>import java.sql.ResultSet;<br>import java.sql.SQLException;<br>import java.util.Map;<br>import java.util.Stack;<br>import java.util.concurrent.ConcurrentHashMap;<br>import java.util.concurrent.locks.Lock;<br><br>@Singleton<br>public class PostgresAdvisoryLockMgr implements LockMgr {<br>    private static final Logger logger = Logger.getLogger(PostgresAdvisoryLockMgr.class);<br>    private final AgroalDataSource lockDatasource;<br>    private Map&lt;Long, PgLockConnection&gt; thread2connection = new ConcurrentHashMap&lt;&gt;();<br>    private Map&lt;Long, Stack&lt;PgAdvisoryLock&gt;&gt; thread2locks = new ConcurrentHashMap&lt;&gt;();<br>    public PostgresAdvisoryLockMgr(@ConfigProperty(name=&quot;quarkus.datasource.jdbc.url&quot;) String jdbcUrl,<br>                                   @ConfigProperty(name=&quot;quarkus.datasource.username&quot;) String username,<br>                                   @ConfigProperty(name=&quot;quarkus.datasource.password&quot;) String password) {<br>        /**<br>         * WE have to do this, (totally separate DS) otherwise all the hibernate<br>         * TX joining magic with all the AOP screws everything up.<br>         */<br>        try {<br>            this.lockDatasource = PgLockDatasource.create(jdbcUrl,username,password);<br>            logger.info(&quot;PostgresAdvisoryLockMgr() dedicated datasource &quot; +<br>                    &quot;created OK: jdbcUrl:&quot; + jdbcUrl + &quot; username:&quot;+username);<br>        } catch(Exception e) {<br>            String msg = &quot;PostgresAdvisoryLockMgr() unexpected error creating dedicated AgroalDataSource: &quot; + e.getMessage();<br>            logger.error(msg,e);<br>            throw new RuntimeException(msg,e);<br>        }<br>    }<br>    protected class PgLockConnection {<br>        protected Connection connection;<br>        protected Integer pgPid;<br>        protected PgLockConnection(Connection conn, Integer pgPid) {<br>            this.connection = conn;<br>            this.pgPid = pgPid;<br>        }<br>        public void close() throws SQLException {<br>            if (this.connection != null){<br>                this.connection.close();<br>            }<br>        }<br>        @Override<br>        public String toString() {<br>            return &quot;[&quot;+getClass().getSimpleName()+&quot;].pg_pid:&quot;+this.pgPid;<br>        }<br>    }<br>    private void registerLock(PgAdvisoryLock lock) {<br>        Long threadId = Thread.currentThread().threadId();<br>        Stack&lt;PgAdvisoryLock&gt; locks = thread2locks.get(threadId);<br>        if (locks == null){<br>            locks = new Stack&lt;&gt;();<br>            thread2locks.put(threadId,locks);<br>        }<br>        locks.push(lock);<br>        logger.debug(&quot;registerLock() registered lock: &quot; + lock);<br>    }<br>    protected void handleUnlock(PgAdvisoryLock lock) {<br>        Long threadId = Thread.currentThread().threadId();<br>        Stack&lt;PgAdvisoryLock&gt; locks = thread2locks.get(threadId);<br>        if (locks == null){<br>            throw new NoPgAdvisoryLocksForThreadException(&quot;handleUnlock() unexpected error: current thread: &quot; + threadId +<br>                    &quot;has no Stack&lt;Locks&gt;!!!&quot;);<br>        }<br>        logger.debug(&quot;handleUnlock() popping lock off stack: &quot; + lock);<br>        locks.pop();<br>        if (locks.isEmpty()) {<br>            PgLockConnection pgConn = thread2connection.get(threadId);<br>            if (pgConn != null){<br>                try {<br>                    logger.debug(&quot;handleUnlock() final lock, closing JDBC conn: &quot; + lock);<br>                    pgConn.close();<br>                } catch(Throwable e) {<br>                    logger.error(&quot;handleUnlock() lock: lessee: &quot; + lock.getLessee() +<br>                            &quot; lockName:&quot;+lock.getLockName() + &quot; lockId:&quot; + lock.getLockId() + &quot;&quot; +<br>                            &quot;unexpected error attempting to close() connection: &quot;+ e.getMessage(),e);<br>                }<br>            }<br>            // smoke the thread tracked<br>            this.thread2locks.remove(threadId);<br>            this.thread2connection.remove(threadId);<br>        }<br>    }<br>    private static int getConnectionId(Connection connection) throws SQLException {<br>        try (PreparedStatement statement = connection.prepareStatement(&quot;SELECT pg_backend_pid()&quot;)) {<br>            try (ResultSet resultSet = statement.executeQuery()) {<br>                if (resultSet.next()) {<br>                    return resultSet.getInt(1);<br>                } else {<br>                    throw new SQLException(&quot;Failed to retrieve connection ID.&quot;);<br>                }<br>            }<br>        } catch(Throwable e) {<br>            logger.error(&quot;getConnectionId() unexpected error: &quot; + e.getMessage(),e);<br>            return -999;<br>        }<br>    }<br>    protected PgLockConnection getConnection(PgAdvisoryLock lock) throws SQLException  {<br>        Long threadId = Thread.currentThread().threadId();<br>        PgLockConnection pgConn = thread2connection.get(threadId);<br>        if (pgConn == null){<br>            logger.debug(&quot;getConnection() initializing JDBC conn for lock: &quot; + lock);<br>            Connection conn = this.lockDatasource.getConnection();<br>            Integer pgPid = getConnectionId(conn);<br>            pgConn = new PgLockConnection(conn,pgPid);<br>            thread2connection.put(Thread.currentThread().threadId(),pgConn);<br>        }<br>        return pgConn;<br>    }<br>    public Lock getLock(String lockName, String lessee) throws LockUnavailableException {<br>        lessee = String.format(&quot;%s:%s&quot;, lessee, Thread.currentThread().getName());<br>        PgAdvisoryLock lock = null;<br>        try {<br>            lock = new PgAdvisoryLock(this, lockName, lessee);<br>            registerLock(lock);<br>            return lock;<br>        } catch (Throwable e) {<br>            String msg = &quot;requestLock() Failed to establish lock lessee: &quot;+lessee+&quot; lockName: &quot; +<br>                    lockName + &quot; -&gt; lockId :&quot;+ (lock != null  ?lock.getLockId():null) + &quot; msg:&quot;+e.getMessage();<br>            logger.error(msg);<br>            throw new LockUnavailableException(msg,e);<br>        }<br>    }<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XBqPMPrVoKrKlZvI" /></figure><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2024/05/30/poor-mans-distributed-locking/"><em>http://bitsofinfo.wordpress.com</em></a><em> on May 30, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd99a2dde36d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/poor-mans-distributed-locking-bd99a2dde36d">Poor man’s distributed locking</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple pattern for an asynchronous processing service]]></title>
            <link>https://levelup.gitconnected.com/simple-pattern-for-an-asynchronous-processing-service-18554cd5eee8?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/18554cd5eee8</guid>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Sun, 03 Mar 2024 06:51:20 GMT</pubDate>
            <atom:updated>2026-04-10T02:19:08.928Z</atom:updated>
            <content:encoded><![CDATA[<p>The use case was straightforward: a backend service needed to receive document submissions and forward them to one or more document manager endpoints. Document managers accepted documents, processed them, and stored them. Documents could have been submitted by other systems independently, so the service had to account for the possibility that any given document was already known to a document manager. Submissions had to happen in the background, transient failures had to be retried, and every document needed to ultimately land in a terminal state; either successfully managed or definitively failed.</p><h3>Status tracking via Postgres</h3><p>The foundation was a postgres table. Each document row carried a status field tracking where the document was in its lifecycle:</p><pre>// has received the document<br>RECEIVED,<br>// scheduled for processing in the future<br>SCHEDULED,<br>// queued, awaiting processing<br>QUEUED,<br>// actively being processed<br>PROCESSING,<br>// submitted to the document manager; manager acknowledged reception<br>SUBMITTED,<br>// document manager returned a rejection/error response<br>REJECTED,<br>// document manager accepted the document, or a lookup confirmed it is already managed<br>MANAGED,<br>// document manager returned a &quot;not managed&quot; response on status query<br>NOT_MANAGED,<br>// document manager is busy; retry later<br>TRY_AGAIN_LATER,<br>// an unanticipated error occurred; handling failed<br>FAILED,<br>// a previously pending document was cancelled<br>CANCELLED</pre><p>The statuses cover both sides: states reflecting what the service itself is doing ( RECEIVED, SCHEDULED, QUEUED, PROCESSING) and states reflecting what the document manager reported back ( SUBMITTED, REJECTED, MANAGED, NOT_MANAGED, TRY_AGAIN_LATER). Keeping these in one enum gives a clear audit trail of the full document lifecycle in a single column.</p><p>One thing worth noting: there is an argument to be made that this design co-mingles two different concerns in a single status field. States like RECEIVED, SCHEDULED, QUEUED, and PROCESSING are really queue and processing pipeline states, while SUBMITTED, REJECTED, MANAGED, NOT_MANAGED, and FAILED reflect the document lifecycle and outcomes from the document manager. Keeping them together was a pragmatic choice that kept the implementation simple for the use case at hand; a single column, a single query, straightforward state machine logic. In a future iteration these could reasonably be split into two separate status fields or tables, one tracking where the document is in the processing pipeline and one tracking the document manager outcome.</p><h3>Abstracting documents to WorkItems</h3><p>Once the pattern was working it became clear the underlying mechanics had nothing document specific about them. The queue, status transitions, retry behavior; all generic. So the document concept was abstracted up to a WorkItem with the document implementation sitting behind it, making it straightforward to externalize the whole pattern into a reusable library.</p><p>A WorkItem represents anything that can be queued and processed:</p><pre>public interface WorkItem {<br>    WorkItemType getWorkItemType();<br>    String getWorkItemId();<br>    String getWorkItemStatus();<br>    Boolean workItemIsQueuable();<br>    WorkItemRef toWorkItemRef();<br>}</pre><p>WorkItemRef is a lightweight reference passed around the queue; just enough to identify and route the item without dragging the full entity through every layer:</p><pre>public class WorkItemRef {<br>    private String id;<br>    private WorkItemType type;<br>    private String status;<br>}</pre><p>WorkItemProcessor handles the actual work. The notable thing is TryProcessingWorkItemLaterException; throwing this signals the item should be re-queued rather than failed, which is how transient errors and TRY_AGAIN_LATER responses get handled:</p><pre>public interface WorkItemProcessor {<br>    void processWorkItem(WorkItemRef workItemRef) throws TryProcessingWorkItemLaterException;<br>}</pre><p>Persistence sits behind WorkItemStore, keeping the queue machinery decoupled from any particular storage implementation:</p><pre>public interface WorkItemStore {<br>    List getQueueableWorkItems();<br>    void setWorkItemStatusQueuedAndSave(WorkItem workItem, String message) throws SaveWorkItemFailureException;<br>    WorkItem getWorkItemById(String workItemId);<br>}</pre><h3>What makes a WorkItem queueable</h3><p>The getQueueableWorkItems() contract is open ended; each implementation decides what &quot;queueable&quot; means for its entity. For the document implementation this was backed by a streamActionableRecords() query that covered several scenarios. The logic was not simply &quot;give me everything in status X&quot;; it also had to pick up items sitting in intermediate states too long, and items whose scheduled or retry time had arrived:</p><pre>public Stream streamActionableRecords() {<br>    /*<br>        get everything that is RECEIVED<br>            OR<br>        SCHEDULED and documentScheduledAt &lt; NOW()<br>            OR<br>        QUEUED and statusQueuedAt &lt; NOW-60 minutes<br>            OR<br>        PROCESSING and statusProcessingAt &lt; NOW-60 minutes<br>            OR<br>        TRY_AGAIN_LATER and documentTryAgainLaterAt &lt; NOW<br>    */<br>    Date now = new Date();<br>    return find(&quot;((status IN (?1)) OR &quot; +<br>                        &quot;(status = ?2 AND (documentScheduledAt IS NULL OR documentScheduledAt &lt; ?3)) OR &quot; +<br>                        &quot;(status = ?4 AND (statusQueuedAt IS NULL OR statusQueuedAt &lt; ?5)) OR &quot; +<br>                        &quot;(status = ?6 AND (statusProcessingAt IS NULL OR statusProcessingAt &lt; ?7)) OR &quot; +<br>                        &quot;(status = ?8 AND (documentTryAgainLaterAt IS NULL OR documentTryAgainLaterAt &lt; ?9)))&quot;,<br>                    actionableStatus, // 1<br>                    DocumentStatus.SCHEDULED, // 2<br>                    now, // 3<br>                    DocumentStatus.QUEUED, // 4<br>                    DateUtil.computeAdjustedDate(-60), // 5<br>                    DocumentStatus.PROCESSING, // 6<br>                    DateUtil.computeAdjustedDate(-60), // 7<br>                    DocumentStatus.TRY_AGAIN_LATER, // 8<br>                    now // 9<br>             ).stream();<br>}</pre><p>The QUEUED and PROCESSING thresholds act as a safety net. If an item was placed in either state but never advanced due to a crash or restart, the next scheduled sweep picks it back up rather than leaving it stuck. The TRY_AGAIN_LATER case lets the document manager throttle retries by setting a future timestamp; the item will not be picked up again until that time has passed.</p><h3>The queue</h3><p>The WorkQueue interface is intentionally thin:</p><pre>public interface WorkQueue {<br>    String getLockId();<br>    void enqueue(WorkItemRef workItemRef);<br>    WorkItemRef dequeue(long timeout, TimeUnit unit) throws NoSuchElementException, InterruptedException;<br>}</pre><p>The initial implementation was an in process LinkedBlockingDeque wrapped behind InProcessQueue. Keeping the queue behind an interface means swapping to SQS, Kafka, or anything else requires only a new implementation without touching publisher or consumer logic.</p><pre>public class InProcessQueue implements WorkQueue {<br>    private final BlockingQueue queue = new LinkedBlockingDeque&lt;&gt;(1000);<br>    ...<br>    @Override<br>    public void enqueue(WorkItemRef workItemRef) {<br>        if (workItemRef != null &amp;&amp; !this.queue.contains(workItemRef)) {<br>            this.queue.add(workItemRef);<br>        }<br>    }<br>    @Override<br>    public WorkItemRef dequeue(long timeout, TimeUnit unit) throws InterruptedException {<br>        return this.queue.poll(timeout, unit);<br>    }<br>}</pre><h3>Publisher and consumer</h3><p>WorkQueuePublisher has two entry points: populateWorkQueue(), called on a schedule to sweep for queueable items, and publish(WorkItemRef), which queues a specific item directly. Both acquire a distributed lock before doing anything; more on that below.</p><p>Before queuing an item, publish() re-fetches the WorkItem from the store and calls workItemIsQueuable(). This guards against race conditions where an item&#39;s status may have changed between the time it was identified as queueable and the time publish() actually runs. If no longer queueable it is silently skipped.</p><pre>public final void publish(WorkItemRef workItemRef) {<br>    ...<br>    workItem = this.workItemStore.getWorkItemById(workItemRef.getId());<br>    if (!workItem.workItemIsQueuable()) {<br>        // status changed since we last looked; skip it<br>        return;<br>    }<br>    workItemStore.setWorkItemStatusQueuedAndSave(workItem, ...);<br>    this.workQueue.enqueue(workItem.toWorkItemRef());<br>    ...<br>}</pre><p>WorkQueueConsumer runs as a Runnable on a ManagedExecutor (Quarkus&#39;s CDI aware managed thread pool). It loops continuously, blocking on dequeue() with a 30 second timeout. When an item comes off the queue it is handed off to a separate executor for processing, keeping the consumer loop free to pull the next item. If the processor throws TryProcessingWorkItemLaterException the item is re-published rather than discarded:</p><pre>public void consumeFromQueue() throws InterruptedException {<br>    WorkItemRef workItemRef = this.workQueue.dequeue(30, TimeUnit.SECONDS);<br>    if (workItemRef != null) {<br>        this.processorExecutor.execute(new Runnable() {<br>            @Override<br>            public void run() {<br>                try {<br>                    workItemProcessor.processWorkItem(workItemRef);<br>                } catch(TryProcessingWorkItemLaterException e) {<br>                    republish(workItemRef);<br>                }<br>            }<br>        });<br>    }<br>}</pre><h3>Distributed locking</h3><p>Both the publisher and consumer rely on a distributed locking mechanism to prevent duplicate processing and queue population races. The locking abstraction implements the standard java.util.concurrent.Lock contract so the underlying implementation can be swapped out; database backed locks, Hazelcast, or anything else; without touching the publisher or consumer code. This is covered in more detail in a follow-up post.</p><h3>The resulting structure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*v3WEzwD-B0HVKyw_" /></figure><p>A scheduler periodically calls populateWorkQueue() which sweeps the store for any items in a queueable state and publishes them. Separately, any new item received by the service can be published directly. The consumer loop pulls items off the queue and dispatches them to the processor. The processor calls the document manager, updates the status, and either completes the item or throws TryProcessingWorkItemLaterException to trigger a retry.</p><h3>Summary</h3><p>A postgres status column, a thin queue abstraction, a publisher/consumer pair with distributed locking, and a processor interface with a retry exception covers the majority of asynchronous background processing use cases without pulling in a full message broker. The higher level abstractions ( WorkItem, WorkItemStore, WorkQueue, WorkQueuePublisher, WorkQueueConsumer) were extracted into a shared library; concrete implementations of the queue and store live in the service itself. New projects pick up the library and provide their own implementations of the relevant interfaces as needed.</p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2024/03/02/simple-pattern-for-an-asynchronous-processing-service"><em>http://bitsofinfo.wordpress.com</em></a><em> on March 3, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18554cd5eee8" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/simple-pattern-for-an-asynchronous-processing-service-18554cd5eee8">Simple pattern for an asynchronous processing service</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecture for a data lake REST API using Delta Lake, Fugue & Spark]]></title>
            <link>https://levelup.gitconnected.com/architecture-for-a-data-lake-rest-api-using-delta-lake-fugue-spark-3f1fb9077c41?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f1fb9077c41</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Tue, 15 Aug 2023 18:01:28 GMT</pubDate>
            <atom:updated>2023-09-03T15:32:03.317Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>“Hey, we need some kind of a REST API over all our data lakes to let analysts and other integrations query records on demand . Can we please get this done?”</em></strong></p><p>That was the use-case laid out that needed a solution. If you’ve had any experience with data lakes you know that they can be very complicated, very large and utilize unique file formats. In any case, this project sounded interesting, so here we go.</p><h3>CURRENT STATE OF THE OCEAN</h3><p>We needed to do this for N data lakes, not just one. Thankfully the current architecture of the data lakes in question followed fairly consistent pattern.</p><p>Each lake followed the <a href="https://delta.io/">Delta Lake Architecture</a> following the patterns of bronze, silver and gold zones each one progressing from more primitive data (bronze) to more refined data (gold). For each lake the gold data held the final gold copy of the data; the official consumable data or curated data if you’d like to call it that. The “gold” location of each lake was generally the data we wanted to make consumable via a REST API, however at the end of the day, despite each silo of data holding different refinement levels of data, they all still followed the same pattern inside each silo: a top level root folder, followed by a set of sub-folders organizing the data in a categorization tree of directories, and finally the last folder being the delta table that represented some “entity”.</p><p>For example it would look something like this:</p><p>/[data_lake_name]/[project]/some/path/to/the/data/[delta-table-name]/</p><p>The above is pretty typical in lakes anyways and is not anything unique. The lakes themselves generally resided in <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction">Azure ADLS storage (Azure Data Lake Storage)</a> accounts. The end delta tables were also often cataloged in <a href="https://www.databricks.com/">Databricks</a> and utilized via various Spark jobs/notebooks etc. Some tables were small, others medium sized and others were pretty large. Individual records within each table generally were not that big and well within the margin of size reasonable to be returned over an API response.</p><h3>REQUIREMENTS</h3><p>The API requirements were pretty basic, as most the integration use-cases were simple. The primary use was being able to select any subset of data from a single entity (delta table), joining across tables was not needed (yet). Give than wanted to expose the following functionality via rest endpoints:</p><ul><li>Get a schema for any target entity in the lake</li><li>Retrieve data for any single entity (delta table) in the lake and return the results. Support the ability to filter the data and specify the specific columns to return. Sorting/grouping etc should all be supported and paging through results.</li><li>API clients need to be restricted to only parts of a given lake they have access to, however row level security was not necessary.</li></ul><p>Ok the <strong>R</strong> in C<strong>R</strong>UD; cool, pretty simple.</p><h3>DELTA-RS &amp; DUCKDB</h3><p>Given the variety of entity/table sizes residing across the lake it became apparent that we would need a variety of execution methods to access the data. For example when accessing smaller tables it would be ideal (for speed) to simply access the data locally or direct from the storage account, while for larger ones we might have to formulate queries that execute over Databricks/Spark.</p><p>Given The first goal was to start writing some Python code to see what libraries might work well to access the Delta Tables natively just using Python and not having to rely on something like Spark.</p><p>The first library looked at was <a href="https://github.com/delta-io/delta-rs">delta-rs</a> which provides low level access to Delta tables in Rust but with higher level bindings for Python. Using this you could get at the tables and load up data into pyarrow tables and go from there, but would not be an ideal layer to directly bind code that is mediating a REST API. With delta-rs you could also feed the pyarrow tables directly into <a href="https://duckdb.org/">DuckDB</a> which is in-process OLAP DBMS (database management system)… which would much better suited for this kind of use-case: load the target delta table with delta-rs, feed it into duckdb, then use plain sql to query it. Prototyping with this proceeded and it worked quite well. For smaller to medium sized tables this was pretty performant when the delta tables were accessible locally on the filesystem or via and Azure ABFSS URI and supporting driver library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*OngrCG28mdBDSB_q" /></figure><p>However, when using this method over large delta tables it was super slow but eventually worked… and would not be acceptable for real-time API requests. The size of the delta tables was simply too much for an in-process query engine to both load and process in a timely manner.</p><h3>HANDLING LARGE DELTA TABLES</h3><p>For the large tables the only practical solution was to field the queries to be executed over the wire to Spark. Given the data being targeted in the lakes was already in the “gold” zone, these tables were already cataloged and queryable via Spark sql, so it would be the natural choice. Some initial testing was done with this, it worked, but ultimately would force the access pattern of the API for these types of target tables to be asynchronous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*Em9rt3wgECOtGiun" /></figure><h3>MEH, NOT GREAT</h3><p>While both of these options could work, we really didn’t like having to work directly with two different underlying chains of libraries depending on what table in any particular data lake was being targeted. We’d much prefer to simply be able to create a query, send it off to a single library, give it some kind of “execution preference/hint” and then let that library take care of the rest. At the end of the day, we could write that abstraction layer ourselves however we’d like to avoid that. That is when we came across the <a href="https://github.com/fugue-project/fugue">Fugue Project</a>.</p><h3>FUGUE PROJECT</h3><p>As stated by the <a href="https://github.com/fugue-project/fugue">Fugue project</a> themselves:</p><p><em>“</em><strong><em>Fugue is a unified interface for distributed computing that lets users execute Python, Pandas, and SQL code on Spark, Dask, and Ray with minimal rewrites</em></strong><em>.”</em></p><p>What is omitted in that statement, is the capability of being executable in process via DuckDB. The two key abilities of Fugue are an API of functions that can take your function and execute over all these distributed backends as well as a SQL interface (supporting ANSI) with extensions to the do the same using a SQL “workflow” syntax; i.e. load dataframe from any INPUT, execute some SQL against dataframe, then do something with the OUTPUT.</p><p>So now we could build something more like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*cNegGxlqligXEw0Y" /></figure><p>We did a little bit of prototyping with this model and it worked pretty well. We could focus our efforts on producing valid Fugue SQL workflow statements along with some small functions to permit Fugue to get at the raw delta tables and let Fugue do the rest. Based on which data lake entity/table was being targeted, we could simply switch which Fugue engine type utilize (duckdb or spark) and Fugue would handle the rest. By doing this were then able to focus more on all the logic sitting above such as the API itself, security and creating the queries themselves.</p><h3>CHANGE OF PLANS</h3><p>First off we made a decision that all executed queries, regardless of local execution or remote via spark would be treated as asynchronous. The caller should make a request and get an immediate result containing a result_id that could be used to get the result at a later time when the query was finished (or check the status). We had to do this given any query could be executed over Spark OR local IO and it was good design to hide that from the callers and keep the API contract consistent whether the target was a small or large table. Secondly, we realized that now we had a bit more power available to us and we wanted to expose that query capability all the way to the callers. We did not want to re-invent the wheel of creating yet another abstraction in an API to allow users to express things like what to get, what columns to return, how to sort it, paging etc. Instead we would let them pass raw ANSI SQL queries that targeted only the table/entity they were requesting. (yes I can already hear readers screaming <em>“sql injection! OWASP! sql injection!”</em>…. and yes we addressed that, read on on).</p><p>Hence the requirements changed a bit for. the API:</p><ul><li>Get a schema for any target entity in the data lake</li><li>Query data for any single entity (delta table) in the lake and return the a result_id that could be used to check the processing status as well as to fetch the completed results. Callers can provide any valid ANSI SQL query that only targets the named table/entity and nothing else, no joins across data lake entities.</li><li>Get the status for any given result_id</li><li>Retrieve the results for any completed result_id, specifying an offset and limit. Query results would be persisted and retained until expired.</li><li>API clients need to be restricted to only parts of a given lake they have access to, however row level security was not necessary.</li></ul><h3>ENGINE IMPLEMENTATION</h3><p>So how did the implementation evolve?</p><p>The core goal of the system was to ultimately craft a generated Fugue SQL workflow statement that consisted of these 3 key parts:</p><ul><li>optionally create a data frame from an Fugue SQL <a href="https://fugue-tutorials.readthedocs.io/tutorials/fugue_sql/syntax.html#input-and-output">INPUT</a></li><li>execute a SQL query against the data frame</li><li>optionally write the result data frame to an Fugue SQL <a href="https://fugue-tutorials.readthedocs.io/tutorials/fugue_sql/syntax.html#input-and-output">OUTPUT</a></li></ul><p>Fugue SQL <a href="https://fugue-tutorials.readthedocs.io/tutorials/fugue_sql/syntax.html#input-and-output">INPUT/OUTPUT statement</a>s are fugue SQL extensions that let you load resources into and out of dataframes in the local process. You can load natively from the local filesystem via path references or provide custom function callouts to provide the data as PyArrow tables. We took the later approach by creating custom python functions that could load the data from local filesystem locations or ABFSS URIs to Delta Tables in Azure storage accounts using the methods previously described above (w/ DuckDB as the engine). This was then further abstracted away as IOStatementBuilders (with a number of implementations) in our project which could be configured per data lake entity/table to permit mixing/matching of input sources and output targets. Which Fugue engine to use per table was also configurable (Spark or DuckDB). For example you could configure any particular Data Lake/Table to read a table from local io, execute the query and then output it to some other storage account via <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri">abfss uris</a> (duckdb engine), or execute the query against a hive table and write to DBFS (bound to a storage account for results under the covers); spark engine.</p><p>Note that the INPUT statements could be optional. Why? Because we created the capability to configure the engine of execution to be different per data lake target. If the execution was targeted for Spark, we had no need for the INPUT statement as the input dataframe would be inherent in the SQL query as it would reference a Hive cataloged schema.tablename. Likewise OUTPUT might optional as the query being executed might be a query targeting the retrieval of a previously stored set of query results stored by result_id, hence we had no need to re-store the retrieval query’s results, but instead just return them directly to the caller.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*RsfGah7rmhJ-xfj6" /></figure><p>So how did the implementation evolve? We broke it down into the following areas:</p><ul><li>API contract definition</li><li>authorization and security</li><li>SQL query parsing, whitelisting enforcement and rewriting</li><li>Fugue IO statement builders for different INPUT/OUTPUTs and SQL query enrichment, for example we would inject a row_number on all query result dataframes before persistence to permit for paging of the results on retrieval.</li><li>Database for query results metadata and auditing</li><li>Data lake configuration to drive behavior changes per lake/table as well as permit mixing/matching INPUTs/OUTPUTs for different lake table sources and where their query results would be persisted.</li><li>Query result expiration and purging engine</li><li>Schema service implementation</li><li>Whitelisting engine for restricted columns per lake/table driven by data lake config</li><li>Data Lake replication processes for mirroring target tables to local or more proximal network storage for faster access.</li></ul><h3>CLIENT API FLOW</h3><p>The API specification was pretty straightforward and naturally could grow as a data lake evolved and new paths were added over time as the structure of the data lake was reflected in the API paths themself. Any part of an exposed data lake path could be aliased with pointers (in configuration) to other locations to further limit what could be queried.</p><ul><li>Client acquires a JWT that identifies it</li><li>Client POSTs a SQL query for the entity they wish to egress data for at /&lt;data-lake-id&gt;/&lt;data-lake-project-id&gt;/[path/to]/&lt;entity&gt;. The query executes asynchronously and returns an result that can be tracked by its result_id</li><li>Client inquires the result status via GET against /&lt;data-lake-id&gt;/&lt;data-lake-project-id&gt;/[path/to]/&lt;entity&gt;/&lt;result-id&gt;/status (SUBMITTED,PROCESSING,FAILED,COMPLETED)</li><li>Once COMPLETED the client can retrieve their results by issuing one or more GETs for /&lt;data-lake-id&gt;/&lt;data-lake-project-id&gt;/[path/to]/&lt;entity&gt;/&lt;result-id&gt;/results?offset=[n]&amp;limit=[n]</li><li>Query results are no longer available after their expiration time and are purged.</li></ul><h3>GET SCHEMA</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*zk7HYNXc8oV_zebd" /></figure><h3>POST QUERY</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*WF_MAOXY9s36AKQl" /></figure><h3>GET STATUS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*PiVzdaWsk4cZw0UU" /></figure><h3>GET RESULTS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mFoMKQmHzo8B1ZPJ" /></figure><h3>SECURITY</h3><p>There was a fair amount of configuration behind the scenes that enforced which clients could access which data lakes, data lake projects, folders within the hierarchies as well as end tables. Security was enforced both within the service engine as well as at bindings at the driver levels utilizing different underlying identities when accessing different data lakes or locations within them as well as specific spark clusters for different security contexts.</p><p>Regarding the potential for SQL injection this was locked down in various ways. Much of this was inherent negated via the underlying identities being utilized for lower level operations but another large set of it was restricted via the usage of tools like <a href="https://github.com/tobymao/sqlglot">SQLGlot</a>. Tools like SQLGlot could parse all SQL statements into traversable graphs that could then be analyzed for any exceptions against a set rules that defined which kinds of SQL constructs were whitelisted. For example, reject INSERT and permit SELECT. This extended into analysis of targeted table names even via aliasing to ensure no cross joins were permissible. SQLGlot made this quite easy and all statements ran through multiple passes before submission to any execution engine.</p><h3>SUMMARY</h3><p>In summary this project opened up a world of possibilities and there is more work going into the platform as more use-cases emerge. The <a href="https://github.com/fugue-project/fugue">Fugue Project </a>is definitely a pretty cool piece of work that is worth a look if you do any work in this space. The number of possible use cases this could be thrown at to help solve is quite large. Kudos to the Fugue project team members and the community as well; they are quite helpful and responsive in their slack channels.</p><p><a href="https://github.com/fugue-project/fugue">https://github.com/fugue-project/fugue</a></p><p><a href="https://fugue-tutorials.readthedocs.io/tutorials/quick_look/ten_minutes.html">https://fugue-tutorials.readthedocs.io/tutorials/quick_look/ten_minutes.html</a></p><p>originally posted at: <a href="https://bitsofinfo.wordpress.com/2023/08/14/data-lake-rest-api-delta-lake-fugue-spark/">https://bitsofinfo.wordpress.com/2023/08/14/data-lake-rest-api-delta-lake-fugue-spark/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f1fb9077c41" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/architecture-for-a-data-lake-rest-api-using-delta-lake-fugue-spark-3f1fb9077c41">Architecture for a data lake REST API using Delta Lake, Fugue &amp; Spark</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecture for generative Terragrunt & Terraform infrastructure as code (IaC)]]></title>
            <link>https://levelup.gitconnected.com/architecture-for-generative-terragrunt-terraform-infrastructure-as-code-iac-70a41e2be960?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/70a41e2be960</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terragrunt]]></category>
            <category><![CDATA[terraform]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Sun, 26 Feb 2023 19:35:56 GMT</pubDate>
            <atom:updated>2023-09-03T15:32:13.051Z</atom:updated>
            <content:encoded><![CDATA[<p>This article covers a specific scenario where despite trying to leverage as many DRY (don’t repeat yourself) principles made available to us by the underlying IaC (infrastructure as code) frameworks, sometimes we still need to elevate the abstraction to another level to fully reduce code duplication and gain larger economies of scale deploying large platforms using IaC.</p><h3>Evolution 1</h3><p>Sometimes when we first get into DevOps automation using infrastructure as code tools (IaC) like <a href="https://www.terraform.io/">Terraform</a>, we start out with monolithic projects that define multiple logical environments (i.e. dev, test prod) and end up with a lot of copy pasted code between them as resource definitions first get created and tested in a lower environment and then once working, get copy pasted up the chain. Its doesn’t end up very DRY (don’t repeat yourself)</p><p>For example, here we we have a simple project for 3 environments, each have the same Terraform code that defines N resources in main.tf, the code is identical in each folder, what varies at runtime is the value of the variables. The more environments we need, the more copied code we end up with. For each environment you need to do a separate terraform plan/apply/destroy. Each environment also has its own provider, vars, outputs configuration and backend state definition, again each basically the same with minor differences pertaining to the context of the environment they are for.</p><pre>my-infra/<br>     dev/<br>        variables.tf<br>        main.tf <br>        outputs.tf<br>        providers.tf<br>     test/<br>        variables.tf<br>        main.tf <br>        outputs.tf<br>        providers.tf<br>     prod/<br>        variables.tf<br>        main.tf<br>        outputs.tf<br>        providers.tf</pre><h3>Evolution 2</h3><p>The next level of evolution towards being DRY is to get rid of the copy-pasted resource definitions defined in main.tf and to start using Terraform modules that live in a dedicated &quot;modules&quot; folder somewhere in the IaC project or even remotely in a git repository. In this scenario you still have some copy pasted code in each main.tf file but it contains <em>much less code</em> as it is now invoking your Terraform modules via source references which abstract away the resource details while just passing the required input variables through down to the underlying module referenced via source. Still... each time you have a new logical environment folder you want to target you end up copying your main.tf file, and the variables.tf ... and the outputs.tf.</p><pre>my-infra/<br>     modules/<br>        database/<br>          main.tf<br>          variables.tf<br>          outputs.tf<br>        vm/<br>          ...<br>        vnet/<br>          ...<br>        storage/<br>          ...<br>     dev/<br>        variables.tf<br>        main.tf (references modules/[N+] via source refs)<br>        outputs.tf<br>        providers.tf<br>     test/<br>        main.tf (references modules/[N+] via source refs)<br>        ...<br>     prod/<br>        main.tf (references modules/[N+] via source refs)<br>        ...</pre><h3>Evolution 3</h3><p>What about storing our variables? tfvars work, but even for those we might end up with a fair amount of copy pasting of variables that have standard values that we want to keep configurable but don&#39;t necessarily deviate that much from environment to environment.... i.e. we really always end up identifying variables that can be classified at a higher level as shared across N environments or common to other characteristics such as location/regions etc., we have a need for global/common/shared variables at different levels of a hierarchy.</p><p>Once we really start to look at the problem, outside of minor differences related to underlying resource capacity settings and other minor feature flags, the majority of our resources per environment are pretty much configured identically (as they should be!) Also, how each resource component relates to one another at a network level; in other words, the general topology or “footprint” of each environment is often the same.</p><p>So what else is different between environments? Well often the major thing that is different is outside of those things described above, is really the target cloud account that our infrastructure is being deployed to and something in the logical <strong><em>“name/context”</em></strong> for the instance of our topology being deployed. I.E. the “name” we assign to everything and the convention for it. Man… this all sounds like it can be further templated…. at this point we start looking at other solutions, like <a href="https://terragrunt.gruntwork.io/">Terragrunt</a></p><h3>Terragrunt</h3><p>I’m not going to go too much into the full details of <a href="https://terragrunt.gruntwork.io">Terragrunt</a> but the key take away of it is that it really allows you to take your Terraform IaC to <a href="https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/">the next level of DRY</a>. Terragrunt is a wrapper around Terraform that really lets you fully isolate your Terraform code into modules that can be independently executed or executed as part of a larger footprint that exists within a hierarchy. For example:</p><pre>my-infra-modules/<br>       database/<br>          main.tf<br>          variables.tf<br>          outputs.tf<br>       vm/<br>         ...<br>       vnet/<br>         ...<br>       storage/<br>         ...<br> <br>my-infra/<br>     terragrunt.hcl (common TF backend, provider, global vars declarations)<br>     useast/<br>       terragrunt.hcl (common vars for region)<br>       apphosting/<br>         terragrunt.hcl (common vars for apphosting footprint)<br>           dev/<br>              terragrunt.hcl (common vars for dev)<br>              database/<br>                 terragrunt.hcl (git ref to my-infra-modules/database + input vars)<br>              vnet/<br>              storage/<br>              vm/<br>           test/<br>              ...<br>           prod/<br>              ...<br>            <br>        ...</pre><p>There is a lot to Terragrunt and its a bit à la carte in regards to which features you wish to use. Point being is to get the next level of DRY and economies of scale with your Terraform code, you will need to use something else like Terragrunt.</p><h3>Evolution 4</h3><p>Awesome. Now that we are using Terragrunt everything feels much cleaner and DRY. We have a place for common/shared variables in the hierarchy, core Terraform constructs like backend for our state and provider are not repeated and still customizable. Things are pretty good. However the one less than ideal thing is that we still have some code repetition within each separate footprint&#39;s &quot; <em>environment</em>&quot; folders for each sub-module being called ( <em>i.e. what is in each environment&#39;s vm/database etc terragrunt.hcl files</em>). Its not too bad and we can live with it as the other gains described above make up for it.</p><p>But now lets imagine you need to be able to take the above and treat that as an “template” for a platform that can be deployed for N number of customers? For example, not only do we need 3 instances of our “apphosting” footprint (dev, test, prod), but now need to create N number of “apphosting” footprint instances for various clients? If you are a platform provider now you would be faced with copy pasting each Terragrunt “footprint” project for each of these instances and each of those has 3 different environments! <em>“apphosting-teamA-[env]”, “apphosting-teamB-[env]”, “apphosting-teamC-[env]” etc.</em></p><p>This is the kind of scenario the team was faced with. What we started out building as a single services platform and all of its infrastructure worked pretty good; … but then we decided to make this a much larger platform with many instances of it, each of those with their own requirement of 3 of environments with the management of them delegated to other teams.</p><h3>Footprints overview</h3><p>At this point you’ve recognized a key term that I’ve thrown out there: <strong>footprints</strong></p><p>When you create a set of IaC that produces a set of infrastructure that is fulfilling some particular use-case/role, we call that a <strong>footprint</strong>. Regardless of environment or “where” it is provisioned to, outside of some naming, feature flags and capacity related settings, any given instance of a <strong>footprint</strong> is the basically the same as any other, at least from an IaC driven management perspective. Knowing this we can start to think about making these footprints “templates” themselves, so that when given an even more limited set of parameters they can be stamped out on-demand and new ones easily created.</p><h3>Platform overview</h3><p>For the team in question we boiled it down to 3 distinct footprints that were already codified in Terragrunt and Terraform and successfully deploying into 3 different environments dev, test and production. The instances of these footprints happened to be deployed for one platform instance, but given the footprint’s architecture was generic enough it could be used for many other use-cases…. so we needed to blow this out to be able to easily create N platform instances.</p><p>At a high level the three footprints that together constitute a single “platform instance” were as follows:</p><ul><li><strong>control (CNTRL)</strong>: set of isolated cloud network infrastructure, vms, dns, storage etc a core Kubernetes cluster to be used for key services (running in k8s) to support all peer level <em>APP[N]</em> footprints. Here we are talking about things that support telemetry, monitoring, dns, CDN services, service discovery etc.</li><li><strong>application</strong>: <strong>(APP[N]): </strong>set of isolated cloud network infrastructure, vms, dns, storage etc and one or more dedicated Kubernetes cluster(s) to be used for running N number of application and service workloads. Certain elements in this footprint depended on services provided in the CNTRL footprint and each APP[N] network was peered with the central CNTRL network. There could be N instances of an APP[N] footprint within a platform instance depending upon the requirements.</li><li><strong>admin (ADM)</strong>: set of isolated cloud network infrastructure, vms, dns, storage and CI/CD execution infrastructure intended for DevOps personnel to use to deploy and administer the CNTRL and APP[N] footprint instances under its governance.</li></ul><p>Each of these 3 footprints would have their own dev, test, and production instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*buGCRajYw-YYx3D_" /></figure><p>For each of these 3 footprints, constituting a single platform instance, the desire was to have N instances of it as such, each of the logical instances had environment specific instances for dev, test, prod. So given 4 logical platforms, there were actually 12 physical instances materialized in the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-G2xOVVRw53aNKmv" /></figure><h3>Generating Terragrunt projects</h3><p>So how was this platform ultimately materialized? As stated earlier, Terragrunt could only take us so far, given the requirements it was unmanageable to proceed with DevOps staff copy-pasting Terragrunt projects over and over. What if footprints evolved? How would be introduce new combinations of footprints?</p><p>Ultimately it was decided to do the following:</p><ul><li>Take each Terragrunt “footprint” and templatize each using <a href="https://jinja.palletsprojects.com/en/3.0.x/">Jinja templating</a>. As stated earlier, when you really break it down, much of the actual deviation between environments boils to down to a few dozen feature flag changes, capacity related settings, target cloud accounts, backend state targets and primarily the naming of of all resources managed under IaC control.</li><li>Consolidate and express all configurable variables that each footprint would need into a simplified single file per platform instance/environment combination. Express the configuration using YAML to make it much clearer for an human operator to curate and make changes to.</li><li>Write a python program that could be run via CLI invocation that could take in a target Terragrunt footprint template, marry it with a platform environment YAML file and output a executable Terragrunt project that depends on nothing else other than any underlying Terraform modules it sources from versioned git URLs.</li><li>Once generated each Terragrunt project could completely fork and live on its own, or ideally simply be updated as need be by the same python program that initially generated it, applying any new changes that materialize as a result of changes to the YAML configuration and/or the underlying Terragrunt footprint template projects.</li><li>Using all of the above in combination with solid Git workflow development practices using branching, PRs and tagging…. each platform instance could be fully customizable, versioned and isolated from breaking changes over time.</li></ul><p>The implementation broke down into the following separate projects:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*olaOoyqkVWBkNK15" /></figure><p>By separating the framework into the individual projects, as well as the executable Terragrunt project outputs themselves, this permitted a high level of granularity when rolling out changes at any level, while permitting for numerous types of release lifecycles and version combinations. We didn’t have to worry as much about “will this change break something” as the separation of projects, each with their own release cycles permitted for a very flexible and de-coupled change management process.</p><h3>Final Thoughts</h3><p>Overall this was a challenging project with a unique solution that ultimately met the project requirements. By compartmentalizing and abstracting patterns of infrastructure into footprints that could then be templatized and stamped out into individual executable Terragrunt projects, this gave much larger economies of scale to be able to create N number of platform instances.</p><p>Since the original inception of this project, some newer features in Terraform itself and other 3rd party frameworks, such as <a href="https://terraspace.cloud/">Terraspace</a> have come online which are also worthy of consideration if venturing down a path like this. Terragrunt itself is pretty nice, however it does have some issues that we struggled with, particularly around Terragrunt’s “dependencies” syntax…. it was often problematic and un-reliable.</p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2023/02/26/architecture-for-generative-terragrunt-terraform-infrastructure-as-code-iac/"><em>http://bitsofinfo.wordpress.com</em></a><em> on February 26, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70a41e2be960" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/architecture-for-generative-terragrunt-terraform-infrastructure-as-code-iac-70a41e2be960">Architecture for generative Terragrunt &amp; Terraform infrastructure as code (IaC)</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bridging Auth0 with Legacy IdPs]]></title>
            <link>https://levelup.gitconnected.com/bridging-auth0-with-legacy-idps-ad03bcd1588a?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad03bcd1588a</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[auth0]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Fri, 09 Sep 2022 04:57:18 GMT</pubDate>
            <atom:updated>2026-04-10T02:19:29.092Z</atom:updated>
            <content:encoded><![CDATA[<p><em>“We know we need to move to Auth0 and OAuth standards eventually. But we can’t just flip a switch. Can we figure out a path where both worlds can coexist?”</em></p><p>That was the challenge from a client with a long running custom platform that had its own bespoke authentication and authorization system. Their existing custom login mechanism issued its own proprietary access tokens; not JWTs, not anything standards based, just a custom opaque token format that had been woven into the fabric of their application ecosystem over many years. The desire to move to Auth0 was there; the problem was how to get there without forcing every application to migrate at the same time.</p><p>What was needed was a bridge: a way to let users log in via Auth0, while still being able to obtain a legacy token for applications that were not yet ready to consume a JWT. New applications could adopt the Auth0 JWT directly. Older ones could continue getting the legacy token they already understood; or some mix of the two during migration. Both from the same login event.</p><p>This post documents a proof of concept that explored that approach. This was a prototype running in an isolated test environment and was not intended to be production ready. Several shortcuts were taken that would need to be addressed before anything like this was deployed for real. That said, it was a useful exercise in validating the pattern and identifying which Auth0 primitives could anchor the solution.</p><h3>The legacy platform</h3><p>The legacy platform had its own user credential store, its own login endpoint, and its own token issuance logic. When a user authenticated, the platform issued a proprietary opaque token that downstream services validated by calling back to the platform. No JWT, no standard claims, no JWKS endpoint.</p><p>The platform also supported service to service authentication; a service could log in with a service account and obtain a token, then use that token to make privileged API calls on behalf of users. This capability turned out to be a key ingredient in the bridging design.</p><h3>The bridging concept</h3><p>The idea was fairly straightforward once the pieces were in view. Auth0 would become the front door for authentication. After a user logged in through Auth0, there needed to be a mechanism to go fetch an equivalent legacy token for that user, so that applications still dependent on the legacy auth system could continue to function.</p><p>The question was: how do you prove to the legacy platform that a user legitimately authenticated through Auth0, in a way the legacy platform could verify, without fundamentally changing its core auth model?</p><p>The answer was a nonce. A nonce (number used once) is a one time use random value; the idea being that it can be generated at a known point in a flow, stored, and then consumed exactly once to verify that a specific event occurred. When a user authenticated through Auth0, the login flow would generate a nonce, store it against the user’s account in the legacy platform, and surface it as a claim in the Auth0 access token. An application receiving that token could then present the nonce to the legacy platform to say “this user just authenticated through Auth0; here is the proof, now issue their legacy token.” The legacy platform would validate the nonce, mark it consumed so it could not be replayed, and return the legacy token.</p><p>Neither system had to fundamentally change how it worked. Auth0 handled the login. The legacy platform handled legacy token issuance. The nonce was the handshake between them.</p><h3>Auth0 database action scripts</h3><p>Auth0 provides <a href="https://auth0.com/docs/authenticate/database-connections/custom-db">Database Action Scripts</a>; hooks written in JavaScript that plug into Auth0’s authentication pipeline. When using Auth0’s Universal Login with a Custom Database connection, you can provide a login(email, password, callback) function that Auth0 invokes when a user attempts to authenticate. This is the hook used in the prototype.</p><p>The login script would take the user’s email and password, hash the password using the same algorithm the legacy platform used, and query the legacy credential store to validate the match. On success, it would generate a nonce, write it to the legacy platform’s database associated with the user’s account, and invoke the Auth0 callback with the user’s profile information including the nonce as a custom attribute. Auth0 would then include that nonce as a claim in the issued access token.</p><p><strong>One important caveat:</strong> in a real production scenario you would never have an Auth0 action script talking directly to a database. You would go through an API or a secure intermediary. For a POC running in an isolated test environment, the direct database call was acceptable to prove the concept, but this would be one of the first things to rework on any path to production.</p><h3>The legacy platform additions</h3><p>A test branch of the legacy platform was created that exposed a new endpoint: login_auth0. This endpoint was protected by the service account authentication mechanism; it would only accept calls from a caller presenting a valid service token. The endpoint accepted a user email and a nonce value. It would look up the nonce for the given user, validate that it was active and matched what had been stored during the Auth0 login flow, mark it inactive to prevent replay, and return a legacy token issued on behalf of that user.</p><p>This was a contained addition to the legacy platform. The core authentication model was not changed and no user facing behavior was altered.</p><h3>The test application</h3><p>To wire everything together and validate the flow end to end, a small Node.js/Express application was built using the <a href="https://github.com/auth0/express-openid-connect">express-openid-connect</a> library. This test app stood in for any application in the ecosystem that needed to be Auth0 enabled while still depending on a legacy token during the transition period.</p><p>The test app required a handful of configuration parameters: the Auth0 application CLIENT_ID, the ISSUER_BASE_URL for the Auth0 tenant, a session secret, and a service account identity and credential for the legacy platform. Once running, navigating to the login route would redirect the user to Auth0&#39;s Universal Login page. After authentication, the user was redirected back with an authorization code, which the app exchanged with Auth0 for an access token containing the nonce claim.</p><p>From there the app would authenticate against the legacy platform using its service account credentials to get a service token, then call the login_auth0 endpoint with the user&#39;s email and nonce. On success it received back the legacy token for that user. A /debug debug endpoint would display both tokens side by side; the Auth0 JWT and the legacy token, demonstrating that a single login event had produced both. From that point the user could present either token to downstream services as appropriate.</p><p>In a production version of this architecture, the token exchange logic embedded in the test app would be extracted into a dedicated intermediary authorization service. Rather than every application implementing this exchange themselves, they would delegate to that service. The test app was sufficient to validate the concept.</p><h3>The end to end flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*jEybKBsA60DSTnZ3xwxl7Q.jpeg" /></figure><h3>Migration flexibility</h3><p>One of the more useful properties of this design is that it decouples the authentication migration from the application migration. Once Auth0 is the front door, individual applications can choose when they cut over to consuming the JWT directly. An application that is ready can use the Auth0 access token. One that is not can still call the legacy token exchange to get what it needs. Both tokens are available from the same login event, so there is no hard cutover date that forces every application to move at once. Teams can migrate on their own schedule as they are ready.</p><h3>What would still be needed for production</h3><p>As noted up front, this was a POC and a fair amount of work would remain before any version of this could be considered production ready. At a minimum:</p><ul><li>The Auth0 login action script would need to talk to the legacy credential store via a REST API rather than a direct database connection.</li><li>The login_auth0 endpoint and surrounding nonce handling logic would need a thorough security review; including nonce generation entropy, expiration windows, and edge cases around concurrent requests.</li><li>The token exchange logic would need to be extracted into a proper intermediary authorization service rather than living inside each consuming application.</li><li>All credentials and secrets would need to go through a proper secrets management system.</li><li>The nonce mechanism itself could be made more robust; for example, replacing the plain nonce with a signed nonce (e.g. a short lived signed JWT) to provide additional tamper evidence without requiring a database lookup to validate integrity before consumption.</li></ul><p>None of these are blockers to the pattern; they are the expected hardening work that comes with taking any proof of concept toward production.</p><h3>Summary</h3><p>Overall this was a useful exercise in validating how Auth0’s extensibility primitives, specifically its db action acripts and custom database connection, can be used to bridge a modern oauth based identity provider with a legacy proprietary auth system. The nonce based handshake kept the surface area of changes small on both sides, and the dual token result from a single login event gives a reasonable incremental migration path.</p><p>Auth0’s documentation on database action scripts is a good starting point if you want to dig further into this space: <a href="https://auth0.com/docs/authenticate/database-connections/custom-db">https://auth0.com/docs/authenticate/database-connections/custom-db</a></p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2022/09/08/bridging-auth0-with-legacy-idps/"><em>http://bitsofinfo.wordpress.com</em></a><em> on September 9, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad03bcd1588a" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/bridging-auth0-with-legacy-idps-ad03bcd1588a">Bridging Auth0 with Legacy IdPs</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fully automated Lets Encrypt TLS certs with ACME-DNS on Kubernetes]]></title>
            <link>https://levelup.gitconnected.com/fully-automated-lets-encrypt-tls-certs-with-acme-dns-on-k8s-6d79ec23e7bf?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d79ec23e7bf</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[ssl]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[lets-encrypt]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Thu, 08 Sep 2022 19:39:28 GMT</pubDate>
            <atom:updated>2023-09-03T15:34:36.456Z</atom:updated>
            <content:encoded><![CDATA[<p>This article covers fully automating DNS and the issuance of TLS certificates of Kubernetes for Ingress based workloads (both public and private) utilizing <a href="https://cert-manager.io/docs/">cert-manager</a>, <a href="https://github.com/kubernetes-sigs/external-dns">external-dns</a>, and <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/"><strong>kubernetes-acme-dns-registrar</strong></a></p><h3>Scenario</h3><p>You are a busy DevOps professional. You want to setup an Kubernetes platform that can accept any typical HTTP based workload ( Ingress based) with minimal management overhead. You don&#39;t want to be bothered with day-to-day requests for new DNS hostnames and TLS certificates... nor all the management overhead that can come with that long term. Workloads from your team, particularly in lower dev/qa environments, come and go all day long; new apps are released, old ones go away; these workloads sometimes can be publicly accessible (public dns zones), other times internal only (private dns zones). At the end of the day you simply want to create an automated ecosystem to support this and <em>&quot;set it and forget it&quot;</em> ...so to speak.</p><h3>Reality</h3><p>There are numerous solutions and tool sets to help you solve this type of problem out there. They generally range from some kind of manually process using various tools, to a full blown <a href="https://avinetworks.com/glossary/kubernetes-service-mesh/">service mesh</a> or something in between. This article covers a solution that sits sort of in-between, who through the years across multiple variations has served me quite well in addressing the scenario above.</p><h3>Tooling</h3><p>As stated above there are a plethora of existing open source tools out there to aid in building a solution to address this on top of Kubernetes. Here are some of the key ones to make all the automation actually happen.</p><p><a href="https://github.com/kubernetes-sigs/external-dns"><strong>external-dns</strong></a><strong>:</strong> this project is is fantastic. Basically one or more <a href="https://github.com/kubernetes-sigs/external-dns">external-dns</a> instances runs on your cluster, detect the presences of new objects (such as <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress</a> objects) and automatically creates DNS entries in a target DNS server/service that point to that’s Ingress’ controller’s backing service’s IP. As you would expect each <em>external-dns</em> instances is fully configurable to filter what types of objects it will listen for and which target DNS backend service it will be managing entries in.</p><p><strong>Workloads and Ingress objects</strong>: these are the actual application deployments, pods, services and Kubernetes Ingress objects that define your running application and which host(s) it can be reached by as well as which cert-manager Issuer should be used when executing TLS certificate operations for it.</p><p><a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/"><strong>Ingress Controllers</strong></a><strong>:</strong> you are going to need one or more Ingress controllers to actually service the traffic for each defined Ingress object its selector’s are applicable for. The IP of this ingress controller’s backing k8s service is the target of each distinct Ingress “host” which gets an A record created for it via external-dns. You can have N number of ingress controllers as you see fit which, as with external-dns, align to different ingress objects based on label/annotation (or other) based selector filters.</p><p><a href="https://github.com/cert-manager/cert-manager"><strong>cert-manager</strong></a><strong>:</strong> Next up we are going to want <a href="https://github.com/cert-manager/cert-manager">cert-manager</a> running on our cluster, which itself can also react to <a href="https://kubernetes.io/docs/concepts/services-networking/ingress/">Ingress objects</a> and automatically request <a href="https://letsencrypt.org/">Lets Encrypt</a> CA signed TLS certificates for the hosts defined by each Ingress. Once issued these TLS certificates (and backing keys) are stored as kubernetes secrets on the cluster and are then used by the <a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/">Ingress Controller </a>for TLS handling/termination for each Ingress. cert-manager also handles automatically renewing these as they expire. One key config item here is <a href="https://cert-manager.io/docs/configuration/acme/#configuration">configuring an ACME Issuer </a>for cert-manager that targets Lets Encrypt (staging or prod) AND defines an <a href="https://cert-manager.io/docs/configuration/acme/#solving-challenges">Solver</a> for handling the ACME challenges using <a href="https://cert-manager.io/docs/configuration/acme/http01//">HTTP01</a> or <a href="https://cert-manager.io/docs/configuration/acme/dns01//">DNS01</a>; for the architecture in this article we would use DNS01</p><p><a href="https://github.com/joohoi/acme-dns"><strong>acme-dns</strong></a><strong>:</strong> When cert-manager initiates the process to automatically, create and acquire a new TLS certificate with the Lets Encrypt CA, under the covers it is using the <a href="https://cert-manager.io/docs/configuration/acme/#solving-challenges">ACME protocol</a>; part of this protocol involves the CA validating a challenge to prove that the requester actually has authority over the domain name the host certificate is being requested for. The ACME CA (Lets Encrypt) will attempt to verify the existence of this challenge in various ways, the two most common being <a href="https://cert-manager.io/docs/configuration/acme/http01//">HTTP01</a> which checks for the challenge via http at the requested host&#39;s URL, or <a href="https://cert-manager.io/docs/configuration/acme/dns01//">DNS01</a> which checks for the challenge via a DNS TXT record for the domain (we will be using DNS01). There are various reasons why you might want to use DNS01 vs HTTP01 (such as wildcard names, or an internal non-public http service), its your choice based on your scenario. That said, if you utilize DNS01 the project provides a use-case specific DNS server you can use to serve up these challenge TXT records so that you don&#39;t have to (or if you can&#39;t) provision them in your primary DNS server yourself, or for example if your DNS zone is private. <a href="https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation">A good article for the reasons on this is here.</a> In this use-case <a href="https://cert-manager.io/docs/configuration/acme/dns01/acme-dns/">cert-manager would be configured to utilize </a><a href="https://cert-manager.io/docs/configuration/acme/dns01/acme-dns/">acme-dns with the DNS01 solver</a>, and it places the TXT record challenge values there for each host according to the configuration of the Solver using the acme-dns.json host/fqdn mapping to acme-dns zone file ( <a href="https://cert-manager.io/docs/configuration/acme/dns01/acme-dns/">see documentation here</a>)</p><h3>So how does this work?</h3><p>So give the above components, how does this all work? This article is not intended to provide you with a step by step of setting this all up, but I will provide you with the high level, AND most importantly where the gaps are in the automation that need to be filled in. Once all the above is configured, lets walk through how it all works together in a typical use-case:</p><ol><li><em>DevOps staff are informed by the Dev team of an new k8s application and </em><em>Ingress </em><em>host fqdn that will need to be accessible on the k8s cluster, this triggers the DevOps team to manually do the items listed in #2 in preparation prior to app deployment:</em></li><li><em>DevOps first manually registers a new domain/host with the </em><em>acme-dns server&#39;s API and get back a acme-dns &quot;registration&quot;. Second they manually update the &quot;</em><em>acme-dns.json&quot; secret bound in the </em><em>cert-manager</em> Issuer <em>with a new entry for the domain/hostname that contains this registration information. 3rd they manually update the primary DNS server with the CNAME record that will tell the verifier how to locate the challenge TXT record. This is obtained from the </em><em>fulldomain field in the registration. </em><a href="https://cert-manager.io/docs/configuration/acme/dns01/acme-dns/">Read here for the manual steps</a></li><li>Now that things are ready, the application is deployed and a k8s Ingress is created/updated on Kubernetes for the app`s hostname</li><li><strong>external-dns</strong> detects the Ingress and creates an A record in the Ingress&#39; host FQDN DNS zone that points to the IP of the Ingress controller Service&#39;s load balancer&#39;s IP address</li><li><strong>cert-manager</strong> detects the Ingress and initiates the configured Issuer&#39;s ACME TLS certificate request against the Lets Encrypt CA. Assuming the Issuer&#39;s Solver is configured w/ DNS01 and acme-dns (<a href="https://cert-manager.io/docs/configuration/acme/dns01/acme-dns/">see document here</a>) cert-manager will proceed to lookup the hostname/domain&#39;s registration info in the acme-dns.json secret via the solver&#39;s configuration that was prepared manually by DevOps (in step #2 above) and use the original registration info to authenticate against the acme-dns DNS server to set the challenge TXT record properly for that domain.</li><li><strong>lets-encrypt</strong> eventually successfully resolvest the challenge TXT record via DNS and validates that domain/host is owned by the requestor and issues the certificate to cert-manager</li><li><strong>cert-manager</strong> stores the issued TLS certificate as a k8s secret and the Ingress Controller uses this for TLS termination for the app’s Ingress definition.</li><li>The app can be accessed via its Ingress host name w/ valid TLS security! boom!</li><li>All is good! … except for the manual steps at #2….</li></ol><h3>What can be improved</h3><p>Well ideally we would like this to be fully automated and <strong>eliminate the manual steps #1 and #2</strong> in the process described above. Doing so would permit new applications w/ Ingress definitions to simply be deployed to the k8s cluster and automatically have everything described previously with no human intervention. How can we achieve this?</p><p>During my time working with a prior team I lived through this scenario. It was annoying and made things slower. When presented an opportunity to create a new Kubernetes platform for a new group, I decided to solve this problem and ended up developing <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/">kubernetes-acme-dns-registrar</a> to provide the glue that fills this gap in the automation. So what exactly does this project do?</p><p>There are 3 key things that <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/">kubernetes-acme-dns-registrar</a> automates for you, that cover what was described in <strong>#1 and #2 above</strong></p><ol><li>Automatic detection of Ingress objects and ensuring a acme-dns registration exists in the acme-dns server</li><li>Automatic creation of the required DNS CNAME in a target DNS server/service for the host/domain per the acme-dns registration</li><li>Automatic updating of the cert-manager acme-dns.json secret with the new registration info</li></ol><p>Below is a diagram that helps to explain the overall flow of automation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/0*zyvXu463j0Rs0QPk" /></figure><p>With <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/">kubernetes-acme-dns-registrar</a> running, it provides that extra little bit of automation glue that can make end to end TLS automation for Ingress based workloads a bit more seamless. Now when an Ingress appears on a cluster, within about 1-3 minutes it is fully accessible at its defined hostname with no human intervention. This project is currently in use on some of the team&#39;s production clusters.</p><p>Please checkout the <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/">kubernetes-acme-dns-registrar</a> project and i <a href="https://github.com/bitsofinfo/kubernetes-acme-dns-registrar/">ts README for full information on install, setup, configuration etc</a>. I hope this helps others out there with this kind of need!</p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2022/09/08/fully-automated-lets-encrypt-tls-certs-with-acme-dns-on-k8s/"><em>http://bitsofinfo.wordpress.com</em></a><em> on September 8, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d79ec23e7bf" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/fully-automated-lets-encrypt-tls-certs-with-acme-dns-on-k8s-6d79ec23e7bf">Fully automated Lets Encrypt TLS certs with ACME-DNS on Kubernetes</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reacting to K8s Events with k8s-watcher]]></title>
            <link>https://levelup.gitconnected.com/reacting-to-k8s-events-with-k8s-watcher-3a3fd2ea82d6?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a3fd2ea82d6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Mon, 23 May 2022 19:42:23 GMT</pubDate>
            <atom:updated>2023-09-03T15:34:46.222Z</atom:updated>
            <content:encoded><![CDATA[<p>As part of a recent project which needs to automatically issue new TLS certificates for hosts defined in Kubernetes Ingress objects, I ended up having to create a library that would let me detect such events in a simplified manner for part of a larger Python program which needs to react to such events.</p><p>My goal was to be able to define a simple specification that lets the calling code define what type of Kubernbetes object to get events for, as well as any label/field filters to further refine the result.</p><p>The native Kubernetes API provides a low-level mechanism for this via the “watch” capability for different k8s resource types (see <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes">“efficient detection of changes”</a>) and the <a href="https://github.com/kubernetes-client/python">Python k8s library</a> reflects this same capability. That said, the functionality is low level, and I wanted a simplified abstraction over this that lets the calling code just react to events it cares about and not having to worry about the details.</p><h3>k8s-watcher</h3><p><a href="https://github.com/bitsofinfo/k8swatcher">k8s-watcher</a> is a Python module that simplifies watching anything on a kubernetes cluster. You can utilize this module in your own python application to fulfill the typical <em>“list then watch”</em> functionality as described in the “ <a href="https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes">Efficient detection of changes</a> “ section of the kubernetes API documentation. (without having to write that code yourself!).</p><p>The library provides a few different ways to invoke it.</p><p>First off you can define one or more K8sWatchConfig instances, which simply define what you want to watch. Such as:</p><pre>watch_config = K8sWatchConfig(**{ \<br>                    &quot;namespace&quot;: &quot;my-namespace&quot;, \<br>                    &quot;kind&quot;: &quot;Pod&quot;, \<br>                    &quot;sdk_client_class_name&quot;: &quot;CoreV1Api&quot;, \<br>                    &quot;sdk_list_function_name&quot;: &quot;list_namespaced_pod&quot;, \<br>                    &quot;field_selector&quot;: None, \<br>                    &quot;label_selector&quot;: &quot;mylabel=x,myotherlabel=y&quot;,<br>                    &quot;include_k8s_objects&quot;: True<br>                })</pre><h3>Basic direct usage</h3><p>Next you can start watching directly inline by creating a K8sWatcher for each K8sWatchConfig and invoke its watcher() method that returns a generator you can iterate over as events become available that match your criteria.</p><p>Note that iterating over the results as they become available is blocking… so this may not be ideal depending on your use-case…</p><pre>k8s_watcher = K8sWatcher(watch_config).watcher()<br> <br>for event in k8s_watcher:<br>    print(json.dumps(event.dict(),default=str,indent=2))</pre><h3>Queues usage</h3><p>A more realistic approach would be to let k8s-watcher work in the background, and write events to a queue that your code can consume and react to on its own terms... well k8s-watcher supports this scenario as well.</p><p>If you are not interested in writing your own consumer Thread code and would like each K8sWatchEvent to be delivered via python Queues you can use the K8sWatcherService.queuing_watch(K8sWatchConfig, unified_queue=True|False) method. Each time you call queuing_watch, K8sWatcherService creates a new Thread bound to a unique K8sWatcher instance to automatically capture all events emitted from it. Each event will be placed on a Queue that is returned to you. If you pass unified_queue=True, the same Queue instance will be returned for every call to queuing_watch() so you only have to monitor a single Queue that will contains different K8sWatchEvents across all the different K8sWatchConfigs you define.</p><pre>import json<br>from k8swatcher import K8sWatchConfig, K8sWatchService, K8sWatchEvent<br>from threading import Thread<br><br>class MyConsumerThread(Thread):<br><br>    def __init__(self, event_queue_to_monitor, *args, **kwargs):<br>        kwargs.setdefault(&#39;daemon&#39;, True)<br>        super().__init__(*args, **kwargs)<br><br>    def run(self):<br>        while True:<br>            watch_event:K8sWatchEvent = self.watch_event_queue.get()<br>            print(json.dumps(watch_event.dict(),default=str,indent=2))<br><br>pod_watch_config = K8sWatchConfig(**{ \<br>                    &quot;id&quot;: k8s_kind,<br>                    &quot;namespace&quot;: k8s_namespace, \<br>                    &quot;kind&quot;: &quot;Pod&quot;, \<br>                  ...})<br><br>ingress_watch_config = K8sWatchConfig(**{ \<br>                    &quot;id&quot;: k8s_kind,<br>                    &quot;namespace&quot;: k8s_namespace, \<br>                    &quot;kind&quot;: &quot;Ingress&quot;, \<br>                  ...})<br><br>watch_service = K8sWatcherService()<br><br>&quot;&quot;&quot;<br>With `unified_queue=False` (the default):<br><br>... each distinct call to queuing_watch() returns<br>a dedicated Queue per K8sWatchConfig<br>&quot;&quot;&quot;<br>pod_event_queue:Queue = watch_service.queuing_watch(pod_watch_config)<br>ingress_event_queue:Queue = watch_service.queuing_watch(ingress_watch_config)<br><br>pod_consumer = MyConsumerThread(pod_event_queue)<br>pod_consumer.start()<br><br>ingress_consumer = MyConsumerThread(ingress_event_queue)<br>ingress_consumer.start()<br><br>pod_consumer.join()<br>ingress_consuner.join()<br>watch_service.join()<br><br><br>&quot;&quot;&quot;<br>However with `unified_queue=True`:<br><br>... each distinct call to queuing_watch() returns<br>a the same Queue that will get events for all K8sWatchConfigs <br>&quot;&quot;&quot;<br>global_event_queue:Queue = watch_service.queuing_watch(pod_watch_config,unified_queue=True)<br>watch_service.queuing_watch(ingress_watch_config,unified_queue=True)<br><br>all_events_consumer = MyConsumerThread(global_event_queue)<br>all_events_consumer.start()<br>all_events_consumer.join()<br><br>watch_service.join()<br>...</pre><h3>Asyncio Usage</h3><p>If you don’t want to manage any Threads at all, you can utilize the K8sEventHandler method. In this usage pattern you simply provide a class instance that implements the K8sEventHandler method async def handle_k8s_watch_event(self, k8s_watch_event:K8sWatchEvent) and your handler class will be called every time a new K8sWatchEvent is created. Internally K8sWatcherService manages a consumer thread automatically for you that captures all events and calls your async handler and then awaits it&#39;s finish.</p><pre>import json<br>from k8swatcher import K8sWatchConfig, K8sWatchService, K8sWatchEvent<br><br>class MyCustomHandler(K8sEventHandler):<br>    async def handle_k8s_watch_event(self, k8s_watch_event:K8sWatchEvent):<br>        watch_event:K8sWatchEvent = self.watch_event_queue.get()<br>        print(json.dumps(watch_event.dict(),default=str,indent=2))<br>        await doMyCustomStuff(k8s_watch_event)<br><br><br>pod_watch_config = K8sWatchConfig(**{ \<br>                    &quot;id&quot;: k8s_kind,<br>                    &quot;namespace&quot;: k8s_namespace, \<br>                    &quot;kind&quot;: &quot;Pod&quot;, \<br>                  ...})<br><br>ingress_watch_config = K8sWatchConfig(**{ \<br>                    &quot;id&quot;: k8s_kind,<br>                    &quot;namespace&quot;: k8s_namespace, \<br>                    &quot;kind&quot;: &quot;Ingress&quot;, \<br>                  ...})<br><br>watch_service = K8sWatcherService()<br><br>watch_service.asyncio_watch([pod_watch_config,ingress_watch_config],MyCustomHandler())<br><br>watch_service.join()</pre><h3>Run locally w/ built in CLI</h3><p>In addition to being able to utilize this module inline in your code, the module also includes a simple CLI you can use for testing out the functionality. The CLI is not intended for any production use. You can see more details here at <a href="http://github.com/bitsofinfo/k8swatcher">github.com/bitsofinfo/k8swatcher</a></p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2022/05/23/reacting-to-k8s-events-with-k8s-watcher/"><em>http://bitsofinfo.wordpress.com</em></a><em> on May 23, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a3fd2ea82d6" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/reacting-to-k8s-events-with-k8s-watcher-3a3fd2ea82d6">Reacting to K8s Events with k8s-watcher</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecture for non-deterministic mass data collection: part 2: dynamic data lake schemas]]></title>
            <link>https://levelup.gitconnected.com/architecture-for-non-deterministic-mass-data-collection-part-2-dynamic-data-lake-schemas-c89a59abef34?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/c89a59abef34</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 19:41:29 GMT</pubDate>
            <atom:updated>2023-09-03T15:34:54.480Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Note, this is the final part of a two part series about this project; </em><a href="https://bitsofinfo.wordpress.com/2022/01/23/non-deterministic-mass-data-collection-dynamic-data-lake-schemas-aws-serverless-glue-spark/"><em>article #1 is here.</em></a></p><p>Continuing on from where we last left off, now that we had a functioning collection engine producing full graphs of crawled data all the way down to interrogable dataset_items, it was now time to get down to work and figure out how to get the relevant target endpoint data out of each dataset_item and into a format usable by downstream automation and analysts for further refinements as <em>Layer 2 Type Specific Items (L2TS).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qD1BksEqbLEGtwSF" /></figure><h3>L2 Type Specific Items (L2TS)</h3><p>From the initial project goals, we already identified that we were going to want to create individual “records” that represented the actual real data we were fetching from all of the collected endpoints. Meaning, we needed to take the actual data embedded in the dataset_items (PAGED, SIMPLE or KEY_VALUE) data fields and turn the into a usable record. For example, search results from a search engine that might be in HTML or some API response format for example. We needed to explode them out of their raw response format and flatten them out into a more usable/queryable (at scale) &quot;record&quot; that would be consumable by both humans using big data analysis tooling as well as automation. If an API expressed some parent child relationship within a response JSON document we wanted to reflect those natural relationships in our L2TS items.</p><p>For the example “search engine” use case, we wanted to be able to reflect the actual entities you might have from a set of search results as an actual model such as:</p><ul><li>search_result (search_query, total_pages, …)</li><li>search_result_page: (search_id, page_num, total_items, …)</li><li>search_result_item: (search_result_page_id, page_position, title, short_desc, url, …)</li></ul><p>Once dataset_items could be converted into type specific entities such as this at scale, we would then be able to do real meaningful logic and analysis over such entities for building more and more refined models downstream in a pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CUqLsIpjNS-Pqw13" /></figure><h3>Where to start?</h3><p>The first pass at this began where the dataset_items originally left off. We already had the model for dataset_items and many of those attributes could carry forward to each individual L2TS record. Some of the names might change slightly and a few additional lineage attributes would need to be added for L2TS records, such as parent lineage keys back to the original source dataset_item etc. But, at the end of the day we really needed to do the work... per collector.. to look into each different collector&#39;s actual response data embedded in each dataset_item and begin to model each of them appropriately. There really was no way (at least in the first pass) to avoid having human take a look at the structure of the response samples. So.. thats what we did.</p><h3>Analysis to prototype</h3><p>The first pass at the data yielded some key patterns:</p><p>From these patterns we started prototyping a CLI driven tool written in Python that given a human curated config file could parse a raw JSON structure embedded in a dataset_item and basically work on the following principle.</p><p><em>“For each config.output given an expression, fetch that value and assign it to a named key in a target dictionary. If config.output has children, recurse; no? return all dicts”</em></p><p>The <em>“target dictionary” </em>in this case represented an object that would be marshaled out to an actual record to be persisted in the data lake. The actual behavior was a bit more involved than that, but it worked quite well. Given a single dataset_item input, the result would yield an array of new strongly typed L2TS records related together with appropriate keys from relationships within the data. After each run we ended up with a set of &quot;tables&quot; (or single table) that pretty accurately reflected the data contained in any JSON document.</p><h3>Phase 1 implementation</h3><p>After iterating on the prototype we decided to formalize this into a Python module that could be utilized by both a CLI and automation jobs in a pipeline, such as Spark. The CLI version would primarily be used by developers or analysts as they developed <em>“transformation configs”</em> for each collector’s output that would would drive the L2TS converter.</p><p>So what was a transformation config? This was a YAML file that defined a hierarchy of target outputs, where each output cloud have child outputs as well. Each top level output definition had a matcher configuration that could be used to evaluate a dataset_item to see if it would qualify being parsed by bound output definition. The idea was that a stream of dataset_items would be handed off for evaluation and processing... and they needed to be sorted out.</p><p>If a dataset_item qualified the matcher rules parsing could begin. Each output definition had one schema definition that contained one or more datasource definitions, where each datasource yielded an arbitrarily named item who&#39;s value pointed to a JSONPath expression that yielded an object retrieved from a named attribute on the current dataset_item. For example this would be utilized to load a dataset_items a search result &quot;page&quot; data (from the dataset_item_page1_json attribute) into memory for consumption by further output config directives.</p><p>At this point the parser would have a context established containing the schema and datasources and current state information about everything going on including <em>&quot;where&quot;</em> in the current stack hierarchy anything <em>&quot;was&quot;</em> to aid in relationship pointer automation. Each schema had attributes definitions. Each attribute definition defined a column name plus other directives using JSONPath expressions, simple paths, or Jinja2 expressions that could be used to extract a value (or object/array values) from items available relative to the context and/or datasources. Each schema definition could be augmented with common attribute definitions for different profiles that were defined in other core config files. Each extracted value(s) would then be assigned to a new type specific record. From since each output could haves child outputs the parsing could recurse until there were no more outputs left. At that point all data would have been extracted dataset_item yielding N number of records, ready to be flushed to the data lake; all of which was strongly typed from primitives to lists/kv values and each record related to one another via parent/child inference if necessary. There were also numerous standard computed attributes automatically decorated on all records such as content hashes which would aid in drift/change detection for various record types as well as data obfuscation and sanitization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*h2Jng5LgKPdEspSX" /></figure><p>All of this would occur when invoked by the CLI against an export of dataset_items from an Athena query targeting a sample set of data for a specific collector. Developers and or analysts would iterate on this process as a transformation config was being developed and curated into its final state. Once a transformation config was finalized it was promoted into the automation pipeline where L2TS transformations could be run automatically on demand or a scheduled basis via AWS Glue Spark jobs. The output of these jobs would then be catalogued via AWS Glue yielding tables for each L2TS record type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*5VWmPqd7PXK8MKGR" /></figure><h3>Phase 2 improvements</h3><p>After phase 1 was rolled out we now had lots… and I mean lots of tables in the catalog that could be used for analysis and downstream processing. Things were strongly typed and the table schemas closely matched collector endpoint API documentation which made it convenient for analysts to read actual endpoint API documentation and clearly be able to align it to the attributes in the type specific tables. The models naturally aligned with API responses.</p><p>The main issue remaining was the creation of transformation configurations which were different per collector and all its different output documents. Each one was different and one single collector might be capable of emitting N different L1 result documents (and subsequent dataset items). Each of these ultimately required a human to manually analyze and craft a transformation config to explode the dataset item into the L2TS records. Ultimately we wanted to simplify this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ktbknNIavN5WX6Eb" /></figure><p>Given this we developed a higher level CLI tool that could generate the transformation configs given a raw dataset item JSON sample and a much simplified level of configuration that would simply define the high level output types that were expected and their locations within the document (again via expressions). Once this tool ran it could auto analyze and generate the more verbose YAML transformation config specification as well as supporting documentation on the L2TS types which would be useful to analysts.</p><p>At this point we had a pretty good end to end flow that was permitting us to collect and mass amount of data for given targets and get what was otherwise a gigantic amount of documents into a very strongly typed set of table structures that aligned w/ the source endpoint APIs/sites and was self documenting in various ways. Downstream consumers of this data were now able to query and utilize it using standard big data tooling as it moved forward though phase 4 where it would begin the long process of being enriched, cross compared, subtle relationships identification, leveraged for model training and all sorts of interesting things which I’ll write about in part 3.</p><h3>Summary</h3><p>Overall so far this was a super fun and challenging project, and it was great to see it start to be used and get off to a solid start after phase 2 was complete. There are a lot of details not covered in these articles yet because it would just be overwhelming. Besides the code, there was also a lot of DevOps automation to bring this all together that I haven’t even touched upon! In any case, hope you picked something up of use, and idea or two etc. Thanks for reading. To be continued in part 3.</p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2022/02/14/architecture-for-non-deterministic-mass-data-collection-part-2-dynamic-data-lake-schemas/"><em>http://bitsofinfo.wordpress.com</em></a><em> on February 14, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c89a59abef34" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/architecture-for-non-deterministic-mass-data-collection-part-2-dynamic-data-lake-schemas-c89a59abef34">Architecture for non-deterministic mass data collection: part 2: dynamic data lake schemas</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Architecture for non-deterministic mass data collection: part 1: collection engine]]></title>
            <link>https://levelup.gitconnected.com/architecture-for-non-deterministic-mass-data-collection-part-1-collection-engine-8c6567022da5?source=rss-f1b4cea7e056------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c6567022da5</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[bitsofinfo]]></dc:creator>
            <pubDate>Sun, 23 Jan 2022 15:58:39 GMT</pubDate>
            <atom:updated>2023-09-03T15:35:09.434Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Note, this is part one of a two part series about this project; </em><a href="https://bitsofinfo.wordpress.com/2022/02/14/architecture-for-non-deterministic-mass-data-collection-part-2-dynamic-data-lake-schemas/"><em>article #2 is here.</em></a></p><p>One of my more recent projects was spawned from a pretty interesting idea. The team wanted to build a system that would permit them to scour the Internet for information regarding a particular set of targets; a “target” being defined as product, company, topic, person, location etc… anything. Once collected, any particular target as well as collections of targets could be used for historical and predictive analysis for various goals.</p><p>For each target, a select curated set of very specific endpoints would serve as the entry points on the Internet to find information on the target, and then from there, linkages could be followed for supported endpoints and the data from those linkages collected.. and on an on… similar to a crawler but on more of a leash. The goal was that data being collected on a target would be further curated with custom logic into more enriched data sets for each target as well as how any particular target relates to other targets. The desire was that new targets would also be discovered throughout this process and then promoted to first class targets that could subsequently be monitored going forward. Again, all of this data was to be utilized for historical and predictive analysis.</p><h3>Where to start? Logical architecture</h3><p>As we started, some of the decisions were already made for the team such as the cloud provider that this would execute in: AWS. From there, given a project of this scale it would be easy to get overwhelmed so we ended up breaking it up into some high level design goals :</p><ul><li>First design a data lake structure that would store all the raw data collected in an organized manner. This lake would also need to be able to store additional permutations of the data as it gets refined through a lifecycle. All data would be immutable.</li><li>Create a generic collection/crawling engine that would collect raw target data from various endpoints, lightly wrap it in structure, decorate it with relevant meta-data and store it in the data lake as documents. The ability to add support for new endpoints should be relatively easy. The engine needed to be recursive, but on a tight leash. The raw data stored from the engine at this phase would be deemed <strong><em>level 1</em></strong> data as it would be stored “as-is” (document format) could serve as a permanent snapshot in time of what was collected on that run.</li><li>Each collector “result” document would contain a meta-data such as its own unique identifier, timestamps, parent lineage information,</li><li>Utilize a secondary database that the collection engine would use to store additional meta-data about what was collected as well as to track lineage relationships between all things collected. Ideally this would permit us to build a map of what was crawled and serve as a high level graph for each collection run for a given target that could be referenced as need be in subsequent steps.</li><li>Develop a secondary process that would consume the raw result documents and convert them into a more structured flat representation of the raw format (more like a table). These converted results were to be stored in the data lake in one or more format(s) (columnar and/or row oriented) that could be queried by standard big data tooling (both by humans and automation) for further analysis and processing. The data at this phase would be deemed data.</li><li>Next create a tertiary process that could consume the structured data and convert it into highly structured concrete types with many additional attributes specifically relevant to the actual endpoint data embedded within the dataset wrapper. This would be called <strong><em>Layer 2 Type Specific (L2TS)</em></strong> data. This would also likely be the most challenging portion to make a reality at scale. <strong>L2TS</strong> data would be the core data set that would be consumed both data analysts, ML engineers and ultimately a logic layer for enrichment and promotion to first class representations of targets <strong>(L3 and beyond)</strong> that would ultimately be used downstream for other goals.</li></ul><p>The list above represented the first set of goals that would serve as the foundation of a multi-staged process that would continually feed the data lake with consumable, structured data that was sourced from highly varying underlying endpoints. The diagram below shows the logical architecture and shows how the data for any given raw result would start out quite coarse and then explode into much more granular refinements through the different layers of transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*zchQ9AY58ASmthjp" /></figure><p>So what might this actually look like in reality and what might be an example “target” and an “endpoint” we want to collect information from? Lets walk through a use case:</p><p>In this case a <strong>collector</strong> implementation for the Twitter APIs would be need to be developed. It would support taking a configurable payload with arguments, one of which being a string target (in this case <em>“kayaking in florida”</em>). With this target string its implementation could execute a twitter search API invocation for the phrase <em>“kayaking in florida”</em>, capturing the raw JSON API response as it paginates through results and finally wrapping the paged results in a larger collector result object (containing collector metadata). This result object would contain up to N “pages” of tweets from the search for example. The embedded raw results from the Twitter API call would be left as-is, and the overall result object wrapping it would be marshaled to JSON and flushed to the data lake. Collector meta-data about the result would also be stored in a secondary database.</p><h3>Collection engine</h3><p>The first logical place we started was to design and build the collection engine itself. Since the implementation would utilize AWS it was decided to implement all functionality of the collector engine using Lambda functions in combination with SNS/SQS so that the crawler engine could be fully reactive, event driven and take advantage of the massive amount of parallel processing needed to efficiently complete a collector engine execution. Each execution could be different each time it ran due to the non-deterministic nature a target collection run could take on any given invocation.</p><p>The collector engine itself was originally prototyped in Python as a <a href="https://www.serverless.com/">Serverless Framework</a> project leveraging numerous plugins and could be fully mocked locally on developer machines using <a href="https://github.com/dherault/serverless-offline">serverless offline</a>. From this prototype patterns were identified and much of the implementation was moved into a core Python module that provided much of the overall flow implementation while relying on callers of the module to provide a handful of concrete hook implementations to fulfill the particulars of any given collector.</p><p>What was a collector? A collector was simply a Serverless project created for a specific supported endpoint that provided a handful of Lambda functions that would invoke the core Python module&#39;s suite of functions who&#39;s arguments where instances of the collectors hook implementations that fulfilled the specific work particular to a collector. When support for a new collector was needed, a new Serverless project was scaffolded from a Serverless Project template made specifically for collectors that provided much of the boilerplate while letting collector developers to focus on the few required implementation details to actually pull, process and react to data fetched from the collectors endpoint (i.e. an 3rd party api or website etc)</p><p>Here is a high level view of the engine to provide some context for the paragraphs that will follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*A8mktcLK6z4mNsQl" /></figure><h3>Collector types</h3><p>There could be dozens of different collector implementations.. an ever growing number. The decision on what kinds of collector&#39;s to implement was driven by the organization and was very specific as we were not trying to implement a super generalized crawler in the traditional sense. For example, collectors were generally implemented for the following categories of endpoints that target data could be ingested from.</p><ul><li>public/private search engines</li><li>public/private 3rd party APIs</li><li>public/private social media APIs</li><li>specific public/private content/news outlets and asset repositories</li></ul><p>In addition to dozens of the above, there were a few special collectors developed for specific use cases.</p><ul><li><strong><em>generic raw HTTP</em></strong>: this implementation was specifically used for exploration and collecting data from linkages to yet-to-be developed endpoints. This collector could return content and utilized various libraries internally to parse raw response data and make its best attempts to identify relevant text, further linkages and remove any code. Once ingested, automation/analysis could be used to determine relevancy and whether or not certain linkages should be promoted to first class collectors for future monitoring.</li><li><strong>orchestrator</strong>: this implementation was used to “bootstrap” one or more other collectors to begin the crawl for given topic. The payload for the orchestrator would contain an instruction set of child payloads for other collectors that needed to be invoked. The orchestrator invocation would have a special collector id that would be used as the root lineage parent for all other child collectors spawned from it. Orchestrator invocations were run on a scheduled or on-demand basis in reaction to new topics/targets that were identified real-time or by analysts that were worthy of their own collector crawls.</li></ul><p>Given the bulk of the workflow logic was implemented in a separate Python module that each Serverless collector project could import; each collector implementation only had to fulfill about three (3) functions. Generally the first to fetch the data and publish an array of named datasets to a result consumer, the second to react to another payload and lastly one to convert results into more structure dataset item records. (more on this below)</p><p>Given a Serverless Framework Template was created and used to to instantiate each individual collector’s own Serverless project. All of the individual collector Serverless projects resided within a single monorepo along with the core module and project template itself.</p><h3>Collectors</h3><p>Each collector project was specific to a particular supported endpoint. The collector&#39;s responsibility when invoked was to make a request as instructed by the payload arguments, parse the response (paging if necessary), write the L1 result documents out to the data lake and publish a collector event. Secondarily each collector was responsible for reacting to collector events received on its own queue as well as well as converting L1 result documents to L2 dataset items. Conversion into L2TS items would occur via additional downstream processes.</p><h3>Collectors: collect()</h3><p>The collect() function was the primary entry point to instruct a collector to go and fetch the data it is instructed to do so via a payload that contains arguments that would vary per collector. Each collector&#39;s implementation was responsible for providing the implementation for a <strong>DataFetcher</strong> which would do the specific request logic particular to that collector&#39;s supported endpoint as configured by the payloads arguments. When collect() was invoked, the collector would construct an instance of the core module&#39;s <strong>Collector</strong> class and passing the following to it&#39;s execute() method:</p><ul><li><strong>AWSInvocation</strong>: bound to the inbound payload and abstracts away the details of invocations from various Lambda invocation methods and their differences (SQS, api-gateway http, lambda direct). Invocation objects yielded the relevant payload + header info in a consistant format without having to worry about the lambda request/context differences.</li><li><strong>DataFetcher</strong>: does the actual request logic and then write to a ResultConsumer</li><li><strong>ResultConsumer</strong>: would write an L1 result document to S3, secondly write metadata to a graph database and lastly publish a <strong>CollectorEvent</strong> to SNS</li></ul><p>The call to Collector.execute() would utilize the above arguments and orchestrate the overall flow as shown in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/0*uTirMpve74YtaYxM" /></figure><h3>Collectors: collect(): data sets</h3><p>Upon invocation of collect(), each collector&#39;s DataFetcher.do_fetch() implementation produces an array of one or more named datasets that is passes to the ResultConsumer; a dataset is simply an arbitrarily named object that contains a id, name, tags and metadata with key/values relevant to its type. Most importantly addition to these fields, each dataset contains the actual data that was fetched from some endpoint, and the structure of this varies based on the dataset&#39;s &quot;type&quot;. We settled on 3 different &quot;types&quot; based on the patterns of access that surfaced across the dozens of different endpoints.</p><ul><li><strong>SIMPLE</strong>: contains a data property that yields the actual data collected and a data_url property. For example this typically contained the raw API response body JSON &quot;as is&quot; or simply a large string of response content. This type was used when a single logical request yielded a single response. (non-paged)</li><li><strong>PAGED</strong>: contains a map of pages data to store paginated results from a single logical collector request. Each page object contained an id, metadata, the page number, and the data element holding the actual data. Each page object was located in the pages map keyed by its individual page id. The metadata field also contained keys to get all pages by ordinal position numbers. This type was used when a single logical request yielded some kind of paged response from the endpoint.</li><li><strong>KEY_VALUE</strong>: contains a map of key_values to store bucketed arbitrarily named result data split up however the collector needs to categorize it. Each key was the value&#39;s id. Each value was a special kv object containing an id, key name, metadata map, and the actual data containing the result data for that key. This type was used when a collector needed to arbitrarily categorize and break up a large/single collector response in some custom way.</li></ul><p>For all of these, there was a loose “shared knowledge” based contract between the collector’s do_fetch() implementation and the consumer of a given collector&#39;s results, i.e. the react() method was often implemented in the same collector, or some other consumer. The collector documentation would explain the contents of each collectors datasets produced so that consumers could then utilize it effectively and generally the same developer who was writing a particular collector was also implementing the other consumption methods as well as working with any other analysts/logic that would ultimately make use of the collector&#39;s output downstream.</p><h3>Collectors: react()</h3><p>The react() function was a reactive entry point to instruct a collector to go and fetch the persisted L1 result doc from the data lake, analyze it, and then take further action such as invoking additional collector.collect() invocations either on itself or other related collector types; these invocations were over SQS. Each collector&#39;s implementation was responsible for providing the implementation for a <strong>AggregateEventReactorPlugin</strong> which would do the specific logic particular to that collector &#39;s requirements for reacting to other collected results. For example, a reactor might analyze a result, pick out one or more particular identities and then use a secondary API to go fetch the details on that identity; or simply following all links.</p><p>One key point with the react() implementation was that without some sort of leash, things could get out of control depending on how a collector was coded, for example it might follow all links, creating more events, and on and on. The way this was controlled was via a reactor config passed in all collect payloads (and global settings) that controlled the max branch depth any reactor/collector could traverse. This was enforced at the core module level that managed the production of all event messages to SQS. This would ensure we could limit the size of a crawl within some estimation for any given target.</p><p>When react() was invoked, the collector would construct an instance of the core module&#39;s <strong>Reactor</strong> class and passing the following to it&#39;s execute() method:</p><ul><li><strong>AWSInvocation</strong>: see the collect() section above for details</li><li><strong>AggregateEventReactorPlugin</strong>: does the actual processing logic</li></ul><p>The call to Reactor.execute() would utilize the above arguments and orchestrate the overall flow as shown in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*IFlQ5WIYg8p6anKz" /></figure><h3>Collectors: convert()</h3><p>The convert() function was an additional reactive entry point to instruct a collector to go and fetch the persisted a result in the data lake and convert it to one or more dataset items, each as <a href="https://www.databricks.com/glossary/what-is-parquet">Parquet</a> and <a href="https://avro.apache.org/">Avro</a>.</p><p>Regarding datasets, recall from above that each JSON result document contains N number of datasets each potentially varying by type and each containing one or more sets of actual data. In order to efficiently get at each chunk of data embedded within the datasets, the conversion process would create a separate dataset_item for each data element within a dataset. So for a PAGED dataset with 10 pages, the conversion process would create 10 dataset_items each with full copies of the top level result document attributes and lineage information and additional attributes specific to their individual page. Likewise for a KEY_VALUE dataset with 2 KV pairs, would yield 2 dataset_items. For SIMPLE datasets one dataset_item would be created. Once this was done, now queries could be issued such as<em> &quot;get all dataset_items for search-result-a&quot;</em> which would yield 10 results, each representing the data from one page or<em> &quot;get all dataset_items for search-result-a where page_number = 7&quot;</em>. Once converted to dataset_items now all results were effectively queryable by all their attributes and now routable by downstream automation. (note that the L2TS data embedded inside was yet to be processed or accessible in a structured way).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*0na4V692slf2Cj1m" /></figure><p>The whole point of converting each JSON result document into both Parquet and Avro formats was two fold.</p><ul><li>First one of not being able to predict the exact future. The main priority was to get the data into a format that could be a bit more machine queryable, routable and readable as well as more easily usable by human analysts. By having fully expressed in both column and row oriented formats, this gave us more flexibility.</li><li>Second being that the JSON result docs were intended to be an immutable long term snapshot of the crawled data. In order to be able to properly discover and route it by high level attributes for further processing in an automated way, we needed to give it some more structure, flatten it and explode it into the individual dataset items. Think of dataset items as little cargo boxes with a manifest on them for routing. Inside the boxes was the actual data we cared about, but didn’t quite need to look inside the box just yet… downstream processes would be responsible for that.</li></ul><p>To support the convert() function, each collector&#39;s implementation was responsible for providing the implementation for a <strong>AggregateEventConverterPlugin</strong> which would do the specific conversion logic which was primarily implemented in a core module library. When convert() was invoked, the collector would construct an instance of the core module&#39;s <strong>Converter</strong> class and passing the following to it&#39;s execute() method:</p><p>The call to Converter.execute() would utilize the above arguments and orchestrate the overall flow as shown in the diagram below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*K3GHEUxnEgoQ_1R8" /></figure><h3>Collectors: orchestrator</h3><p>As mentioned in a previous section, the orchestrator was a special collector implementation was used to &quot;bootstrap&quot; one or more other collectors to begin the crawl for given topic. After a given crawl for a target the graph of results would look something like the below. By organizing target crawls under a single parent, the entire graph of meta-data, and JSON results in the data lake could now be treated as a single package or &quot;snapshot&quot; of an entire crawl that could be processed as a single unit over an over as need be and compared with other &quot;snapshots&quot; to detect drift over time for various data points. Target collections could might be run hourly, daily, weekly etc. depending on the size of the target base.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*eaoYy8Mn4Cc1V7z6" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/0*UiZLNXsHOkoJAmJF" /></figure><h3>Data lake</h3><p>collectors would ultimately need to save their results data JSON documents into a data lake. The storage for this was to be S3, but the &quot;folder&quot; structure needed to be scalable and compatible with big data tooling taking into account partitioning.</p><p>We ended up coming up w/ a few different storage path options under target buckets so that collector implementations themselves could dictate the hierarchy their output files would reside in. There were numerous options but ultimately most collectors ended up writing output into the following lake structure, where ROOT_DIR_NAME generally was a S3 bucket specific to a individual collector or a local directory when running in development w/ AWS service mocks.</p><pre>[ROOT_DIR_NAME]/year=[year]/<br>       month=[month][/day=[day]]/[/hour=[hour]]/<br>       clct_uuid=[clt_uuid]/<br>       clct_version=[version]/<br>       org=[org]/<br>       project=[project]/<br>       result_context_name=[resultContext]/<br>       result_name=[resultName]/[result_file_document].json</pre><p>Note that clct_uuid represented the top level root orchestrator collector id that would bootstrap an entire crawl for a given target/topic.</p><h3>Metadata database</h3><p>Although a large amount of meta-data, lineage information and audit data was recorded within each collector result, some out of band database was needed for the system to track meta-data about collector runs at a high level. This would serve useful for both admins, analysts, developers and downstream automation when processing a collector run. We ended up going with a graph database for this, <a href="https://neo4j.com/">Neo4j</a> in particular. The graph database was a key element to be able to visually and programmatically traverse the lineage graph for a large set of collector runs. From these high level relationships, more granular relationships that were embedded deep within the data within results where able to have more context.</p><h3>AWS Glue</h3><p>As noted previously, when a collector’s convert() method is invoked in response to a CollectorEvent, the full result document is fetched from the S3 data lake and each dataset embedded within it is converted into both Parquet &amp; Avro dataset_item and then written to the data lake.</p><p>All dataset_item &#39;s are written to a dedicated location in the lake partitioned as follows:</p><pre>[CONVERT_ROOT_DIR]/[parquet|avro]/  <br>  compression=[compression type]/  <br>  dataset_items/<br>  p_schema_version=[version]/<br>  p_environment=[env]/<br>  p_producer_name=[collector name]/<br>  p_year=[yyyy]/<br>  p_month=[mm]/<br>  p_day=[dd]/<br>  p_hour=[hh]/collector_[env]_dataset_items.[parquet|avro]</pre><p>Structuring it this way permits an AWS Glue catalog to be created for the target bucket and then subsequently crawled after any new collection execution or on an otherwise scheduled basis. Once the data is catalogued it can then be queried using Athena or Spark jobs against via the partitions and any top level dataset item attributes. It can exported in part as well as other higher level automation run against it.</p><h3>High level view</h3><p>At this point you should have a pretty good feel for how the data collection side of this system worked. After each collector executed for a given target, we had a full set of JSON result documents, queryable graph of related metadata about the crawl, a far more structured/queryable version of the data contained in all the individual dataset_items, and finally all of this was catalogued in AWS Glue&#39;s catalog. Now things are bit more friendly to process and sort through at a high level vs going directly to millions of big JSON files.</p><p>Why even have the big JSON result documents to begin with? Well by storing in this format, we ensure we have a permanent snapshot for a point in time of an entire target’s “graph of existence” within the scope of the crawled endpoints. Given JSON is fairly ubiquitous, if at some point in the future we need to re-process the data or change the format of dataset_items we can just go back to the JSON and re-process from there.</p><p>Great, wow that was a lot of work… but we haven’t even started to look at the meaningful data contained within each one of the dataset_items yet! To get to that point we have more challenges on our hands, which you can read about in part 2 of this series which covers the <strong>Layer 2 Type Specific Items (L2TS)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*affgOFpPCaQfa1wx" /></figure><h3>To be continued</h3><p>The next big part of this architecture was actually digging into the data contained within each dataset_item. For more. on this please continue on to part 2 of this series. Thanks for reading.</p><p><em>Originally published at </em><a href="https://bitsofinfo.wordpress.com/2022/01/23/non-deterministic-mass-data-collection-dynamic-data-lake-schemas-aws-serverless-glue-spark/"><em>http://bitsofinfo.wordpress.com</em></a><em> on January 23, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c6567022da5" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/architecture-for-non-deterministic-mass-data-collection-part-1-collection-engine-8c6567022da5">Architecture for non-deterministic mass data collection: part 1: collection engine</a> was originally published in <a href="https://levelup.gitconnected.com">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>