<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Veronika Durgin on Medium]]></title>
        <description><![CDATA[Stories by Veronika Durgin on Medium]]></description>
        <link>https://medium.com/@durginv?source=rss-3ed77638d9c0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PcsQllWF_rNm_gzQBxN6Ag.jpeg</url>
            <title>Stories by Veronika Durgin on Medium</title>
            <link>https://medium.com/@durginv?source=rss-3ed77638d9c0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:41:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@durginv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Data Testing]]></title>
            <link>https://medium.com/@durginv/data-testing-37f85518ba99?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/37f85518ba99</guid>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Sat, 06 Sep 2025 12:07:19 GMT</pubDate>
            <atom:updated>2025-09-06T12:08:54.207Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*uq24QPESBHXR2tz-QMjc6g.jpeg" /><figcaption>meme generator still delivers</figcaption></figure><p>In a typical organization, the software development life cycle (SDLC) includes multiple lower environments: dev, test, and stage/pre-prod. These contained, end-to-end environments work really well for software delivery.</p><p>Data teams operate differently. We don’t just deliver code, we deliver <strong>data</strong>. That means our testing requirements go beyond whether code compiles or pipelines run. We also have to validate the quality, accuracy, and performance of the data itself.</p><p>And this is where I’ve often had to pause and explain. The SDLC model with lower environments feels complete to many, but for data teams it isn’t. Most of what really matters, like performance, accuracy, cost, and trust, can only be validated with production-like data. By production-like data, I mean a recent copy of production that mirrors its scale and messiness without exposing sensitive information.</p><p>That said, there are two broad categories of testing:</p><ul><li><strong>Early development checks</strong>: code compiles, runs, and referential integrity is intact. These can be done without production data if that’s all you have.</li><li><strong>Production-like validation</strong>: the real test, where we see if pipelines behave under scale and deliver accurate, trustworthy outputs.</li></ul><blockquote>Testing with production-like data is <strong>NOT</strong> the same as testing in production.</blockquote><p>The first is a DataOps best practice. The second is for teams that like living on the edge.</p><h3>What We Can Test in Lower Environments</h3><p>Lower environments are not strictly required, but they can be useful when production-like data is not available. They give us a safe place to run early checks:</p><ul><li><strong>Code Validation</strong>: making sure the code compiles, deploys, and runs as expected.</li><li><strong>Data Integrity Checks</strong>: validating data types, primary/foreign key constraints, NULLs, and data grain.</li><li><strong>Referential Integrity</strong>: ensuring relationships between tables are intact.</li><li><strong>End-to-End Testing</strong>: verifying that data inserted into a source system flows correctly through to analytics.</li></ul><p>These steps are helpful for catching structural issues and making sure pipelines are deployable. Lower environments are also a good place to validate lineage and schema changes. You don’t just want the pipeline to run, you want to confirm that data can be traced end to end and that downstream jobs will not break if a column is added, renamed, or dropped.</p><h3>What Requires Production-Like Data</h3><p>I have seen this more times than I want to admit. A pipeline looks perfect in dev, runs in minutes, no errors, everything checks out. Then it’s pointed at production-scale data and suddenly it takes hours and the compute bill spikes. Nothing is “wrong” with the code, but the scale exposed problems that dev could never show.</p><p>Even back when I was working with transactional databases, I always stress tested SQL before rolling it to production. A query that looked harmless in dev could lock tables or crawl under real workloads. Data pipelines are no different. The only way to know if they hold up to SLAs is to test them against production-like scale. (More about <a href="https://medium.com/@durginv/data-slas-c04e63460b87">Data SLAs</a>)</p><p>This is why lower environments alone aren’t enough. To validate performance, accuracy, and cost, you need production-like data:</p><ul><li><strong>Performance and Scale</strong>: Queries that succeed on a test dataset may become prohibitively expensive or slow at production scale.</li><li><strong>Accuracy and Trust</strong>: Dev datasets are often too clean. Pipelines may “work,” but once exposed to production-like messiness of late-arriving records, unexpected nulls, odd edge cases, the answers stop matching business reality.</li><li><strong>Cost Management</strong>: True resource consumption can only be evaluated with realistic data volumes.</li><li><strong>Data Quality SLAs</strong>: Completeness, timeliness, and freshness checks depend on messy, real-world data. Synthetic or toy datasets won’t surface late-arriving records or outliers.</li><li><strong>Freshness and Staleness in Streaming</strong>: In batch, a delay might just mean your dashboard is a day late. In streaming, freshness is part of the contract. Out-of-order events or broken watermarks don’t show up in clean test datasets.</li><li><strong>Backward Compatibility</strong>: Schema evolution testing is not optional. Queries that worked yesterday should either still work today or fail loudly and predictably.</li><li><strong>Data Drift</strong>: Pipelines can be technically “working” while the distributions of the data shift under the hood. Without production-like data, you won’t notice when a partner system starts reporting currencies in a new format, or when intake forms quietly add new codes that downstream jobs don’t expect.</li></ul><h3>The Infrastructure Reality</h3><p>“Automated production snapshots” sounds simple, but it isn’t. Moving production data into lower environments raises compliance and PII challenges. You often need masking, anonymization, or subsetting to make snapshots safe and usable. Full-scale prod replicas can be expensive too, and not every team can afford to keep them around.</p><p>If your platform supports zero-copy cloning, that can be a game changer. It lets you spin up production-like datasets with minimal cost and storage overhead, while still applying masking or anonymization policies as needed. This avoids keeping multiple heavy replicas around and makes it easier to give teams realistic test environments.</p><p>Some teams also lean on synthetic data generation. It’s never perfect. Synthetic data is usually too clean to catch messy edge cases, but it can be a good middle ground when compliance or cost prevents you from using actual production data.</p><p>In practice, a strong DataOps strategy blends approaches:</p><ul><li>Automated snapshots or clones where feasible, with masking, anonymization, and retention policies as needed.</li><li>Zero-copy clones when the platform allows it, to reduce cost and overhead.</li><li>Representative subsets or ephemeral environments to control costs.</li><li>Synthetic data to supplement production copies when compliance requires it.</li></ul><h3>Other Tests Worth Calling Out</h3><p>There are a couple other areas that don’t always get attention but deserve it:</p><ul><li><strong>Retention and Archival Testing</strong>: verifying that data retention policies are enforced correctly. Data that should expire is dropped on schedule, sensitive data is archived or anonymized properly, and nothing critical is deleted or lost in the process.</li><li><strong>Model Validation</strong>: for teams supporting data science, testing doesn’t stop at the pipeline. You also need to validate models against drift, stale training data, and changing feature distributions.</li></ul><h3>Because Debugging in Production Isn’t Everyone’s Adventure</h3><p>Lower environments can be useful for early checks, but they are never sufficient. Reliable data pipelines require production-like validation.</p><p>Without it, costs spiral, data stops tying out, and confidence in data disappears. If you want analytics the business can rely on, you have to test against reality.</p><p>If you don’t have a strategy for production-like testing, you’re not really testing. Testing in prod is only fun if you like your job to feel like an extreme sport.</p><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37f85518ba99" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self Recovering Data Pipelines]]></title>
            <link>https://medium.com/@durginv/self-recovering-data-pipelines-c1e4e6b7fbce?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1e4e6b7fbce</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Sun, 17 Aug 2025 13:01:39 GMT</pubDate>
            <atom:updated>2025-08-17T13:01:39.379Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*S-MaBEnwGIZFxd5_jHJVJg.jpeg" /><figcaption>good old meme generator</figcaption></figure><p>It’s 2:07 a.m. Your phone buzzes. Pipeline down. Again.<br>One task choked on a bad record, and everything downstream just… stopped. Now you’re squinting at logs, trying to remember which restart sequence won’t accidentally wipe the data you already processed.</p><p>Welcome to traditional <strong>ETL </strong>(Extract Transform Load) life: sequential jobs, hard dependencies, and always one malformed record away from total system failure. I’ve been there — whole teams dedicated to babysitting a handful of critical pipelines. Those <em>weren’t</em> the good old days.</p><p>Back then, you built for the nightly batch. If it failed, you had a full day to fix it before the next run. Now streaming data, near-real-time dashboards, and stakeholders who want everything yesterday are just the norm. That “we’ll fix it tomorrow” buffer is gone. Along with your sleep schedule.</p><h4>ELT Changed the Game… Sort Of</h4><p>I’m team <strong>ELT</strong> (Extract Load Transform). It broke the dependency chain by decoupling extraction (E) from transformation (T). Ingestion runs asynchronously, transformations happen in parallel, throughput goes up.</p><p>But flexibility alone doesn’t equal self-healing. Recovery requires intentional architecture. If you want pipelines that fail gracefully without waking you up, bake these in from day one:</p><h4>1. New Data Check / High Water Mark</h4><p>Pipelines break, it happens. The key is knowing where to restart. A High Water Mark (HWM) is your “last safe checkpoint,” telling you the most recent successfully processed data so you can pick up right where you left off instead of starting from scratch.</p><p>Pair it with a new-data check so you only process when something new has actually arrived. No new records? Pipeline sleeps instead of reprocessing the same old rows like a hamster on a wheel.</p><p>High water marks used to be table stakes because storage was expensive and reprocessing hurt. Then storage got cheap, compute got faster, and people stopped bothering… until a “quick” backfill took 12 hours instead of 15 minutes. Cloud bills will remind you why you cared in the first place. And no, CTAS (create table as) isn’t your friend here.</p><h4>2. Re-route Bad Records</h4><p>Bad data is inevitable. But one bad row shouldn’t hold an entire dataset hostage. Send troublemakers to an error queue to be dealt with later while the good data keeps moving.</p><p>I’ve seen one malformed timestamp in a billion-row file freeze an entire pipeline. That’s like shutting down a highway because one car ran out of gas.</p><h4>3. Retry Mechanisms</h4><p>Not every failure is catastrophic. Sometimes it’s just a network hiccup, a database lock, or a file that showed up late. That’s where automatic retries are useful. A solid retry strategy reprocesses failed tasks without human intervention, keeping pipelines reliable even when external systems misbehave.</p><p>But retries have to be controlled. Blindly hammering the same request over and over can turn a small glitch into a bigger outage. Set sensible limits so retries stop after a point, and make sure failures are logged clearly. That way, you don’t wake up to a mystery, just a well-documented issue waiting to be addressed.</p><h4>4. Alerting and Monitoring</h4><p>“Job failed” doesn’t tell me much. I need to know <em>what</em> broke, <em>where</em> it broke, and <em>why</em> so I can fix it without playing detective at 2 a.m.</p><p>Good alerting should provide context and enough details. A message that points directly to the failing component saves hours of digging and prevents frustration for whoever is on call.</p><p>It’s just as important to avoid noise. If every minor blip generates an alert, people stop paying attention. Alert fatigue is real, and it’s how serious issues slip through unnoticed.</p><p>The goal of alerting and monitoring is to provide a clear signal when something is going wrong. Real-time systems should catch failures and performance issues early enough that you can respond quickly, reduce downtime, and limit the impact on the business.</p><h4>5. Data Validation</h4><p>Validate data at every stage: ingestion, transformation, and before final delivery. The earlier you catch errors, the less costly and disruptive they are to fix. I’ll go deeper into Data Testing in another blog, but the principle is simple: prevention beats cleanup.</p><p>The goal is to ensure that only clean, reliable data moves forward. By building validation into multiple stages of the pipeline, you reduce rework, minimize the risk of incorrect results, and strengthen trust in the data your stakeholders depend on.</p><h4><strong>Why This Matters</strong></h4><p>Self-recovering pipelines mean fewer emergencies and more time spent on work that actually moves the business forward. <a href="https://medium.com/@durginv/the-forgotten-bucket-of-data-work-2ab6bb89458e">https://medium.com/@durginv/the-forgotten-bucket-of-data-work-2ab6bb89458e</a></p><p>In data engineering, things will break. That’s a given. The difference is whether you’re stuck fixing them at the most inconvenient times or the system handles it for you.</p><p>These aren’t shiny new ideas. They’re the basics. And like most basics, you don’t notice them until they’re missing… Usually at the worst possible time.</p><p>Build resilience in early. Your future self will be glad you did. So will your team, your on-call rotation, and anyone who’s had to explain to leadership why a report is late because of “technical difficulties.”</p><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1e4e6b7fbce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dates: Handle with Care]]></title>
            <link>https://medium.com/@durginv/dates-handle-with-care-e1f19ab0b40b?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1f19ab0b40b</guid>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 12:41:59 GMT</pubDate>
            <atom:updated>2025-07-29T12:41:59.114Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*44FbX1M8hngNgC71R0P__w.png" /><figcaption><a href="https://xkcd.com/2867/">https://xkcd.com/2867/</a></figcaption></figure><blockquote>Hey Veronika, why don’t the numbers from system A match what we’re seeing in the data platform?</blockquote><p>Ugh, the dreaded question. And like always, it came out of nowhere.</p><p>The KPI in question was very important and widely used. But no alerts had gone off. No one raised a red flag. Very odd. Hmm. Is this one of <em>those</em> situations where the data’s fine but someone used it in the wrong context? Maybe?</p><p>I spent an hour validating filters and metrics. Everything matched and aligned. Another hour digging deeper. Still nothing. Why don’t the numbers match?</p><p>And then I saw it!!!!! A tiny line in the corner of the system-generated report:</p><p><em>Generated on XX/XX XX:XX</em><strong><em> PST</em></strong></p><blockquote>WHAT THE H.E. double 🏒 ?!?!?!?</blockquote><p>All our internal reports run in <strong>Eastern Time</strong>. Of course the numbers didn’t match. I’ve been breaking my brain. But the real culprit? Time zones being their usual sneaky selves.</p><h3>You Haven’t Worked in Data If You Don’t Have a Date Story</h3><p>This is one of those things <em>everyone</em> knows about and yet, somehow, we keep forgetting. Until things break. Until the numbers don’t match. Until someone panics and raises a red flag.</p><p>Dates and times are deceptively simple. But they’re not. There are many formats, precisions, and time zones. And every one of them matters.</p><p>Instead of deep-diving into every standard and rule, here’s a practical list of <strong>DOs and DON’Ts</strong> that will save you hours of debugging and head-scratching.</p><h4>✅ DOs:</h4><h4>- Use the correct precision (e.g., DATE vs. TIMESTAMP)</h4><p>Sometimes, just storing a date is enough. <em>DATE</em> data type takes up less space on some platforms and could be perfectly fine for some use cases. Other times, pun intended, you need higher precision: seconds, milliseconds, or even nanoseconds. If you pick the wrong type, you risk losing information or confusing your downstream users. Precision should always match the business need and the grain of data.</p><h4>- Store in UTC</h4><p>UTC (<a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">Coordinated Universal Time</a>) is the most stable andconsistent way to store time. It avoids all the messiness of daylight saving time and local timezone shifts. That said, just storing in UTC isn’t enough if the original timezone had business meaning. When you convert to UTC, make sure to preserve the original offset if you’ll ever need to reconstruct or display the original time.</p><h4>- Match timezones when joining</h4><p>This one catches everyone eventually. Two datasets look like they should line up… but they don’t. You spend hours digging into filters and business logic only to realize: one table was in Eastern Time, the other in Pacific. When joining tables, check the timezone first. This is one of those sneaky, hard-to-spot issues that can completely invalidate your results without throwing any errors.</p><h4>- Display in the business’s local timezone</h4><p>Just because UTC is great for storage doesn’t mean it’s great for end users. Most people don’t live in “+0000” or read military time fluently. They want to see when something happened in their world, not do timezone math every time they look at data. If your company or customer base mostly lives in one time zone, convert to that business timezone before displaying timestamps. It reduces friction, confusion, and misinterpretation.</p><h4>- Use clear naming conventions like “created_dt_utc”</h4><p>Naming conventions are a way to self document. A column called <em>created_dt_utc</em> tells you everything you need to know: it’s a datetime, and it’s stored in UTC. No guesswork. No diving into source code to figure it out. Meanwhile, something like <em>created_date</em> is vague. Is it a full timestamp? Just a date? In local time? The more explicit you are in naming, the less ambiguity there is across teams. No-one needs to assume anything.</p><h3>❌ DON’Ts:</h3><h4>- Store ambiguous values like EST or EDT</h4><p>This is one of the biggest offenders. And it keeps coming back. Time zone labels like EST and PST aren’t fixed; they shift with daylight saving time. What’s EST in January becomes EDT in July. If you store just “EST,” your data won’t reflect that shift. It’ll look right at first glance… until someone notices hourly metrics are off by one hour for half the year. Always use timezone-aware formats like <em>America/New_York</em> or include exact UTC offsets (e.g., -0500).</p><h4>- Assume users can convert UTC mentally</h4><p>Your pipeline speaks UTC. Your logs speak UTC. But your business users don’t. If you hand them a dashboard with raw <em>2025–07–28T14:00:00Z</em> timestamps, they’ll either misread it or ask you (again) if that’s before or after the flight departure. Don’t make them translate time in their heads. It leads to confusion and bad decisions made on misunderstood data.</p><h4>- Truncate time without checking grain</h4><p>It’s tempting to cast <em>TIMESTAMP</em> to <em>DATE</em> to simplify things. But if the data is event-based or transactional, that little truncation can introduce major errors like undercounting events, misaligning hourly trends, or breaking window functions. Always check the intended grain of the data before you truncate or round off time. The wrong transformation can silently change the meaning of your metrics.</p><h4>- Ignore Daylight Saving Time shifts</h4><p>Daylight saving time adds a layer of chaos twice a year. If your business operates in a region that observes it and your systems don’t adjust, you’ll see weird one-hour gaps or overlaps in your data during those spring/fall transitions. That messes up anything time-sensitive: scheduling, forecasting, even basic aggregations. Be DST-aware in your logic and reporting. It’s a pain. But ignoring it only makes things worse.</p><h4>- Use unclear or inconsistent names like “created_date”</h4><p>We already touched on this, but it bears repeating: vague names lead to bad assumptions. A column named <em>created_date</em> could be a date, a timestamp, UTC, local time. There’s no way to tell without digging into the code or lineage. You can avoid so much confusion by just having a precise naming convention. Pick one and stick to it. And favor verbosity over cleverness. Your future teammates (and your future self) will thank you.</p><h3>A Little Parting Advice from Someone Who’s Been Burned by Time</h3><p>… and loves shoes 👠</p><p>Timezone handling is like walking in heels on cobblestones.<br>Yeah, you <em>can</em> do it. But if you’re not paying attention, you’re gonna twist something. Probably your business logic, your sanity, or both.</p><p>Dates and times seem simple, but they’re anything but. You haven’t really worked in data until you’ve spent hours chasing down a number mismatch, only to discover your systems were speaking in different timezones all along.</p><p>So do yourself (and your future self) a favor:</p><ul><li>Store in UTC.</li><li>Use names that say exactly what you mean.</li><li>Convert for your users.</li><li>Don’t forget about daylight saving… ever!!!</li></ul><p>Treat time like tricky terrain: don’t walk in blindly.</p><blockquote>Choose your footwear, and your time format, wisely.</blockquote><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1f19ab0b40b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Business Seasonality: Why It Should Be on Your Data Team’s Radar]]></title>
            <link>https://medium.com/@durginv/business-seasonality-why-it-should-be-on-your-data-teams-radar-852d000accdc?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/852d000accdc</guid>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 12:03:20 GMT</pubDate>
            <atom:updated>2025-07-09T12:03:20.838Z</atom:updated>
            <content:encoded><![CDATA[<p>From the 10 Most Neglected Data Engineering Tasks <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">series</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f47CZbxqna8cruHLxUU-wQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@foodjetsexpress?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">logan jeffrey</a> on <a href="https://unsplash.com/photos/cooked-rice-with-green-vegetable-on-white-ceramic-plate-7UYmkS315JU?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>There’s a tradition in the U.S. of ordering Chinese food on New Year’s Eve. It’s one of the busiest nights of the year for Chinese restaurants. My local spot closes the dining room, switches to a preset menu, and focuses completely on volume and speed. Why? Because they know it’s going to be chaos, and they plan for it.</p><p>Many businesses have their own version of New Year’s Eve.</p><p>In agriculture, planting and harvesting are short and very critical windows. If the systems handling schedules, weather data, or equipment go down, it’s a huge problem. You can’t just “try again in a few weeks.”</p><p>Retail and e-commerce hit peak load from Thanksgiving through the winter holidays. One broken data pipeline during that time can mess up inventory, personalization, or fulfillment. And nobody wants to explain why a package didn’t make it under the tree, so now the customer thinks Santa put them on the naughty list.</p><p>In accounting, tax season is non-stop and running on a tight timeline. A reporting delay there isn’t just annoying. It can cause real damage.</p><p>Not all peak seasons are tied to the calendar or easy to spot. Some sneak up on us. A product launch, a marketing campaign, an earnings call, a fashion drop, back-to-school, a critical demo. Anything that brings a lot of attention or traffic can become a big deal for us on the data side.</p><p>These are the times when everything needs to work. Staying ready means knowing what matters to the business and when. Whether it’s a holiday rush, a campaign kickoff, or a major event, being in sync with business timelines helps us avoid surprises. For data engineers, that means planning ahead and being ready, no last-minute scrambling.</p><h3>Code Freezes Still Catch Data Teams By Surprise</h3><p>In the past, the data side of the house wasn’t on the same release cycle as software teams. We weren’t on the critical path, so we weren’t treated as such. That’s changing, but the habits haven’t caught up yet.</p><p>Code freezes happen pretty regularly, but they still manage to sneak up on data teams.</p><p>One reason is that data work flows differently than software development. We’re not shipping tidy features on a set schedule. We’re constantly migrating dashboards, cleaning up schemas, adding and modifying tables, tuning models. So when someone suddenly says, “We’re freezing production,” it feels disruptive. Because it is.</p><p>That’s exactly why we need to plan ahead. The data function is now following DataOps practices and is part of the critical production path. We can’t afford to wait or react last minute anymore.</p><h3>Code Freeze Periods</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*Y0rJOtL0mk7URGJ1vg4Mww.jpeg" /></figure><p>Code freezes are meant to ensure stability during times when business cannot afford downtimes. That makes sense. But it does not mean that development stops. It just shifts.</p><p>A few things to check:</p><ul><li>Can your team keep working in dev or test environments? (More on Data Testing in a <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">future blog</a>)</li><li>Do your source control branching strategies and CI/CD pipelines support that?</li><li>Do you have a plan to roll out all pending changes post-freeze cleanly and safely?</li></ul><p>Code freezes reduce risk during critical business windows. But if we’re not careful, all the changes held back during the freeze will pile up and go out at once. Bulk deployments like that are messy, and that’s when things tend to break. That’s why we need a plan.</p><h3>Emergency Fixes Need a Clear Path</h3><p>No matter how well we plan for critical business seasons, something will break. When it does, we don’t want to scramble.</p><p>Before the busy season starts, we need to have:</p><ul><li>A clear escalation path</li><li>Defined ownership and on-call schedules</li><li>A lightweight way to patch things quickly without five approvals, three meetings… and a goat sacrifice (I promise that no goats were hurt during writing of this blog)</li></ul><p>The simpler we make this ahead of time, the faster we can respond when the pressure’s on.</p><h3>Expect a Spike in Load</h3><p>During peak season, data volume can double, or even more.</p><p>We need to ask ourselves:</p><ul><li>Can our systems scale up to handle that load?</li><li>Do we have enough monitoring to catch slowdowns before they turn into outages?</li><li>Are our queries and jobs tuned to perform well under pressure?</li></ul><p>If we’ve only tested for average load, we’re probably not ready for peak load.</p><p>Historical data can help here. What was the volume increase last year? What broke? What alerts went off? Use that info to get ready this time around.</p><p>And don’t forget: once the peak is over, our systems should be able to scale back down too. No need to keep everything cranked up if demand is back to normal.</p><h3>Strict SLAs</h3><p>Code may be frozen, but SLAs aren’t. Data pipelines still have to deliver on time. Prioritize like your job depends on it. Make sure a P1 is really a P1 and treat it that way. (More on Data SLAs <a href="https://medium.com/@durginv/data-slas-c04e63460b87">here</a>)</p><p>Dashboards need fresh data. Models need updated inputs. Alerts need to fire when something breaks.</p><p>Sharp prioritization makes all the difference because when everything feels urgent, nothing gets fixed fast enough. Everyone has to be on the same page about what’s truly critical. A P1 should mean “drop everything and fix this,” not “someone just yelled loudest.”</p><h3>Stay in Sync With the Business</h3><p>If you’re expecting a freeze, make sure stakeholders are notified early. They need to be clear on what deliverables might slow down and what’s still on track.</p><p>This goes beyond just project management. It’s trust-building. A quick heads-up and clear expectations go a long way when things get stressful.</p><h3>Checklist for Data Teams</h3><p>☑ Know your business’s critical seasons</p><p>☑ Align on freeze dates early</p><p>☑ Make sure dev and test environments are solid</p><p>☑ Document how to escalate and patch quickly</p><p>☑ Load test before things get busy</p><p>☑ Review on-call coverage and SLAs</p><p>☑ Loop in stakeholders early</p><p>Every industry has its make-or-break moments. Plan ahead so your data systems can keep up without breaking under pressure. Build with intention. When the peak hits, you’ll be ready and maybe even calm enough to enjoy it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=852d000accdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data SLAs]]></title>
            <link>https://medium.com/@durginv/data-slas-c04e63460b87?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c04e63460b87</guid>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Fri, 27 Jun 2025 16:45:40 GMT</pubDate>
            <atom:updated>2025-06-27T21:20:34.711Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQ75vq-yt7AqlQz7s6pFwQ.png" /><figcaption>Yes, I made this. No, I will not be taking design feedback at this time.</figcaption></figure><h4><strong>Data SLAs</strong></h4><p>If you’ve ever gotten paged at 3 a.m. over a missing table that no one documented properly, this one’s for you!</p><p>There are plenty of articles about data SLAs. But this one is for the data engineers who keep getting burned because they forgot to write one or never had one in the first place.</p><p>This blog is part of the <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054"><em>10 Most Neglected Data Engineering Tasks</em></a> series. I’ll keep it simple and focus on two SLA dimensions data engineers tend to overlook: data <em>freshness</em> and data <em>completeness</em>.</p><h4>Why SLAs Matter</h4><p>I was a DBA for a big chunk of my career. My specialty was performance tuning and optimization. Basically, I made sure systems didn’t catch on fire under pressure. That role gave me a deep appreciation for precision, uptime, and accountability. When a database slows to a crawl or goes down, there’s no ignoring it in hopes that it will magically fix itself. The SLA is the rulebook. And if you miss it, people notice really fast.</p><p>That mindset stuck with me. As data engineers, we don’t always think of our work in terms of SLAs, but we should. They’re not just some process checklist. They’re how we stay sane.</p><p>SLAs (Service Level Agreements) set clear expectations for when data should arrive and how complete it needs to be. Defining them upfront helps manage costs, prioritize fixes, and avoid 3 a.m. firefighting.</p><p>SLAs also help cut the noise. Without them, we end up optimizing for everything and solving the wrong problems. One team wants real-time, another is fine with daily, and suddenly every pipeline feels like the most important thing in the world. A good SLA helps us pick our battles and not spread out too thin. It’s the difference between focused urgency and pure chaos.</p><h3>Data Freshness</h3><p>How late is too late? Five minutes? An hour? Next morning by 9 AM?</p><p>Not every data pipeline needs to run in real time. Trying to do that everywhere wastes time and money. But we need to define what “fresh enough” means for each data artifact.</p><p>For example, maybe your sales dashboard can show yesterday’s totals first thing in the morning but must update every 30 minutes during business hours. If the pipeline fails at 2 a.m., a delayed update at 9 a.m. might be fine with no emergency page. But if it fails at noon, that’s a showstopper.</p><p>Also, if a job fails, what do users see? Stale data? An error? No data at all? Ask your data consumers, document their expectations, and share them widely. Clear expectations reduce panic and unnecessary alerts.</p><h3>Data Completeness</h3><p>Bad records happen. The question is whether they should stop the entire pipeline or be logged for later async troubleshooting and skipped.</p><p>If a vendor sends a bad file, do you:</p><ul><li>Stop the pipeline?</li><li>Skip the bad rows and alert someone?</li><li>Process what you can and move on?</li></ul><p>There’s no single right answer. Talk to your users, make a call, and document it.</p><p>We don’t just monitor for missing data either. A sudden spike in records can be just as problematic. For example, if your vendor’s daily file suddenly doubles in size, it could mean duplicated or corrupted data. Catching these issues is just as important as missing data.</p><p>Some teams send bad records to an error log but keep processing good data. Others stop the pipeline and fix the source before continuing. Both approaches are fine as long as users’ expectations are clear and communicated.</p><h4>Late-Arriving Records</h4><p>Beyond completeness, timing can also cause issues especially when records arrive late but still need to be accounted for.</p><p>If you work with queues or event data, late-arriving records are normal. The key question is whether users expect continuous updates (as-is) or consistent, point-in-time snapshots (as-was).</p><p>Neither option is better. Just pick one and be clear. Nobody likes surprises.</p><h4>Referential Integrity</h4><p>While timeliness is one part of the picture, consistency across tables is the other.</p><p>Modern data pipelines often load data in parallel. That means child records can arrive before parent records. If referential integrity matters, make sure your joins handle this properly. Use inner joins or other methods to avoid orphaned records.</p><p>And don’t just check one table and assume it’s okay. Always validate relationships across tables to keep data clean.</p><h3>Make SLAs Easy to Find</h3><p>Write your SLAs down and put them somewhere easy to find. If people don’t know what normal looks like, every issue feels like an emergency. It’s like opening a container expecting surprise brownies and realizing someone already ate them all. No heads up, just disappointment. (I❤️🍫)</p><h4>Best Practices</h4><p>Here’s a quick summary you can use as a checklist:</p><p><strong>Define and Align</strong></p><ul><li>Set SLAs for freshness and completeness. Be conservative for high-impact data since missed SLAs can hurt the business.</li><li>Document vendor expectations. If your data depends on others, track their SLA performance and set clear internal standards.</li></ul><p><strong>Make It Real</strong></p><ul><li>Don’t set SLAs you can’t meet.</li><li>Monitor by SLA, not just job status. A job can succeed but still miss an SLA, or fail without breaking an SLA. Monitor accordingly.</li></ul><p><strong>Make It Known</strong></p><ul><li>Put SLAs front and center. If SLAs live buried in old tickets, they don’t help anyone. Put them front and center.</li><li>Communicate clearly when you miss one and how you’re fixing it. When you miss an SLA, share what happened, why, and your fix plan. This builds trust.</li></ul><h3>Final Thought</h3><p>SLAs are meant to reduce noise, set expectations, and prevent chaos when something breaks.</p><p>Most importantly, SLAs help us avoid getting blindsided. If we don’t define normal, we can’t tell when something’s wrong.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c04e63460b87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Definition of Done]]></title>
            <link>https://medium.com/@durginv/definition-of-done-eea52c472cc3?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/eea52c472cc3</guid>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 13:07:30 GMT</pubDate>
            <atom:updated>2025-06-17T13:07:30.791Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*Lgz0g_Imt-vaZ-AyGT48XA.png" /><figcaption><strong>Mission Impossible 6 — Fallout</strong></figcaption></figure><blockquote>“It’s done. Dev is complete.”</blockquote><p>I’ve lost count of how many times I’ve heard that and had to follow up with, “Done for who?”<br>If the answer is “the person who wrote the code,” I hate to break it to you, but it’s <strong>NOT</strong> done.</p><p>Here’s the reality: dev complete is a milestone, not the finish line. If no one’s actually using it or it’s breaking something downstream, it’s not done. It’s nothing more than work in progress (WIP) left out in the wild.</p><p>This is one of those quietly critical things data engineers tend to forget, buried under the pressure to move fast. So let’s talk about it.</p><p>I really like this definition from <a href="https://www.leadingagile.com/2017/02/definition-of-done/">LeadingAgile</a></p><blockquote>“The definition of done is when all conditions or acceptance criteria are met and ready to be accepted by a user, customer, team, or system.”</blockquote><p>What I appreciate about it is the emphasis on <strong>someone</strong> or <strong>something</strong> needing what we’ve delivered. It’s a good reminder that we don’t build just for the sake of building. There’s always a consumer, whether human or machine, on the other side.</p><h3>What Goes Into a Real Definition of Done</h3><p>Every ticket, task, or story needs explicit <strong>success criteria </strong>written down. Not a vague “build the thing,” but a clear understanding of what “done” looks like and who it’s for.</p><p>In data work, this matters even more. We can build a perfect pipeline that no one knows about, or create a dashboard that looks great but tells the wrong story.</p><p>Here’s what I suggest everyone write down as part of their definition of done:</p><h4>1. Complete Development Based on Requirements</h4><p>Spell this out in your success criteria. It might involve:</p><ul><li>Ingesting a new data set</li><li>Modeling and Integrating new data into a data warehouse</li><li>Building a report</li><li>Running analysis</li><li>Deploying an ML model</li></ul><p>But dev complete does not equal done.</p><blockquote>Dev Complete <strong>≠ </strong>Done</blockquote><p>What we build has to align with what was asked for and be usable by someone else. Write this acceptance clearly in the criteria.</p><h4>2. Validate the Data</h4><p>This one is especially critical for data folks: validating that your work is <strong>logically correct</strong>, not just functionally complete. This is <strong><em>not</em></strong> the same as user acceptance testing.</p><p>Ask yourself:</p><ul><li>Do the transformations match the business rules?</li><li>Are there any inconsistencies, duplicates, or null values that need to be handled or flagged?</li><li>Are edge cases documented?</li><li>Did anything unexpected happen in the process?</li></ul><p>It’s easy to assume our logic is solid when the query runs. But just because it runs doesn’t mean the output is right. Otherwise it’s just garbage in, garbage out, plain and simple.</p><p>And don’t go solo. A PR review, arch review, or even a quick sanity check from a teammate can catch blind spots. Fresh eyes help us see what we missed and confirm we’re on the right track.</p><h4>3. Understand Downstream Impact</h4><p>Include in our success criteria who’s consuming the data and how the changes will be communicated.</p><ol><li>If we’re making a change that will break a dashboard, pipeline, or ruin someone’s morning, we must write down how and when we’ll loop them in. Especially for backwards-incompatible changes: plan them and put it in writing. <em>Don’t be the cause of surprise 7 a.m. fire drills.</em></li><li>Define SLAs. Document whether this is a one-time use or a recurring deliverable. If people will depend on it regularly, clearly state expectations:</li></ol><ul><li>How fresh the data should be</li><li>How often it runs</li><li>What to do if things go wrong</li></ul><p>“SLAs” don’t mean “we promise perfection.” They mean “we all agree on what’s reasonable.” And make sure to put that agreement down in writing.</p><h4>4. Monitoring and Alerting</h4><p>Make it part of our success criteria to set up monitoring and alerts for production jobs.</p><p>Who will get notified if something fails? Write that down and make sure there’s a handoff if it’s not you. Silent failures are the worst. No alert is just asking for more unexpected fires to put out.</p><h4>5. Handover</h4><p>Don’t close tickets without writing down a proper handoff plan. Work doesn’t end when our JIRA status changes.</p><p>If our output feeds another team, process, or deliverable, be crystal clear on:</p><ul><li>What’s done</li><li>Where to find it</li><li>What happens next</li></ul><p>Don’t be that person who just tosses something over the fence without caring if anyone’s there to catch it.</p><h3>Write It Down. Seriously.</h3><p>I know some of this can (and should) be automated. But putting it in writing matters. It forces us to think about the purpose of the work and makes sure everyone is aligned on what success looks like.</p><p>It also saves time. I’ve seen entire sprints wasted on “surprise” rework because the original definition of done wasn’t clear.</p><h3>Final Thought</h3><p>A good definition of done isn’t a checklist for checklist’s sake. It’s how we:</p><ul><li>Build trust in the work</li><li>Deliver value, not just lines of code</li><li>Avoid chaos downstream</li></ul><p>So the next time someone says “it’s done,” take a moment and ask:</p><blockquote><em>Done for who? Is anyone actually using it? Are we sure it’s not going to blow up tomorrow?</em></blockquote><p><strong>If the answers aren’t clear, we’re not done.</strong></p><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eea52c472cc3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hidden Cost of Build vs Buy]]></title>
            <link>https://medium.com/@durginv/hidden-cost-of-build-vs-buy-744d077716ce?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/744d077716ce</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[build-vs-buy]]></category>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Thu, 29 May 2025 01:31:13 GMT</pubDate>
            <atom:updated>2025-05-29T01:39:50.974Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*WhMYzWzUwbScG_JUDUc6Ug.jpeg" /></figure><p>Build vs. buy conversations happen all the time. We have favorite platforms. We get attached to tools or develop a deep dislike for them. We are influenced by vendors or just get tired of fighting with them. I’ve been on all sides of this.</p><p>Build vs Buy topic stirs up emotions. It doesn’t take long for the room to split into camps. I’m going to exaggerate and oversimplify a bit here because it’s fun and it makes the point.</p><p>When it comes to build vs buy, usually, there are two camps of thought:</p><ul><li>One camp says, “If there’s a turnkey solution, we buy it.”</li><li>The other camp, often driven by engineers, leans toward, “Let’s just build it ourselves.”</li></ul><p>And let’s be honest: sometimes we all suffer from the Lake Wobegon effect, that tendency to overestimate our own capabilities.<br><a href="https://www.oxfordreference.com/display/10.1093/oi/authority.20110810105237549">Lake Wobegon effect</a></p><p>But it’s not really about which is better. It’s about taking an honest look at <strong>total cost of ownership</strong> (TCO), what it really takes to implement, build, run, support, and scale a solution over time.</p><p>Here’s a high-level decision diagram to help us think through whether build or buy makes sense for our teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zb6z-QkkeuOhdMRo" /><figcaption>created by me</figcaption></figure><p>But beyond the diagram, it’s critical to consider all the factors that add up to the total cost of ownership. That’s what should really drive the right choice.</p><h3>Build</h3><p>I’m an engineer. I love building. It’s how I learn. But building isn’t just about writing code.</p><p>Let’s break down the hidden costs of building that add up to the total cost of ownership:</p><ul><li><strong>Platform setup</strong> Development needs tools: source control, CI/CD, secrets management, ticketing, documentation, observability. We need to integrate these tools, manage access, and make sure they all work together seamlessly. Someone has to maintain it. It does not come for free</li><li><strong>Team skills and capacity</strong> Let’s say the best solution should be written in C++, but the team only knows Python. Do we stick with what we know and potentially compromise the architecture? Or do we invest in training or hiring for the right skills? Even if the team has the right skills, do they have the bandwidth? If we don’t already have engineers available to do the work, we will need to find them by deprioritizing something else. Either way: cost adds up.</li><li><strong>Support</strong> Once a solution is built, who owns it? Our team? Another team? Someone has to keep it alive, answer questions, fix bugs, make improvements, stay on top of issues. That work needs to be planned, staffed, and supported.</li><li><strong>Operations</strong> We’ll need environments: dev, test, staging, prod. We will need monitoring, alerting, on-call. We will need high availability and a plan for disaster recovery. All of these are essential for production systems. All of these also add to the total cost of a solution</li><li><strong>Maintenance</strong> Nothing is ever really done. Bugs will show up. Users will want enhancements. Things will break. The long tail of maintenance is real and it’s part of the total cost of ownership.</li></ul><h3>Buy</h3><p>Buying might feel easier until we dig deeper. The software price is just the start.</p><ul><li><strong>Pricing</strong> In today’s world of consumption-based fotware pricing, we have to make sure that the pricing model will scale with our needs. What happens if our usage spikes or drops? Do we get hit with surprise overages or minimums? This gets even trickier with multi-year contracts. Some vendors lock us into thresholds, and we end up paying penalties whether we over- or under-consume. It’s not just about the bill, it can actually change how we use the tool, how fast we grow, or even make us make bad decisions just to use up extra credits before we lose them.</li><li><strong>Hosting model</strong> Is the tool SaaS, or does it require installation in our own infrastructure? If it’s not SaaS, now we’re responsible for uptime, patching, upgrades, monitoring. And don’t forget that hosting something ourselves means someone has to own that environment, keep it secure, scale it, and fix it when things break. That infra isn’t free, and neither is the time it takes to manage it.</li><li><strong>Customizations</strong> Every business is unique. Off the shelf tools are very rarely a perfect fit out of the box. We’ll almost always need some tweaks and customizations. That means using internal resources or hiring outside help or professional services. Either way, it’s more time, more people, and more cost.</li></ul><p>All these costs add up.</p><h3>Opportunity Cost</h3><p>I’m putting this last on my list of costs to consider, but it might be the most important.</p><p>Buying the right solution can accelerate delivery, unlock new capabilities, and help us move faster to meet changing business needs.</p><p><strong>Time to value matters.</strong> That’s a real cost and a real advantage.</p><h3>Sometimes, the answer isn’t Build or Buy — it’s Bridge</h3><p>In a fast-moving world we currently live in, speed to value matters. Sometimes we buy a solution temporarily to unlock value quickly while giving our team time to build the right long-term solution.</p><blockquote>That’s a <strong>Bridge</strong>: a short-term, intentional choice that delivers value now while we build for the future.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qIXO3wmnnn4nkk7Jc_KWeA.png" /><figcaption>created by me</figcaption></figure><h3>Design for Change</h3><p>Nothing is forever. Business needs evolve. So does technology. So should our architecture.</p><p>That’s why modular design matters. We need to focus on building or buying flexible, swappable components so we can adapt quickly and easily to new business needs, new tech, and the growing maturity of our systems and teams.</p><p>The right decision isn’t always permanent, it’s right for <em>now</em>.</p><p>So we have to stop thinking in binaries. It’s not build <em>or</em> buy.</p><p>It’s <strong>build, buy, or bridge</strong>.</p><p>And whatever we choose, we need to make sure we can swap it out when the time comes.</p><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=744d077716ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Most Neglected Data Engineering Tasks]]></title>
            <link>https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ee65b271054</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Fri, 23 May 2025 15:37:54 GMT</pubDate>
            <atom:updated>2025-09-06T12:08:03.570Z</atom:updated>
            <content:encoded><![CDATA[<p>(A blog series based on a talk I gave)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cTSJ2xnvEeUg4PXlMWkaFw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dengxiangs?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Choong Deng Xiang</a> on <a href="https://unsplash.com/photos/graphical-user-interface--WXQm_NTK0U?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Every data team I’ve ever worked with or been part of is <em>busy</em>. Really busy.<br>There’s always more work than time: dashboards, pipelines, ML models, bugs, meetings, and last-minute requests. Most days, it feels like we’re just trying to stay afloat.</p><p>In this blog series, I want to talk about the things that don’t usually make it to the top of the data team’s to-do list or sometimes don’t make the list at all. These are the overlooked and often forgotten tasks in data engineering.</p><p>I’ll share why they matter and how we can bring them back into focus.</p><h4><strong>Series Overview</strong></h4><ol><li><a href="https://medium.com/p/2ab6bb89458e">The Forgotten Bucket of Data Work</a></li><li><a href="https://medium.com/@durginv/hidden-cost-of-build-vs-buy-744d077716ce">Hidden Cost of Build vs Buy</a></li><li><a href="https://medium.com/@durginv/definition-of-done-eea52c472cc3">Definition of Done</a></li><li><a href="https://medium.com/@durginv/data-slas-c04e63460b87">Data SLAs</a></li><li><a href="https://medium.com/@durginv/852d000accdc">Business Seasonality</a></li><li><a href="https://medium.com/@durginv/dates-handle-with-care-e1f19ab0b40b">Dates</a></li><li><a href="https://medium.com/p/c1e4e6b7fbce/">Self Recovering Pipelines</a></li><li><a href="https://medium.com/@durginv/data-testing-37f85518ba99">Data Testing</a></li><li>Environmental Impact</li><li>Walk a Mile in Their Shoes</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ee65b271054" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Forgotten Bucket of Data Work]]></title>
            <link>https://medium.com/@durginv/the-forgotten-bucket-of-data-work-2ab6bb89458e?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ab6bb89458e</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Fri, 23 May 2025 15:37:22 GMT</pubDate>
            <atom:updated>2025-05-23T15:42:12.413Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>(And Why Smart Teams Don’t Skip It)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stcQFd82JeWM_qf6aqkcaQ.png" /><figcaption>created by me</figcaption></figure><p>I like to think of data work falling into three buckets. Two get plenty of attention either by choice or out of necessity. The third? It’s often overlooked, and that’s a big mistake.</p><p>Let’s break them down.</p><h3>1. 🪣 Business Projects</h3><p>This is the work everyone talks about and the one we all want to be part of. It’s about building new data capabilities: dashboards, pipelines, models, and products. This type of work is tied to company goals. It drives strategy, improves operational efficiency, and unlocks new capabilities and enhances customer experience. This is what people usually mean when they say “delivering business value.”</p><p>This work is important. It is a high-visibility, high-impact type of work. It’s exciting. It gets attention, funding, and celebration.</p><h3>2. 🪣 Unplanned Work</h3><p>This work shows up as bug fixes, pipeline failures, broken dashboards, late-night alerts, and unexpected Slack messages that start with “this does not look right”. This is the work no one plans for but everyone has to deal with.</p><p>It’s disruptive. It’s urgent. And it always shows up at the worst possible time.</p><p>Unplanned work is distracting, it pulls us away from what we planned to do. It breaks focus, and eats up hours we don’t have. And no matter how frustrating it is, it’s not going away. Every system has failures. Every dataset has edge cases. Every software has bugs.</p><p>This type of work is unavoidable.</p><h3>3. 🪣 The Forgotten Bucket</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/204/1*5LyN5NvwsaK2fI-HH1Vz5Q.png" /><figcaption>author unknown</figcaption></figure><p>The third bucket is the quiet, often forgotten one. It’s the work no one explicitly asks for, but everyone benefits from.</p><p>It includes things like:</p><ul><li>Cleaning up tech debt</li><li>Refactoring messy code</li><li>Upgrading tools and libraries</li><li>Improving tests and documentation</li><li>Streamlining pipelines</li><li>Exploring new tools or approaches</li><li>Keeping the lights on</li></ul><p>This is the time we spend fixing inefficiencies and shortcuts we took to deliver faster. It’s when we iterate to make improvements, upgrade software, and integrate new tools. It’s also when we experiment and test new approaches to stay ahead.</p><p>This work is often not flashy. It doesn’t come with product launches or executive shout-outs. But it’s what keeps the team moving and the platform healthy.</p><p>The problem? It’s easy to ignore. Until it’s too late. When we skip this bucket for too long, systems get fragile, tech debt piles up, outages happen more often, and suddenly most of our time is spent fighting fires in unplanned work.</p><p>This bucket is about slowing down so we can move faster later. It’s how we build stability, reduce burnout, and ensure our future work doesn’t come with unnecessary baggage.</p><p>Great data teams don’t just dedicate time to this work — <strong>they fiercely protect it</strong>. They know small, consistent investments pay off over time. They don’t wait for things to break to start fixing them.</p><p>So how do we make sure this critical bucket doesn’t stay forgotten? We bake it into our process.</p><p><strong>Iterating to Enhance</strong></p><p>Before any project begins, there should be a <strong>data architecture blueprint.</strong> This is a north star that guides development. But we all know things change. New requirements, tighter timelines, and unexpected discoveries can push us off track.</p><p>That’s okay as long as we document those decisions and plan enhancements to realign post-launch.<br><strong>Enhancement sprints</strong> shouldn’t be optional or vague “someday” items. They should be planned and prioritized, just like feature work.</p><p>Every sprint should include time for bug fixes.<br>Every launch should be followed by a round of refinements.<br>This is part of agility, not an afterthought.</p><p><strong>Making room for innovation</strong></p><p>Google’s <strong>70/20/10 Rule (</strong><a href="https://www.forbes.com/sites/quentinhardy/2011/07/16/googles-innovation-and-everyones/"><em>70/20/10 framework</em></a><em>) </em>is a great way to think about how to spend data team’s time:</p><ul><li><strong>70%</strong> on core business projects</li><li><strong>20%</strong> on innovation, experimentation, and testing new ideas</li><li><strong>10%</strong> on personal growth and learning</li></ul><p>For most teams, dedicating 30% of capacity to non-core work feels like a luxury and might not be feasible. But even 10–20% can make a huge difference.<br>That’s one day every two-week sprint. Not enough to disrupt project timelines. But a full 8 hours to experiment, improve, or learn. Other options:</p><ul><li>Quarterly hackathons</li><li>Monthly Innovation days</li><li>Rotating maintenance sprints</li><li>“Keep the lights on” rotations</li></ul><p>The method doesn’t matter as much as the <strong>consistency</strong>. Make it a habit, not a hope.</p><p>Because at the end of the day, if we don’t take care of our systems, we’re the ones who end up paying for it again and again</p><p><em>Inspired by one of my favorite books </em><a href="https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592"><em>The Phoenix Project</em></a></p><p>Back to <a href="https://medium.com/@durginv/10-most-neglected-data-engineering-tasks-3ee65b271054">10 Most Neglected Data Engineering Tasks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ab6bb89458e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The door is open — Data Vault Resources]]></title>
            <link>https://medium.com/snowflake/the-door-is-open-data-vault-resources-35096d36a56?source=rss-3ed77638d9c0------2</link>
            <guid isPermaLink="false">https://medium.com/p/35096d36a56</guid>
            <category><![CDATA[data-superhero]]></category>
            <category><![CDATA[snowflake-data-cloud]]></category>
            <category><![CDATA[data-vault]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Veronika Durgin]]></dc:creator>
            <pubDate>Thu, 03 Mar 2022 00:36:31 GMT</pubDate>
            <atom:updated>2022-03-04T02:15:59.090Z</atom:updated>
            <content:encoded><![CDATA[<h3>The door is open — Data Vault Resources</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GZyzSF-gRCdcjPklzZtMjA.png" /><figcaption><a href="https://spongebob.fandom.com/wiki/Just_One_Bite">https://spongebob.fandom.com/wiki/Just_One_Bite</a></figcaption></figure><p>My journey into the world of Data Vault started in 2019 when I first read Kent Graziano’s blog that mentioned Data Vault methodology and its ability to be successfully implemented on Snowflake. Being the typical engineer that I am, I googled it. To my disappointment I did not find all that much information about Data Vault. As if all of it was truly hidden … in a vault.</p><p>Here is a list of resources that I came across over the past 3 years. This is not a comprehensive list of everything. This is just my go to list that has supported my learning journey of Data Vault.</p><h3><strong>Online resources</strong></h3><p><a href="https://datavaultalliance.com/"><strong>Data Vault Alliance</strong></a><strong> </strong>This is the home base. This is where <a href="https://danlinstedt.com/">Dan Linstedt</a> keeps everyone up to date with the latest Data Vault advancements. You will find that some content is available for a fee. There is also a lot of free content like blogs and some sessions from the previous conferences. A few of last year’s <a href="https://datavaultalliance.com/product/wwdvc-2021/">WWDVC</a> presentations are available for viewing, just click on the sessions with hyperlinks.</p><p><a href="https://resources.snowflake.com/2/data-vault-resources-kit"><strong>Snowflake</strong></a><strong> </strong>Data Vault and Snowflake are better together. Snowflake’s team put together a list of resources about running Data Vaults on Snowflake with some best practices and user stories.</p><p><a href="https://www.accelerate.world/dvinnovatorscommunity"><strong>Data Vault Innovators</strong></a><strong> — </strong>is a very experienced team of Data Vault practitioners. I have a challenge making the meetings run by this team due to the time difference as they are in New Zealand/Australia area I am on the East Coast in North America. But I’ve heard Nols speak before and I highly recommend signing up anyway.</p><p><a href="https://www.scalefree.com/"><strong>Scalefree</strong></a><strong>. </strong>You will find webinars, blogs, and newsletters from Michael Olschimke and his team. I was especially excited to see a series of blogs covering modern data stack with <a href="https://www.scalefree.com/scalefree-newsletter/open-source-production-grade-data-integration-part-1/">meltano singer taps, and dbt</a>.</p><h3><strong>Books</strong></h3><p><a href="https://www.amazon.com/Building-Scalable-Data-Warehouse-Vault/dp/0128025107">Building a Scalable Data Warehouse with Data Vault 2.0</a></p><p>Written by <a href="https://www.amazon.com/s/ref=dp_byline_sr_book_1?ie=UTF8&amp;field-author=Daniel+Linstedt&amp;text=Daniel+Linstedt&amp;sort=relevancerank&amp;search-alias=books">Daniel Linstedt</a> and <a href="https://www.amazon.com/s/ref=dp_byline_sr_book_2?ie=UTF8&amp;field-author=Michael+Olschimke&amp;text=Michael+Olschimke&amp;sort=relevancerank&amp;search-alias=books">Michael Olschimke</a>. It’s a little bit outdated and some standards have evolved since it was published, but I would still recommend this book as a must read for anyone exploring or working with the Data Vault methodology</p><p><a href="https://www.amazon.com/Data-Vault-Guru-pragmatic-building/dp/B08KJLJW9Q">The Data Vault Guru</a></p><p>Written by Patrick Cuba. This is the newest and very comprehensive book with great illustrations and code examples. I found it easier to have this book in a digital form that allowed me to zoom into images.</p><p><a href="https://www.amazon.com/Elephant-Fridge-Success-Building-Business-Centered-ebook/dp/B07RH9C2JK">The Elephant in the Fridge</a></p><p>Written by John Giles. This book is focused on the modeling aspect of the Data Vault methodology and provides a lot of practical advice.</p><p><a href="https://www.amazon.com/Introduction-Agile-Engineering-Using-Vault/dp/1796584932">An Introduction to Agile Data Engineering Using Data Vault 2.0</a></p><p>Written By Kent Graziano. This book is small but mighty. It is incredibly easy to read because it’s very succinct and all of the important parts and key words are highlighted. Kent is a pro at the art of visual story telling and writing.</p><h3><strong>User Groups</strong></h3><p><a href="https://www.ukdatavaultusergroup.co.uk/">Data Vault UK</a></p><p>Run by Neil Strange and his team from <a href="https://www.data-vault.co.uk/">Datavault UK.</a> I have not missed a single meeting since this group went virtual due to Covid and I can tell you that I learn something at every single one of them.</p><p>Previously recorded meetups can be found on <a href="https://www.youtube.com/channel/UCdhSsZWTwkX1-v8EX84WLYg">youtube</a></p><p>Follow them on <a href="https://www.linkedin.com/company/data-vault-user-group/">LinkedIn</a></p><p><a href="https://www.meetup.com/Data-Modeling-DM3/">Data Modeling Meetup Munich (DM3)</a></p><p>Run by Christian Kaul from <a href="https://obaysch.net/">Obaysch</a>. While this user group is not dedicated exclusively to the Data Vault methodology, understanding modeling approaches is important and this group discusses Data Vault modeling amongst others. This group is based in Germany but many meetups are in English.</p><p>You can find previously recorded meetups on <a href="https://www.youtube.com/channel/UCewghQgxBZsyJEgxn7wR0qw">youtube</a></p><p><a href="https://usergroups.snowflake.com/data-vault/">Snowflake Data Vault User Group</a></p><p>Maybe I am a tiny bit biased because I help out with this user group. But it’s my most favorite one both because Data Vault and Snowflake are better together and because the team at Snowflake that supports the community are some of my favorite people. Even if you are not a Snowflake user, don’t let it stop you from joining this group. Data Vault methodology is platform agnostic and many topics discussed are applicable to other platforms as well.</p><p><a href="https://www.meetup.com/data-vault-north-american-user-group/">North American Data Vault User Group</a></p><p>The newest Data Vault user group with a very passionate team behind it. I am very excited for this one and the content they are delivering.</p><h3><strong>Forums</strong></h3><p><a href="https://forum.ukdatavaultusergroup.co.uk/">https://forum.ukdatavaultusergroup.co.uk/</a></p><p>It’s managed by the same amazing team from Datavault UK. The forum is completely free and you will see many Data Vault thought leaders and practitioners participating in discussions. It might be a little daunting to ask questions in a public forum. But always remember that “the only stupid question is the one that is not asked”.</p><h3><strong>Slack</strong></h3><p>There are a few slack communities that have channels dedicated to Data Vault.</p><p><a href="https://getdbt.slack.com/#modeling-datavault">dbt</a></p><p>This community is managed by the team from dbt labs. And as a result there is a decent amount of dbt specific content. However, this slack is also a community of data professionals discussing everything data related from tools, to different types of modeling, leading data teams, and the most important subject of all — memes. This is one of my favorite places to lurk around and learn about modern data stack.</p><p><a href="https://snowflakecommunity.slack.com/#data-vault">Snowflake Slack Community</a></p><p>Did I mention that Data Vault and Snowflake are better together? Maybe a few times. This community could help answer questions about Data Vault and Snowflake in general and Data Vault running on Snowflake in particular. Join it, don’t be shy. We are better, stronger, and smarter as a community.</p><p><a href="https://dbtvault.slack.com/#data-vault-method">dbtvault</a></p><p>Brought to us by the developers of dbtvault also known as the amazing Datavault UK team. While it’s focused on Data Vault implementations using dbtvault, Neil often provides very thoughtful responses that are related to Data Vault methodology. I am sure you observed a pattern in my recommendations to join all communities. Every single one is a place to learn.</p><h3><strong>Thought Leaders</strong></h3><p><a href="https://www.linkedin.com/in/kentgraziano/"><strong>Kent Graziano</strong></a> — the man, the legend, the Data Warrior. Kent recently retired from his role as the Chief Technical Evangelist at Snowflake. But can the Data Warrior really retire? I selfishly hope not.</p><p>Kent blogs at <a href="https://kentgraziano.com/">https://kentgraziano.com/</a>, shares valuable content on <a href="https://www.linkedin.com/in/kentgraziano/">linkedin</a> and answers forum questions.</p><p><a href="https://www.linkedin.com/in/patrickcuba/"><strong>Patrick Cuba</strong></a> — It’s hard to express the amount of Patrick’s contribution to the Data Vault community in a short paragraph. He wrote a book and many blogs on the subject. He answers questions on slack and forums, and his Snowflake Data Vault demos are some of the best I’ve seen.</p><p>Find Patrick on <a href="https://medium.com/@patrickcuba">Medium</a>, <a href="https://www.linkedin.com/company/cuba-bi-consulting/">linkedin</a>, and more <a href="https://www.linkedin.com/in/patrickcuba/">linkedin</a>.</p><h3><strong>Tools</strong></h3><p>Data Vault methodology is pattern based. To truly see its potential, a Data Vault implementation will require automation. There are a few automation tools both open source and commercial. And as with any tool, make sure to evaluate more than one and pick what fits your use case the best.</p><h4>Dbtvault</h4><p><a href="https://www.data-vault.co.uk/dbtvault/">https://www.data-vault.co.uk/dbtvault/</a></p><h4>Diepvries</h4><p>An open source automation tool created by the team at <a href="https://blog.picnic.nl/picnics-lakeless-data-warehouse-8ec02801d50b">Picnic</a></p><p><a href="https://github.com/PicnicSupermarket/diepvries">https://github.com/PicnicSupermarket/diepvries</a></p><h4>Unnamed :)</h4><p>Roeland Vos wrote a few articles about templates and automation.</p><p><a href="https://roelantvos.com/blog/simple-etl-generation-series/">https://roelantvos.com/blog/simple-etl-generation-series/</a></p><h4>Wherescape</h4><p><a href="https://www.wherescape.com/solutions/automation-software/">https://www.wherescape.com/solutions/automation-software/</a></p><h4>DataVault Builder</h4><p><a href="https://datavault-builder.com/">https://datavault-builder.com/</a></p><h4>Vaultspeed</h4><p><a href="https://vaultspeed.com/">https://vaultspeed.com/</a></p><h4>Ellie</h4><p>Logical Data Modeling tool.</p><p><a href="https://ellie.ai/">https://ellie.ai/</a></p><h3><strong>Training</strong></h3><p>Data Vault trainers go through a very rigorous process before they get to teach. Be rest assured that you will get the best training possible when you sign up for any of these bootcamps</p><p>Data Vault Alliance<strong> </strong>keeps a very handy <a href="https://datavaultalliance.com/event-directory/.">calendar</a> with all of the upcoming Data Vault training. But let me break each out out into their own links.</p><h4>Datarebels</h4><p>I took this training. Cindi’s energy, passion, positive attitude, and knowledge are equal to none.</p><p><a href="https://datarebels.net/course-schedule/">https://datarebels.net/course-schedule/</a></p><h4>PerformanceG2</h4><p><a href="https://www.performanceg2.com/">https://www.performanceg2.com/</a></p><h4>Scalefree</h4><p><a href="https://www.scalefree.com/datavaulttrainings/">https://www.scalefree.com/datavaulttrainings/</a></p><h4>Certus Solutions</h4><p><a href="https://www.certussolutions.com/datavault/training">https://www.certussolutions.com/datavault/training</a></p><p>Did I miss anyone or anything? Let me know in comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35096d36a56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/the-door-is-open-data-vault-resources-35096d36a56">The door is open — Data Vault Resources</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>