<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Debasis Biswal on Medium]]></title>
        <description><![CDATA[Stories by Debasis Biswal on Medium]]></description>
        <link>https://medium.com/@debasisbiswal?source=rss-3c6a1ddce23f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*uT3AJbjANPIYes02.</url>
            <title>Stories by Debasis Biswal on Medium</title>
            <link>https://medium.com/@debasisbiswal?source=rss-3c6a1ddce23f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:36:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@debasisbiswal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Deep Dive into Kubernetes HPA Scaling Behaviour]]></title>
            <link>https://medium.com/@debasisbiswal/deep-dive-into-kubernetes-hpa-scaling-behaviour-3f8ff592763d?source=rss-3c6a1ddce23f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f8ff592763d</guid>
            <category><![CDATA[scaling-behavior]]></category>
            <category><![CDATA[kubernetes-scaling]]></category>
            <dc:creator><![CDATA[Debasis Biswal]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 11:21:02 GMT</pubDate>
            <atom:updated>2026-07-01T03:46:59.441Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nms3lOq52A6iyW4TplvtUQ.png" /></figure><p>In my previous article <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c">Decoding Kubernetes HPA </a>, I focused on why HPA sometimes does not scale as expected, especially in sidecar-heavy workloads. In this article, I want to focus on something equally important, and in many cases even more misunderstood.</p><h4>HPA Scaling Behaviour</h4><p>In real-world Kubernetes clusters, the hard part is often not whether HPA can calculate a number. The hard part is understanding:</p><ul><li>Why HPA scales up the way it does</li><li>Why HPA scales down much more cautiously</li><li>Why replica count does not instantly jump to the calculated number</li><li>and most importantly, how desired replica count directly affects HPA behaviour in both directions</li></ul><p>This is the layer where HPA stops being “just a formula” and starts becoming a <strong>control system</strong>.</p><h4>HPA is not just a metric trigger</h4><p>A lot of engineers think of HPA like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*LXrmG_sc0fVZ8ISVHHdCIQ.png" /></figure><p>That is too simplistic.<br>What really happens is closer to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*F9-hJGW0r2niAFhaFXvOLg.png" /></figure><p>That middle step — <strong>desired replica calculation</strong> — is important. But the part that really determines cluster behaviour is what comes after it.</p><p>Because HPA does not blindly apply the number it calculates.</p><p>It evaluates:</p><ul><li>Whether scale-up or scale-down is happening</li><li>How far current state is from desired state</li><li>Whether the recommendation should be stabilised</li><li>Whether scaling should be rate-limited</li><li>Whether min/max bounds should override it</li></ul><p>That is what I mean by <strong>scaling behaviour</strong>.</p><h3>The most important concept: Desired Replica count</h3><p>If there is one term to understand deeply in HPA, it is this one: <strong>Desired Replica</strong></p><p>This is the number of replicas the HPA controller believes the workload should be running, based on current observed metrics.</p><p>For CPU or utilisation based metrics, the formula is:</p><blockquote>Desired Replicas = CEIL( Current Pod count × (Average CPU_used_vs_request% of all pods / Target CPU% ))</blockquote><p>Let us take a simple example.</p><p>Current replicas = 5 , Target CPU = 50% , Observed CPU = 80%</p><p>Then:<br><em>Desired Replicas = CEIL ( 5 * (80 / 50 ) = CEIL(8) = 8</em></p><p>This means HPA wants 8 replicas.<br>But here is the critical point:</p><blockquote><strong>Desired Replicas</strong> does not always match the next actual replica count</blockquote><p>It is a recommendation.</p><p>And Scaling behaviour determines how Kubernetes moves from:</p><blockquote><strong>Current Replicas</strong> to <strong>Desired Replicas</strong></blockquote><p>That journey is what we need to understand.</p><h3>Scaling Behaviour at a Glance</h3><p>Lets first see how scaling behaviour looks like in actual implementation :</p><pre>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: hpa-test<br>  namespace: hpa-test<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: hpa-test-Deployment<br>  minReplicas: 5<br>  maxReplicas: 50<br>  behavior:<br>    scaleUp:<br>      stabilizationWindowSeconds: 0<br>      policies:<br>      - type: Percent<br>        value: 100<br>        periodSeconds: 15<br>      - type: Pods<br>        value: 4<br>        periodSeconds: 15<br>      selectPolicy: Max<br>    scaleDown:<br>      stabilizationWindowSeconds: 300<br>      policies:<br>      - type: Percent<br>        value: 50<br>        periodSeconds: 60<br>      - type: Pods<br>        value: 5<br>        periodSeconds: 60<br>      selectPolicy: Min<br>  metrics:<br>  - type: ContainerResource<br>    containerResource:<br>      name: cpu<br>      container: main-application<br>      target:<br>        type: Utilization<br>        averageUtilization: 50</pre><p>In the above sample code we can see a “behavior” section with two subsection “scaleUp” and “scaleDown”.</p><p>“scaleUp” and “scaleDown” behavior gets triggered when desiredReplicas count meets the criteria. Each behavior follows the policies defined within it.</p><p>Now lets understand each policy step by step.</p><h3>Scaling Policy Types</h3><p>In Kubernetes we have two types of policies</p><h4>Percent Policy</h4><pre>policies:<br>- type: Percent<br>  value: 100<br>  periodSeconds: 15</pre><p>Percent policy defines how much percentage of current replica will be added in the specified time interval . In above example the replica count will double(in scaleUp) in every 15 seconds when the metrics require it.</p><p>So if we have 2 replicas, it can scale to 4, then 8, then 16. Percent policies are effective when we need rapid pod scaleup but we have to use it cautiously as it can overshoot on large deployments.Similarly for scaleDown behavior , the percent policy acts to remove pods on percentage basis .</p><h4>Pod Policy</h4><pre>policies:<br>- type: Pods<br>  value: 5<br>  periodSeconds: 30</pre><p>Pod policy defines how many number of pods will be added or removed. In above example this policy allows up to 5 pods to be added or removed every 30 seconds. Pods policies give predictable linear scaling and work well when we know our capacity needs precisely.</p><h3>Policy Selection Strategy</h3><p>Till now we learned that we have two types of policies available . But which one to select ?</p><p>The answer is <em>“selectPolicy”</em> . We have 3 policy selection strategy</p><h4>Min Selection Strategy</h4><pre>selectPolicy: Min</pre><p>Min policy decides the minimum value of replicas to add or remove from the given policies. Normally we use Min for scaleDown behavior.</p><h4>Max Selection Strategy</h4><pre>selectPolicy: Max</pre><p>Max policy decides the maximum value of replicas to add or remove from the given policies. Normally we use Max for scaleUp behavior.</p><h4>Disabled Selection Strategy</h4><pre>selectPolicy: Disabled</pre><p>This policy disables the scaling up/down . Normally we use this when we perform certain maintenance work.</p><blockquote>When we define <strong>multiple policies</strong> in your behavior spec (e.g., one Percent-based + one Pods -based), selectPolicy : Max tells HPA to pick the policy that <strong>allows the largest number of replicas to be added/removed</strong> in that period.</blockquote><blockquote><em>The behavior policy always acts as a </em><strong><em>ceiling</em></strong><em> (upper bound) for scaling in that period. The metric-desired is the </em><strong><em>target</em></strong><em>.</em></blockquote><blockquote><strong><em>HPA never overshoots the metric-desired.</em></strong></blockquote><h3>Stabilization Window</h3><pre>stabilizationWindowSeconds: 300</pre><p>It defines after how much duration Pods will be added or removed . It ensures that the HPA avoids abrupt scale-up or scale-down(thrashing) actions due to transient metric spikes.</p><p>We can use below stanbilizationWindows values based on requirements :</p><ul><li>Shorter windows (0–60s): Highly responsive, risk of thrashing</li><li>Medium windows (60–300s): Balanced for most applications</li><li>Longer windows (300–600s): Very stable, slower to react</li></ul><p>During stabilizationWindowSeconds , kubernetes collects the desired-replica count on each sync period. Then calculates the MAX of those values.</p><p>This MAX rule is applied in both scale-up and scale-down event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*NdvpyPiHDMJlvnWz9oX-1w.png" /></figure><h3>Default Values</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*VYKICVIyqw7RRwu45Ur-iQ.png" /></figure><p>Note that in Scale-down , there is no default values for Pod policy.</p><h3>Scaling Patterns</h3><p>Till now we have understood each part of scaling behaviour . Now lets see how we can use it for different scenarios.</p><h4>Fast ScaleUp and Slow ScaleDown</h4><pre>behavior:<br>  scaleUp:<br>    stabilizationWindowSeconds: 0<br>    policies:<br>    - type: Percent<br>      value: 100<br>      periodSeconds: 15<br>    - type: Pods<br>      value: 4<br>      periodSeconds: 15<br>    selectPolicy: Max<br>  scaleDown:<br>    stabilizationWindowSeconds: 300<br>    policies:<br>    - type: Percent<br>      value: 10<br>      periodSeconds: 60<br>    selectPolicy: Min</pre><p>This is the most common pattern. Scale up quickly to handle load spikes, but scale down slowly to avoid thrashing.</p><p><strong>Scale-up behavior:</strong></p><ul><li>No stabilization (reacts immediately)</li><li>Doubles capacity every 15 seconds OR adds 4 pods, whichever is more</li><li>Chooses the most aggressive option</li></ul><p><strong>Scale-down behavior:</strong></p><ul><li>Waits 5 minutes before scaling down</li><li>Removes at most 10% of replicas per minute</li><li>Chooses the least aggressive option</li></ul><h4>Burst Scaling</h4><pre>behavior:<br>  scaleUp:<br>    stabilizationWindowSeconds: 0<br>    policies:<br>    - type: Pods<br>      value: 10<br>      periodSeconds: 10<br>    - type: Percent<br>      value: 200<br>      periodSeconds: 10<br>    selectPolicy: Max<br>  scaleDown:<br>    stabilizationWindowSeconds: 600<br>    policies:<br>    - type: Pods<br>      value: 1<br>      periodSeconds: 120</pre><p>Handle sudden traffic bursts with immediate capacity, then gradually scale down.<br>This configuration:</p><ul><li>Adds up to 10 pods or up to 200% of the current replica count every 10 seconds during scale-up</li><li>Waits 10 minutes after load decreases before scaling down</li><li>Removes only 1 pod every 2 minutes during scale-down</li></ul><p>Perfect for applications with bursty traffic patterns .</p><h4>Cost-Optimised Scaling</h4><pre>behavior:<br>  scaleUp:<br>    stabilizationWindowSeconds: 60<br>    policies:<br>    - type: Percent<br>      value: 100<br>      periodSeconds: 30<br>    selectPolicy: Max<br>  scaleDown:<br>    stabilizationWindowSeconds: 180<br>    policies:<br>    - type: Percent<br>      value: 25<br>      periodSeconds: 30<br>    - type: Pods<br>      value: 5<br>      periodSeconds: 30<br>    selectPolicy: Max</pre><p>Minimise costs by scaling down aggressively during low traffic, but maintain availability.</p><p>This configuration scales down more aggressively than typical patterns. We can use it for:</p><ul><li>Development/staging environments</li><li>Batch processing workloads</li><li>Non-critical services with relaxed latency requirements</li></ul><h4>Scheduled Scaling</h4><p>We can write a scheduler which gets triggered on certain specific business hours and set the Min replicas to desired value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*OH_5eHGwhxIYi-vzvB2f1g.png" /><figcaption>Running pods at full scale 24/7</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*hl-yTxxYM_cYrEy36chCMQ.png" /><figcaption>Time based scaling using a Scheduler to set Min HPA</figcaption></figure><p>So we have understood how to apply Scaling behaviour . And we are good to go .. right ?<br>NO<br>Still there is something left.</p><p>Lets understand some basic formulas.</p><h3>Magic Formula for Final Replica count</h3><h4>Scale-up Formula</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QM3on0NqyyTM220LgAnJsA.png" /><figcaption>Scale-up Formula</figcaption></figure><h4>Scale-down Formula</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*liZ1f8ZMewApRCrbtNpPgQ.png" /><figcaption>Scale-down Formula</figcaption></figure><h3>Scaling Behaviour in Action with Scenario</h3><p>Lets go through some scenarios and consider the existing HPA as our example where :</p><p>Min=5 , Max=50 and we have scaleUp and scaleDown behavior defined.</p><h4>Scenario-1 ( scaleUP ) :</h4><p>If Min=5 , Max=50 , currentReplicas count is 5 and desiredReplicas formula gave target pod count is 7 then :</p><p>Whats going to be my final pod count ? is it 7 (desiredReplica) or 10 ( 100% of current replica) ?</p><p>Answer :</p><p>A Percent: 100 policy means: <em>“in one scaling period, you can add/remove at most 100% of the </em><strong><em>current</em></strong><em> running replicas.”</em></p><p>It limits the <strong>rate of change</strong>, NOT the final target.</p><pre>Min Replica = 5<br>Max Replica = 100<br><br>Current Replica = 5<br>Desired Replica (from metric) = 7<br><br>Percent Policy =&gt; Max allowed increase  = 100% × current (5) = 5 pods<br>Pod Policy =&gt; 4 pods<br>Selection Policy = MAX =&gt; max(5,4) =&gt; 5 pods<br><br>Max pods for this sync period  = 5(current replica) + 5(selection policy output) = 10</pre><p>Now we have desiredReplica is 7 and scaling policy says 10 . So which one to select.<br>Here comes the hidden formula for scaleUP behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RSqFnkuRISDPyKQQw98JaA.png" /></figure><p>So our final replica count is going to be 7 .</p><ul><li>The 100% Percent policy allows up to doubling current replicas per period — it’s a ceiling, not a target.</li><li>Since 7 &lt; 10 (allowed ceiling), no throttling occurs =&gt; it reaches 7 in one step.</li><li>If desired were, say, 40 with current at 5: allowed ceiling = 10, so it would be throttled to 10 in the first sync period, then re-evaluated next sync period.</li></ul><h4>Scenario-2 ( scaleUP ) :</h4><p>If Min=5 , Max=50 , currentReplicas count is 5 and desiredReplicas formula gave target pod count is 20 then :</p><p>Whats going to be my final pod count ? is it 20 (desiredReplica) or 10 ( 100% of current replica) ?</p><p>Lets follow the formula which we went through in previous section .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OeHuRKglbJb-Rx3rmty0g.png" /></figure><p>Here we saw that desiredReplica was 20 , but it got throttled to 10 by Allowed Ceiling for that sync period.</p><p>So does that mean in next sync period it will add remaining 10 pods ?</p><p>Answer :</p><p>HPA does NOT remember the previous period’s pending or unfulfilled scale target. Each sync period is a completely fresh evaluation from scratch.</p><p>In each sync period what happens is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*72BA5YfNicRJsm6QyQiQAQ.png" /></figure><p>The only “memory” carried forward is the <em>actual running pod count </em>— because pods scaled in period 1 are physically running in period 2.</p><p>In next sync period , what happens is :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*72BA5YfNicRJsm6QyQiQAQ.png" /></figure><h4>Scenario-3 ( scaleDown ) :</h4><p>Lets consider the scaleDown behavior from our example :</p><pre>    scaleDown:<br>      stabilizationWindowSeconds: 300<br>      policies:<br>      - type: Percent<br>        value: 50<br>        periodSeconds: 60<br>      - type: Pods<br>        value: 5<br>        periodSeconds: 60<br>      selectPolicy: Min</pre><p>Lets say Min = 5 , Max = 50 , current_replica is 20 , desired_replica is 12 , so whats the final replica count</p><p>Lets follow the scaleDown formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y5NCfYxv2kDlEeo7PrcwIQ.png" /></figure><h3>Conclusion</h3><p>HPA is often treated as a set-and-forget mechanism — but as we have seen, the journey from metric observation to actual pod count is far more nuanced.</p><p>The key takeaways to carry forward:</p><ul><li><strong>Desired Replica</strong> is a recommendation, not a command — scaling behaviour determines how fast you reach it</li><li><strong>Policies act as a ceiling</strong> on rate of change, never as an overshoot beyond the metric-desired</li><li><strong>stabilizationWindowSeconds</strong> protects against thrashing but delays reaction — tune it based on your workload’s tolerance</li><li><strong>HPA has no memory between sync periods</strong> — every evaluation starts fresh from the current running pod count</li></ul><p>The next time your HPA doesn’t behave as expected, ask three things:</p><ul><li>What was the desired replica count?</li><li>What was the allowed ceiling for that period?</li><li>Was the stabilization window still active?</li></ul><p>Those three questions will explain almost every HPA behaviour you encounter.</p><p>If you found this useful, do check out my previous article <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c"><strong>Decoding Kubernetes HPA</strong></a>, where I cover why HPA sometimes fails to scale altogether — especially in sidecar-heavy workloads. Together, the two articles give you a complete picture of HPA from formula to production behaviour.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f8ff592763d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Kubernetes HPA-Part2]]></title>
            <link>https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part2-f457676026d9?source=rss-3c6a1ddce23f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f457676026d9</guid>
            <category><![CDATA[kubernetes-hpa]]></category>
            <category><![CDATA[kubernetes-pod-scaling]]></category>
            <category><![CDATA[kubernetes-multicontainer]]></category>
            <dc:creator><![CDATA[Debasis Biswal]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 04:38:50 GMT</pubDate>
            <atom:updated>2026-06-09T06:29:57.301Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1Zv2Ie-9T-ZPcT-u9oBag.png" /></figure><p>In <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c">Part-1</a> we have discovered the problem of Pod Average Resource usage in HPA. And the problem is we are using average of all containers in the pod. But actually we are more interested in resource usage of main/application container rather than sidecar container .</p><p>Here comes the rescuer <strong>“Container Resource”</strong> .<br>Since Kubernetes version 1.30 , a new HPA type is available which is known as “ContainerResource” . This targets the container average resource instead of pod average resource.</p><p>So , I have modified the existing hpa(described in <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c">Part-1</a>) to implement container specific average .</p><pre>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: hpa-test<br>  namespace: hpa-test<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: hpa-test-Deployment<br>  minReplicas: 5<br>  maxReplicas: 20<br>  metrics:<br>  - type: ContainerResource<br>    containerResource:<br>      name: cpu<br>      container: main-application<br>      target:<br>        type: Utilization<br>        averageUtilization: 50<br>  - type: ContainerResource<br>    containerResource:<br>      name: memory<br>      container: main-application<br>      target:<br>        type: Utilization<br>        averageUtilization: 50<br>  - type: ContainerResource<br>    containerResource:<br>      name: cpu<br>      container: sidecar<br>      target:<br>        type: Utilization<br>        averageUtilization: 50<br></pre><p>You can see the HPA , here the 2 most important thing is “type: ContainerResource” and “container: main-application”. Here we are setting the type of HPA and the container name which will be monitored .</p><p>Once I deployed this new HPA type , then I started the load . This time I used same load injector as <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c">Part-1</a>. No change in any load profile.</p><h3>Test &amp; Result Analysis</h3><p>Now lets analyse the test result .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kmf5Cf-EiAd2C_tsOWV1nQ.png" /><figcaption>Figure-1 : Pod Scaling , Pod cpu usage, Main-container cpu usage , Sidecar cpu usage</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3A6b6tlY-yuJMtLmVZ7GVA.png" /><figcaption>Figure-2 : Time wise Pod CPU usage and Scaling details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9UJ0Y531EIJN-2GHSFw2VA.png" /><figcaption>Figure-3 : Combo Chart depicting Application and Sidecar container cpu usage trend(in LINE Left Y axis), with Pod count(in BAR, Right Y axis)</figcaption></figure><p>From Figure-2 and Figure-3 , we can clearly see how the scaling happened based on application container’s cpu usage. When application containers average cpu usage reached 56.7% , hpa approached its trigger point and by the it reached 59.3 , the hpa kicked in and 2 pods added making pod count from 5 to 7.</p><p>Through out the test which ran for 14 minutes , the application containers average cpu usage remained within 56% to 60% . Till we reach the end of test , hpa has added 9 pods making pod count from 5 to 14 .</p><p>Lets have a side by side comparison of Figure3 from <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c">Part-1</a> and Part-2(this article)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXlB6cf8B4Im_fbhepqscg.png" /><figcaption>Figure-4: Side by Side Comparision</figcaption></figure><p>Here we can see that HPA based on ContainerResource is clear winner. It trigered scaling on time and saved the application container . The cpu usage remained unders 60%. Where as in the Pod Resource level hpa , the application container’s cpu usage reached 90%.</p><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cxLJkP1r2XMJHABlaxVkqQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f457676026d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding Kubernetes HPA-Part1]]></title>
            <link>https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part1-88324821a16c?source=rss-3c6a1ddce23f------2</link>
            <guid isPermaLink="false">https://medium.com/p/88324821a16c</guid>
            <category><![CDATA[kubernets-pod-scaling]]></category>
            <category><![CDATA[multi-container]]></category>
            <category><![CDATA[kubernetes-hpa]]></category>
            <dc:creator><![CDATA[Debasis Biswal]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 04:34:55 GMT</pubDate>
            <atom:updated>2026-06-22T16:31:56.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xbG3WFQaNmuPMCcxiGuCYw.png" /></figure><p>From last many years I was struggling to fix a problem related to Pod Scaling. Every time I try to adjust the HPA values it was never perfect. If I am running a pod with single container then scaling works perfectly. But in my case my pod is having multiple containers , which creates the problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*D22XVcPl3zs-RyBARTedyA.png" /><figcaption>A sample POD</figcaption></figure><p>There are multiple solutions available to address this problem:</p><p>1 . KEDA( Kubernetes Event-Driven Autoscaling) : Keda can be used with external metrics server to perform scaling . These metrics can be cpu/memory utilisation or message/event lags from event streaming apps like Kafka etc.</p><p>2. VPA ( Vertical Pod Autoscaler) in AKS : VPA automatically sets the resource request and limit on container per workload based on past usage.It reduces the scaling need .</p><p>I did not choose above approaches , here is WHY :</p><ul><li>KEDA needs extra component to be maintained and I was looking for a native solution .</li><li>All our applications are in JAVA and as per <a href="https://learn.microsoft.com/en-us/azure/aks/vertical-pod-autoscaler#limitations-and-considerations">Microsoft</a> , VPA will not work properly in JVM based applications .</li></ul><p>Apart from these there is also most important thing to consider , Scaling Behaviour. I will publish a separate article on it to explain briefly.</p><p>Being said that , to address the pod scaling issue most of the developers run extra pods to handle any burst of request so that their application will be safe. Those extra set of pods run 24/7 and adds cost.</p><p>In this series of 2 articles I will explain the existing problem and its solution.</p><p>All my application pods are having a main application container and a sidecar container which reads the log of application container and push the log to the server. We are mostly concern with the main application containers resource usage and we set out HPA based on that. But we never succeed as the sidecar containers resource usage dilutes the HPA target resource utilisation which we set considering main application container.</p><p>Before proceeding further I want to explain few basics which will help to understand the analysis better.</p><p><strong>Desired Replica:</strong> <br>HPA calculates the desired replica count with the below formula(example: cpu)</p><blockquote>Desired Replicas = CEIL( Current Pod count × (Average CPU_used_vs_request% of all pods / Target CPU% ))</blockquote><p><strong>The Tolerance Band :<br></strong>HPA does not fire on every tiny fluctuation. It has a built-in ±10% tolerance band around the target. This prevents rapid scale up and down — what Kubernetes calls thrashing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*OMTDuUQ-mM09CKSCizzAMg.png" /><figcaption>Tolerance Band</figcaption></figure><blockquote>So If our Target = 50% then<br><strong><em>Upper Trigger</em></strong><em> = 50% + 10% of 50 = 55% =&gt; Scale UP if CPU exceeds this <br></em><strong><em>No Action Zone</em></strong><em> = 45% to 55% =&gt; HPA does nothing here <br></em><strong><em>Lower Trigger</em></strong><em> = 50% — 10% of 50 = 45% =&gt;Scale DOWN if CPU drops below this</em></blockquote><p><strong>Sync Period:</strong><br>Kubernetes implements horizontal pod autoscaling as a control loop that runs intermittently . The interval is set by the --horizontal-pod-autoscaler-sync-period parameter to the kube-controller-manager and the default interval is 15 seconds .</p><p><strong>How Pod-Level Resource usage% Is Calculated:<br></strong>When HPA uses type: Resource (pod-level), Kubernetes calculates pod resource utilisation as(taking CPU for example):</p><blockquote><strong><em>Pod CPU%</em></strong><em> = ( Container1 CPU % + Container2 CPU % ) / Number of Containers</em></blockquote><p>Consider I am running 5 pods with 2 containers in a pod .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*3wDyyAcJobMU6rYoz0x0tg.png" /></figure><p>Main-container : CPU Request is 800m and average usage is 784.8m <br>Sidecar-container : CPU Request is 500m and average usage is 10.9m</p><p>Now lets look at the Pod CPU usage %:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*8ZG3NXykK_PQgdesCPNGhA.png" /></figure><p>Main-container actual CPU usage = 784.8m (from 5-pod average) <br>Sidecar-container actual CPU usage = 10.9m (from 5-pod average) <br>Main-container utilisation = 784.8 / 800 × 100 = 98.1% <br>Sidecar-container utilisation = 10.9 / 500 × 100 = 2.17% <br>Pod CPU usage% = ( 98.1 + 2.17 ) / 2 = 100.27 / 2 = 50.13%</p><p>Now with the above basics lets head towards our test.<br>I wrote a simple deployment which runs a pod with 2 containers.<br>The main-container mimics the cpu usage when a load-injector calls its exposed api. And the sidecar-container mimics the logger. Its cpu usage is very low and does not depend on load(I purpose fully did this to replicate the production scenario where main-container cpu goes high as compared to sidecar).</p><h3><strong>The Test Bed</strong></h3><p>My deployment contains 2 containers . Below are the resource request and limit values .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*I0qozYqjs6zh1-4n3VXDsA.png" /></figure><p>Main-container(Application) :<br>CPU =&gt; Request = 800m , Limit = 4000m<br>Memory =&gt; Request = 64mi , Limit = 1028Mi<br>Sidecar-container(Logger):<br>CPU =&gt; Request = 500m , Limit = 800m<br>Memory =&gt; Request = 32mi , Limit = 64Mi</p><p>You can see that I have kept the Limit of Main-container to a high range , so that I can run my test when cpu usage goes beyond 100% and the pod will not crash or throttle . It will help me to capture the metrics on monitoring dashboard clearly.</p><p>My Load injector is a simple load generator which continuously injects load in every 1 second gap. There is no limit or stable state. Its just a simple load injector.</p><h3><strong>Replicating the Real issue</strong></h3><p>Lets first set up our HPA. Its a simple HPA targeting CPU utilisation of 50% at pod level.</p><pre>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: hpa-test<br>  namespace: hpa-test<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: hpa-test-Deployment<br>  minReplicas: 5<br>  maxReplicas: 20<br>  metrics:<br>  - type: Resource<br>    resource:<br>      name: cpu<br>      target:<br>        type: Utilization<br>        averageUtilization: 50</pre><p>Then I deployed the HPA and waited for 5 pods(minReplicas) to come up.</p><p>Once pods are in ready state , I started the load injector and ran it for almost 15minutes. I setup the load injector to inject load slowly so that I can see the cpu usage going high step by step or slowly and pod scaling also in same way.</p><h3>Test &amp; Result Analysis</h3><p>Now lets see the test result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U2b_sJfpflYkOyLXKgSO0Q.png" /><figcaption>Figure-1 : Pod Scaling , Pod cpu usage, Main-container cpu usage , Sidecar cpu usage</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5vqRlHXSotyLo43iWJxVA.png" /><figcaption>Figure-2 : Time wise Pod CPU usage and Scaling details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pjc7IL0LAUK4Y5HRv1mXRA.png" /><figcaption>Figure-3 : Combo Chart depicting Application and Sidecar container cpu usage trend(in LINE Left Y axis), with Pod count(in BAR, Right Y axis)</figcaption></figure><p>From Figure-2 , we can clearly see that our first scaling occurred at 9.09–9.10. At that point Pod Avg cpu usage was 46.5% but our application/main container avg cpu usage was 91% and sidecar/logger container cpu usage was 2.13% .</p><p>The Pod avg cpu usage remain between 46%-44% till the load stopped. But in between that our application containers cpu usage remained between 87%-91% . And scaled till 9 pods . Which is not acceptable and it creates many problems in production scenarios .</p><p>Now you might be thinking , we have our HPA target set at 50% and we already know that 45%-55%(+/-10% of 50%=+/-5) is no Scale zone , then why pod scaled at 46.5%?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lf_mGU22fflb5d_ebBAqjw.png" /><figcaption>Figure-4 : HPA Sync/Sampling in 15 seconds and scaling impact</figcaption></figure><p>We know that kube-controller-manager performs a sync operation of resource usage metrics in every 15 seconds . And our monitoring tool captures metrics in every 1 minute(we get metrics for every 1 minute duration). So there is a high chance that in that 1 minute in any of the sync slot(15 seconds) the average cpu usage of application container went beyond its no scaling zone: i.e :55%.</p><p>For understanding purpose in Figure-4 I explained with some dummy data . At 09:08:00(it may be between 09:07–09:08 also)the usage was 45.3% and at 09:09:00 the usage was 46.5%. Then some where in between 09:08:00 and 09:09:00 the pod usage went to 56%(&gt; 55% , assuming for explanation) , causing pod scaling. And then cpu usage came down to 46.5%.</p><p>Till now we saw how HPA based on Pod average resource usage did not help us in scaling while having multiple containers in a pod . So In <a href="https://medium.com/@debasisbiswal/decoding-kubernetes-hpa-part2-f457676026d9">Part-2</a> of this article we will see how to solve this .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88324821a16c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uneven Request Distribution in Kubernetes]]></title>
            <link>https://medium.com/engineered-publicis-sapient/kubernetes-uneven-traffic-iptable-keepalive-b6a4a1043bc8?source=rss-3c6a1ddce23f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6a4a1043bc8</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Debasis Biswal]]></dc:creator>
            <pubDate>Thu, 12 Jun 2025 06:52:07 GMT</pubDate>
            <atom:updated>2025-06-12T06:52:07.550Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*2crKq-c4E2EYzW7j_wGUCw.png" /></figure><p>In a world where applications are expected to scale elastically and instantly, even subtle inefficiencies can create bottlenecks that quietly erode performance. One such hidden challenge is <em>Uneven request distribution in Kubernetes </em>— an issue I recently encountered in a production environment.</p><p>At a first glance, it seemed like Kubernetes services were failing to distribute traffic fairly across pods. Some pods were flooded with requests, while others remained frustratingly idle. But digging deeper revealed a surprising truth: the culprit wasn’t Kubernetes; it was the behavior of the clients and TCP keep-alive connections, combined with how Kubernetes leverages Iptables for routing.</p><p>In my latest whitepaper, I unpack this phenomenon — and it’s far more intricate (and fascinating) than it appears.</p><h4>Key Highlights from the Whitepaper:</h4><ul><li><strong>What Happens When You Create a Kubernetes Service:</strong> Behind the scenes, Kubernetes dynamically creates virtual IPs, sets up NAT rules, and injects complexity that’s often invisible until problems arise.</li><li><strong>The Hidden Impact of Client Keep-Alives:</strong> Persistent client connections “lock” into an initial routing decision, creating sticky sessions that bypass re-evaluation of IPTables rules.</li><li><strong>First Request vs. Subsequent Requests:</strong> I walk through what happens <em>under the hood</em> when the very first request hits a Kubernetes service — and how subsequent requests behave very differently.</li><li><strong>Deep Dive into IPTables Evaluation:</strong> The whitepaper demystifies how Kubernetes programs IPTables rules and how connection tracking influences packet routing.</li></ul><h4>Who Should Read This?</h4><p>If you’re working with Kubernetes, designing microservices, optimizing service meshes, or building resilient platforms — understanding this behavior can save you hours of debugging and make your architecture more robust.</p><h4>📄 Read the Full Whitepaper</h4><p>🔗 <a href="https://github.com/debbiswal/K8sWhitePaper/raw/main/WhitePaper-K8s-RequestDistribution.pdf"><strong><em>Click here to download the Whitepaper PDF</em></strong></a></p><h4>Let’s Connect!</h4><p>Have you faced weird load-balancing issues in Kubernetes too?<br>Drop a comment here or reach out — I’d love to exchange ideas and learn from your experiences.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6a4a1043bc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/kubernetes-uneven-traffic-iptable-keepalive-b6a4a1043bc8">Uneven Request Distribution in Kubernetes</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>