<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alan Mosca on Medium]]></title>
        <description><![CDATA[Stories by Alan Mosca on Medium]]></description>
        <link>https://medium.com/@nitbix?source=rss-4f87c2293d63------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zpFaBEU0kFo-grMdRlJmcw.png</url>
            <title>Stories by Alan Mosca on Medium</title>
            <link>https://medium.com/@nitbix?source=rss-4f87c2293d63------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:48:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nitbix/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[The Prophet’s Dilemma: When Predictions Change the Future]]></title>
            <link>https://medium.com/nplan/the-prophets-dilemma-when-predictions-change-the-future-05806fa9c83b?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/05806fa9c83b</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[risk]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Mon, 10 Feb 2025 15:29:29 GMT</pubDate>
            <atom:updated>2025-02-11T12:07:53.945Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Prophet’s Paradox: When Predictions Change the Future</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_aE0eaJwvVk7zB63ttrMSQ.jpeg" /></figure><p>Lately, I’ve been diving deep into the world of predictions and uncertainty — an interest sparked by my work at <a href="https://www.nplan.io/">nPlan</a>, where we build AI to forecast the outcomes of megaprojects. This experience has given me a unique perspective on how our forecasts can sometimes change the very events we’re trying to predict. Today, I want to explore what I call <strong>the prophet’s paradox</strong>, a paradox that challenges the very nature of prediction when human behavior comes into play.</p><h3>Defining the Prophet’s Paradox</h3><p>Imagine two roles in a prediction scenario: the <strong>predictor</strong> and the <strong>receiver</strong>. The predictor offers a forecast about the future. Now, consider a situation where the predictor announces a negative outcome — say, “You’re going to fail your exam.” Instead of accepting this as inevitable, the receiver might work extra hard to avoid that fate. In doing so, the prediction is rendered false.</p><p>This leads us to the essence of the prophet’s paradox:</p><blockquote><strong><em>A prediction can only be considered truly correct if it’s ignored — and even then, only if natural uncertainty allows the predicted event to occur.</em></strong></blockquote><p>In many cases, especially with negative predictions, the very act of forecasting prompts actions that defy the prediction.</p><h3>Real-World Examples and Applications</h3><p>The prophet’s paradox isn’t just a theoretical concept — it has practical implications in various fields:</p><ul><li><strong>Academic and Personal Life:</strong><br> Imagine a teacher predicts that a student will struggle in class. The student, determined not to confirm this negative forecast, might invest extra effort and ultimately excel, effectively defeating the prophecy.</li><li><strong>Economic Forecasts:</strong><br> Consider an analyst who predicts that a particular stock is doomed to crash. Investors, acting on that forecast, might sell their shares. Their collective action could either trigger the crash (a self-fulfilling prophecy) or, in some cases, lead to market overreactions that later stabilize, contradicting the initial prediction.</li><li><strong>Behavioral Predictions:</strong><br> In politics or social dynamics, if a pundit forecasts violent reactions to a new policy, community leaders might take preemptive measures to ease tensions. By doing so, they avert the predicted outcome, thereby invalidating the original forecast.</li><li><strong>Megaproject Risk Mitigation at nPlan:</strong><br>At nPlan, our AI is designed to forecast potential risks in large-scale projects — such as delays caused by supply chain issues or unexpected cost overruns — by analyzing historical data and project-specific parameters. When our model signals a high risk (for example, a significant schedule delay), the project team is alerted and takes proactive measures: they might adjust timelines, secure additional resources, or find alternative suppliers. These actions often prevent the predicted issue from materializing. Consequently, the forecast, although statistically sound, appears less accurate because the very prediction spurred behavior that averted the problem. This is a classic example of the prophet’s paradox, where the predictor ends up being <strong>more right than chance would suggest when ignored, or more wrong than expected when the prediction influences behavior.</strong></li></ul><h3>The Underlying Mechanism: Feedback Loops and Self-Reference</h3><p>The key to understanding the prophet’s paradox lies in the concept of <strong>feedback loops</strong>. Predictions are not made in a vacuum — they interact with the behavior of those who hear them. This interaction creates a self-referential loop:</p><ol><li><strong>The Prediction Is Made:</strong> A negative forecast is announced.</li><li><strong>Behavioral Response:</strong> The receiver changes their behavior in response.</li><li><strong>Altered Outcome:</strong> The change in behavior modifies the conditions that would have led to the predicted outcome.</li></ol><p>Only if the prediction is ignored does it stand a chance of being “verified.” Even then, the inherent uncertainty of future events means it might still turn out to be wrong.</p><h3>The Accuracy Paradox: Being More Right or More Wrong</h3><p>One of the most intriguing implications of the prophet’s paradox is how it affects the predictor’s accuracy. In traditional forecasting, you might expect that any error in your prediction reflects a straightforward miscalculation of probabilities. However, in a system influenced by the very act of prediction, the predictor ends up in a sort of double bind:</p><p><strong>If the Prediction Is Heeded</strong><br>When a forecast — especially a negative one — is taken seriously, the receiver often alters their behavior to avoid the outcome. In this case, the prediction ends up being <strong>more wrong than it should be</strong>, not because the predictor misjudged the initial probability, but because their forecast actively prompted a change that prevented the outcome.</p><p><strong>If the Prediction Is Ignored</strong><br>On the other hand, if the forecast is largely ignored and the outcome still occurs, then the prediction might be <strong>more right than chance would suggest</strong>. Here, the forecast stands as a rare instance where, despite being public knowledge, the outcome unfolds exactly as predicted.</p><p>This asymmetry means that the predictor is forced into a situation where their accuracy is skewed by the very influence of their prediction. They are either penalized by a self-defeating prophecy — making them more wrong than expected — or they achieve an extraordinary level of accuracy when their forecast goes unheeded.</p><h3>Measuring Prediction Accuracy and the Role of Superforecasting</h3><p>In many fields, especially those dealing with uncertainty, we rely on quantitative measures like the <strong>Brier Score</strong> or <strong>log scoring rules</strong> to assess prediction accuracy. These metrics help us gauge how closely a forecast aligns with the eventual outcome — a critical tool in both human judgment and AI-driven predictions (like those at nPlan).</p><p>This is where the concept of <strong>Superforecasting</strong> enters the discussion. In <em>Superforecasting: The Art and Science of Prediction</em> by Philip Tetlock and Dan Gardner, we learn about forecasters who consistently achieve remarkable accuracy. Their success isn’t just a matter of intuition; it’s built on systematic methods, continuous belief updating in the face of new evidence, and rigorous probabilistic reasoning.</p><p>However, the prophet’s paradox adds a twist to this narrative. Superforecasters typically excel in environments where their predictions don’t significantly alter the outcome. In many real-world scenarios — such as forecasting megaproject outcomes or economic shifts — the very act of making a prediction can influence the situation. If a forecast becomes widely known, it may trigger actions that either mitigate or completely negate the predicted result.</p><p>Thus, while traditional scoring methods like the Brier Score are incredibly valuable, they can be challenged in dynamic settings where predictions themselves impact behavior.</p><h3>Implications for Forecasting in Dynamic Environments</h3><p>Understanding the interplay between prediction and behavioral response has profound implications:</p><ul><li><strong>Economic and Political Forecasting:</strong><br> When forecasters predict significant economic shifts or political upheavals, their forecasts can sometimes act as catalysts for change. For example, if an AI predicts delays in a megaproject, the forecast might prompt project managers to adjust resource allocation or implement new strategies to avoid the predicted issues.</li><li><strong>The Challenge of Dynamic Systems:</strong><br> In these dynamic, human-influenced systems, the relationship between forecast and outcome isn’t straightforward. While rigorous forecasting methods remain essential, we must also account for the fact that our predictions can change the very environment they aim to predict.</li></ul><p>Superforecasting teaches us that forecasts should be seen as evolving tools rather than fixed endpoints. By continuously updating our beliefs and planning for multiple scenarios, we can navigate the inherent uncertainty — even when our predictions are part of the feedback loop.</p><h3>Conclusion</h3><p>The prophet’s paradox challenges us to reconsider the nature of predictions in a world where human behavior plays a critical role. It underscores the idea that our forecasts, especially negative ones, can prompt actions that avert the very outcomes they predict.</p><p>Moreover, this paradox forces predictors into an accuracy paradox: if their forecast influences behavior, they risk being <strong>more wrong than they should be</strong>, yet if it is ignored and the outcome occurs, they might be <strong>more right than chance would allow</strong>. This inherent asymmetry highlights the complex dance between expectation, action, and chance in dynamic systems.</p><p>I’d love to hear your thoughts — have you ever seen a prediction influence behavior in unexpected ways? Share your insights and experiences in the comments below!</p><p><em>If you found this discussion thought-provoking, please consider sharing it or following for more insights on forecasting, AI, and the fascinating interplay between prediction and human behavior.</em></p><p>Happy forecasting!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=05806fa9c83b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/the-prophets-dilemma-when-predictions-change-the-future-05806fa9c83b">The Prophet’s Dilemma: When Predictions Change the Future</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Forward-lookingProactive Culture in Project Controls]]></title>
            <link>https://medium.com/nplan/a-forward-lookingproactive-culture-in-project-controls-c77465293fbc?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/c77465293fbc</guid>
            <category><![CDATA[proejct-controls]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[project-management]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Sun, 14 Apr 2024 17:31:37 GMT</pubDate>
            <atom:updated>2024-04-18T20:52:30.972Z</atom:updated>
            <content:encoded><![CDATA[<h3>A forward-looking, proactive culture in project controls</h3><p><em>This article was written with valuable contributions from John Hollmann, </em>author of “<em>Project Risk Quantification” (2016) and lead author of AACE® International’s TCM Framework (2006). John is the owner of </em><a href="http://www.validest.com/"><em>Validation Estimating, LLC</em></a><em> since 2005, and works with capital program managers and project leaders in a variety of industries from process to infrastructure to improve their cost engineering practices; i.e., estimating, risk quantification, project control, etc.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*JDaU7dmFe5EQIUvh.jpg" /></figure><p>Time and time again we have seen projects be affected by delays, cost overruns, bad practices, and all sorts of high consequence issues. One might argue these are some of the reasons why Project Controls exists in the first place. Rather than talk about some specific ad-hoc solution to particular problems (we all do a lot of this), we thought that a frank and honest reflection on the culture of Project Controls with a focus on risks might be useful, together with a vision for how mindsets and attitudes can change for the betterment of our projects. This reflection includes a borrowed idea, the HSE Culture Ladder, and adapts it Project Controls and the Risk Maturity model.</p><h3>The HSE Culture Ladder</h3><p>The HSE Culture Ladder is a model used to assess and improve the maturity of an organisation’s Health, Safety, and Environment (HSE) culture, which has been consistently used over the years, as a framework for teams to change from within. This ladder has multiple steps in it, which an organisation may aspire to climb.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/0*7hqRX3juH_KAJxrf.png" /><figcaption>The HSE Culture Ladder</figcaption></figure><p>The ladder consists of five progressive stages:</p><ul><li>Pathological: Safety is disregarded, and blame is placed on individuals for incidents</li><li>Reactive: Safety is only addressed after incidents occur</li><li>Calculative: Organizations actively measure and manage risks through systems and procedures</li><li>Proactive: HSE is embedded in the organization’s culture and continuous improvement is sought</li><li>Generative: HSE is an intrinsic value, and the organization demonstrates exceptional performance and learning.</li></ul><p>It is typically thought of as a progression (hence the name “ladder”), where an organisation can only get to a certain step by having improved and reached the previous one. For example, it is impossible to go from reactive to generative without changing the organization to also be calculative and then proactive. By identifying their position on the ladder, organizations can develop strategies to enhance their HSE culture, ultimately reducing incidents and fostering a safer work environment.</p><h3>Adapting the HSE culture ladder for project controls</h3><p>We can attempt to adapt this culture ladder for Project Controls, by keeping the names of the levels and then describing what this means for our specific domain:</p><p><strong>Pathological<br></strong>Delays and cost overruns are masked, blame is placed on individuals or groups, and everyone’s primary directive is to avoid being the one that gets blamed. This type of organisation lacks accountability for project outcomes. Project controls is of questionable value to the project and team. There is no explicit risk management role in project controls or the team.</p><p><strong>Reactive<br></strong>Project controls are implemented sporadically, mainly as a response to project failures or significant deviations from the plan. The focus remains on firefighting and resolving immediate issues and issuing the required monthly report without addressing underlying causes.<br>Project controls are somewhat valued members of the team. There is no risk management role in project controls, but there may be a risk function they coordinate with.</p><p><strong>Calculative<br></strong>The organisation has established systems and procedures for project controls, including scope, schedule, cost, and risk management. Metrics and performance indicators are tracked and analyzed regularly beyond just the required monthly report to ensure projects stay on track, and corrective actions are taken as needed.<br>Project controls are valued members of the team. There is no risk management role in project controls but they work closely with the risk management function.</p><p><strong>Proactive<br></strong>The organisation is actively gathering intelligence about the causes of performance issues and looking ahead to identify emerging risks, and creates plans to mitigate future risks. Metrics and KPIs all have future projections that guide decision making.<br>Project controls are highly valued members of the team. Project controls and risk management are integrated.</p><p><strong>Generative<br></strong>The organisation is envisioning and preemptively discussing many different versions of the future — event-based scenarios, “what-if”s, design adjustments, and alternative methods of delivery. Each version of the future comes with its current expected projection for metrics and KPIs.<br>Project controls are essential, sought out members of the team. Project controls and risk management are highly integrated.</p><p>It is generally thought that most project controls functions operate at the reactive level currently, though we aspire to become more proactive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*Wp4gmpNXA7LOmGI2" /><figcaption>The IRM Risk Maturity framework is somewhat similar to a direct application of the Dreyfus model to Risk Management</figcaption></figure><p>By the way, the Institute of Risk Management’s <a href="https://www.theirm.org/media/4512/irm-charities-sig-risk-maturity-framework-final.pdf">Risk Maturity Framework</a> provides what I think is a complementary framework for assessing and improving an organisation’s risk management capabilities across people, process, and technology dimensions. They also provide <a href="https://www.theirm.org/media/9167/irm-getting-better-flyer-final-march-2016.pdf">guidelines</a> on how to improve on that scale.</p><p>Any organization seeking to enhance its project controls culture can leverage frameworks like the Risk Maturity Framework to ensure their risk management capabilities mature in parallel. For instance, a reactive project controls culture maps roughly to a “conscious” risk maturity. Progressing towards proactive project controls requires reaching the “expert” level of risk maturity (and integration of risk management in project controls during execution), where anticipating and mitigating risks is second nature.</p><p>By considering both cultural and capability maturity, organizations can take a holistic approach to transforming project controls into a proactive function focused on early risk identification and resilience. In the following sections, we will explore what it means to have reactive versus proactive project controls, including the limitations of reactive controls and recommendations for cultivating a more proactive discipline.</p><h3>Reactive Project Controls — where most of us are</h3><p>The existing culture in Project Controls is predominantly geared towards reporting, often emphasising the generation of detailed progress reports, cost and schedule analyses, and other performance metrics. This approach, while invaluable for keeping stakeholders informed, has inadvertently fostered a mindset that places more importance on capturing and documenting issues after they occur rather than working to prevent them in the first place. Risk registers are often not updated once construction begins. This has shifted the focus away from proactively identifying risks and implementing strategies to address them before they impact project performance.</p><p>I know it’s not because we choose to do things this way. I am of the opinion that this is due, at least in part, to the pressure faced by project teams to meet tight deadlines and satisfy (realistic? unrealistic? I’m not going to take a position on this particular debate right now) stakeholder expectations. In an attempt to balance the competing demands of time, cost, and quality, project teams often find themselves grappling with unforeseen challenges and deviations from the project plan. Getting the monthly report done on time too often becomes as much as one can do. This is when the primary focus of Project Controls becomes managing issues and their consequences, leaving little to no room for prevention and proactive risk management.</p><p>The time is now to implement more proactive risk management strategies integrated in project controls. This change should involve a greater emphasis on risk identification and mitigation not only during the planning but also the execution phases of projects. If we integrate risk management strategies and proactively address potential issues, Project Controls can evolve from a reactive reporting function to a forward-thinking, prevention-oriented discipline that anticipates and mitigates risks before they materialize. In turn, this will lead to more successful projects.</p><h3>Proactive Project Controls — where we want to be</h3><p>What does it mean to be proactive about risk? Being proactive about risk means anticipating, identifying, and addressing potential threats and uncertainties before they materialise and negatively impact a project or organisation. Proactive risk management is all about taking a forward-looking approach to risk, focusing on prevention and mitigation, rather than merely reacting to issues as they arise.</p><p>There are a few ways in which proactivity can be applied to the different areas of Risk Management:</p><p><strong>Risk Identification</strong>: Proactively identifying risks involves systematically scanning the internal and external environment for potential threats and uncertainties that could affect the project or organisation. Participate in regular status meetings and get first-hand intelligence. This process includes analysing historical data, industry trends, and other relevant information to uncover potential risks.</p><p><strong>Risk Assessment</strong>: Once potential risks are identified, proactive risk management involves assessing the likelihood and impact of these risks. This includes not only event risks, but also system attributes (e.g., weak teams) which result in greater uncertainty throughout the project. This requires evaluating the probability of each risk occurring, as well as the potential consequences if the risk materializes. Risk assessment helps prioritize risks and allocate resources effectively. When decisions need to be made (e.g., major changes), that is when the quantitative risk analysis (QRA) step of assessment comes in.</p><p><strong>Risk Mitigation</strong>: Developing and implementing risk mitigation strategies is a critical aspect of proactive risk management. This may involve taking preventive measures to reduce the likelihood of risks occurring or developing contingency plans to minimize the impact if they do materialize. Risk mitigation strategies should be flexible and adaptable, allowing for adjustments as new information becomes available or as conditions change.</p><p><strong>Continuous Monitoring</strong>: Proactive risk management requires ongoing monitoring of risks and the effectiveness of mitigation strategies. This involves tracking risk indicators, evaluating the performance of risk mitigation measures, and adjusting strategies as needed based on new data or changes in the project or organisational environment.</p><p><strong>Communication and Collaboration</strong>: Being proactive about risk necessitates open and effective communication among team members, stakeholders, and other relevant parties. This fosters a shared understanding of risks and the importance of proactive risk management, ensuring that all parties are engaged in the process and aligned on the risk mitigation strategies.</p><p><strong>Learning and Improvement</strong>: A proactive approach to risk management emphasises continuous learning and improvement. This involves learning from past experiences, as well as benchmarking against industry best practices, to refine risk identification, assessment, and mitigation processes over time.</p><p>By embracing a proactive approach to risk management, organizations can minimize the potential negative impacts of risks, improve project performance, and increase their overall resilience in the face of uncertainty.</p><h3>Why forecasts alone are not enough</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vslQZ7rnn40XKXyPe2SCKw.png" /><figcaption>An example end date forecast: we reported it, so now what? The decisions that descend from this are more important than the forecast itself or even its accuracy.</figcaption></figure><p>Forecasting plays a crucial role in identifying potential risks and predicting future events. It is a key project control responsibility, but as with risk management, it is often poorly done. Reactive cultures use magic thinking assuming performance will recover because it has to. Calculative cultures often just extrapolate current performance. But even if done well, we don’t think it’s enough on its own to ensure the success of a project. Forecasting serves as a valuable input for decision-making, but the real value comes from taking appropriate actions based on analysis to address the potential risks and uncertainties that the forecasts reveal. After all if you don’t make a change, well, nothing changes.</p><p>Even worse, focusing solely on forecasting without analysis and taking corresponding actions can lead to a false sense of security, as the project team may believe that through working harder the forecast will come true without actually addressing risks at all! Of course, the decisions of whether and how to act are not the responsibility of project controls, but strong project controls make effective decision making more likely.</p><p><em>Prevention strategies </em>(sometimes referred to as “risk treatmens”), on the other hand, focus on reducing the likelihood of risks occurring in the first place. These actions can involve improving elements of the project system such as processes, enhancing communication and collaboration, or investing in training and development to strengthen the project team’s ability to identify and address potential issues before they escalate. By emphasizing prevention, project teams can proactively reduce the chances of encountering major problems, ultimately improving project performance and success.</p><p><em>Mitigation strategies </em>(sometimes referred to as “risk responses”), contingency or contingent risk response planning, aim to reduce the impact of potential risks on a project, should they occur. These strategies can include reallocating resources, adjusting schedules, and/or implementing alternative solutions to minimize the negative consequences of risks. By incorporating planned mitigation actions into forecasting, project teams can ensure they are better prepared to manage risks and recover from potential disruptions.</p><p>In conclusion, while forecasting is an essential aspect of project controls, it must be complemented by a focus on actions like prenvention and mitigation to truly enhance the resilience and success of a project. By using forecasting insights to inform decision-making and develop proactive strategies, project teams can navigate uncertainties more effectively and improve overall project outcomes.</p><h3>Pre-mitigation</h3><p>“Pre-mitigation” (risk treatment), refers to the act of proactively identifying and addressing potential risks and issues before they materialise or escalate, in order to minimise their impact on a project. This approach emphasises prevention and early intervention, as opposed to responding to issues only after they have occurred. There are three ways you can address risks before they materialise:<br>1) Reduce the likelihood (prevention)<br>2) Reduce the severity (mitigation)<br>3) Reduce the impact (cushioning)</p><p>Pre-mitigation strategies encompass various actions and processes designed to reduce only some or all three of these variables. Here are some examples of pre-mitigation strategies, in descending order of effectiveness:</p><ul><li>Find workarounds or changes that neutralise the causes of some of the major risks, if plausible.</li><li>Plotting alternative courses of action and multiple possible responses for the major risks, and decide what the trigger points would be to change course.</li><li>Implementing redundancy in resource allocation and scheduling to absorb potential disruptions or ease up on aggressive strategies if work is complex and fragile.</li><li>Agree additional contingency or have insurance in place</li><li>Monitoring early warning lead-indicators / KPIs to start interventions at the first signs of trouble</li></ul><p>By focusing efforts on pre-mitigation as opposed to post-incident crisis management, as common in proactive project controls cultures, project teams can enhance resilience, reduce expensive firefighting, and improve the chances of project success.</p><h3>Improving contingency planning through forecasting and proactivity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*17QiEx69TlfyzdPpkfg1qg.png" /><figcaption>An example insight from nPlan Insights: use the forecast to highlight the areas of highest expected impact on the project outcomes.</figcaption></figure><p>Contingency planning, a crucial aspect of project controls, involves developing alternative strategies or plans to address potential residual risks or unforeseen events that may impact a project’s objectives. Improving contingency planning through better forecasting, analysis, and a proactive mindset can significantly enhance a project’s resilience and overall success.</p><p>Better forecasting plays a vital role in enhancing contingency planning by providing more accurate and timely predictions about the impact of potential risks and uncertainties considering risk response actions. By leveraging advanced tools and techniques, such as AI-led forecasting and predictive analytics, project teams can gain deeper insights into potential challenges and more effectively anticipate the likelihood and impact of various risks. With more accurate forecasts, project teams can develop more targeted and efficient contingency plans, ensuring that resources are allocated appropriately and potential issues are addressed effectively.</p><p>A proactive mindset, on the other hand, focuses on anticipating and addressing potential problems before they occur, rather than simply reacting to issues as they arise. By adopting a proactive approach to contingency planning, project teams can systematically identify potential risks, assess their impacts, and develop and test appropriate strategies to mitigate them. This forward-looking project controls approach not only helps project teams to better prepare for uncertainties but also fosters a culture of prevention and risk mitigation, ultimately reducing the likelihood and severity of delays and budget overruns.</p><p>Incorporating better forecasting and a proactive mindset into contingency planning can lead to more robust and effective plans, improving a project’s ability to adapt to unforeseen events and changes. As a result, projects are more likely to stay on track and achieve their objectives, ensuring greater overall success in the face of uncertainty.</p><h3>Scenario analysis for proactive project controls</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_S0WmOdMGscjkHWrk2xINg.png" /><figcaption>Scenario Analysis in nPlan Insights: comparing outcomes between different choices or possible future events</figcaption></figure><p>Scenario analysis is a powerful tool for proactive risk-aware project controls that helps project teams identify, assess, and manage potential risks and uncertainties. By exploring various plausible future scenarios, project teams can develop flexible strategies and contingency plans that account for a wide range of potential outcomes. This forward-looking approach enables project controls to anticipate and adapt to changes, improving project resilience and performance. And ultimately, better project controls analyses makes more effective decision making by the project manager more likely.</p><p>Scenario analysis involves steps such as identifying key drivers and uncertainties, developing diverse and plausible scenarios based on those factors, assessing the potential impact of each scenario, and formulating adaptive strategies that can be adjusted as conditions change. Continuous monitoring and updating of scenarios, as well as collaboration among stakeholders, are also critical for effective implementation. Such analysis calls for risk management to be integrated with project controls.</p><p>Incorporating scenario analysis into project controls can help project teams anticipate potential risks, develop adaptive strategies, and improve overall project performance. By embracing this proactive approach, project controls can shift from a reactive, reporting-focused function to a proactive, forward-thinking, prevention-oriented discipline.</p><h3>Robust Decision Making for proactive project controls</h3><p><em>If you want to know more, there is a fantastic discipline called </em><a href="https://www.deepuncertainty.org/"><em>DMDU</em></a><em> which I will cover in a future post. For now, we’ll look at one of the techniques in DMDU, called Robust Decision Making. The founders of the group also published an </em><a href="https://www.amazon.com/Decision-Making-Under-Deep-Uncertainty/dp/1013275594"><em>introductory book</em></a><em> to the topic that I thoroughly recommend.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/0*CoeG5LkyxH8bkpsS.png" /><figcaption>An overview of all the steps in Robust Decision Making</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Robust_decision-making">Robust Decision Making</a> (RDM) is a systematic approach to Decision-Making under Deep Uncertainty (DMDU), where the goal is to identify strategies that can perform well across a wide range of possible future conditions. Incorporating RDM into Project Controls can help transform the discipline into a more proactive function by enabling project teams to anticipate potential risks and develop flexible strategies to address them.</p><p>RDM enhances proactivity through comprehensive risk identification, scenario planning, adaptive management strategies, enhanced stakeholder collaboration, and continuous monitoring and learning. By incorporating RDM principles, project teams can improve the resilience and adaptability of their projects, allowing them to proactively respond to emerging risks and uncertainties. This ultimately increases the likelihood of successful project outcomes.</p><h3>Conclusion and Recommendations</h3><p>To foster a more proactive risk culture in Project Controls, organizations must focus on enhancing both cultural and capability maturity. Based on the analysis in this piece, we recommend the following actions that can progress Project Controls up the cultural ladder while also elevating risk management maturity:</p><ul><li>Provide training on proactive mindsets to instill a forward-looking culture focused on early risk identification and mitigation. This cultural shift from reactive to proactive aligns with progressing from novice to normalized risk maturity.</li><li>Implement robust systems and procedures for real-time risk management integrated with change control while tracking metrics to monitor effectiveness. Moving from calculative to proactive project controls requires achieving normalized and natural risk maturity.</li><li>Leverage advanced tools like AI and predictive analytics to enable data-driven forecasting and scenario planning. Adopting these technologies demonstrates natural risk maturity and a proactive project controls culture.</li><li>Incorporate adaptable decision frameworks like Robust Decision Making that perform well under uncertainty. This enhances resilience, fitting a proactive culture and natural risk maturity.</li><li>Structure processes to facilitate continuous risk monitoring, updating, and learning. Natural risk maturity and generative project controls culture are defined by embedded learning.</li><li>Promote collaboration between project teams, business management, leadership, and stakeholders on risk strategy. Cross-functional alignment on risk management supports natural maturity and proactive culture.</li><li>Implement performance incentives tied to proactive project controls integrated with risk management. Rewards for early identification and mitigation encourage desired cultural and maturity shifts.</li></ul><p>We have published <a href="https://www.nplan.io/knowledge-base/from-qsra-to-ai-sra">our own framework</a>, called AI-SRA, which leverages AI to improve already-existing QSRA methodologies in project schedule risk management, which is inspired by the principles in this document. We’d be humbled if you were to take a look.</p><p>With a combination of mutually reinforcing improvements to culture and capabilities, organizations can achieve higher levels of project controls and risk management maturity, enabling successful delivery amidst uncertainty. The key is approaching both transformations holistically rather than in silos.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c77465293fbc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/a-forward-lookingproactive-culture-in-project-controls-c77465293fbc">A Forward-lookingProactive Culture in Project Controls</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Barry update: A more sophisticated AI agent]]></title>
            <link>https://medium.com/nplan/barry-update-a-more-sophisticated-ai-agent-bea4ae6ff089?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/bea4ae6ff089</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[project-controls]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[construction-management]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Mon, 14 Aug 2023 10:36:59 GMT</pubDate>
            <atom:updated>2023-08-14T10:36:59.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*5nVV6WjaHzP3LdE8PQu8ow.png" /><figcaption>A new, more grounded, way of answering hypotheticals</figcaption></figure><p>We introduced Barry in May this year, and we’ve been updating its capabilities ever since. I don’t often talk about the internals, but this week I’d like to highlight some of what goes on under the hood.</p><p>Barry has always been an “AI agent”, not just an LLM. In fact, the LLM that underpins Barry is interchangeable, and we often make tweaks and updates to how the LLMs are used. One core component of this agent architecture is the “planner”. This is an LLM that plans out a series of actions that are required to answer the question or task posed by the user. For example, an action may be to search the forecast for the details of certain activities, or a predecessor/successor network.</p><p>This is a massively complicated task, and until now we have been calling the planner once per question, and then executing its plan. This week, we made some changes that bring a massive improvement to the quality of responses you can see from Barry. The planner is now recursive, meaning that after executing each action, we ask it again what it should do next. One way we’ve already seen how this improves is when asking about network effects. Previously, the planner would have only been able to say “search for the successors of activities X,Y, and Z”. However, the planner does not know about all the activities in a schedule, so that search would be limited. Now, it is able to look at the successors of certain activities, reason about the impact, and follow the network in the next step of the loop.</p><p>Even more importantly, we’ve managed to do so without affecting the response time of simple questions — the planner knows when it’s done and it has sufficient data to answer a question.</p><p>The question was “how does a 1 month delay to CON-9800 affect the rest of the project?”</p><p>Before this change, we were getting a lot of hallucinations caused by the fact that the relevant data was not provided to the “frontend” LLM. See below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m5vocvDYwjtRqRase_HBHA.png" /><figcaption>The previous version was answering in a suboptimal way because it was not able to ground itself with sufficient data about the schedule network.</figcaption></figure><p>(Yes, we deliberately created a terrible schedule so there is lots of room for improvement)</p><p>I’ve included some screenshots of our development environment that highlight the change in reasoning capability. There were more steps than those in the screenshots, but you can already see how the new planner reasons about the results of each step to generate the next step.</p><p>Step 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHjmLBqf4kx8U1NVBZcKBQ.png" /></figure><p>Notice how the planning agent also “talks to itself”. This provides a couple of advantages: we always have a reason for why it created a specific plan. This is very useful for debugging, but also in the future we expect much richer communication to happen between the planner agent (or agents?) and the LLM that generates the output that you see in the platform.</p><p>Step 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnY5ydsoOy-R4UHUZyNCSA.png" /></figure><p>Notice how it is running some of the same commands again. This is ok, because we also instruct the planner to be thorough and err on the side of completeness, so it builds some redundancy in its plans. You or I would call it “being on the safe side”.</p><p>Answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q7teIRD7T56XUeOOs8cjRg.png" /></figure><p>As you can see, Barry is now gathering much more information and data from the platform, and the resulting reasoning is getting better and better. You can expect us to continuously make improvements like this. This one is easy to explain, so I decided to open the hood and show the difference in outcome, but there are plenty more — we update Barry’s engine on average about once or twice a day to make it more powerful!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bea4ae6ff089" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/barry-update-a-more-sophisticated-ai-agent-bea4ae6ff089">Barry update: A more sophisticated AI agent</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your startup is not a tech company]]></title>
            <link>https://medium.com/@nitbix/your-startup-is-not-a-tech-company-9a94a8c355b8?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a94a8c355b8</guid>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[venture-capital]]></category>
            <category><![CDATA[investment]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 15:25:52 GMT</pubDate>
            <atom:updated>2022-03-24T15:25:52.257Z</atom:updated>
            <content:encoded><![CDATA[<p>I came to this conclusion while re-reading Stewart Butterfield’s <a href="https://medium.com/@stewart/we-dont-sell-saddles-here-4c59524d650d">We don’t sell saddles here</a> — if you haven’t read it, it’s a great take on what startups have to do when they are creating a new category. It’s all about selling a vision rather than a product, and I believe this to be fundamentally true. All the greatest companies in the world started by having to create a new category. To mock Silicon Valley a bit (both the place and the show), if you want to <em>change the world</em> you can’t be doing the same thing as someone else. In his post, Stewart refers to another oft-quoted article by Marc Andreessen of A16Z about <a href="https://pmarchive.com/guide_to_startups_part4.html">Product/Market Fit</a>.</p><p>In these two posts we find some of the roots of the current thinking that most people have when looking at the phases of the life of startups:</p><ol><li>Find Product/Market Fit</li><li>Then, Scale up</li></ol><p>At some point there has even been a switch from calling these companies <em>startups</em> to <em>scaleups</em> to emphasise the difference between the two modes of operation. However, if you look at how the best tech companies grew, it was never in this two-phase approach. All the major tech companies followed a different recipe, where they had to create a new technology, a new category, and then continued creating new technology.</p><p>Most SaaS startups today that follow Andreessen’s two-phase model, are not in fact what I would call tech companies. They use existing technologies to solve a valuable business problem, by building a product. I would call these <em>product companies</em>.</p><p>Please don’t misinterpret what I’m trying to say, these companies are very valuable and produce solutions that have a huge impact on the world. The majority of the VC industry is built on these types of companies scaling quickly and exiting at huge valuations. I love that they exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Kdz6hoctAXI-ho8S" /><figcaption>Space is an industry where we have already started seeing the emergence of the next wave of tech companies like SpaceX</figcaption></figure><p>If we look at the tech Giants like Apple, Google, and others, they are undisputedly <em>technology-first companies.</em> This feels very different from “build a product then scale the sales”, and if we look at each of their individual journeys, they aren’t likely to have followed this path. The ones that initially did, like Facebook, quickly mutated after becoming successful, into technology-first companies — as demonstrated that it renamed itself to Meta and is now focusing all its energy on becoming the platform that runs the Metaverse.</p><p>So, if you want to build a tech company, it’s likely that you will have to first build a <em>startup, </em>but when you manage to scale there will be another shift that your company will have to make, into producing new technology by itself.</p><p>The other option is to start by being a tech company from the beginning. This is a far more risky proposition: you will have to spend time and money building something new, which nobody knows how it will work, at a time when you have neither time nor money. You will need deep pocketed investors with an even deeper belief in your ability to build something that most likely includes scientific discovery and novel engineering solutions.</p><p>In conclusion, I believe that there already are many valuable product startups which are producing huge value for VCs and founders, and we need to start finding ways to create more tech companies from it. These are the companies that will force the largest leaps in human evolution, especially when it comes to lifespan and life quality improvement, as well as becoming a spacefaring civilisation. We have seen this happen in telecoms and artificial intelligence already, so we know how to build these companies. We just need to shift the incentives, and what better than starting to aim for trillion dollar exits instead of billions?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a94a8c355b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why AI is more accurate than humans at probabilistic risk forecasting]]></title>
            <link>https://medium.com/nplan/why-ai-is-more-accurate-than-humans-at-probabilistic-risk-forecasting-2cbe2baae02a?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cbe2baae02a</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[construction-management]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[project-controls]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 22:30:59 GMT</pubDate>
            <atom:updated>2021-04-19T22:30:59.638Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1: Machines vs People</p><p><em>Many thanks to my fantastic colleague </em><a href="https://medium.com/@georgia.stillwell"><em>Georgia Stillwell</em></a><em>, who co-authored this article with me.</em></p><p>nPlan is in the business of forecasting risk on projects, by using machine learning to predict the outcomes of construction schedules. We spend a lot of time thinking about subjectivity, forecasting accuracy, schedule quality, and the way people think about uncertainty and artificial intelligence.</p><p>People often ask two questions during the early phases of an engagement with us:</p><ul><li>How can an algorithm be better than an expert at forecasting risk?</li><li>How can your algorithm work on my unique project?</li></ul><p>We’ll look at both these questions, providing insight from two angles: my technical experience and Georgia’s risk analysis background.</p><p>In part 1, we’ll address the first question.</p><h3>How can an algorithm be better than an expert?</h3><p>To understand why a machine could outperform project experts, we first need to know how professionals have an impact on a ‘traditional’ risk management process. Today, most professional bodies (including the IRM, ISO, Prince2, API) will prescribe a risk management processing using the following steps:</p><ul><li>Identification</li><li>Evaluation/assessment</li><li>Analysis</li><li>Treatment</li><li>Review (rinse and repeat)</li></ul><p>We can start by looking at the team’s role and influence in the first two steps: identification and assessment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*rYXmLh5NJs9aJtg_kqZGPw.png" /><figcaption>A typical risk assessment process, where specific risk events are identified and subjectively quantified.</figcaption></figure><p>This process is pretty much the same everywhere no matter the project size. From the small (£20M) to the mega (£10B+) schemes, this means that the project teams are potentially making the same mistakes over and over again. We believe this to be one of the reasons why we still see project delays, even though delays in projects have been ‘a thing’ for decades. The problem is that we are asking humans to identify risks, the likelihood of those risks occurring, and the impact of those risks if they materialise. This creates not just one, but three points of subjectivity for each risk item, and many stirling opportunities to mess up the inputs to a risk analysis.</p><p><em>But I’ve got the best tunnelling team ever, and they are estimating these risks, not guessing them!</em></p><h4>Superforecasting</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3XPV4sR3iAIoLRpu.jpg" /></figure><p>This is probably as good a time as any to mention that humans are terrible at estimating probabilities and risk. In the book <a href="https://hbr.org/2016/05/superforecasting-how-to-upgrade-your-companys-judgment">Superforecasting</a>, the authors talk at length about several years of research into understanding what makes some people incredibly good at creating forecasts, whilst most are not better than randomly guessing outcomes. We recommend that if you are serious about risk and forecasting, you put this book at the top of your reading pile. The tl;dr of it is that it is all about estimating <em>uncertainty, </em>and doing so in a way that is consistent with your own track record of estimation. It turns out that, for example, experts are terrible at this, and are consistently overconfident and ultimately wrong. However, the way we appraise experts makes us vulnerable to confirmation bias, and we ultimately find signal in the noise by praising people for ultimately just guessing right many times in a row.</p><h4>Uncertainty, Forecasting and Risk</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*1i6rHcRj29oh2XYs.png" /><figcaption>How distributions change with changes in Volatility and Variance, whilst keeping the same mean value.</figcaption></figure><p>When we talk about any events in the future, we must consider all three. They exist together and are forever entangled.</p><p><strong>Risk</strong>, defined as broadly as possible, is the potential for deviation from an intended plan or result.</p><p><strong>Forecasting </strong>is the act of producing a probabilistic view over any future events.</p><p><strong>Uncertainty </strong>is the variability of any outcome, and is a direct consequence of Risk. It is also included as a parameter when producing forecasts.</p><p>The higher the Risk, the higher the Uncertainty, the wider the forecasts.</p><h4>Human perception of probability</h4><p>Humans, as we previously stated, are not good at estimating probabilities. Especially small probabilities. If something is 90% probable, we all tend to feel very secure about it, and are not able to internalise the fact that one in ten times that something will not occur. It is important to accept it as a fact, which comes from our own species’ evolution, that humans will always be overconfident forecasters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*isKoJQEyxgl90RtZ" /><figcaption>Humans perceive probabilities very differently, leading to conflicting and inaccurate estimations of uncertainty. See more in <a href="https://www.linkedin.com/pulse/human-perception-probability-mirko-mazzoleni/">Mirko Mazzoleni’s post</a> on the matter.</figcaption></figure><p>Many of you may recall sitting in a risk workshop with a risk manager/analyst/engineer probing your mind for a probability of some sort of event like sprayed concrete lining collapse, as if you were meant to know that intuitively. “<em>I’ve seen it once in the five major tunneling jobs I’ve worked on, so there’d probably be a one in five or 20% chance”</em> is the best anyone can do here. The probability may vary, but that doesn’t make it any more accurate. With effects like group think and anchoring (and all other cognitive biases too), the team is only likely to adjust in relation to that starting 20%, which is another flaw in the way we make judgements. This process goes is repeated until we have assessed all the risks we guessed, and married them up with guessed probabilities and impacts.</p><p>These “probabilities” are then added to a monte carlo simulation, on which billion dollar decisions are made. Are you gawking? You should be. Big financial decisions are based on subjective inputs like this every day! Also in this process, we completely ignored all catastrophic risks as well as the low-likelihood high-impact risks which were too hard to estimate — <a href="https://medium.com/nplan/on-the-effect-of-black-swans-on-projects-532c887bfe04">the Black Swans</a>, plus any others that were just unknown to the project team.</p><h4>Machine Learning and Forecasting</h4><p>Machine Learning, in short, is the act of taking data with known outcomes, following a certain set of algorithms, and producing a model that has somewhat internalised the nature of the problem at hand as a mathematical model. These models are used to create predictions about new data which was previously unseen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*McrKkE6tUiyDkm9r.png" /><figcaption>Probabilistic Forecasting assigns varying probabilities to future states (or in fewer cases events), on a continuous gradient, so that practitioners can reason about how the estimated variance and volatility have an effect on outcomes.</figcaption></figure><p>ML models can also be created in such a way that they are producing forecasts, with the associated uncertainty, guaranteeing <em>true calibration. </em>A calibrated model is a model whose uncertainty is correct. So if a model says there is 10% probability of some event occurring, when we measure over all of the same predictions, that event happens 10% of the time.</p><h4>The machine is your friend</h4><p>If we now think about the process that professionals follow today, two issues appear:</p><ul><li>The teams of experts thinking about risk are doing so by coming up with a limited set of “potential” events, instead of thinking about all possible occurrences and taking and “end to end” view of (in nPlan’s case) project delay. It is guaranteed that there will be “risks” that have been missed.</li><li>Humans are not good at forecasting because of their inherent overconfidence, which leads to traditional forecasting exercises to have severely underestimated risk.</li></ul><p>We trained our ML models to be the Risk Manager’s friend. We trained on over half a million project schedules of projects ranging from datacentres to railways, and have taught our models to provide calibrated forecasts with accurate estimates of uncertainty (if you’ve read Superforecasting, we actually measure the Brier Score of our models).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*3rX46MPOghJpMfSK7TVVIQ.png" /><figcaption>nPlan uses the outputs of the ML models and simulation engines to recommend improvements that reduce the volatility of a project schedule.</figcaption></figure><p>Rather than competing with the Risk Manager, this approach empowers her with risk forecasts that are derived from data instead of opinions. This allows her then to focus on the actions that the model will never be able to take, which in our view are a far more valuable use of one’s time: the exploration, and the decisions. By being able to understand how schedule changes and progress updates affect the uncertainty of a project’s outcomes, we want to help everyone be proactive in reducing said uncertainty.</p><p>Once the uncertainty is below acceptable thresholds, we observe that activities stop interfering with each other and the schedule becomes “stable”. It is then truly effective to adopt the existing and well-known project management techniques such as Critical Path Methodology (CPM), and focus on improving the efficiency of the project, because we are no longer affected by wild changes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cbe2baae02a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/why-ai-is-more-accurate-than-humans-at-probabilistic-risk-forecasting-2cbe2baae02a">Why AI is more accurate than humans at probabilistic risk forecasting</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AoC 2020 Day 1]]></title>
            <link>https://medium.com/nplan/aoc-2020-day-1-ad1e78203dd?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad1e78203dd</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[advent-of-code]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[advent-of-code-2020]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 20:16:29 GMT</pubDate>
            <atom:updated>2021-01-13T20:17:16.883Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This is part of a series we did on the Advent of Code 2020 problems. You can see a list of all the problems we wrote about <a href="https://medium.com/nplan/advent-of-code-2020-at-nplan-1c4ab0f8b8fa">here</a>. <br>If you are a fan of AoC, and had fun working on these problems, it is likely that you would be a good fit with our team. Our engineering team had a blast with it, as we like to be challenged with hard problems. If you like what you see, we are constantly <a href="https://nplan.io/careers">looking for great talent</a>, please consider joining us!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*PSZM4flQmzf-PVkfDIhJeg.png" /><figcaption>This story is part of our series on AoC 2020</figcaption></figure><p>This exercise, being the first in the series, was designed to warm everyone up. I’m writing about it because it gives a nice introduction to the format, and there is a non-obvious optimisation to speed up the solution.</p><p>The problem, like all AoC problems, is divided in two parts. You will only get to see the second problem after solving the first one correctly. I’ll be using OCaml for this particular problem, because it is fun to use new languages, and I’ve been a fan of functional programming for a very long while. After getting set up with the basic toolchain I was ready to go and experiment with finding a solution.</p><h3>Part 1</h3><p>There is a <a href="https://adventofcode.com/2020/day/1">nice description</a> on the AoC site which involves a story Elves in accounting. Ultimately, the problem boils down to:</p><ul><li>We have a text file with one number per line</li><li>We are asked to find two numbers that sum to 2020 and return their product</li></ul><p>I saved the input into a input.txt file so I wouldn’t have to write boilerplate code to read straight from the URL. I think this is an acceptable shortcut given that the fun lays in finding the algorithm.</p><p>Reading the file is then pretty trivial, and we store it in a entries variable of time Int list.</p><pre>open Core;;</pre><pre>let filename = &quot;input.txt&quot; in<br>let flines = In_channel.read_lines filename in<br>let entries = List.map flines ~f:Int.of_string in</pre><pre>() (* print the list however you want)</pre><p>Once we can confirm that we can read the file correctly, the naive strategy in an imperative paradigm would be to have two nested loops and check the sum of the two numbers. We would want to terminate the loops early when a solution is found. We can translate this to a functional solution by making use of tail recursion, list unpacking, and mapping the result to an Int Option so that we can represent whenever a solution is not found. The full solution looks like this.</p><pre>open Core;;</pre><pre>let filename = &quot;input.txt&quot; in<br>let flines = In_channel.read_lines filename in<br>let entries = List.map flines ~f:Int.of_string in<br>let target = 2020 in</pre><pre>let rec check target a b =<br>  match a with<br>  | [] -&gt; None<br>  | hd_a :: tl_a -&gt;<br>      match b with<br>      | [] -&gt; check tl_a entries<br>      | hd_b :: tl_b -&gt;<br>          match hd_a + hd_b with<br>          | target -&gt; Option.some (hd_a * hd_b)<br>          | _ -&gt; check target a tl_b<br>in</pre><pre>let r = match (check target entries entries) with<br>| Some x -&gt; Int.to_string x<br>| None -&gt; &quot;not found&quot;<br>in</pre><pre>print_string r</pre><p>This is also O(n²), and OCaml has very good tail call optimisation, so we do not need to worry about blowing the stack. However, there is a trick that may not be obvious to make the solution O(n). We can search a hash for membership of (target — a) , which is a O(1) operation if we use the right type of Hash. In OCaml this is called a Hash_set. In the imperative world, this is equivalent to looping only once, which confirms that our solution is now O(n). Notice how we have to do an imperative operation to pre-populate the Hash_set.</p><pre>open Core;;</pre><pre>let filename = &quot;input.txt&quot; in<br>let flines = In_channel.read_lines filename in<br>let entries = List.map flines ~f:Int.of_string in</pre><pre>let seen =<br>  Hash_set.create ~size:(List.length entries) (module Int)<br>in<br>let () =<br> List.iter entries ~f:(fun elem -&gt; Hash_set.add seen elem)<br>in</pre><pre>let rec check a =                                                                                                                                      match a with<br>  | [] -&gt; None<br>  | hd_a :: tl_a -&gt;<br>    if (Hash_set.mem seen (2020 - hd_a)) then<br>      Some (hd_a * (2020 - hd_a))<br>    else check tl_a<br>in</pre><pre>let r = match (check entries) with<br>  | Some x -&gt; Int.to_string x<br>  | None -&gt; &quot;not found&quot;                                                                                                                                in<br>print_string r</pre><h3>Part 2</h3><p>Once we complete part 1, we are presented with a slightly more complex version of the same problem. Now the number of entries we need to check is three. The solution doesn’t change much, even from the optimised version above. However, we do have to iterate the list twice, which means adding a helper function to do the “outer loop”.</p><pre>open Core;;</pre><pre>let filename = &quot;input.txt&quot; in<br>let flines = In_channel.read_lines filename in<br>let entries = List.map flines ~f:Int.of_string in</pre><pre>let seen = <br>  Hash_set.create ~size:(List.length entries) (module Int)<br>in<br>let () =<br>  List.iter entries ~f:(fun elem -&gt; Hash_set.add seen elem)<br>in</pre><pre>let rec check a b =<br>  match a with<br>  | [] -&gt; None<br>  | hd_a :: tl_a -&gt;<br>      if (Hash_set.mem seen (2020 - hd_a - b)) then<br>        Some (hd_a * (2020 - hd_a - b) * b)<br>else<br>  check tl_a b<br>in</pre><pre>let rec check2 a =<br>  match a with<br>    | [] -&gt; None<br>    | hd_a :: tl_a -&gt;<br>      match check entries hd_a with<br>        | Some x -&gt; Some x<br>        | None -&gt; check2 tl_a<br>      in</pre><pre>let r = match (check2 entries) with<br>  | Some x -&gt; Int.to_string x<br>  | None -&gt; &quot;not found&quot;<br>in<br>print_string r</pre><p>I hope you enjoyed reading about this problem. I had fun working on it, and I would encourage you to take a look at <a href="https://medium.com/nplan/advent-of-code-2020-at-nplan-1c4ab0f8b8fa">the other entries in this series</a>. If you are passionate about software engineering and solving difficult problems, we are <a href="https://nplan.io/careers">constantly hiring</a> for talented engineers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad1e78203dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/aoc-2020-day-1-ad1e78203dd">AoC 2020 Day 1</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advent of Code 2020 at nPlan]]></title>
            <link>https://medium.com/nplan/advent-of-code-2020-at-nplan-1c4ab0f8b8fa?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c4ab0f8b8fa</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[advent-of-code]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 19:51:17 GMT</pubDate>
            <atom:updated>2021-01-28T00:21:20.366Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*PSZM4flQmzf-PVkfDIhJeg.png" /><figcaption>This story is part of our series on AoC 2020</figcaption></figure><p>I’m sure many people in software engineering are familiar with <a href="https://adventofcode.com/2020">Advent of Code</a>. For those of you that aren’t, it’s a yearly competition that happens in December, with 25 open problems that need to be solved, usually with code. The problems look a lot like <a href="https://projecteuler.net/">Project Euler</a>, where the method used doesn’t matter, you only have to get to the correct answer.</p><p>Naturally, AoC quickly became a craze in our engineering team, especially because we “unoficially sanctioned” it as a way to feed some healthy competitiveness in the engineering team. We gave the freedom to everyone to choose what language they used, and some of us took the offer more seriously than others with some problems solved in Excel.</p><p>We’ll be writing up our solutions over the next few weeks, and you can find an up-to-date index to all the individual articles at the bottom of this page. We are not going to publish solutions to all problems; only those that we found particularly entertaining or we solved in an usual way (yes, we will talk about the solution in Excel). If you don’t see a particular problem that you’re interested, keep checking in, we might still be writing the article. Or drop us a line!</p><p>The purpose of this series is to inspire others to participate more in these fun exercises. Our engineering team had a blast with it, as we like to be challenged with hard problems. If you like what you see, we are constantly <a href="https://nplan.io/careers">looking for great talent</a>, please consider joining us!</p><h3>All our AoC solutions</h3><p><a href="https://medium.com/nplan/aoc-2020-day-1-ad1e78203dd">Day 1</a><br><a href="https://medium.com/nplan/aoc-2020-day-3-line-animation-using-visx-and-react-motion-9e6297910bf8">Day 3</a><br><a href="https://medium.com/nplan/aoc-2020-day-4-a-factory-of-enums-112742953f94">Day 4</a><br><a href="https://medium.com/nplan/aoc-2020-day-18-shunting-yard-algorithm-with-rust-6439cb02dbad">Day 18</a><br><a href="https://medium.com/nplan/aoc-2020-day-24-representing-hexagonal-tiling-with-cube-coordinates-bf425a8d6ee4">Day 24</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c4ab0f8b8fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/advent-of-code-2020-at-nplan-1c4ab0f8b8fa">Advent of Code 2020 at nPlan</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Deep Tech part of building a Deep Tech company (part 2)]]></title>
            <link>https://medium.com/nplan/the-deep-tech-part-of-building-a-deep-tech-company-part-2-7c192f658d7d?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c192f658d7d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Sun, 23 Feb 2020 01:23:02 GMT</pubDate>
            <atom:updated>2020-02-26T14:19:54.235Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/nplan/the-deep-tech-part-of-building-a-deep-tech-company-part-1-e671ea7a212a">previous episode</a>, I told the story of how we collected and created a giant dataset to prove that our problem was solvable, and that construction schedules were a learnable input. In this episode, I’ll tell you more about how we built our first models that led us to show that it works and raise capital to build the company.</p><h3>Can we do some ML now please?</h3><p>We gave ourselves the mammoth task of creating a pipeline we could use to run experiments, and to create a finalised model that would go into the product, whilst facing the same scale problems that affected all our other endeavours. We were in fact not building one single model, but a structure made up of multiple modules, such as word and paragraph embeddings, and multiple classifiers on various targets.</p><p>We had to train one component at a time, and find an objective for each one that would correlate to the predictive ability of its successors in the pipeline. We struggled for a while on what would be the best architectures and the best strategy, and finally converged on a technique that I call “proxy evaluation”. For every step that is not being currently trained, we optimise hyper-parameters by attaching the rest of the trained architecture and seeing if that improves the results of the final output. It’s indirect, it’s inefficient, and it’s just plain wrong if you are looking for a truly optimal solution. However, at this particular point in our history we were looking for something that would <em>just do the job, </em>and this did the job just fine. We’ve since found ways of improving this simple greedy strategy, and we continue to constantly work on improving our pipeline.</p><p>Some of the scaling problems we had to face on training individual models were (in no particular order):</p><ul><li>The training set did not fit in main memory, so we had to stream from disk</li><li>H5 does not deal well with shuffling instances, so we had to find a workaround. Not shuffling our dataset was not an option, we tried it and the models stopped learning almost completely. We settled initially on pre-shuffling into shards and then shuffling the shards themselves, which is similar to how shards work in TFRecords.</li><li>Making a change to the dataset took over 2 days and was very expensive.</li><li>Optimising and training a model took another 3 days of GPU time, which meant that experiments had to be carefully designed.</li></ul><p>We did fortunately manage to overcome these obstacles, and create a model that predicts the outcomes of construction activities very well. Sometimes I am still surprised by how well our latest model performs, and I can only thank our amazing engineering team for this.</p><p>There are some other scale-related issues that we had to resolve later on, which we consciously decided to file under “acceptable tech debt for now”:</p><ul><li>Switching to data streaming to avoid using hundreds of GB of memory</li><li>Using serverless training (we eventually did this on GCP AI Platform)</li><li>Automating the calculation of metrics over different slices of the dataset (e.g. getting per-client performance)</li><li>Automatically detecting data leakage between the training and test set — we had a special case where it was possible to get repeated instances in both.</li></ul><h3>Building a product</h3><p>Having a model that can predict the outcomes of construction activities is, by itself, particularly useless. No matter how well it predicts those outcomes, there are plenty of reasons why nobody will be interested in that output alone. Top of this list would be the fact that construction companies, and their clients, are not interested in buying a trained model saved as an HDF5 file.</p><p>What follows naturally is that we should build something that is useful to a project director or planner or portfolio manager. When asking our prospective clients what information they would be interested in seeing and then figuring out how to generate that information, the answer we heard the most was around risk. More specifically, they wanted to know what was the risk of the entire project overrunning and how did every activity impact that risk, or how their project portfolio might become disproportionately affected by one of the many projects happening concurrently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chOZC2PItc30she1MhlxuA.png" /><figcaption>Anonymised typical output from our project overview</figcaption></figure><p>To produce accurate risk measures, we had to build an entire simulation engine based on the inputs of the ML model. This was <em>not</em> an easy task. If one thinks of a schedule as a Directed Acyclical Graph, one can see how simulation can be done with Monte Carlo — style simulation. This is also, quite conveniently, what the industry already does — but using subjective inputs and some incorrect assumptions about tail risk. I wrote more about that <a href="https://medium.com/nplan/on-the-effect-of-black-swans-on-projects-532c887bfe04">here</a>.</p><p>We built a Monte Carlo engine that worked in a similar way, and we added in our own risk measures in the meanwhile. The one that we like the most we call “Risk Share”. It’s the independent potential that each individual activity has to change the end date of the entire project, expressed as a percentage of all the total possible deviations. So, for example, an activity that will consistently move the end date by a small amount, will have a lower Risk Share than one that will consistently move it by a lot, and will also have a similar Risk Share to one that infrequently moves the end date, but when it does it moves it by a lot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8VFNnS2GDscce392it244g.png" /><figcaption>Anonymised typical output from our milestone predictions</figcaption></figure><p>Our clients use this information to find the most relevant mitigation strategies or plan changes that reduce the risk of delaying the project. This means that less risk margin is required, and it is less likely that the project will be over budget because of any potential delays.</p><p>We now run regular workshops and learning sessions to better understand what our clients need at every step of using the product. It has been incredible to see how, by engaging the community in the right way, we have been able to focus on building a valuable product from the very beginning.</p><h3>Conclusion</h3><p>When I set out to write this series of blog entries, I didn’t really have a clear objective. I wanted to tell a story about how many difficult problems we have had to overcome, both as engineers and as a company. There are many more obstacles, and many more that will come in the future, and I fully trust our team to take these head on and always do our best to solve them as best we can.</p><p>If I have inspired you to work on something difficult, start your own company, and rise up to whatever your challenges may be, I have done more than I expected to with this post.</p><h3>Get in touch</h3><p>If you think that doing machine learning at scale is exciting, we are hiring! Look at <a href="https://nplan.io/careers">nplan.io/careers</a> for current opportunities.</p><p>You can also get in touch at <a href="http://twitter.com/nplanHQ">twitter.com/nplanHQ</a> and <a href="http://twitter.com/nitbix">twitter.com/nitbix</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c192f658d7d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/the-deep-tech-part-of-building-a-deep-tech-company-part-2-7c192f658d7d">The Deep Tech part of building a Deep Tech company (part 2)</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On the effect of Black Swans on projects]]></title>
            <link>https://medium.com/nplan/on-the-effect-of-black-swans-on-projects-532c887bfe04?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/532c887bfe04</guid>
            <category><![CDATA[project-controls]]></category>
            <category><![CDATA[construction]]></category>
            <category><![CDATA[megaprojects]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Fri, 03 Jan 2020 10:08:30 GMT</pubDate>
            <atom:updated>2020-01-03T10:08:30.634Z</atom:updated>
            <content:encoded><![CDATA[<p>In Project Risk Analysis it is very common to use the Program Evaluation and Review Technique (PERT) and other similar methods to estimate risk. This methodology utilises a probability distribution to represent the possible duration of an activity in a project. This is in direct contrast with the Critical Path Methodology (CPM), where durations and sequences are entirely deterministic and only have one possible value. By using probability distributions and combining them appropriately based on the project schedule, PERT allows practitioners to model uncertainty in a way that wasn’t previously possible.</p><p>However, a certain set of assumptions is made in PERT about these distributions, when they are manually picked and assigned by experts on a project. This has the consequence that all events with low probability and high impact are ignored and therefore considered impossible. Such “Black Swan” events are all around us, and happen with incredible frequency, both in projects and everyday life. I try to illustrate the problem, and explain how the nPlan framework is different.</p><h3>Black Swans</h3><p>In his seminal book “The Black Swan”, Nassim Taleb describes a Black Swan as an event with three fundamental properties:</p><ul><li>It is an event with very low probability of occurring</li><li>It is an event with a disproportionate amount of impact and consequence</li><li>It is an event that cannot be predicted, but is nonetheless often post-rationalised and explained as a higher probability phenomena</li></ul><p>A low probability event is by definition, something that is not very likely to happen. Think of the likelihood of bumping into William Shatner on the street. Humans are particularly bad at estimating such low probabilities, and they do so by several orders of magnitude. Just ask people why they play the lottery (it is much more likely to bump into William Shatner!).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*49xq-R6vZoOa0SCwlyr80Q.png" /><figcaption>Black Swans happen in the tail of probability. For an activity, the black swans are the low probability events that can cause a lot of delay.</figcaption></figure><p>This inability to correctly estimate low probabilities, combined with a total blindness to the fact that there are many more unknown low probability events than there are known ones, is one of the reasons why current project planning techniques are not designed to survive a Black Swan event. The very moment a potential Black Swan is identified, something will be done about it. A plan will be changed, additional resources will be allocated, or simply more attention will be paid to a certain section of a project. When we attempt to mitigate the occurrence or the effect of a Black Swan, it stops being a black swan because we are limiting its impact, and reducing its probability even further.</p><h4>A thought experiment on tails</h4><p>Now consider the probability distribution of the duration of an activity at the time of its planning. This probability distribution will tell us that the expected remaining duration is, say, 50 days. After 10 days have passed, this expectation may not, in fact, be 40 days. It is likely to be slightly higher. The reason for this lies in the fact that when we pass the 50 day mark, if the activity is still occurring, it cannot have a negative expectation of duration, so we have to incorporate this fact in our modelling of the expectation. In the real world, this just means that factors which increase duration are learned during the execution of the activity and not before.</p><p>If this activity were <em>well behaved</em> (what Taleb calls “Mediocristan”) what we would see is that the expectation of remaining duration after 60 days is less than the expectation of remaining duration after, say, 55 days. The opposite is true for activities (and indeed projects) that are <em>badly behaved </em>(what Taleb calls “Extremistan”). While this may seem counterintuitive for a moment, this actually means that as an activity is more delayed, we should expect it to be even more delayed. In practice, this can be thought of like this: after one day of delay, we would think “this might still be done soon”, while after a very large (say 400%) delay, we will start asking ourselves “will it ever be done?”.</p><p>If you want a real life example of just this type of problem, think of Crossrail, a new underground railway that is being build in London, originally planned for delivery in 2018 at a cost of £14.8bn and which is now projected to open in March 2021 (not by nPlan), at a cost of over £18bn — and with one station (Bond Street) delayed even further.</p><p>As Taleb describes in his book, and we have corroborated this in our dataset, projects and activities belong to this latter category, which is subject to the appearance of Black Swans, and forces the practitioner to consider the effects of probability tails.</p><h4>A note about ‘tunnelling’ and the three point estimate</h4><p>Some practitioners do indeed think about Black Swans and consider them to be out of scope for the purpose of risk estimation. This is a symptom of a phenomenon described in the book itself as ‘tunnelling’, where problems outside the direct control of the practitioner are discarded entirely.</p><p>On top of this the “three point estimates” (best, most likely, worst) picked for activity distributions in PERT analysis, follow the guideline that anything worse than the worst case should be around 1% likely. If we follow this definition from a strict statistical point of view, on a project with 4000 activities, 40 activities will perform worse than the worst estimate.</p><p>Combining these two notions, we find that the 1% estimates are too optimistic and several experiments, both in the literature, but also from observations made on our dataset, confirm that humans tend to vastly underestimate these tails because they are not considering external factors. Our data shows that the “worst” estimates used for PERT analysis appear to be excluding around 5%~10% of duration probability rather than the target 1% (which we believe to already be too big a number).</p><h3>The Norm’s fallacy</h3><p>A Black Swan usually lies in what is called “the tails” of a probability distribution — that part that is far away from the most likely events. The rest of the probability mass (the majority of all events) is what is normally considered while planning a project. These events do indeed happen the majority of the time, and we should plan for them to happen in this way, because they represent the bulk of the execution of a modern project.</p><p>Only considering typical outcomes as part of a simulation or a forecast, creates a huge blind spot. If we repeatedly sample from a distribution with a long and/or fat tail, it becomes increasingly likely that we will have low probability events materialising. When such events are unexpected and of high consequence (imagine an electrical transformer exploding), the outcome of the project itself becomes more likely to be affected.</p><p>Let’s imagine that all activities on a project only had a minuscule 0.1% to become wildly delayed by, let’s say, over 200% of the original planned duration. That’s a 99.9% probability of being under that duration. If our projects has 5000 of these activities, we should expect that, on average, 5 activities will take above 200% of their original duration. If any of these are on or near the critical path, they will have a massive impact, causing a very large cascade effect on everything planned to happen after them.</p><p>This pattern is clearly visible in the data that we have collected over the last few years. For every project that was significantly delayed, most of its activities were indeed completed successfully within their planned duration. However, the very few activities that did happen to be badly delayed, had a tremendous effect on the entire project.</p><h3>The limits of PERT analysis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*ZeSVKegSrHklw3OlJcp-yA.png" /><figcaption>A PERT distribution with limits at -10% and +40% delay, and “most likely” as no delay.</figcaption></figure><p>PERT analysis is what is currently used to perform risk analysis on schedules. This is done by assigning (more on why that is bad later) a probability distribution to each activity, based on the PERT distribution. This distribution is a curve that is fitted between two limits (a minimum and a maximum) and has a peak on a “most likely” outcome.</p><p>By defining such a minimum and a maximum possible delay value for each activity using a PERT distribution, we are assigning all activities a zero probability of becoming Black Swans. This assumption creates a vulnerability, and is reflected in the fact that it is almost impossible for a large project to go according to plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*KzPap3UQruI5roun8VRTkA.png" /><figcaption>Lots of PERT distributions: No version of PERT is able to give a probability for any event beyond the given bounds</figcaption></figure><p>Deleting all tail probabilities also makes Black Swans more likely, because we are de facto ignoring all of them, therefore making them all possible.</p><p>PERT was a fantastic tool in the age of slide rules and mainframes, and we had to concentrate the bulk of a simulation on the core central events. Nowadays however, we are able to run much larger simulations; ones that produce correct and calibrated outputs on the tails as well.</p><p>By including the probability of each activity becoming a Black Swan, we are effectively building the mechanisms that create resilience in our plans. Black Swans will not stop happening, and we usually can’t tell which ones will happen, because the moment a prediction is made, mitigation steps are taken. Instead, we believe that the correct way is to forecast the probability of each item becoming a Black Swan individually, learning from past data and in a way that is as granular as possible, and create plans that try to minimise the impact of any Black Swan occurring.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*wpA8ZgDWowsQ7LsbsaX7nA.png" /><figcaption>PERT vs a simple Gaussian. In this case, we set the standard deviation of the Gaussian to be the same as the limits of the PERT and we can see that a lot of tail risk of delay is being ignored (the shaded area in blue), amounting to 15.9% of the total probability mass.</figcaption></figure><h3>Learning the activity outcome</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rGiQRbdMQ4VN9J0NFZnb0w.png" /><figcaption>How our engine learns to output a custom distribution for each activity, based on learning from the historical dataset.</figcaption></figure><p><em>Assigning</em> a probability distribution, even one with tails, is still however not sufficient. As much as one can believe that they have full knowledge of the possible outcomes of an activity, it is very unlikely that they will have witnessed the potential tails of the specific activity at hand. Think back to the whole array of low probability events (say 0.1%) that could have an impact in a schedule built up of thousands of activities.</p><p>By collecting a large dataset of all past schedules, it is possible to create distributions based on historical data which contain low probability events. This will then form the basis of a better estimation which does not rely on any assumptions, not even the shape of the distribution.</p><h3>A different framework — confidence vs probability</h3><p>Our models have learned how these events behave and are able to provide the tools to improve the resilience of plans, by carefully forecasting the probabilities of delay, rather than assigning a <em>confidence</em> value to a plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*AdAzB6POBYEWRmyG1tAhSQ.png" /><figcaption>What a good result might look like when using PERT: The dashed line represents the planned end date. Most of the simulations end before the planned end and there is high confidence of completing on or before it.</figcaption></figure><p>When we do this, we no longer refer to <em>confidence</em> in the schedule, because the result is independent of how we feel about it. When we give an estimate of the probabilities for each activities, we are expressing an opinion which contains an unknown amount of epistemic uncertainty. This estimate directly dictates the results of a Monte Carlo simulation, resulting in end date values that are more expressive of our own knowledge than the actual potential outcomes. Instead, we can now refer to an actual <em>probability </em>of the target date which includes the effects of probabilities based on historical data, which includes Black Swans, external factors (for example client interventions), and uncertainty due to the execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*dMoPG8WCRd61esPr7Qqv7g.png" /><figcaption>What an equivalently good schedule might look like when using tails and learned distributions. The planned end date is near the most likely simulated outcome, but there is still a probability of small delays and a certain amount of tails to the simulation results. The goal of the planner will be to structure the schedule so that the tail of this graph is contained as much as possible and the planned end is as close as possible to the most likely end date.</figcaption></figure><p>These <em>probability</em> values will have a much wider base than the confidence values provided by a standard PERT simulation, giving a more accurate estimate of the actual outcome of the project. This new information will help make informed decisions about the risk of a project. For example, a programme that gives 90% confidence in PERT may be a good programme with 70% probability of completing the project on time, or a bad one, where the probability of delivering on time is much lower.</p><p>This, and a complete change in the way the industry thinks about probabilities, will help us shake the impact of Murphy’s Law, allowing us to build larger projects with less risk.</p><h4>Get in touch!</h4><p>If you are interested in talking about the effects of Black Swans on large projects, please get in touch through <a href="https://nplan.io">https://nplan.io</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=532c887bfe04" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/on-the-effect-of-black-swans-on-projects-532c887bfe04">On the effect of Black Swans on projects</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Deep Tech part of building a Deep Tech company (part 1)]]></title>
            <link>https://medium.com/nplan/the-deep-tech-part-of-building-a-deep-tech-company-part-1-e671ea7a212a?source=rss-4f87c2293d63------2</link>
            <guid isPermaLink="false">https://medium.com/p/e671ea7a212a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[entrepreneurship]]></category>
            <category><![CDATA[startup]]></category>
            <dc:creator><![CDATA[Alan Mosca]]></dc:creator>
            <pubDate>Mon, 25 Nov 2019 10:14:08 GMT</pubDate>
            <atom:updated>2019-11-25T22:35:32.753Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>For many people at nPlan, the story starts <a href="https://www.youtube.com/watch?v=C8E2NjRkgtU">here</a>. A 3 minute pitch by my co-founder <a href="https://medium.com/u/eb2e897e9aae">Dev Amratia</a>, in front of over five hundred of europe’s best venture capitalists. We told the world that construction programmes (or schedules if you’re not in the UK) had a low accuracy, and that we knew how to fix it. After all it’s not news that most large construction projects are delayed and over budget.</p><p>Thanks to our efforts, a new solution was possible that allowed a planner (or scheduler) to understand the sort of risk that the timeline of their project represented. And that very risk could be summarised to the project manger, to the person that is responsible for submitting a bid, to the CEO, and even to the clients of a construction company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-WUqYZBkO7n7EixK1l0R-A.jpeg" /><figcaption>Our first team picture!</figcaption></figure><p>A not-so-small problem kept going round and round in my head. “How the hell are we supposed to be doing all of this?”. That, in essence, was the meaning of my job as technical co-founder.</p><p>We had a working prototype that demonstrated that the problem was tractable. We had also collected a substantial dataset (over one hundred thousand schedule files) that would allow us to train a good enough machine learning model.</p><p>Because we had about 1TB of data already, and a very complex problem at hand, I was constantly thinking about how to build the <strong>real</strong> model. You know, the one that you actually put into the product. The one that you sell to customers.</p><p>This story is about what happened before that day. It’s about what we did that allowed us to claim that we knew how to solve the $2T/year problem in construction. It’s a story that continues today and that we’re constantly writing.</p><h3>Part one: Our first dataset</h3><p>It was a very exciting day. I had just downloaded our first batch of construction schedules from a client. About ten thousand of them, to be precise. Each file has an average of two thousand activities, for a total twenty million activities, give or take. Anyone that is familiar with machine learning, on receiving such a dataset, would sigh in relief at the fact that there is, in fact, enough data available to at least prove the learnability of the problem.</p><p>Fortunately enough the ecosystem of software used to create a very large GANTT chart of a construction project is quite small. The consequence of this is that there are only a couple of file formats that we needed to understand. This invited its own second sigh of relief.</p><p>Building a preprocessor that put everything into a database was therefore conceptually easy, if not slightly time consuming. I wasn’t sure how stable our database schema was going to be, so for the time being I used MongoDB.</p><p>Meanwhile, the dataset was already growing. By the time we had some rudimentary tools that could read this format, there were over one hundred thousand XML files that took up 1TB of disk space, and a resulting 800GB database.</p><p>Surprisingly (or not, if you think about it), the actual amount of disk space taken only shrunk by 20% even though we were discarding a lot of variables and pruning aggressively.</p><p>The first pipeline run took about five days just to load all the data. It was at this moment that it became clear to me that we wouldn’t be able to fix up a quick solution (an MVP?) that we could roll out quickly to our prospective clients that were giving us this data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*KtbM_1D9s-YoIdVt8JevTw.png" /><figcaption>The first sign that we were processing close to full speed!</figcaption></figure><p>This was (and still is) scale problem no.1 for nPlan: As we get data from more clients, the time requirement to reprocess everything from scratch grows linearly with the number of schedules.</p><p>I couldn’t tell Dev to slow down traction — that’s not how you build a successful startup. Parallel processing only helped a little (about 50% speedup) because most of these operations were disk bound.</p><p>Upgrading the disks on our cloud provider’s instances was painful, but at least we could now process our entire dataset in about 2 days. By then João, Josh, and Vahan had joined (we finally had an engineering team!), and I was very glad that they did.</p><p>We first realised that the preprocessing was embarrassingly parallel and we could shard the source data across multiple disks to speed up the loading. We could even split the work across several machines, or use tools like Apache Storm (TFX and Beam weren’t around at the time).</p><p>This shifted the bottleneck to MongoDB write operations, so we had to become smarter about writing into the database. We only needed to update a record if something changed, so we introduced schedule fingerprinting and checksum-based caching. This got us down to about half a day, which was an acceptable penalty to pay every time we had to reprocess the entire dataset.</p><p>We still had a lot of processing to do before we got to the actual learning. We had a toy model (a random forest), trained on a small subset of the data, that we used to learn the outcomes of an activity, but we did not have a complete dataset.</p><p>Transforming the data from MongoDB into numeric form is truly an embarrassing parallel problem as well. We had no idea if any line of code we were writing would survive for longer than a week, so we wrote our own little parallel implementation of map/reduce jobs in Python to get data transformations done quickly, and managed to extract a 100GB file that looked like a supervised ML dataset. This is usually the starting point for an ML project, and we were already a few months in and wrote an epic amount of code to get us there.</p><h3>Up next</h3><p>In part two, I’ll talk about the part of the journey where we had to build ML models based on this dataset, and all the subsequent evolutions of the pipeline and the dataset.</p><p>In part three, I’ll cover product. ML startups don’t talk about product enough, and it’s usually a very difficult part of the job to figure how to deliver value to clients.</p><h3>Get in touch</h3><p>If you think that doing machine learning at scale is exciting, we are hiring! Look at <a href="https://nplan.io/careers">nplan.io/careers</a> for current opportunities.</p><p>You can also get in touch at twitter.com/nplanHQ and twitter.com/nitbix</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e671ea7a212a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/nplan/the-deep-tech-part-of-building-a-deep-tech-company-part-1-e671ea7a212a">The Deep Tech part of building a Deep Tech company (part 1)</a> was originally published in <a href="https://medium.com/nplan">nPlan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>