<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Paco Nathan on Medium]]></title>
        <description><![CDATA[Stories by Paco Nathan on Medium]]></description>
        <link>https://medium.com/@pacoid?source=rss-ed409c5b95e3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*g49ziRKg1DozTYM_HuOp6g.jpeg</url>
            <title>Stories by Paco Nathan on Medium</title>
            <link>https://medium.com/@pacoid?source=rss-ed409c5b95e3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 18 May 2021 13:47:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pacoid/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Downturns, and after]]></title>
            <link>https://pacoid.medium.com/downturns-and-after-9444322124d2?source=rss-ed409c5b95e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9444322124d2</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[economic-downturn]]></category>
            <dc:creator><![CDATA[Paco Nathan]]></dc:creator>
            <pubDate>Wed, 07 Apr 2021 06:38:00 GMT</pubDate>
            <atom:updated>2021-04-08T14:18:09.167Z</atom:updated>
            <content:encoded><![CDATA[<p>I think that people tend to get the wrong idea about technology, particularly software.</p><p>We make new capabilities.</p><p>To have a defensible strategy in business, the “newness” of what you’re making cannot be minor. Firms must aim higher, provide more than simply incremental advances. They must push far enough ahead to drive value, and give themselves runway to grow.</p><p>So, about downturns… I’ve participated in a few: 1987, 2001, 2008, and now 2020. Understand that most of the “disruption” occurs <em>after</em> business begins to pick up again.</p><p>The reason for that belies what people tend to get wrong about technology. Lots of people in the tech industry think they understand their gig, how to earn big bucks doing a <em>thing</em>. Until a downturn happens, then afterward their thing isn’t especially valuable anymore. Oops.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4p8W6C9h3kGJZGT8j2N6Rg.png" /></figure><p>For example…</p><p>In 1986, lots of people with tech jobs believed they understood how to rake in big bucks. Frankly, I was interested in something obscure at the time called “neural networks”. Plenty of people around the industry thought that was stupid and told me so. Hey, if a person really wanted to earn $50K/yr starting salary (good at the time) they should just do the obvious thing and become a COBOL programmer. Or maybe get a corporate job at HP building printers.</p><p>When the market crashed in 1987, tech jobs underwent a relatively subtle transformation. On the other side of that recovery, instead some work in Unix and languages such as C++ were considered valuable. People who hadn’t tracked that shift found it increasingly difficult to find good work. Once the WWW thing hit, just a few years later, changes in tech only accelerated.</p><p>In 1999, lots of people with tech jobs believed they had their cash cow figured out: launch web sites for ecommerce, count on $80k/yr. Then the Dot Com Boom crashed in 2000 and the worldwide economy crashed even further a year later.</p><p>Over the course of the crash and heading into its recovery, things in tech had shifted. Knowing how to write HTML didn’t land you interviews any more. Websites had meanwhile become more complex due to webapps, MVC, Java, Java, Java. I was working in network security, and teaching that plus a new thing we called “Ajax” and the shift in tech became apparent. Web 2.0 lurched forward, and all those “dark fiber” lines suddenly became valuable going into the mid-aughts as YouTube and smartphones prevailed.</p><p>I met a lot of people who were convinced, circa 2008, that being a certified Oracle DBA or Business Intelligence analyst was their ticket to a cushy retirement. Except that the tech industry had oversold the “promise” of enterprise data warehouses, and the economy crashed later that year. On the other side of that recovery cycle you needed “data science” on your resumé instead. BTW, did you know how to write any MapReduce? (If HR could even spell the term correctly, but I digress…)</p><p>I met a bunch of people who were quite convinced circa 2019 that they had the tech industry figured out, they were set for life. SF was an idyllic paradise for their lifestyle, etc., etc. For many of these people, 2021 is turning out to be rather rude.</p><p>Here’s the thing: in tech, we create new capabilities —<em> dramatically improved newer capabilities</em>. Often that involves identifying what had previously been considered “too expensive” then finding ways to change the rules so that it’s not.</p><p>You know what’s really expensive in tech? People. Loaded salaries. Expectations that don’t keep apace with the ever-changing reality of what is actually valuable — and what’s no longer quite so valuable.</p><p>If you want to do tech well, find a way to make a bunch of expensive headcount unnecessary, while creating an entirely new set of affordances for your product/service/thing which makes it in hot demand. Of course, more people will get hired— they always do. But those will be different people, with different skills. That’s the game of it.</p><p>PS, as a note to people who want to manage tech teams well: be wary about hiring people who entered the industry during a boom, and seek out people who entered during a downturn. You’ll thank me later.</p><p>The other thing is about timing: companies that have enough capital to double-down on the whole “aim higher” technology investment strategy during a downturn will tend to be among the ones that race into leading positions after the recovery happens. Other companies, not so much. Ergo, even more jobs get disrupted. You see, the economics of aiming high to develop <em>much better capabilities</em> generally don’t work so well during a boom. Not many firms do it. The ones with deep pockets and good leadership tend to leverage downturns as a time to wipe out their competition.</p><p>2021 is turning out much better than the previous year. Vaccines are on their way, business is picking up. It’s just that, if you’ve been expecting “things to go back to normal” in the tech industry … they won’t. That’s what we do for a living.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9444322124d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decades]]></title>
            <link>https://pacoid.medium.com/decades-47b9cb7dd2a?source=rss-ed409c5b95e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/47b9cb7dd2a</guid>
            <category><![CDATA[communication-skills]]></category>
            <category><![CDATA[coping-mechanisms]]></category>
            <category><![CDATA[digital-media]]></category>
            <dc:creator><![CDATA[Paco Nathan]]></dc:creator>
            <pubDate>Fri, 19 Feb 2021 13:13:46 GMT</pubDate>
            <atom:updated>2021-03-23T16:06:04.407Z</atom:updated>
            <content:encoded><![CDATA[<p>Something I’ve learnt from 39.5 years of using email every day for work:</p><ol><li>If you are the kind of person who communicates with another person to tell them that you need to communicate with them…</li><li>Then you are a dreadfully <strong>stupid</strong> person.</li></ol><p>There, I’ve said it. Time to exhale.</p><p>Let’s put this into business terms: extremely poor ROI and low LTV.</p><p>Let’s put this into scientific terms: consider the existence of a Bayesian prior indicating how another individual is likely to be a complete flake.</p><p>You’re welcome.</p><p>Not clear on this yet? We’ll illustrate with an example.</p><p>Alice is one person, Bob is another person. They are separated, physically, although multiple means of digital communication exist between them, including: texts, email, phone, video chat, tweets, Instagram comment threads, robotic carrier pigeons, and the like.</p><p>Bob sends Alice an email message: “Hey Alice, I need to talk with you.” Meaning, on some implied level, “Alice, call me. I have something to say.” However — and here comes strike two — then our friend Bob has a little fucking freak-out and starts to text or message Alice with nonsense along the lines of “Alice, why haven’t you answered my email about giving me a call!?!”</p><p>All the while, Bob could have simply said exactly what he felt so compelled that he needed to say <em>right there during the initial communication.</em> He could have saved them both time and <strong>grief</strong>.</p><p>Bob did not, because Bob is a terribly, horribly, dreadfully sad excuse for a person.</p><p>We could dwell on the media-theoretic implications of this scenario, in case you didn’t bother to read McLuhan in detail? The esteemed philosopher from Winnipeg <em>did</em> spell out the problem quite <strong>luridly</strong> six decades ago. Moreover, he elaborated on those themes throughout the rest of his life.</p><p>Do not be a Bob. The next time you go to communicate, ask yourself: “Am I truly saying anything? Or am I merely projecting anxiety? Do I need to abuse a half dozen different modes of communication against some hapless victim, merely to demonstrate the state of being <strong>an arse</strong>?”</p><p><strong>To wit:</strong> I’ve noticed over the past few decades how this phenomenon becomes amplified when many people feel anxious and have dire needs for emotional support and human connection. In other words, as the fecal matter prepares to confront an HVAC, <strong>people in aggregate seem to communicate less well</strong>. Perhaps the effect can be used as a leading indicator for, say, global financial crises. Or something.</p><p>My “email anniversary” is coming up this August. If you have similar depth of experience with the medium, I’m eager to <a href="https://github.com/ceteri/ceteri/blob/main/CONTACT.md">hear</a> your perspectives. If you have less than, say, 30 years experience with email —<strong> just put a pin in it</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47b9cb7dd2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Review: Metadata Day 2020]]></title>
            <link>https://medium.com/knowledge-technologies/review-metadata-day-2020-e38c28c4cf1a?source=rss-ed409c5b95e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e38c28c4cf1a</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[metadata-management]]></category>
            <dc:creator><![CDATA[Paco Nathan]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 21:23:33 GMT</pubDate>
            <atom:updated>2021-02-03T19:49:21.440Z</atom:updated>
            <content:encoded><![CDATA[<p>For a full day on December 14, 2020, LinkedIn sponsored a virtual workshop called Metadata Day, followed by a public online meetup called Metaspeak. View <a href="https://metadataday2020.splashthat.com/">videos of the event</a> on their website including a set of lightning talks provided by several of the speakers.</p><p>The gist is that circa 2018, the pending regulatory compliance for GDPR, CCPA, etc., drove urgency in enterprise organizations to have a better understanding and accountability about datasets and their use cases. Several firms, about a dozen, responded in strikingly similar ways — which signals that a kind of “design pattern” is emerging for how to handle these compliance challenges. Given how the datasets in question are connected to one or more use cases, and how each use case requires use of one or more datasets to train machine learning models, when auditors begin asking difficult questions about data provenance, the answers usually require graph-based approaches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XVT8cuI5wXIsBYkh" /></figure><p>Of course, <em>data governance</em> is not a new topic — it’s been around for decades. Good products exist for managing data catalogs, tracking lineage/provenance, and so on. What changed was (1) urgency since 2018; and (2) a crucial role of graphs for understanding metadata. How are <em>connected datasets</em> getting used? Which team is responsible for producing and updating each dataset? Who manages the metadata? What customer products rely on a given dataset?</p><p>A 2017 paper out of UC Berkeley’s <a href="https://rise.cs.berkeley.edu/">RISELab</a> has explored these questions formally, <a href="https://rise.cs.berkeley.edu/wp-content/uploads/2017/03/CIDR17.pdf">“Ground: A Data Context Service”</a> by <a href="https://twitter.com/joe_hellerstein">Joseph M. Hellerstein</a>, <a href="https://twitter.com/shirshanka">Shirshanka Das</a>, et al., CIDR ’17, which predicted the needs of what the industry would begin to recognize the following year. Shirshanka (co-organizer Metadata Day 2020) as chief architect of Data at LinkedIn led the metadata strategy for LinkedIn to drive data discovery and compliance. In the article “<a href="https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained">DataHub: Popular metadata architectures explained</a>” he describes the evolution of frameworks for this kind of use case, particularly regarding the <a href="https://datahub.io/">DataHub</a> platform developed at LinkedIn. Early solutions for data catalog involved relatively static, represented by database tables. A second generation followed that, providing metadata API services. However, more contemporary needs in enterprise (since GDPR, etc.) demand graph-based solutions.</p><p><a href="https://twitter.com/mark_grover">Mark Grover</a> held a similar role at Lyft, as product manager for the <a href="https://www.amundsen.io/">Amundsen</a> project, coordinating their solutions for GDPR compliance needs. In the article <a href="https://mark-grover.medium.com/production-consumption-gap-75080253244f">“The data production-consumption gap”</a>, Mark describes how Lyft recognized this as an organizational problem — where understanding enterprise use of connected datasets requires much more than a data catalog. There’s an inherent producer/consumer problem where HR data about the organization is needed (“Who’s responsible for curating XYZ?”) as well as data from operations and other departments. Graph-based representation allowed Lyft to collect and integrate metadata at a layer above the typical departmental data silos. Coupled with discovery and search UX, the experiences with Amundsen led to a stark discovery: “Over 30% of analyst time was wasted in finding and validating trusted data.” Lyft has 250 data analysts, so their 30% overhead translates to roughly $20M/year. Given this new “360 view” of their data assets, they also began to discover relations among their connected datasets that could open new business opportunities. Two major enterprise costs — compliance as a risk and the overhead of impedance across teams — were converted into business upside. Thirty firms now use the Amundsen platform, including ING Bank, Square, Instacart, Workday, Edmunds, etc. Mark also presented in the session <a href="https://www.knowledgeconnexions.world/talks/from-discovering-data-to-trusting-data/">“From discovering data to trusting data”</a> at Knowledge Connexions.</p><p>Approximately a dozen tech companies reached similar realizations about GDPR-era data context and graph-based metadata approaches: LinkedIn, Lyft, Netflix, Airbnb, Uber, PayPal, and so on. Most of these efforts resulted in open source projects. <a href="https://twitter.com/eugeneyan">Eugene Yan</a> authored an excellent survey article <a href="https://eugeneyan.com/writing/data-discovery-platforms/">“Data Discovery Platforms and Their Open Source Solutions”</a> to compare/contrast these projects. The common ground is that these projects tend to combine graph-based overlays atop organizational data silos, along with serious investments in UX work, to provide local search plus recommender systems. This design affords <em>understanding</em> and <em>trust</em> about connected datasets, their use cases, and also about the relationships within an organization — since one lesson learned is that root problems in this area often require executives to intercede.</p><p>Another key observation: three tech start-ups have recently spun out based on this new category of data context, with these use cases and open source projects as a core: <a href="https://stemma.ai/">Stemma</a>, <a href="https://metaphor.io/">Metaphor Data</a>, and one other to be announced. Venture capital investors describe this kind of situation in terms of <a href="https://mashable.com/2015/02/04/category-defining-company/">category-defining</a> companies. In other words, as certain start-ups begin to reshape the industry, equity growth typically follows.</p><p>Recognizing that an industry “moment” was in progress, LinkedIn organized the Metadata Day event to pull together the people leading these projects. The event was structured in two parts. First, an invitation-only virtual workshop (Foo Camp style) which split into one track for the systems implementations (i.e., issues about production and scale) and another track for articulating use case issues (i.e., expertise about metadata and knowledge graphs). The latter track included: Joe Hellerstein; <a href="https://scholar.google.com/citations?user=FE08ALAAAAAJ&amp;hl=en">Natasha Noy</a> (Google) and <a href="https://scholar.google.com/citations?user=PLJ0L4QAAAAJ&amp;hl=en">Deborah McGuinness</a> (RPI) who co-authored the famous <a href="http://www-ksl.stanford.edu/people/dlm/papers/ontology-tutorial-noy-mcguinness-abstract.html">“Ontology 101”</a> paper while at Stanford; data governance pioneer <a href="https://twitter.com/satyx">Satyen Sangani</a> from Alation; <a href="https://twitter.com/AxSaucedo">Alejandro Saucedo</a> from Seldon, plus metadata experts <a href="https://twitter.com/DaniLowenberg">Daniella Lowenberg</a> (UC), <a href="https://metadatagamechangers.com/">Ted Habermann</a> (Metadata Game Changers), <a href="https://twitter.com/IanMulvany">Ian Mulvany</a> (BMJ). The workshop track discussions worked to articulate key points about their practices, quickly assembled into slide decks. Then the tracks reconvened to compare notes.</p><p>The second part of the Metadata Day event was a public meetup online, introduced with a keynote talk by LinkedIn CDO <a href="https://www.linkedin.com/in/igorperisic/">Igor Perisic</a>. Both tracks presented their slide decks as panel discussions, open to much audience Q&amp;A. The full video is at <a href="https://youtu.be/LS2LxEsj-94">https://youtu.be/LS2LxEsj-94</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4jFLssqgFTwas4XI" /></figure><p>One of the quintessential insights out of Metadata Day was articulated by Natasha Noy during our discussion about “What is a knowledge graph?” Natasha described how Google’s Knowledge Graph project, while it has well-known use within the search engine results, also fulfills another crucial purpose to help cohere Google’s data context across many different data silos. To paraphrase Natasha, there was a realization that no “one data framework to rule them all” would ever happen, and instead it was important to decentralize services for translating schema and identifiers (e.g., “foreign keys”) across different data frameworks, allowing many teams to work independently on the kinds of producer/consumer challenges described above by LinkedIn and Lyft.</p><p>To be fair, this is certainly not a “solved problem”, nor are graph-based methods a panacea for resolving tech debt and cross-team impedance in enterprise IT. The challenges of data silos and enabling cross-team collaboration are <em>organizational problems</em> at their core which require Leadership to resolve. These data context platforms introduce open source solutions and related practices which can help greatly, when the corporate leadership and will to resolve is there first. Graph-based approaches are not a magic solution, though they are proving to be the more effective approach.</p><p>Also recognize that using knowledge graphs about metadata as an “overlay” to make an organization’s data silos <em>cohere</em> is not specific to Google — we’re hearing about this kind of practice across many enterprise organizations. Mainstream IT may not be referring to these kinds of practices as knowledge graphs — not yet, at least if one listens to Gartner Research and their “Hype Cycle”. However, the reality on the ground which Metadata Day proves is that Silicon Valley is driving and recognizing a category-defining moment.</p><p>From within the KGC community, there are two related articles:</p><ul><li><a href="https://medium.com/derwen/metaspeak-meetup-dec-14-2020-220b7af5e4cc">”Introduction to Metadata Day”</a> by <a href="https://derwen.ai/paco">Paco Nathan</a> (before the event)</li><li><a href="https://guitton.co/posts/dbt-search/">“Rethinking the search over your dbt models”</a> by <a href="https://twitter.com/louis_guitton">Louis Guitton</a> (reviewing the event)</li></ul><p>Also check out the closely related article:</p><ul><li><a href="https://gradientflow.com/the-growing-importance-of-metadata-management-systems/">“The Growing Importance of Metadata Management Systems”</a> by <a href="https://twitter.com/bigdata">Ben Lorica</a> (Gradient Flow) and <a href="https://twitter.com/bigdata_ml">Assaf Araki</a> (Intel Capital)</li></ul><p>To add more context, there have also been notable, long-term efforts in research towards similar ends which focused on delivering research results along the code and data required to <em>reproduce</em> them. That’s the academic research analog for what data science teams perform in industry. Notably, <a href="https://www.research.manchester.ac.uk/portal/carole.goble.html">Carole Goble</a> and colleagues at University of Manchester have pushed toward better tooling and practices for data context in the sciences, over the past two decades — e.g., the <a href="https://www.research.manchester.ac.uk/portal/en/publications/towards-fair-principles-forresearchsoftware(0bea2ca4-c609-48a0-8c13-3a52ce174880).html">FAIR Guiding Principles</a> for data governance. Their work has been foundational for semantic technologies. Another good example is the work by <a href="https://www.janelia.org/freeman-lab">Jeremy Freeman</a> at HHMI Janelia and their <a href="http://codeneuro.org/">CodeNeuro</a>, weaving a web of open source support for understanding and leveraging connected datasets in neuroscience worldwide. Project Jupyter’s <a href="https://binderhub.readthedocs.io/en/latest/">Binder</a> was a direct result. Part of Metadata Day’s pretext was to bring experts from the academic side who’ve had decades of work in metadata management and connected datasets, together with open source project leads from leading tech firms, to find and articulate common ground to help guide other organizations.</p><p>One thing to watch: as more companies turn to graph-based solutions for their data context, i.e., addressing their needs to understand and trust metadata about connected datasets, that creates pervasive opportunities for <em>leveraging</em> this new category of graph data with even more sophisticated graph-based approaches. Most of the solutions mentioned above are relatively light in terms of leveraging semantic technologies, inference, graph embedding, advanced approaches for interactive graph visualization at scale, etc. That absence was quite apparent from Metadata Day discussions on the “production” side.</p><p>One wonders what will happen when the new graph acolytes in Silicon Valley — and the many enterprise firms who follow their lead and adopt their open source frameworks — begin to discover KGE, UMAP, SHACL, PSL, and so on. Considering the point by Natasha Noy at Google, the foundations for knowledge graph work are becoming widespread across enterprise IT; this new abstraction layer of discovery and trust, driven by regulatory compliance, may catalyze industry adoption of enterprise knowledge graphs. And that practice may happen years before Gartner Research decides to give it a name 😀</p><p>###</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e38c28c4cf1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowledge-technologies/review-metadata-day-2020-e38c28c4cf1a">Review: Metadata Day 2020</a> was originally published in <a href="https://medium.com/knowledge-technologies">Knowledge Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metaspeak Meetup, Dec 14 2020]]></title>
            <link>https://medium.com/knowledge-technologies/metaspeak-meetup-dec-14-2020-220b7af5e4cc?source=rss-ed409c5b95e3------2</link>
            <guid isPermaLink="false">https://medium.com/p/220b7af5e4cc</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[metadata]]></category>
            <dc:creator><![CDATA[Paco Nathan]]></dc:creator>
            <pubDate>Tue, 01 Dec 2020 03:09:16 GMT</pubDate>
            <atom:updated>2020-12-01T05:14:54.169Z</atom:updated>
            <content:encoded><![CDATA[<p>Registration is open for the <strong>Metaspeak Meetup</strong> (a free online event) on December 14, 2020, hosted by LinkedIn <a href="https://metadataday2020.splashthat.com/">https://metadataday2020.splashthat.com/</a></p><p>This is the public portion of the <strong>Metadata Day 2020</strong> workshop, which convenes leaders from several open source projects for <em>graph-based dataset metadata management</em>, along with thought leaders in metadata management and dataset governance from Google, IBM, UC Berkeley, etc.</p><p><strong>Igor Perisic</strong>, CDO @ LinkedIn, will present our welcome address, followed by two panels of expert practitioners who will summarize outcomes from our workshop earlier in the day — (1) focus on use cases, and (2) focus on production at scale — plus Q&amp;A from the audience.</p><p>A little over a year ago I presented <a href="https://youtu.be/AzpS8nvmQz0">“Overview of Data Governance”</a> at Big Things Conference in Madrid. One of my slides referenced <a href="https://rise.cs.berkeley.edu/wp-content/uploads/2017/03/CIDR17.pdf">“Ground: A Data Context Service”</a> by <strong>Joe Hellerstein</strong>, et al., at <a href="https://rise.cs.berkeley.edu/people/">UC Berkeley</a>, followed by a list of open source projects which focused on <em>graph-based discovery for metadata about datasets</em>. I cited work from LinkedIn, Lyft, Uber, Netflix, etc.</p><figure><a href="https://youtu.be/AzpS8nvmQz0?t=1167"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1mYThBAw8R9SV39Fi1RpQ.png" /></a></figure><p>That DG <a href="https://derwen.ai/s/6fqt">talk</a> had begun as a request by my colleague and co-author <a href="https://gradientflow.com/">Ben Lorica</a> — we’d been conducting a <a href="https://derwen.ai/s/6fqt#4">series of industry surveys</a>, beginning in 2018, about “ABC” (AI, Big Data, Cloud) adoption trends.</p><p>2018 was a banner year for data governance: the launch of GDPR, published findings about Facebook/Cambridge Analytica scandal, and — although not in the headlines as much — a year in which hundreds of millions of people had their private info exposed (i.e., sold) through several large industry failures. Data Governance was becoming top-of-mind for many IT leaders as a consequence — in contrast to what I’d seen in four decades of computing during which DG had largely been swept under the run.</p><p>Earlier in the talk I referenced <a href="https://zenodo.org/record/556504#.X8WSMhNKgWp">“The Case for Open Metadata”</a> by <strong>Mandy Chessell</strong> at IBM, plus her team’s work on the open standard <a href="https://www.odpi.org/projects/egeria">ODPi Egeria</a> and its reference implementation <a href="https://atlas.apache.org/index.html#/">Apache Atlas</a>. I’d also described related US government initiatives, such as the US <a href="https://strategy.data.gov/">Federal Data Strategy</a>, H.R. 4174 <a href="https://www.congress.gov/bill/115th-congress/house-bill/4174">evidence-based policymaking</a>, etc. Plus, I’d mentioned the <a href="https://github.com/jupyterlab/jupyterlab-metadata-service/blob/master/press_release.md">JupyterLab Metadata Explorer project</a> that I’d helped work on with Brian Granger, et al., at Project Jupyter.</p><p>Speaking of those open source projects… I stuck that slide into the deck with a “Watch this space” caveat in the voice-over. It seemed worth tracking carefully, and that’s proven out over time.</p><p>For a particular cohort of organizations, the natural response to GDPR was simple:</p><ol><li>Collect metadata about dataset usage across your organization.</li><li>Build a graph-based platform for metadata representation.</li><li>Build out UX features for dataset discovery and reporting (aka, <em>compliance</em>).</li><li>Recognize savings across your organization since your data scientists finally have a definitive source for collected knowledge about datasets.</li><li>Push the thing public as an open source project and advocate for adoption by other organizations.</li><li>Recognize business upside beyond compliance efforts, since this new lens into dataset usage allows insights about possible new business lines which were previously unavailable.</li><li>Elevate reporting of these results up to executive levels.</li></ol><p>While GDPR compliance was probably all good right around Step #3, these organizations kept pushing the envelope. The found both ROI and increased markets from an activity that began as risk mitigation.</p><p>From my vantage point, the next logical step in business would be for these projects to begin to marry AI methods along with their graph-based metadata discovery practices, leading inevitably into something much larger. That’s on deck now.</p><p>You may have read the excellent, recent (2020–10) article <a href="https://eugeneyan.com/writing/data-discovery-platforms/">“Almost Everything You Need To Know on Data Discovery Platforms”</a> by Eugene Yan at Amazon. If not, please do. The article is filled with brilliant analysis and insights. Eugene curates a list of open source projects within this emerging category: <a href="https://github.com/eugeneyan/applied-ml#data-discovery">https://github.com/eugeneyan/applied-ml#data-discovery</a></p><p>The list of these data discovery projects, courtesy of Eugene, currently includes:</p><ul><li><a href="https://www.amundsen.io/">Amundsen</a> — Lyft, Square, ING, Edmunds, Workday, etc.</li><li><a href="https://engineering.linkedin.com/blog/2019/data-hub">DataHub</a> — LinkedIn</li><li><a href="https://marquezproject.github.io/marquez/">Marquez</a> — WeWork, Stitch Fix</li><li><a href="https://medium.com/airbnb-engineering/democratizing-data-at-airbnb-852d76c51770">Dataportal</a> — Airbnb</li><li><a href="https://eng.uber.com/metadata-insights-databook/">DataBook</a> — Uber</li><li><a href="https://atlas.apache.org/#/">Apache Atlas</a> — IBM, plus several large enterprise firms</li><li><a href="https://netflixtechblog.com/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520">Metacat</a> — Netflix</li><li><a href="https://engineering.atspotify.com/2020/02/27/how-we-improved-data-discovery-for-data-scientists-at-spotify/">Lexikon</a> — Spotify</li><li><a href="https://blog.twitter.com/engineering/en_us/topics/insights/2016/discovery-and-consumption-of-analytics-data-at-twitter.html">Data Platform</a> — Twitter</li><li><a href="https://engineering.fb.com/2020/10/09/data-infrastructure/nemo/">Nemo</a> — Facebook</li><li><a href="https://cloud.google.com/data-catalog/">Data Catalog</a> — Google</li></ul><p>From the perspective of an equity partner at a <a href="https://amplifypartners.com/">Bay Area VC firm</a>, “Something is definitely up!”</p><p>This much activity emerging from several different tech firms indicates both needs and capabilities, i.e., a new market category emerging. Indeed, two of these projects have recently spun out VC-funded tech start-ups.</p><p>Part of the pitch is simple: if you, as an enterprise firm, want to be competitive, you’ll need to leverage AI applications … because your competitors already are doing so. One consistent signal that Ben Lorica have found throughout our industry surveys is that when firms begin to recognize ROI from machine learning investments, they double-down on those investments. Consequently the gap between “have’s” and “have-not’s” in the AI space is accelerating. One major hurdle among the “have-not’s” who aren’t competitive: they must get their data in order. Like, yesterday. Data governance. Data strategy. Effective data engineering at scale. Visibility into what data they have. How their data assets can be leveraged. Taking advantage of their metadata to gain even more leverage.</p><p>Approximately 50% of enterprise is in the “have-not’s” category currently, and they need help. This emerging category of tech start-ups focusing on these graph-based solutions is key.</p><p>If you haven’t been involved much in <a href="https://www.oreilly.com/library/view/fifty-years-of/9781492057512/">graph-based data management</a>, nor its applications which are rapidly spilling over into AI, let me tell you that “Something is definitely up!”</p><p>I’ve been assisting organizers at knowledge graph conferences such as <a href="https://www.knowledgeconnexions.world/">Knowledge Connexions</a> and <a href="https://www.knowledgegraph.tech/">The Knowledge Graph Conference</a>. There’s also the fantastic workshop <a href="https://www.akbc.ws/">AKBC</a>, albeit more about research. The dirty little secret is that KG practices are widespread throughout enterprise — with the caveat that historically it’s been difficult to get those teams to talk about their work in an industry-first forum, outside of academic conferences. The list of related topics goes well beyond graph DBs and knowledge graphs, into more specific technology areas such as interactive graph visualization, graph algorithms, embedding and graph AI (deep learning), statistical relational learning, probabilistic inference on networks, causal reasoning, persistent homology, and so on.</p><p>Full disclosure: I’ve been teaching industry courses about these topics for several years; I’m an advisor for related AI start-ups including <a href="https://www.recogn.ai/">Recognai</a> and <a href="https://primer.ai/">Primer</a>; I’m a committer for a nascent little open source project called <a href="https://github.com/DerwenAI/kglab/">kglab</a> which attempts to blend these application areas into a simple Python-based abstraction layer with <a href="https://gradientflow.com/towards-an-infinite-laptop/">“infinite laptop”</a> scale-out provided by <a href="https://ray.io/">Ray</a> from UC Berkeley and <a href="https://www.anyscale.com/">Anyscale</a>; plus I’m super excited about this general area of <a href="https://arxiv.org/abs/2003.02320">knowledge graphs</a> and math-meets-code-meets-use-cases. But I digress…</p><p>There was another twist. For those of you who haven’t been working much in the federal sector and spending time in DC, let me just say that mention of <em>metadata management</em> or <em>data governance</em> tends to get people excited. There’s work to be done, there’s recent policy to support that work, and the priorities (and consequences) for the public are crystal clear: the 2019 US federal data strategy in response to the 2017 bipartisan H.R. 4174 bill sets the stage for dramatic changes in how approximately 30% of the US economy leverages data in the Age of AI.</p><p>Compliance, privacy, fairness/bias, security, managing pandemic response, etc., are all key concerns, although the larger issue is about potentially missing out on <em>opportunity</em>. One problem: the US federal government — aside from perhaps the Intelligence Community — has not been renowned as an “early adopter” of technology.</p><p>Late last year I was co-chair for the “Rich Context” workshop in DC, which brought together expert practitioners from government, expert practitioners from Silicon Valley, plus top researchers from academia and corporate research labs. <strong>Ian Mulvany,</strong> also a co-chair there, helped wrangle metadata experts in the room into a “Foo Camp” styled unconference, including <strong>Natasha Noy</strong> from Google, <strong>Deborah McGuinness</strong> from RPI, <strong>Daniella Lowenberg</strong> from Dryad (University of California library system), and more. I had invited several colleagues, including Ed Kearns and his team handling dataset management at NOAA, plus my good friend <strong>Mark Grover,</strong> product manager for Amundsen at Lyft.</p><p>During that workshop I noticed something rare: people who had 30+ years working in the data-intensive parts of the US federal government got <em>really excited</em> about talking with leaders from data-intensive projects in Silicon Valley. One side thoroughly understood <em>policy</em>, <em>compliance</em>, <em>security</em>, <em>accountability</em>, <em>social impact.</em> One side thoroughly understood how to deliver technology solutions that scale, based on open source, while leveraging data with machine learning. Ed Kearns in one corner of the room talking fast with Mark Grover — <em>that</em> told me <em>most</em> of what I needed to know!</p><p>To wit, even though <em>decades</em> of data governance products have resulted in a balkanized field of <em>point solutions</em>, the stars appear to be aligning for the next stage of evolution. Indications include:</p><ul><li>Top research institutions (e.g., RISElab at UC Berkeley) are focusing on how to define the hard problems — “Something is definitely up!”</li><li>A flurry of recent open source projects from sophisticated tech firms in response to critical data governance needs, are earning ROI and gaining enterprise adoption — “Something is definitely up!”</li><li>VC firms are jumping into this space, defining a new category of tech start-up s— “Something is definitely up!”</li><li>Federal government recognizes the need and is reaching out to partner with Silicon Valley on solutions at scale — “Something is definitely up!”</li></ul><p>That last item is especially important. The more data-intensive US federal agencies have budgets that would make most Silicon Valley execs scream, or cry, or both. While we may talk about trillion-dollar market valuations for unicorns on NASDAQ, try comparing that with a decade’s worth of the federal budget. Moreover, given the world’s need for better handling of data (AI ethics, anyone? Urgent pandemic responses?) this is a case of <em>need</em> meeting <em>opportunity</em>. These stars are aligning.</p><p>Again, projects such as Amundsen began in response to crucial needs for GDPR compliance, but then led to business upside. Think about that at a much, much larger scale. These stars are aligning. Your work, given the ubiquity of data applications, will probably depend on these innovations, soon.</p><p><strong>Metadata Day</strong> is a workshop of leading practitioners in this field. We’ll hold a closed session in the morning and collaborate to reduce the outcomes into two presentations: one about <strong>use cases</strong>, and the other about <strong>production practices at scale</strong>.</p><figure><a href="https://metadataday2020.splashthat.com/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KGiHFfAtpdgTYtXddNbQA.png" /></a></figure><p>Metadata Day is a workshop of leading practitioners in this field. We’ll hold a closed session in the morning and collaborate to reduce the outcomes into two presentations: one about <strong>use cases</strong>, and the other about <strong>production practices at scale</strong>.</p><p>Several people I mentioned above will be among the expert practitioners participating in this workshop: Joe Hellerstein, Mandy Chessell, Deborah McGuinness, Natasha Noy, Daniella Lowenberg, Ian Mulvany, Mark Grover — and more. Plus we’ll have leads from several of the open source projects mentioned. <strong>Shirshanka Das</strong> (DataHub), <strong>Nadiya Hayes</strong>, and <strong>Kapil Surlaker</strong> from LinkedIn are hosting the event — and I’m helping them.</p><p><strong>Please join us!</strong></p><p>Register (free) for the online event at <a href="https://metadataday2020.splashthat.com/">https://metadataday2020.splashthat.com/</a></p><p>Social media hashtag: <a href="https://www.linkedin.com/feed/hashtag/?keywords=metaspeak2020&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A6739317571595718656">#metaspeak2020</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=220b7af5e4cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/knowledge-technologies/metaspeak-meetup-dec-14-2020-220b7af5e4cc">Metaspeak Meetup, Dec 14 2020</a> was originally published in <a href="https://medium.com/knowledge-technologies">Knowledge Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>