<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bill Chambers on Medium]]></title>
        <description><![CDATA[Stories by Bill Chambers on Medium]]></description>
        <link>https://medium.com/@wachambers?source=rss-9627ead2f75f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zcfRFQydq1PVdkOTC9GH4A.jpeg</url>
            <title>Stories by Bill Chambers on Medium</title>
            <link>https://medium.com/@wachambers?source=rss-9627ead2f75f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:46:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wachambers/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Structured Data Extraction using LLMs and Guardrails AI]]></title>
            <link>https://medium.com/@wachambers/structured-data-extraction-using-llms-and-guardrails-ai-3c1a104d8b87?source=rss-9627ead2f75f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c1a104d8b87</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Bill Chambers]]></dc:creator>
            <pubDate>Tue, 14 Nov 2023 22:09:51 GMT</pubDate>
            <atom:updated>2023-11-14T22:09:51.101Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lm_Z_yO3gX20enCRCSdevA.jpeg" /></figure><p>In this post, we’re going to be extracting structured data from a podcast transcript. We’ve all seen the ability for generative models to be effective <em>generating</em> text.</p><p><strong>But what about extracting data?</strong></p><p>Data extraction is a far more common use case (today) than generation, in particular for businesses. Businesses have to process all kinds of documents, exchange them and so on.</p><p><strong>Emerging Use Case: LLMs for data extraction</strong></p><p>If you think about it, summarization is effectively data extraction. We’ve seen LLMs perform well at summarization, but did you know that they can extract structured data quite well?</p><h3>What you’ll learn from this post</h3><p>In this post, you’ll learn how to extract structured data from LLMs into Pydantic objects using the <a href="https://www.guardrailsai.com/">Guardrails</a> library.</p><h3>Let’s setup our environment and confirm our versions</h3><p>This makes it easy for you to follow along! With this library, I had some serious issues getting the versions right!</p><pre>!python --version # Python 3.9.18</pre><pre>!pip freeze | grep guardrails-ai # guardrails-ai==0.2.4</pre><pre>import pydantic<br>print(pydantic.__version__) # 1.10.9<br>from dotenv import load_dotenv<br>load_dotenv()</pre><h3>Background on Guardrails AI</h3><p><a href="https://www.guardrailsai.com/">Guardrails</a> is a project that appears to be backed by a company. The project, aptly named Guardrails, is centered around the following concept:</p><blockquote><em>What is Guardrails?</em></blockquote><blockquote><em>Guardrails AI is an open-source library designed to ensure reliable interactions with Large Language Models (LLMs). It provides:</em></blockquote><blockquote><em>✅ A framework for creating custom validators ✅ Orchestration of prompting, verification, and re-prompting ✅ A library of commonly used validators for various use cases ✅ A specification language for communicating requirements to LLM</em></blockquote><p>Guardrails shares a similar philosophy with Instructor and Marvin.</p><blockquote><em>Guardrails AI enables you to define and enforce assurance for AI applications, from structuring output to quality controls. It achieves this by creating a ‘Guard’, a firewall-like bounding box around the LLM application, which contains a set of validators. A Guard can include validators from our library or a custom validator that enforces your application’s intended function.</em></blockquote><blockquote>from <a href="https://www.guardrailsai.com/blog/0.2-release">the blog</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5WqxR9E1VOOH7T-ut5dX4g.png" /><figcaption>the guardrails architecture according to the documentation</figcaption></figure><p>Currently, Guardrails is in beta and has not had a formal release, so it may not be suitable for production use cases.</p><p>However, Guardrails has a broader mission. They aim to create a “bounding box” around LLM apps to validate and ensure quality. They plan to achieve this by introducing a .RAIL file type, a dialect of XML. This ambitious approach extends beyond the simple concept of &quot;using AI to extract data&quot;.</p><p>Here’s their explanation of RAIL.</p><blockquote><em>🤖 What is </em><em>RAIL?</em></blockquote><blockquote><em>.RAIL is a dialect of XML, standing for &quot;</em><strong><em>R</em></strong><em>eliable </em><strong><em>AI</em></strong><em> markup </em><strong><em>L</em></strong><em>anguage&quot;. It can be used to define:</em></blockquote><blockquote>- The structure of the expected outcome of the LLM. (E.g. JSON)</blockquote><blockquote>- The type of each field in the expected outcome. (E.g. string, integer, list, object)</blockquote><blockquote>- The quality criteria for the expected outcome to be considered valid. (E.g. generated text should be bias-free, generated code should be bug-free)</blockquote><blockquote>- The corrective action to take in case the quality criteria is not met. (E.g. reask the question, filter the LLM, programmatically fix, etc.)</blockquote><p>Let’s get building with it</p><p><strong>Warning for the reader…🚨</strong></p><p>Another challenge I had was the dependencies, Guardrails requires pydantic version 1.10.9. <a href="https://docs.pydantic.dev/latest/">Pydantic</a>&#39;s latest version is <a href="https://github.com/pydantic/pydantic/releases/tag/v2.4.1">v.2.4.1</a>. This caused a number of issues for me during installation and I had to setup a dedicated environment for it. A challenge is that<a href="https://github.com/pydantic/pydantic#pydantic-v110-vs-v2"> pydantic went through a ground up rewrite from version 1.10</a>, so it may simply be challenging for them to rewrite everything.</p><pre>import feedparser<br>podcast_atom_link = &quot;https://api.substack.com/feed/podcast/1084089.rss&quot; # latent space podcastbbbbb<br>parsed = feedparser.parse(podcast_atom_link)<br>episode = [ep for ep in parsed.entries if ep[&#39;title&#39;] == &quot;Why AI Agents Don&#39;t Work (yet) - with Kanjun Qiu of Imbue&quot;][0]<br>episode_summary = episode[&#39;summary&#39;]<br>print(episode_summary[:100])</pre><pre>from unstructured.partition.html import partition_html<br>parsed_summary = partition_html(text=&#39;&#39;.join(episode_summary)) <br>start_of_transcript = [x.text for x in parsed_summary].index(&quot;Transcript&quot;) + 1<br>print(f&quot;First line of the transcript: {start_of_transcript}&quot;)<br>text = &#39;\n&#39;.join(t.text for t in parsed_summary[start_of_transcript:])<br>text = text[:3508] # shortening the transcript for speed &amp; cost</pre><pre>First line of the transcript: 58</pre><h3>Using Guardrails</h3><p>Guardrails is similar to <a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">other libraries for structured data extraction</a> that I’ve used. We define our pydantic models and then we specify an output class and a prompt.</p><pre>import guardrails as gd<br>from pydantic import BaseModel<br>from typing import Optional, List<br>from pydantic import Field<br>class Person(BaseModel):<br>    name: str<br>    school: Optional[str] = Field(..., description=&quot;The school this person attended&quot;)<br>    company: Optional[str] = Field(..., description=&quot;The company this person works for&quot;)<br>class People(BaseModel):<br>    people: List[Person]<br>guard = gd.Guard.from_pydantic(output_class=People, prompt=&quot;Get the following objects from the text:\n\n ${text}&quot;)</pre><p>The important thing to note, like we mentioned above, is that Guardrails is translating this into a “RAIL” an underlying abstraction for representing these extractions.</p><p>Now a key difference is that this abstraction might allow for structured data extraction from ANY model, not just OpenAI’s function calling interface.</p><pre>import openai<br>import os<br><br>raw_llm_output, validated_output = guard(<br>    openai.ChatCompletion.create,<br>    prompt_params={&quot;text&quot;: text},<br>)<br>print(validated_output)</pre><pre>{&#39;people&#39;: [{&#39;name&#39;: &#39;Alessio&#39;, &#39;school&#39;: &#39;Residence at Decibel Partners&#39;, &#39;company&#39;: &#39;CTO&#39;}, {&#39;name&#39;: &#39;Swyx&#39;, &#39;school&#39;: &#39;Smol.ai&#39;, &#39;company&#39;: &#39;founder&#39;}, {&#39;name&#39;: &#39;Kanjun&#39;, &#39;school&#39;: &#39;Imbue&#39;, &#39;company&#39;: &#39;founder&#39;}]}</pre><p>It did a fine job getting the people, although you can see that it got the company and schools wrong — especially compared to the other libraries we’ve looked at, this is a lower quality result.</p><p>let’s see how it does when we ask for more objects.</p><pre>class Company(BaseModel):<br>    name:str<br><br>class ResearchPaper(BaseModel):<br>    paper_name:str = Field(..., description=&quot;an academic paper reference discussed&quot;)<br>    <br>class ExtractedInfo(BaseModel):<br>    people: List[Person]<br>    companies: List[Company]<br>    research_papers: Optional[List[ResearchPaper]]<br>guard = gd.Guard.from_pydantic(output_class=ExtractedInfo, prompt=&quot;Get the following objects from the text:\n\n ${text}&quot;)<br>raw_llm_output, validated_output = guard(<br>    openai.ChatCompletion.create,<br>    prompt_params={&quot;text&quot;: text},<br>)<br>print(validated_output)</pre><pre>ERROR!<br>incorrect_value={&#39;people&#39;: [{&#39;name&#39;: &#39;Alessio&#39;, &#39;school&#39;: &#39;Decibel Partners&#39;, &#39;company&#39;: &#39;CTO&#39;}, {&#39;name&#39;: &#39;Swyx&#39;, &#39;school&#39;: &#39;Smol.ai&#39;, &#39;company&#39;: &#39;founder&#39;}, {&#39;name&#39;: &#39;Kanjun&#39;, &#39;school&#39;: &#39;Imbue&#39;, &#39;company&#39;: &#39;founder&#39;}], &#39;companies&#39;: [{&#39;name&#39;: &#39;Decibel Partners&#39;}, {&#39;name&#39;: &#39;Residence&#39;}, {&#39;name&#39;: &#39;Smol.ai&#39;}, {&#39;name&#39;: &#39;Imbue&#39;}]} fail_results=[FailResult(outcome=&#39;fail&#39;, metadata=None, error_message=&#39;JSON does not match schema&#39;, fix_value=None)]</pre><p>Unfortunately, the result failed to parse, while the other two <a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">structured data extraction libraries</a> I’ve looked at succeeded just fine. It might be in the implementation or might be a bug in this version, but it didn’t work out of the box.</p><p>While the example didn’t work, you can see how well this <em>should work</em>. These models and libraries grab a whole bunch of structured data for us with basically no work. Right now we’re just working on a excerpt of this data, but with basically zero NLP specific work, we’re able to get some pretty powerful results.</p><h3>Wrapping it all up</h3><p>The sky is really the limit here. There’s so much structured data that’s been locked up in unstructured text. I’m bullish on this space and can’t wait to see what you build with this tool!</p><p>If you found this interesting, you might want to see similar posts on:</p><ol><li><a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">Comparing 3 Data Extraction Libraries: Marvin, Instructor, and Guardrails</a></li><li><a href="https://learnbybuilding.ai/tutorials/structured-data-extraction-with-marvin-ai-and-llms">Structured Data Extraction using LLMs and Marvin</a></li><li><a href="https://learnbybuilding.ai/tutorials/structured-data-extraction-with-instructor-and-llms">Structured Data Extraction using LLMs and Instructor</a></li></ol><p>This post was originally published on <a href="https://learnbybuilding.ai">LearnByBuilding.AI</a>. <em>See the </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/structured-data-extraction-with-guardrails-and-llms.ipynb"><em>notebook</em></a><em> and </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/structured-data-extraction-with-guardrails-and-llms.py"><em>code</em></a><em> on </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/"><em>Github</em></a><em>.</em> Please reach out to me on <a href="https://twitter.com/bllchmbrs">twitter</a> with any questions or feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c1a104d8b87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structured Data Extraction using LLMs and Instructor]]></title>
            <link>https://medium.com/@wachambers/structured-data-extraction-using-llms-and-instructor-89606561e72f?source=rss-9627ead2f75f------2</link>
            <guid isPermaLink="false">https://medium.com/p/89606561e72f</guid>
            <category><![CDATA[gpt-4]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Bill Chambers]]></dc:creator>
            <pubDate>Mon, 06 Nov 2023 17:10:56 GMT</pubDate>
            <atom:updated>2023-11-06T17:13:02.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pWcuF1Xfr6_cuMdcggYVKQ.jpeg" /></figure><p>In this post, we’re going to be extracting structured data from a podcast transcript. We’ve all seen the ability for generative models to be effective <em>generating</em> text.</p><p><strong>But what about extracting data?</strong></p><p>Data extraction is a far more common use case (today) than generation, in particular for businesses. Businesses have to process all kinds of documents, exchange them and so on.</p><p><strong>Emerging Use Case: LLMs for data extraction</strong></p><p>If you think about it, summarization is effectively data extraction. We’ve seen LLMs perform well at summarization, but did you know that they can extract structured data quite well?</p><h3>What you’ll learn from this post</h3><p>In this post, you’ll learn how to extract structured data from LLMs into Pydantic objects using the <a href="https://jxnl.github.io/instructor/">Instructor</a> library.</p><h3>Here’s us setting up on environment</h3><p>We’re using Python 3.11 and instructor==0.2.9 and pydantic==2.4.2</p><h3>Background on Instructor</h3><p><a href="https://jxnl.github.io/instructor/">Instructor</a> is a library that helps get structured data out of LLMs. It has narrow dependencies and a simple API, requiring basic nothing sophisticated from the end user.</p><p>In the author’s words:</p><blockquote><em>Instructor helps to ensure you get the exact response type you&#39;re looking for when using openai&#39;s function call api. Once you&#39;ve defined the </em><em>Pydantic model for your desired response, </em><em>Instructor handles all the complicated logic in-between - from the parsing/validation of the response to the automatic retries for invalid responses. This means that we can build in validators &#39;for free&#39; and have a clear separation of concerns between the prompt and the code that calls OpenAI.</em></blockquote><p>The library is still early, version 0.2.9 has a modest star count (1300) but hits above it’s weight since it’s basically a single person working on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZxHFbJIy5lm_Qk8mD_wLVA.png" /></figure><p>Now just because it’s one person doesn’t mean it’s bad. In fact, I found it to be very simple to use. It just means that from a long term maintenance perspective, it might be challenging. Something to note.</p><h3>Pre-processing the data</h3><p>Here, we are fetching and parsing an RSS feed from a podcast. We specifically target an episode by its title and then retrieve its summary from the podcast description. This is similar to a <a href="https://learnbybuilding.ai/tutorials/rag-chatbot-on-podcast-llamaindex-faiss-openai">tutorial on llamaindex</a> that we recently published.</p><pre>import feedparser<br>podcast_atom_link = &quot;https://api.substack.com/feed/podcast/1084089.rss&quot; # latent space podcastbbbbb<br>parsed = feedparser.parse(podcast_atom_link)<br>episode = [ep for ep in parsed.entries if ep[&#39;title&#39;] == &quot;Why AI Agents Don&#39;t Work (yet) - with Kanjun Qiu of Imbue&quot;][0]<br>episode_summary = episode[&#39;summary&#39;]<br>print(episode_summary[:100])</pre><pre>&lt;p&gt;&lt;em&gt;Thanks to the &lt;/em&gt;&lt;em&gt;over 11,000 people&lt;/em&gt;&lt;em&gt; who joined us for the first AI Engineer Su</pre><p>Now we’re going to extract the shortened episode summary. We’ll also shorten the summary just to speed up our extraction. In the future, we can extend the transcript to cover the whole thing.</p><pre>from unstructured.partition.html import partition_html<br>parsed_summary = partition_html(text=&#39;&#39;.join(episode_summary)) <br>start_of_transcript = [x.text for x in parsed_summary].index(&quot;Transcript&quot;) + 1<br>print(f&quot;First line of the transcript: {start_of_transcript}&quot;)<br>text = &#39;\n&#39;.join(t.text for t in parsed_summary[start_of_transcript:])<br>text = text[:3508] # shortening the transcript for speed &amp; cost</pre><pre>First line of the transcript: 58</pre><h3>Using Instructor</h3><p>Instructor is the “thinnest” of <a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">the libraries that I’ve used for structured data extraction</a>. The interface is super simple.</p><pre>from pydantic import BaseModel<br>from typing import Optional, List<br>from pydantic import Field<br><br>class Person(BaseModel):<br>    name: str<br>    school: Optional[str] = Field(..., description=&quot;The school this person attended&quot;)<br>    company: Optional[str] = Field(..., description=&quot;The company this person works for &quot;)<br>class People(BaseModel):<br>    people: List[Person]</pre><p>All we do is <a href="https://en.wikipedia.org/wiki/Monkey_patch#:~:text=Monkey%20patching%20is%20a%20technique,Python%2C%20Groovy%2C%20etc.">Monkey patch</a>) OpenAI’s SDK.</p><p><strong>Note:</strong> This could be hard to maintain, so take note of versions and be sure to not let either one get upgraded without testing.</p><pre>import openai<br>import instructor<br>instructor.patch()</pre><p>Now we simply call OpenAI but ask for a response_model. This is the instructor patch at work!</p><pre>response = openai.ChatCompletion.create(<br>    model=&quot;gpt-4&quot;,<br>    response_model=People,<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: text},<br>    ]<br>)<br>print(response)</pre><pre>people=[Person(name=&#39;Alessio&#39;, school=None, company=&#39;Decibel Partners&#39;), Person(name=&#39;Swyx&#39;, school=None, company=&#39;Smol.ai&#39;), Person(name=&#39;Kanjun&#39;, school=&#39;MIT&#39;, company=&#39;Imbue&#39;), Person(name=&#39;Josh&#39;, school=None, company=&#39;Imbue&#39;)]</pre><p>Overall the result is high quality, we’ve gotten all the names and companies.</p><p>Now we can take this to the next level by trying to extract multiple objects at the same time.</p><pre>class Company(BaseModel):<br>    name:str<br><br>class ResearchPaper(BaseModel):<br>    paper_name:str = Field(..., description=&quot;an academic paper reference discussed&quot;)<br>    <br>class ExtractedInfo(BaseModel):<br>    people: List[Person]<br>    companies: List[Company]<br>    research_papers: Optional[List[ResearchPaper]]<br>response = openai.ChatCompletion.create(<br>    model=&quot;gpt-4&quot;,<br>    response_model=ExtractedInfo,<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: text},<br>    ]<br>)<br>print(response)</pre><pre>people=[Person(name=&#39;Alessio&#39;, school=None, company=&#39;Decibel Partners&#39;), Person(name=&#39;Swyx&#39;, school=None, company=&#39;Smol.ai&#39;), Person(name=&#39;Kanjun&#39;, school=&#39;MIT&#39;, company=&#39;Imbue&#39;), Person(name=&#39;Josh&#39;, school=None, company=&#39;Ember&#39;)] companies=[Company(name=&#39;Decibel Partners&#39;), Company(name=&#39;Smol.ai&#39;), Company(name=&#39;Imbue&#39;), Company(name=&#39;Generally Intelligent&#39;), Company(name=&#39;Ember&#39;), Company(name=&#39;Sorceress&#39;), Company(name=&#39;Dropbox&#39;), Company(name=&#39;MIT Media Lab&#39;), Company(name=&#39;OpenA&#39;)] research_papers=None</pre><p>You can see how well this works, to grab a whole bunch of structured data for us with basically no work. Right now we’re just working on a excerpt of this data, but with basically zero NLP specific work, we’re able to get some pretty powerful results.</p><h3>Wrapping it all up</h3><p>The sky is really the limit here. There’s so much structured data that’s been locked up in unstructured text. I’m bullish on this space and can’t wait to see what you build with this tool!</p><p>If you found this interesting, you might want to see similar posts on:</p><ol><li><a href="https://learnbybuilding.ai/vs/marvin-ai-vs-guardrails-vs-instructor">Comparing 3 Data Extraction Libraries: Marvin, Instructor, and Guardrails</a></li><li><a href="https://learnbybuilding.ai/tutorials/structured-data-extraction-with-marvin-ai-and-llms">Structured Data Extraction using LLMs and Marvin</a></li><li><a href="https://learnbybuilding.ai/tutorials/structured-data-extraction-with-guardrails-and-llms">Structured Data Extraction using LLMs and Guardrails AI</a></li></ol><p>This post was originally published on <a href="https://learnbybuilding.ai">LearnByBuilding.AI</a>. <em>See the </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/structured-data-extraction-with-instructor-and-llms.ipynb"><em>notebook</em></a><em> and </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/structured-data-extraction-with-instructor-and-llms.py"><em>code</em></a><em> on </em><a href="https://github.com/bllchmbrs/learnbybuilding.ai/blob/main/basic-data-extraction-llms/"><em>Github</em></a><em>.</em> Please reach out to me on <a href="https://twitter.com/bllchmbrs">twitter</a> with any questions or feedback!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89606561e72f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A beginner’s guide to building a Retrieval Augmented Generation (RAG) application from scratch]]></title>
            <link>https://medium.com/data-science/a-beginners-guide-to-building-a-retrieval-augmented-generation-rag-application-from-scratch-e52921953a5d?source=rss-9627ead2f75f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e52921953a5d</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[retrieval-generation]]></category>
            <dc:creator><![CDATA[Bill Chambers]]></dc:creator>
            <pubDate>Thu, 02 Nov 2023 20:46:23 GMT</pubDate>
            <atom:updated>2024-03-12T13:38:24.617Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn critical knowledge for building AI apps, in plain english</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3oplW4uGMLIGs9qs56RVg.jpeg" /></figure><h3>A beginner’s guide to building a Retrieval Augmented Generation (RAG) application from scratch</h3><p>Retrieval Augmented Generation, or RAG, is all the rage these days because it introduces some serious capabilities to large language models like OpenAI’s GPT-4 — and that’s the ability to use and leverage their own data.</p><p>This post will teach you the fundamental intuition behind RAG while providing a simple tutorial to help you get started.</p><h3>The problem with learning in a fast moving space</h3><p>There’s so much noise in the AI space and in particular about RAG. Vendors are trying to overcomplicate it. They’re trying to inject their tools, their ecosystems, their vision.</p><p>It’s making RAG way more complicated than it needs to be. This tutorial is designed to help beginners learn how to build RAG applications from scratch. No fluff, no (ok, minimal) jargon, no libraries, just a simple step by step RAG application.</p><p><a href="https://twitter.com/jerryjliu0/status/1716122650836439478">Jerry from LlamaIndex advocates for building things from scratch to really understand the pieces</a>. Once you do, using a library like LlamaIndex makes more sense.</p><h3>Jerry Liu on Twitter: &quot;If you 1) already know RAG basics, and 2) want to become a superstar AI engineer, then learn to build advanced RAG from scratch 💪Excited to launch a new category of @llama_index tutorials on this exact topic 🧑‍🏫🪐: ✅ learn advanced IR/AI concepts ✅ tackle more complex user... pic.twitter.com/vbY0tjokoM / Twitter&quot;</h3><p>If you 1) already know RAG basics, and 2) want to become a superstar AI engineer, then learn to build advanced RAG from scratch 💪Excited to launch a new category of @llama_index tutorials on this exact topic 🧑‍🏫🪐: ✅ learn advanced IR/AI concepts ✅ tackle more complex user... pic.twitter.com/vbY0tjokoM</p><p>Build from scratch to learn, then build with libraries to scale.</p><p>Let’s get started!</p><h3>Introducing our concept: Retrieval Augmented Generation</h3><p>You may or may not have heard of Retrieval Augmented Generation or RAG.</p><p>Here’s the definition from <a href="https://ai.meta.com/blog/retrieval-augmented-generation-streamlining-the-creation-of-intelligent-natural-language-processing-models/">the blog post introducing the concept from Facebook</a>:</p><blockquote><em>Building a model that researches and contextualizes is more challenging, but it’s essential for future advancements. We recently made substantial progress in this realm with our Retrieval Augmented Generation (RAG) architecture, an end-to-end differentiable model that combines an information retrieval component (Facebook AI’s dense-passage retrieval system) with a seq2seq generator (our Bidirectional and Auto-Regressive Transformers [BART] model). RAG can be fine-tuned on knowledge-intensive downstream tasks to achieve state-of-the-art results compared with even the largest pretrained seq2seq language models. And unlike these pretrained models, RAG’s internal knowledge can be easily altered or even supplemented on the fly, enabling researchers and engineers to control what RAG knows and doesn’t know without wasting time or compute power retraining the entire model.</em></blockquote><p>Wow, that’s a mouthful.</p><p>In simplifying the technique for beginners, we can state that the essence of RAG involves adding your own data (via a retrieval tool) to the prompt that you pass into a large language model. As a result, you get an output. That gives you several benefits:</p><ol><li>You can include facts in the prompt to help the LLM avoid hallucinations</li><li>You can (manually) refer to sources of truth when responding to a user query, helping to double check any potential issues.</li><li>You can leverage data that the LLM might not have been trained on.</li></ol><h3>The High Level Components of our RAG System</h3><ol><li>a collection of documents (formally called a corpus)</li><li>An input from the user</li><li>a similarity measure between the collection of documents and the user input</li></ol><p>Yes, it’s that simple.</p><p>To start learning and understanding RAG based systems, you don’t need a vector store, you don’t even <em>need</em> an LLM (at least to learn and understand conceptually).</p><p>While it is often portrayed as complicated, it doesn’t have to be.</p><h3>The ordered steps of a querying RAG system</h3><p>We’ll perform the following steps in sequence.</p><ol><li>Receive a user input</li><li>Perform our similarity measure</li><li>Post-process the user input and the fetched document(s).</li></ol><p>The post-processing is done with an LLM.</p><h3>A note from the paper itself</h3><p><a href="https://arxiv.org/abs/2005.11401">The actual RAG paper</a> is obviously <em>the</em> resource. The problem is that it assumes a LOT of context. It’s more complicated than we need it to be.</p><p>For instance, here’s the overview of the RAG system as proposed in the paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSa4MllBej2tLx-WThVNRw.png" /><figcaption><a href="https://arxiv.org/abs/2005.11401">An overview of RAG from the RAG paper</a> by Lewis, et al</figcaption></figure><p>That’s dense.</p><p>It’s great for researchers but for the rest of us, it’s going to be a lot easier to learn step by step by building the system ourselves.</p><h3>Working through an example — the simplest RAG system</h3><p>Let’s get back to building RAG from scratch, step by step. Here’s the simplified steps that we’ll be working through. While this isn’t technically “RAG” it’s a good simplified model to learn with and allow us to progress to more complicated variations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVX9IzuCTe6eG3SXnbcxWA.jpeg" /></figure><h3>Getting a collection of documents</h3><p>Below you can see that we’ve got a simple corpus of ‘documents’ (please be generous 😉).</p><pre>corpus_of_documents = [<br>    &quot;Take a leisurely walk in the park and enjoy the fresh air.&quot;,<br>    &quot;Visit a local museum and discover something new.&quot;,<br>    &quot;Attend a live music concert and feel the rhythm.&quot;,<br>    &quot;Go for a hike and admire the natural scenery.&quot;,<br>    &quot;Have a picnic with friends and share some laughs.&quot;,<br>    &quot;Explore a new cuisine by dining at an ethnic restaurant.&quot;,<br>    &quot;Take a yoga class and stretch your body and mind.&quot;,<br>    &quot;Join a local sports league and enjoy some friendly competition.&quot;,<br>    &quot;Attend a workshop or lecture on a topic you&#39;re interested in.&quot;,<br>    &quot;Visit an amusement park and ride the roller coasters.&quot;<br>]</pre><h3>Defining and performing the similarity measure</h3><p>Now we need a way of measuring the similarity between the <strong>user input</strong> we’re going to receive and the <strong>collection</strong> of documents that we organized. Arguably the simplest similarity measure is <a href="https://en.wikipedia.org/wiki/Jaccard_index">jaccard similarity</a>. I’ve written about that in the past (see <a href="https://billchambers.me/posts/tf-idf-explained-in-python">this post</a> but the short answer is that the <strong>jaccard similarity</strong> is the intersection divided by the union of the “sets” of words.</p><p>This allows us to compare our user input with the source documents.</p><h4>Side note: preprocessing</h4><p>A challenge is that if we have a plain string like &quot;Take a leisurely walk in the park and enjoy the fresh air.&quot;,, we&#39;re going to have to pre-process that into a set, so that we can perform these comparisons. We&#39;re going to do this in the simplest way possible, lower case and split by &quot; &quot;.</p><pre>def jaccard_similarity(query, document):<br>    query = query.lower().split(&quot; &quot;)<br>    document = document.lower().split(&quot; &quot;)<br>    intersection = set(query).intersection(set(document))<br>    union = set(query).union(set(document))<br>    return len(intersection)/len(union)</pre><p>Now we need to define a function that takes in the exact query and our corpus and selects the ‘best’ document to return to the user.</p><pre>def return_response(query, corpus):<br>    similarities = []<br>    for doc in corpus:<br>        similarity = jaccard_similarity(query, doc)<br>        similarities.append(similarity)<br>    return corpus_of_documents[similarities.index(max(similarities))]</pre><p>Now we can run it, we’ll start with a simple prompt.</p><pre>user_prompt = &quot;What is a leisure activity that you like?&quot;</pre><p>And a simple user input…</p><pre>user_input = &quot;I like to hike&quot;</pre><p>Now we can return our response.</p><pre>return_response(user_input, corpus_of_documents)</pre><pre>&#39;Go for a hike and admire the natural scenery.&#39;</pre><p>Congratulations, you’ve built a basic RAG application.</p><h4>I got 99 problems and bad similarity is one</h4><p>Now we’ve opted for a simple similarity measure for learning. But this is going to be problematic because it’s so simple. It has no notion of <strong>semantics</strong>. It’s just looks at what words are in both documents. That means that if we provide a negative example, we’re going to get the same “result” because that’s the closest document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7U4L-psR4_XW4dsYuiCfZQ.jpeg" /></figure><pre>user_input = &quot;I don&#39;t like to hike&quot;</pre><pre>return_response(user_input, corpus_of_documents)</pre><pre>&#39;Go for a hike and admire the natural scenery.&#39;</pre><p>This is a topic that’s going to come up a lot with “RAG”, but for now, rest assured that we’ll address this problem later.</p><p>At this point, we have not done any post-processing of the “document” to which we are responding. So far, we’ve implemented only the “retrieval” part of “Retrieval-Augmented Generation”. The next step is to augment generation by incorporating a large language model (LLM).</p><h3>Adding in a LLM</h3><p>To do this, we’re going to use <a href="https://ollama.ai/">ollama</a> to get up and running with an open source LLM on our local machine. We could just as easily use OpenAI’s gpt-4 or Anthropic’s Claude but for now, we’ll start with the open source llama2 from <a href="https://ai.meta.com/llama/">Meta AI</a>.</p><ul><li><a href="https://ollama.ai/">ollama installation instructions are here</a></li></ul><p>This post is going to assume some basic knowledge of large language models, so let’s get right to querying this model.</p><pre>import requests<br>import json</pre><p>First we’re going to define the inputs. To work with this model, we’re going to take</p><ol><li>user input,</li><li>fetch the most similar document (as measured by our similarity measure),</li><li>pass that into a prompt to the language model,</li><li><em>then</em> return the result to the user</li></ol><p>That introduces a new term, the <strong>prompt</strong>. In short, it’s the instructions that you provide to the LLM.</p><p>When you run this code, you’ll see the streaming result. Streaming is important for user experience.</p><pre>user_input = &quot;I like to hike&quot;<br>relevant_document = return_response(user_input, corpus_of_documents)<br>full_response = []</pre><pre>prompt = &quot;&quot;&quot;<br>You are a bot that makes recommendations for activities. You answer in very short sentences and do not include extra information.<br>This is the recommended activity: {relevant_document}<br>The user input is: {user_input}<br>Compile a recommendation to the user based on the recommended activity and the user input.<br>&quot;&quot;&quot;</pre><p>Having defined that, let’s now make the API call to ollama (and llama2). an important step is to make sure that ollama’s running already on your local machine by running ollama serve.</p><blockquote><em>Note: this might be slow on your machine, it’s certainly slow on mine. Be patient, young grasshopper.</em></blockquote><pre>url = &#39;http://localhost:11434/api/generate&#39;<br>data = {<br>    &quot;model&quot;: &quot;llama2&quot;,<br>    &quot;prompt&quot;: prompt.format(user_input=user_input, relevant_document=relevant_document)<br>}</pre><pre>headers = {&#39;Content-Type&#39;: &#39;application/json&#39;}<br>response = requests.post(url, data=json.dumps(data), headers=headers, stream=True)<br>try:<br>    count = 0<br>    for line in response.iter_lines():<br>        # filter out keep-alive new lines<br>        # count += 1<br>        # if count % 5== 0:<br>        #     print(decoded_line[&#39;response&#39;]) # print every fifth token<br>        if line:<br>            decoded_line = json.loads(line.decode(&#39;utf-8&#39;))<br>            <br>            full_response.append(decoded_line[&#39;response&#39;])<br>finally:<br>    response.close()<br>print(&#39;&#39;.join(full_response))</pre><pre>Great! Based on your interest in hiking, I recommend trying out the nearby trails for a challenging and rewarding experience with breathtaking views Great! Based on your interest in hiking, I recommend checking out the nearby trails for a fun and challenging adventure.</pre><p>This gives us a complete RAG Application, from scratch, no providers, no services. You know all of the components in a Retrieval-Augmented Generation application. Visually, here’s what we’ve built.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-fWNo-SVYYmm_tepIoVuw.jpeg" /></figure><p>The LLM (if you’re lucky) will handle the user input that goes against the recommended document. We can see that below.</p><pre>user_input = &quot;I don&#39;t like to hike&quot;<br>relevant_document = return_response(user_input, corpus_of_documents)<br># https://github.com/jmorganca/ollama/blob/main/docs/api.md<br>full_response = []</pre><pre>prompt = &quot;&quot;&quot;<br>You are a bot that makes recommendations for activities. You answer in very short sentences and do not include extra information.<br>This is the recommended activity: {relevant_document}<br>The user input is: {user_input}<br>Compile a recommendation to the user based on the recommended activity and the user input.<br>&quot;&quot;&quot;<br>url = &#39;http://localhost:11434/api/generate&#39;<br>data = {<br>    &quot;model&quot;: &quot;llama2&quot;,<br>    &quot;prompt&quot;: prompt.format(user_input=user_input, relevant_document=relevant_document)<br>}<br>headers = {&#39;Content-Type&#39;: &#39;application/json&#39;}<br>response = requests.post(url, data=json.dumps(data), headers=headers, stream=True)<br>try:<br>    for line in response.iter_lines():<br>        # filter out keep-alive new lines<br>        if line:<br>            decoded_line = json.loads(line.decode(&#39;utf-8&#39;))<br>            # print(decoded_line[&#39;response&#39;])  # uncomment to results, token by token<br>            full_response.append(decoded_line[&#39;response&#39;])<br>finally:<br>    response.close()<br>print(&#39;&#39;.join(full_response))</pre><pre>Sure, here is my response:<br><br>Try kayaking instead! It&#39;s a great way to enjoy nature without having to hike.</pre><h3>Areas for improvement</h3><p>If we go back to our diagream of the RAG application and think about what we’ve just built, we’ll see various opportunities for improvement. These opportunities are where tools like vector stores, embeddings, and prompt ‘engineering’ gets involved.</p><p>Here are ten potential areas where we could improve the current setup:</p><ol><li><strong>The number of documents</strong> 👉 more documents might mean more recommendations.</li><li><strong>The depth/size of documents</strong> 👉 higher quality content and longer documents with more information might be better.</li><li><strong>The number of documents we give to the LLM</strong> 👉 Right now, we’re only giving the LLM one document. We could feed in several as ‘context’ and allow the model to provide a more personalized recommendation based on the user input.</li><li><strong>The parts of documents that we give to the LLM</strong> 👉 If we have bigger or more thorough documents, we might just want to add in parts of those documents, parts of various documents, or some variation there of. In the lexicon, this is called chunking.</li><li><strong>Our document storage tool</strong> 👉 We might store our documents in a different way or different database. In particular, if we have a lot of documents, we might explore storing them in a data lake or a vector store.</li><li><strong>The similarity measure</strong> 👉 How we measure similarity is of consequence, we might need to trade off performance and thoroughness (e.g., looking at every individual document).</li><li><strong>The pre-processing of the documents &amp; user input</strong> 👉 We might perform some extra preprocessing or augmentation of the user input before we pass it into the similarity measure. For instance, we might use an embedding to convert that input to a vector.</li><li><strong>The similarity measure</strong> 👉 We can change the similarity measure to fetch better or more relevant documents.</li><li><strong>The model</strong> 👉 We can change the final model that we use. We’re using llama2 above, but we could just as easily use an Anthropic or Claude Model.</li><li><strong>The prompt</strong> 👉 We could use a different prompt into the LLM/Model and tune it according to the output we want to get the output we want.</li><li><strong>If you’re worried about harmful or toxic output</strong> 👉 We could implement a “circuit breaker” of sorts that runs the user input to see if there’s toxic, harmful, or dangerous discussions. For instance, in a healthcare context you could see if the information contained unsafe languages and respond accordingly — outside of the typical flow.</li></ol><p>The scope for improvements isn’t limited to these points; the possibilities are vast, and we’ll delve into them in future tutorials. Until then, don’t hesitate to <a href="https://twitter.com/bllchmbrs">reach out on Twitter</a> if you have any questions. Happy RAGING :).</p><h3>References</h3><ul><li><a href="https://arxiv.org/abs/2005.11401">Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</a></li><li><a href="https://twitter.com/jerryjliu0/status/1716122650836439478">Jerry Liu on Twitter advocating for users to build RAG from scratch</a></li></ul><p><a href="https://learnbybuilding.ai/tutorials/rag-from-scratch">This post was originally posted on learnbybuilding.ai</a>. <strong>I’m running a course on How to Build Generative AI Products for Product Managers in the coming months, </strong><a href="https://maven.com/forms/90684f"><strong>sign up here</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e52921953a5d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-beginners-guide-to-building-a-retrieval-augmented-generation-rag-application-from-scratch-e52921953a5d">A beginner’s guide to building a Retrieval Augmented Generation (RAG) application from scratch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Coming Rise of the Citizen AI Engineer]]></title>
            <link>https://medium.com/@wachambers/the-coming-rise-of-the-citizen-ai-engineer-896ca60b9d0d?source=rss-9627ead2f75f------2</link>
            <guid isPermaLink="false">https://medium.com/p/896ca60b9d0d</guid>
            <category><![CDATA[code-interpreter]]></category>
            <category><![CDATA[ai-engineer]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[ai̇]]></category>
            <dc:creator><![CDATA[Bill Chambers]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 15:56:07 GMT</pubDate>
            <atom:updated>2023-09-12T15:56:07.052Z</atom:updated>
            <content:encoded><![CDATA[<p>This post is going to introduce a role of the future. <strong>The Citizen AI Engineer.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZWV2LsyKmKI1uCZA.png" /></figure><p>Walking into the future, hand in hand with a robot.</p><p>This role is fundamentally nontechnical but that heavily uses AI to solve business problems.</p><p>100s if not 1000s of new products will be built to service this end user. Let’s review some background, history, and get to know this coming role better. Then we’ll discuss the competitive landscape for serving this user.</p><h3>Background: AI Engineering</h3><p><a href="https://www.latent.space/p/ai-engineer">The Rise of the AI Engineer on Latent Space</a> is an insightful post about the generational shift in applied AI. In short, developers who are not applied researchers (or ML engineers) can now build incredibly powerful AI-powered applications. This development is set to change the game for AI adoption.</p><p>You should read <a href="https://www.latent.space/p/ai-engineer">the post</a>, but here’s my TL;DR: changes in technology are allowing a new role to emerge at companies of all sizes. The two key technologies are:</p><ol><li><strong>Foundation models</strong> — Developers can build with AI without needing to understand AI.</li><li><strong>Foundational AI tools</strong> — Tools like <a href="https://www.langchain.com/">LangChain</a>, <a href="https://www.llamaindex.ai/">LlamaIndex</a>, and others make it easy to integrate foundation models and providers into various applications. These exist across 2 dominant languages, Python and JavaScript.</li></ol><p>The tools mentioned above are shifting the balance from ML engineering and ML researchers (a specialized skillset that will remain important but requires more training and experience) to the vast number of developers who know Python and JavaScript. <a href="https://substack.com/@swyx">Swyx</a> put the consequence concisely:</p><blockquote><em>A wide range of AI tasks that used to take </em><a href="https://xkcd.com/1425/"><em>5 years and a research team</em></a><em> to accomplish in </em><strong><em>2013</em></strong><em>, now just require API docs and a spare afternoon in </em><strong><em>2023</em></strong><em>.</em></blockquote><p>Lastly, while it’s not technology, the shear amount of news, excitement, attention, and money in AI is creating a Cambrian explosion of new projects, tools, and products in the AI domain. Our economy is built around what gets attention. AI has sucked the air out of the room.</p><p>In short, Swyx posits a new role, the “AI Engineer,” is likely to emerge as a dominant role in software engineering over the next decade.</p><p>I agree AI Engineers will rise. However, I argue there’s even more to come that will have larger impact: <strong>The Citizen AI Engineer.</strong></p><h3>The Citizen AI Engineer</h3><p>Just as we’ve seen <a href="https://www.gartner.com/smarterwithgartner/how-to-use-citizen-data-scientists-to-maximize-your-da-strategy">the rise of Citizen Data Scientists</a> over the past decade, I posit that the next decade will see the rise of <strong>the Citizen AI Engineer.</strong> This role will overshadow AI Engineers because of shear numbers. It is a business user, someone non-technical, who gains significant capabilities with LLMs on their own data.</p><h3>What is a Citizen AI Engineer?</h3><p>A Citizen AI Engineer is a role in which a business-focused, but fundamentally non-developer persona, will be able to leverage AI tools to build and ship all kinds of new products within enterprises — without a heavy reliance on developers (day to day).</p><p>Citizen AI engineers will use tools like ChatGPT and tools custom-built for them by AI engineers, to drive large-scale business changes in everything from operations to forecasting and supply chains.</p><h3>Why Now?</h3><p>There are several trends that are going to enable this user to succeed.</p><ol><li><strong>General Purpose Foundation models</strong> — The models aren’t perfect, but as we’ve seen with the rise of ChatGPT, they can handle a number of tasks.</li><li><strong>Foundational tools</strong> — Tools like <a href="https://www.langchain.com/">LangChain</a>, <a href="https://www.llamaindex.ai/">LlamaIndex</a>, and others yet to be created, make it easy to integrate foundation models and providers into different applications. While Citizen AI Engineers won’t use these tools directly, they’re going to enable the proliferation of tools built by AI Engineers that give Citizen AI engineers superpowers.</li><li><strong>Coding isn’t just for Developers</strong> — Tools like <a href="https://www.cursor.so/">Cursor</a>, <a href="https://github.com/features/copilot">Github Copilot</a>, and ChatGPT Advanced Data Analysis can do the work of entire developer teams. They aren’t perfect, but <a href="https://twitter.com/mckaywrigley/status/1693674524208943594">they’re extremely powerful</a>. This is just the beginning and these tools have personally, written devOps pipelines, helped teach me typescript, and administer databases.</li></ol><p>The combination of all these tools (and the growth they’re going to experience in the next 5 years) represents a state change in enterprise applications.</p><h4>What Does the Old World Look Like?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bq4IMX5JVlmMpxmv.png" /></figure><p>The production line, marching along.</p><p>You’re a data analyst at Home Depot. Every week, you’re running a financial forecast against your data lake, bringing together several pieces of data to solve a business problem.</p><p>This report is one that is sent up to upper leadership.</p><p>You know some basic SQL, but have no knowledge of Python, R, or any other data science tools. The report and forecast you build involve bringing together several kinds of data in a way that is particular to your business. Every week, you and your manager get questions about the forecast and how certain factors affect that forecast. You get follow-ups about:</p><blockquote>“What’s happening in this region?”<br>“Why is there so much disparity in XYZ dataset versus another one”?</blockquote><p>You’re putting out 🔥fires🔥 and have to prioritize only the most urgent requests. Your manager has talked about hiring a data analyst to do some of these immediate reports, but you all haven’t been able to get the headcount.</p><p>Your team has started looking into no-code app builders and other automated data analytics tools, but haven’t been able to find something that fits the bill.</p><h4>What Does the New World Look Like?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vc7GNvEPw84ZjAGA.png" /></figure><p>DataChat — working across your organization.</p><p>Your team just adopted <strong>Enterprise</strong> <strong>DataChat</strong> (<em>note: this product doesn’t exist, but it’s not difficult to imagine. I have no insider knowledge here. This is pure speculation.</em>)</p><p>Your AI engineering team just integrated several DataChat plugins to give access to business data sources and defined some auto-function-calls to perform semantic retrieval against corporate presentations and data in your business unit. They’ve also made available some DataChatFunctionExtractors to extract data from various unstructured data sources.</p><p>Now, instead of building presentations, you log into <strong>DataChat</strong> and specify your report requirements. It’s generated automatically. You can inquire about different variations but finalize on your same format, but include some notes for questions you’re sure to get. It’s taken a quarter of the time it used to take.</p><p>Moreover, because your organization has adopted <strong>DataChat</strong>, you’re able to provide a closed-domain chat application as part of your presentation. Leadership can now chat with your presentation and ask and answer their own questions about the content and data feeding it.</p><p><strong>DataChat</strong> enables you to monitor this “conversation” in real time, you’ve got complete observability on the entire conversation about your deliverable and even able to double check and work with your engineering team to add more data sources and functions to make this experience even more full-featured.</p><h4>The Delta from Old to New</h4><p>The old world is unidirectional — content is produced and consumed. It’s a factory.</p><p>The new world enables <strong>interactive</strong> consumption. You’ll not just digest presentations, but actively question them, live. You’ll build your own derivatives with a few clicks and prompts.</p><ul><li><strong>AI Engineers</strong> Build tools, plugins, and integrate AI to your data and ecosystem.</li><li><strong>Citizen AI Engineers</strong> bring AI and tools to the critical business problems that are relevant to your domain.</li></ul><h3>The History — We’ve Seen this Before</h3><p>This is not a novel idea. It’s just a repetition of history (in a new domain). It came up in the past decade with Data Scientists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zbn5X-yLQUS3wbZl.png" /></figure><p>from Gartner…</p><h3>Data Scientists</h3><p>On the ground at Databricks, I watched the rise of the Data Scientist. The problem is, hiring data scientists for so many enterprises was just impossible.</p><blockquote>As the power of data science provides organizations with differentiating competitive advantages, the demand for talent is rising. Supply, meanwhile, remains too scarce to meet that demand. This has led to data science and machine learning (ML) being opened up to nontraditional roles, such as the citizen data scientist.</blockquote><ul><li><a href="https://www.gartner.com/smarterwithgartner/how-to-use-citizen-data-scientists-to-maximize-your-da-strategy"><em>Gartner</em></a><em>, 2021</em></li></ul><h3>Citizen Data Scientists</h3><p>Citizen data science is a version of data science that lowers the barrier to entry and enables non-technical users to take on bigger and bigger company problems.</p><blockquote>A citizen data scientist is a person who creates or generates models that leverage predictive or prescriptive analytics, but whose primary job function is outside of the field of statistics and analytics.</blockquote><ul><li><a href="https://www.gartner.com/smarterwithgartner/how-to-use-citizen-data-scientists-to-maximize-your-da-strategy"><em>Gartner</em></a><em>, 2021</em></li></ul><p>They’re given tools to become more independent and capable. They bridge parts of the enterprise.</p><blockquote>Complementary roles such as business translators, developers, data engineers and machine learning architects together can support citizen data scientists to fill in the skill gaps that they lack.</blockquote><ul><li><a href="https://www.gartner.com/smarterwithgartner/how-to-use-citizen-data-scientists-to-maximize-your-da-strategy"><em>Gartner</em></a><em>, 2021</em></li></ul><p>These Citizen Data Scientists perform “Augmented Analytics”.</p><blockquote>Augmented analytics provides a guided, smart approach to conducting several steps, such as augmented data preparation, augmented data discovery and augmented data science. D&amp;A leaders can add these to the existing toolkit for CDS.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*N-SGKFp9qzs_tEJL.png" /></figure><h3>Move Over Data Science: Citizen AI Engineers and Augmented AI</h3><p>This is going to be a crazy decade. I posit Data Science will not become irrelevant, but will actually reduce in scope. The Age of Citizen AI Engineering is upon us.</p><p>But don’t take my word for it. Forward-looking technologists and enterprises are already seeing these changes too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_ftgKV7ARIExhyIP.png" /></figure><p><a href="https://twitter.com/Suhail/status/1701007875383189939">Suhail puts it perfectly in his recent tweet.</a></p><p>Suhail, I couldn’t agree more.</p><p>Or a recent <a href="https://chatwithpodcast.com/podcasts/hbr-ideacast-2/episodes/leading-a-workforce-empowered-by-new-ai-tools-11">HBR IdeaCast</a> episode on <a href="https://chatwithpodcast.com/podcasts/hbr-ideacast-2/episodes/leading-a-workforce-empowered-by-new-ai-tools-11"><strong>Leading a Workforce Empowered by New AI Tools</strong></a> when talking about the Rise of AI Tools. In the podcast, HBR discusses the rise of AI tools being used by “Citizen Developers”. Those that are there to solve business problems but need help from IT and developers to truly achieve the dream.</p><h3>The Ecosystem is Ready as well</h3><p>Tools like:</p><ul><li><a href="https://retool.com/">Retool</a> (<a href="https://retool.com/products/ai">who launched their AI product in the last week</a>)</li><li><a href="https://www.glideapps.com/">Glide</a></li><li><a href="https://airtable.com/">Airtable</a></li></ul><p>All of these companies are about giving business users access to data and enabling them to build applications without developers.</p><p>These tools are powerful. But if all of a sudden, they’re augmented with AI in major and fundamental ways — you’ve got extremely powerful business users that are less and less reliant on technical users to deliver business value out of data. That’s the Citizen AI Engineer.</p><p>Zapier is probably one of the most forward looking companies here. <a href="https://zapier.com/resources/webinar/zapathon-put-ai-to-work">They recently held a “Zapathon”</a> to dive into all of AI’s potential — across technical and non-technical users.</p><h3>Who’s Ahead, Who’s Behind?</h3><p>The question inevitably comes — where is the competition. I’m not quite ready to call winners but at this time, OpenAI is quite far ahead. The reason being is simple: <a href="https://twitter.com/Suhail/status/1701007875383189939">ChatGPT Advanced Data Analysis</a>.</p><h4>The Leading Edge: ChatGPT Advanced Data Analysis</h4><p>When OpenAI rebranded this product, I knew it was laying a foundation for something greater. It’s hard to understate how powerful this tool is.</p><p>Advanced Data Analysis is far more important than ChatGPT Enterprise. ChatGPT enterprise are just checkboxes. It’s a vehicle. There’s nothing novel.</p><p>But, if OpenAI can deliver on Advanced Data Analysis enabling access to and manipulation of enterprise data, at scale, we’re going to see a lot of roles change overnight across the enterprise.</p><h3>The Competitive Landscape</h3><p>There’s going to be more “data scientist’s in a box” style products. That’s a massive opportunity and the bedrock of the citizen AI engineer opportunity.</p><h3>Where Does the Rest of the Competition Stand?</h3><p>Overall, it’s too early to tell. If there’s one thing we don’t know is how things are going to change as new developments come out. Here are some thoughts in no particular order or relevance — aside from Nvidia.</p><ul><li><strong>Nvidia</strong> — stands alone here. They’re going to sell hardware, doesn’t matter to who. Jensen will sell a lot of it.</li><li><strong>Open Source</strong> — This isn’t a company, but the power is there. Tools like <a href="https://openinterpreter.com/">OpenInterpreter</a> and <a href="https://gpt4all.io/index.html">GPT4All</a> represent, to me, the most likely force to reduce OpenAI’s dominance. Check out the tools.</li><li><strong>Cohere/Anthropic/et al</strong> — These companies aren’t as in the public eye as OpenAI and candidly I don’t know enough about their product plans. They could do extremely well, but the core problem in AI is one of <strong>distribution</strong>. Technology can do amazing things. If you can’t get it into the right hands, it’s irrelevant.</li><li><strong>Google / GCP</strong> — Google allegedly has <a href="https://www.techopedia.com/google-gemini-is-a-serious-threat-to-chatgpt-heres-why">Gemini</a> — “<a href="https://www.techopedia.com/google-gemini-is-a-serious-threat-to-chatgpt-heres-why">a serious threat to GPT-4”.</a> I’m torn. On the one hand, Google has always had great tech. But their lack of customer intimacy has repeatedly hurt them. I’m not saying they can’t win, but it’s not clear to me they will.</li><li><strong>AWS</strong> — AWS is the standard in the cloud and they’ve got great distribution. Their Bedrock partnership strategy seems interesting and keeps them in the game while they play a bit of catchup.</li><li><strong>Azure</strong> — Azure is a natural winner. I’m continually impressed with Satya’s ability to predict the future and forge partnerships in critical areas. From Data to AI, he’s got strong coverage.</li><li><strong>Databricks</strong> is in a great place. They’re <em>generally </em>close to the AI engineer and developer persona. Certainly closer than the immediate competition — assuming the role doesn’t drift too much towards JavaScript land. If you bet on a proliferation of models outside of the OpenAI ones, then Databricks should be in great shape because data (Spark) and model training (Mosaic) will be critical.</li><li><strong>Snowflake</strong> has potential, but I haven’t seen a strong AI offering outside of Frank Slootman and Jensen holding a snowboard together that represents that they’re ready to capitalize on this opportunity.</li><li><strong>HuggingFace</strong> — HuggingFace has an interesting role as a repo for AI, but at some point the rubber has to hit the road and their bias towards the ML engineer / research might not prevent them from succeeding in this domain.</li><li><strong>AI21 Labs</strong> — They’ve got some great metrics for <a href="https://www.linkedin.com/pulse/quantitative-evaluation-llm-responses-rag-based-gary-stafford%3FtrackingId=Q958cLxQJTTPeQnfwovYLg%253D%253D/?trackingId=Q958cLxQJTTPeQnfwovYLg%3D%3D">RAG Based Q&amp;A Bots</a> and they’re hitting the ground running. Early to say, but seems like a great opportunity.</li><li><strong>Anyscale</strong> — Anyscale has a great position around ML tooling to build these kinds of tools. Ray, the open source project Anyscale is built around and created by the founders of Anyscale, <a href="https://thenewstack.io/how-ray-a-distributed-ai-framework-helps-power-chatgpt/">is used to train ChatGPT</a>. Lots of potential and the space is so early.</li></ul><h3>Conclusion</h3><p>In this post, I define a new role in a new world. <strong>The Citizen AI Engineer.</strong> Building on the <strong>foundational</strong> role definition of the AI engineer, I posit that we’re going to see a Cambrian explosion of business users that are AI-enabled.</p><p>The developer role will continue to be critical and see immense gains in productivity and power, but the massive wave coming is for business users. It’s just not evenly distributed.</p><h3>What the Future Brings</h3><p>The next 10 years are going to represent <strong>massive</strong>, <strong>structural</strong> change in the enterprise. How companies are built, grow, and scale. I don’t know all the implications and no one else does, but entrepreneurs have unbelievable opportunity. I believe that general purpose tools will be great, but specialized tools for specific problems will be even better.</p><p>The Citizen AI Engineer (or whatever the role will be called) will be a foundational part of this new world.</p><p><strong>If you’re building in this space, I’d love to talk.</strong></p><p>Special thanks to <a href="https://www.linkedin.com/in/richardliaw/">Richard Liaw</a> and <a href="https://www.linkedin.com/in/max-pumperla-a8099354/">Max Pumperla</a> for their contributions to this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=896ca60b9d0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>