<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jack on Medium]]></title>
        <description><![CDATA[Stories by Jack on Medium]]></description>
        <link>https://medium.com/@jackatunstructured?source=rss-5869fa8f0623------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*cUbgOc6a10y5qACR</url>
            <title>Stories by Jack on Medium</title>
            <link>https://medium.com/@jackatunstructured?source=rss-5869fa8f0623------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 08:08:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jackatunstructured/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[How to Process PDFs in Python: A Step-by-Step Guide]]></title>
            <link>https://medium.com/unstructured-io/how-to-process-pdfs-in-python-a-step-by-step-guide-d0532f795afe?source=rss-5869fa8f0623------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0532f795afe</guid>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[pdf]]></category>
            <category><![CDATA[data-extraction]]></category>
            <dc:creator><![CDATA[Jack]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 17:08:13 GMT</pubDate>
            <atom:updated>2023-11-27T20:43:13.587Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction:</strong></p><p>Let’s be real for a moment: PDFs can be a nightmare, especially for data scientists. While these ubiquitous documents are excellent for ensuring consistency across platforms, when it comes to extracting their data? Not so much. Many of us have faced the maddening task of wrangling with a PDF to get even the simplest dataset out. Locked-in tables, odd formatting, and the sheer inaccessibility of content make them a less-than-ideal source.</p><p>Yet, here’s the twist: PDFs are everywhere. From research papers to financial statements, valuable data is trapped inside these electronic paper prisons. If only we had a straightforward way to get that data out and into our analysis pipelines, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/0*HJ7kj8jdM1NkCseJ" /></figure><p>Enter <a href="https://unstructured.io/">Unstructured.io</a>. Specializing in extracting and transforming complex enterprise data from various formats, including the tricky PDF, Unstructured streamlines the data preprocessing task. This is not just about making the data extraction process less tedious. It’s about unlocking the potential of vast amounts of information hidden in PDFs and other formats, transforming them into AI-friendly JSON files ready for vector databases and large language model (LLM) frameworks. By reducing the preprocessing workload, Unstructured enables data scientists to channel their focus where it matters most: modeling and analyzing the data to drive actionable insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1XqbtVkyMiZCbXrM" /></figure><p>This guide aims to do just that. We’ll walk through the process of processing PDFs in Python, step by step, offering you the tools to wrestle that stubborn data into a structured, usable format. And while we delve into the ‘how’, we’ll also explore the ‘why’ — why opting for Unstructured is a prudent choice for data scientists striving for efficiency, accuracy, and ease in managing their data assets. So, if you’re tired of PDF-induced headaches and ready to take charge, read on. Let’s demystify the world of PDF data extraction together.</p><p><strong>Setting Up Your Environment</strong></p><p>Before diving into the world of PDF data extraction, ensuring that your environment is primed is crucial. PDFs, being a complex format, often require specific libraries to handle their intricacies. Let’s set up our Python environment to ensure smooth sailing:</p><p>Python Version with pyenv: If you don’t have pyenv installed, it’s a great tool to manage multiple Python versions. Install it following the instructions here. Once done, install your desired Python version:</p><pre>pyenv install 3.9.1</pre><p>Setting Local Python Version: Set the installed version as the local version for your project directory:</p><pre>pyenv local 3.9.1</pre><p>Virtual Environment with pyenv-virtualenv: pyenv works seamlessly with pyenv-virtualenv. If you don’t have it yet, check the installation guide here. Create a new virtual environment:</p><pre>pyenv virtualenv 3.9.1 pdf_env</pre><p>Then activate the environment:</p><pre>pyenv activate pdf_env</pre><p>Installation of Unstructured:</p><pre>pip install unstructured</pre><p>To install the full system dependencies follow this <a href="https://unstructured-io.github.io/unstructured/introduction.html#getting-started">link</a>.</p><p><strong>Exploring Customizability with Unstructured:</strong></p><p>Before we jump into the code, it’s worth mentioning the breadth of options Unstructured.io provides. Not only can it process a myriad of document formats like HTML, CSV, PNG, and PPTX, but it also offers 24 <a href="https://unstructured-io.github.io/unstructured/ingest/source_connectors.html">source connectors</a> and counting to effortlessly pull in your data, eliminating the need for you to spend time building data pipelines. This is a testament to Unstructured’s commitment to streamlining data preprocessing tasks for data scientists.</p><p><strong>Unlocking Text from PDFs</strong></p><p>Unstructured makes it very easy to partition PDFs and extract the key elements. With one line our python package can return a list of elements that are found within the document. Each element has a classification and the associated metadata with it.</p><pre><br>from unstructured.partition.pdf import partition_pdf<br><br># Returns a List[Element] present in the pages of the parsed pdf document<br>elements = partition_pdf(&quot;example-docs/layout-parser-paper-fast.pdf&quot;)</pre><p>There are many more customizations you can make. You can optimize for speed, security, and quality. For a full breakdown of our partition function you can explore it <a href="https://unstructured-io.github.io/unstructured/core/partition.html">here</a>.</p><p><strong>Extracting Tables from PDFs</strong></p><p>Now let’s say that your PDF has tables and let’s say you want to preserve the structure of the tables. You will have to specify the <a href="https://unstructured-io.github.io/unstructured/best_practices/strategies.html">strategy</a> parameter as “hi_res”. This will use a combination of computer vision and Optical Character Recognition (OCR) to extract the tables and maintain the structure. It will return both the text and the html of the table. This is super useful for rendering the tables or passing to a LLM.</p><pre># This will process without issue<br>elements = partition_pdf(&quot;example-docs/copy-protected.pdf&quot;, strategy=&quot;hi_res&quot;)</pre><p>Note: For even better table extraction Unstructured offers an API that improves upon the existing open source models (big improvements coming Oct 18th). You can get your API key <a href="https://unstructured.io/api-key">here</a>. Additionally we are also building our own custom image-to-text model, Chipper, that in some cases outperforms computer vision + OCR.</p><p><strong>Wrapping Up and Taking PDF Data Further</strong></p><p>And there you have it — a concise guide to extracting text and tables from PDFs using Python. The world of PDF data extraction can be daunting given the intricacies of the format. But with the right tools and practices in place, it becomes a more manageable task.</p><p>But don’t stop here. The data extracted is only as good as its application. Think about how this data can be integrated into larger datasets, how it can feed into your machine learning models, or even how it can be visualized for insights. The versatility of Python means the possibilities are nearly endless.</p><p>Encountered challenges following this tutorial or have questions? I encourage you to join <a href="https://join.slack.com/t/unstructuredw-kbe4326/shared_invite/zt-20kd2e9ti-q5yz7RCa2nlyqmAba9vqRw">our community Slack group</a>. There, you can connect with fellow users, exchange insights, and stay updated on the latest developments. We’re excited to see the innovations you come up with!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0532f795afe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unstructured-io/how-to-process-pdfs-in-python-a-step-by-step-guide-d0532f795afe">How to Process PDFs in Python: A Step-by-Step Guide</a> was originally published in <a href="https://medium.com/unstructured-io">Unstructured</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Private Retrieval Augmented Generation (RAG) System with Local Llama 2 model and…]]></title>
            <link>https://medium.com/unstructured-io/setting-up-a-private-retrieval-augmented-generation-rag-system-with-local-vector-database-d42f34692ca7?source=rss-5869fa8f0623------2</link>
            <guid isPermaLink="false">https://medium.com/p/d42f34692ca7</guid>
            <category><![CDATA[on-premise]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[llama-2]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Jack]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 17:41:15 GMT</pubDate>
            <atom:updated>2023-11-02T17:35:00.428Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up a Private Retrieval Augmented Generation (RAG) System with Local Llama 2 model and Vector Database</h3><p>In the realm of AI, access to current and accurate data is paramount. The Retrieval Augmented Generation (RAG) model exemplifies this, serving as an established tool in the AI ecosystem that taps into the synergies of large language models with external databases to deliver more precise and up-to-date answers. With RAG, you aren’t just getting black box answers; you’re receiving insights grounded in real-time information, pulled directly from the most recent data sources.</p><p>However, the power of RAG is truly unlocked when it has a reliable and efficient source of data. This is where Unstructured shines. Imagine having a treasure trove of data spread out in various formats and places, and the task of harnessing it feels herculean. Unstructured is purpose-built to bridge this gap. It acts as the ETL pipeline specifically designed for LLMs, connecting to your data regardless of its format or location, transforming and cleansing it into a streamlined and usable format. In essence, Unstructured is empowering your AI models with real-time, actionable data. It’s the first, yet the most crucial step in your AI journey, transforming unstructured data chaos into data-driven insights.</p><p>While many organizations are inclined to cloud-based RAG solutions, there’s a growing sentiment towards local implementations. This shift is driven by concerns over data privacy, desires for reduced latency, and considerations of cost. Opting for a local RAG system means ensuring that your data never leaves your premises, a crucial aspect for entities dealing with sensitive or proprietary information.</p><p>In this article, we’ll explore the specifics of constructing a private RAG system using a local model and vector database. We’ll break down the process and offer a step-by-step guide for those interested in such a setup. We will also spotlight why using Unstructured in your setup is not just a choice but a necessity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*wfPWxrlWelsF1PEO" /></figure><h3>How to use Unstructured in your Local RAG System:</h3><p>Unstructured is a critical tool when setting up your own RAG system. Here’s a step-by-step guide to get you started:</p><p><strong>Prerequisites:</strong></p><ol><li>Unstructured: Grab it from <a href="https://pypi.org/project/unstructured/">PyPI</a> or directly clone its <a href="https://github.com/Unstructured-IO/unstructured">GitHub repository</a>.</li><li>Pyenv: Install it from <a href="https://pypi.org/project/pyenv/">PyPL</a> or via <a href="https://formulae.brew.sh/formula/pyenv">Brew</a></li><li>Docker: Essential for containerized software utility.</li><li>I will run the notebook in Jupyter Notebook. If you haven’t installed Jupyter Notebook yet you can follow these <a href="https://jupyter.org/install">steps</a>.</li></ol><p><strong>Setting up your Environment:</strong></p><p>Now that you have installed the required packages, open up your terminal and run the following commands.</p><p>PYENV Installation: Use pyenv to manage your Python versions seamlessly.</p><pre>pyenv install 3.9.1<br>pyenv virtualenv 3.9.1 local-rag<br>pyenv activate local-rag</pre><p>Clone Repo:</p><pre>git clone https://github.com/jackretterer/local-rag.git</pre><p>Dependencies Installation: Install necessary packages from the provided requirements file.</p><pre>cd local-rag<br>pip install -r requirement.txt</pre><p>Update Hugging Face Hub: To avoid any unforeseen issues, it’s a good practice to ensure that you’re running the latest version of huggingface_hub.</p><pre>pip install --upgrade huggingface_hub</pre><p>Docker Setup: Fire up Docker.</p><p>Creating the model directory: This directory will store the required model files.</p><pre>mkdir model_files</pre><p>Installing the llama-cpp-python package with specific build arguments:</p><pre>CMAKE_ARGS=&quot;-DLLAMA_METAL=on&quot; FORCE_CMAKE=1 pip install llama-cpp-python</pre><p>Downloading a pre-trained model from Hugging Face Hub:</p><pre>huggingface-cli download TheBloke/Llama-2-7b-Chat-GGUF --local-dir <br>model_files --local-dir-use-symlinks False --include=&#39;*Q4_K*gguf&#39;</pre><p>I also made a new kernel for my jupyter notebook via:</p><ol><li>Ipykernel, you can install it via: $<em>pip install ipykernel</em></li><li>Run: $<em>python -m ipykernel install — user — name=local-rag</em></li><li>Restart notebook</li><li>Kernel-&gt;Change Kernel-&gt;Local Rag</li></ol><p><strong>Set Parameters and Constants:</strong></p><p>Before diving into the coding part, let’s first establish the foundation. Here, we define some key parameters and constants (this tutorial was designed to be run on Apple devices).</p><pre>output_dir = &#39;my-docs&#39; # Path to output directory<br>input_dir = &#39;files_used&#39; # Path to input directory<br>weaviate_url = &quot;http://localhost:8080&quot;<br>embedding_model_name = &#39;all-MiniLM-L6-v2&#39;<br>device = &#39;mps&#39;</pre><p>Notice how we’ve set up directories for both input and output files. We also specify the weaviate_url, a crucial endpoint for our upcoming interactions.</p><p><strong>Helper Functions to Ingest Documents and Pre-Process Them:</strong></p><p>Next, we’re entering the realm of document ingestion and pre-processing. These helper functions will be our trusty tools to ensure a seamless data flow.</p><pre>import subprocess<br>import os<br>from typing import List, Dict<br>from userpaths import get_my_documents<br><br><br>def process_local(output_dir: str, num_processes: int, input_path: str):<br>        command = [<br>          &quot;unstructured-ingest&quot;,<br>          &quot;local&quot;,<br>          &quot;--input-path&quot;, input_path,<br>          &quot;--output-dir&quot;, output_dir,<br>          &quot;--num-processes&quot;, str(num_processes),<br>          &quot;--recursive&quot;,<br>          &quot;--verbose&quot;,<br>        ]<br><br>        # Run the command<br>        process = subprocess.Popen(command, stdout=subprocess.PIPE)<br>        output, error = process.communicate()<br><br>        # Print output<br>        if process.returncode == 0:<br>            print(&#39;Command executed successfully. Output:&#39;)<br>            print(output.decode())<br>        else:<br>            print(&#39;Command failed. Error:&#39;)<br>            print(error.decode())<br><br>def get_result_files(folder_path) -&gt; List[Dict]:<br>    file_list = []<br>    for root, dirs, files in os.walk(folder_path):<br>        for file in files:<br>            if file.endswith(&#39;.json&#39;):<br>                file_path = os.path.join(root, file)<br>                file_list.append(file_path)<br>    return file_list</pre><p>Once our data is processed, the get_result_files function comes into play. This utility function scours a directory to fetch a list of all JSON files, providing us with the much-needed collection of processed results.</p><p>Run Desired Pipeline To Generate Clean JSON using Unstructured</p><pre>process_local(output_dir=output_dir, num_processes=2, input_path=input_dir)<br>files = get_result_files(output_dir)</pre><p><strong>Helper Functions to Setup Weaviate Schema and Client</strong></p><pre>import uuid<br>import weaviate<br>from weaviate.util import get_valid_uuid<br><br>def create_local_weaviate_client(db_url: str):<br>    return weaviate.Client(<br>        url=db_url,<br>    )<br><br>def get_schema(vectorizer: str = &quot;none&quot;):<br>    return {<br>        &quot;classes&quot;: [<br>            {<br>                &quot;class&quot;: &quot;Doc&quot;,<br>                &quot;description&quot;: &quot;A generic document class&quot;,<br>                &quot;vectorizer&quot;: vectorizer,<br>                &quot;properties&quot;: [<br>                    {<br>                        &quot;name&quot;: &quot;last_modified&quot;,<br>                        &quot;dataType&quot;: [&quot;text&quot;],<br>                        &quot;description&quot;: &quot;Last modified date for the document&quot;,<br>                    },<br>                    {<br>                        &quot;name&quot;: &quot;player&quot;,<br>                        &quot;dataType&quot;: [&quot;text&quot;],<br>                        &quot;description&quot;: &quot;Player related to the document&quot;,<br>                    },<br>                    {<br>                        &quot;name&quot;: &quot;position&quot;,<br>                        &quot;dataType&quot;: [&quot;text&quot;],<br>                        &quot;description&quot;: &quot;Player Position related to the document&quot;,<br>                    },<br>                    {<br>                        &quot;name&quot;: &quot;text&quot;,<br>                        &quot;dataType&quot;: [&quot;text&quot;],<br>                        &quot;description&quot;: &quot;Text content for the document&quot;,<br>                    },<br>                ],<br>            },<br>        ],<br>    }<br><br>def upload_schema(my_schema, weaviate):<br>    weaviate.schema.delete_all()<br>    weaviate.schema.create(my_schema)<br><br>def count_documents(client: weaviate.Client) -&gt; Dict:<br>    response = (<br>        client.query<br>        .aggregate(&quot;Doc&quot;)<br>        .with_meta_count()<br>        .do()<br>    )<br>    count = response<br>    return count</pre><p><strong>Setup Weaviate Client and Schema</strong></p><p>Before diving into the intricate process of document indexing, the primary step is to set up our Weaviate client. This client will act as a bridge, connecting us to the Weaviate instance.</p><pre>client = create_local_weaviate_client(db_url=weaviate_url)<br>my_schema = get_schema()<br>upload_schema(my_schema, weaviate=client)</pre><p>With this groundwork, we have our Weaviate system primed and ready to accept the upcoming document chunks.</p><p><strong>Helper Functions to Stage Unstructured Documents for Indexing</strong></p><p>Handling unstructured data can be tricky. Hence, we’ll be leveraging a series of helper functions that leverage Unstructured’s library to process the unstructured data, chunk them appropriately, and index them in our system.</p><pre>from unstructured.chunking.title import chunk_by_title<br>from unstructured.documents.elements import DataSourceMetadata<br>from unstructured.partition.json import partition_json<br>from sentence_transformers import SentenceTransformer<br><br>embedding_model = SentenceTransformer(embedding_model_name, device=device)<br><br>def compute_embedding(chunk_text: List[str]):<br>    embeddings = embedding_model.encode(chunk_text, device=device)<br>    return embeddings<br><br>def get_chunks(elements, chunk_under_n_chars=500, chunk_new_after_n_chars=1500):<br>    for element in elements:<br>        if not type(element.metadata.data_source) is DataSourceMetadata:<br>            delattr(element.metadata, &quot;data_source&quot;)<br><br>        if hasattr(element.metadata, &quot;coordinates&quot;):<br>            delattr(element.metadata, &quot;coordinates&quot;)<br><br>    chunks = chunk_by_title(<br>        elements,<br>        combine_under_n_chars=chunk_under_n_chars,<br>        new_after_n_chars=chunk_new_after_n_chars<br>    )<br><br>    for i in range(len(chunks)):<br>        chunks[i] = {&quot;last_modified&quot;: chunks[i].metadata.last_modified, &quot;text&quot;: chunks[i].text}<br><br>    chunk_texts = [x[&#39;text&#39;] for x in chunks]<br>    embeddings = compute_embedding(chunk_texts)<br>    return chunks, embeddings<br><br><br>def add_data_to_weaviate(files, client, chunk_under_n_chars=500, chunk_new_after_n_chars=1500):<br>    for filename in files:<br>        try:<br>            elements = partition_json(filename=filename)<br>            chunks, embeddings = get_chunks(elements, chunk_under_n_chars, chunk_new_after_n_chars)<br>        except IndexError as e:<br>            print(e)<br>            continue<br><br>        print(f&quot;Uploading {len(chunks)} chunks for {str(filename)}.&quot;)<br>        for i, chunk in enumerate(chunks):<br>            client.batch.add_data_object(<br>                data_object=chunk,<br>                class_name=&quot;doc&quot;,<br>                uuid=get_valid_uuid(uuid.uuid4()),<br>                vector=embeddings[i]<br>            )<br>        <br>    client.batch.flush()</pre><p>Now, the spotlight is on add_data_to_weaviate. This function is the captain of our ship, orchestrating the entire process of reading, chunking, embedding, and eventually storing the data into Weaviate.</p><p><strong>Add Chunks to Weaviate</strong></p><p>Using the predefined functions, we’ll ingest our document chunks into the Weaviate system.</p><pre>add_data_to_weaviate(<br>    files=files,<br>    client=client,<br>    chunk_under_n_chars=250,<br>    chunk_new_after_n_chars=500<br>)<br><br>print(count_documents(client=client)[&#39;data&#39;][&#39;Aggregate&#39;][&#39;Doc&#39;])</pre><p>Upon completion, a quick print statement gives us insight into the total number of documents we’ve successfully added.</p><p><strong>LangChain RAG Application</strong></p><p>For those unfamiliar, LangChain provides a suite of tools designed to enhance information retrieval, search, and more.</p><pre>from langchain.llms import LlamaCpp<br>from langchain.vectorstores.weaviate import Weaviate<br>from langchain.callbacks.manager import CallbackManager<br>from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler<br>from langchain.prompts import PromptTemplate</pre><p><strong>Instantiate Local Llama 2 LLM</strong></p><p>The heart of our question-answering system lies in the open source Llama 2 LLM. At its core, it’s an intricate yet powerful model designed to generate human-like text based on the input provided.</p><pre>callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])<br>n_gpu_layers = 1  # Metal set to 1 is enough.<br>n_batch = 100  # Should be between 1 and n_ctx, consider the amount of RAM of your Apple Silicon Chip.<br># Make sure the model path is correct for your system!<br>llm = LlamaCpp(<br>    model_path=&quot;model_files/llama-2-7b-chat.Q4_K_S.gguf&quot;,<br>    n_gpu_layers=n_gpu_layers,<br>    n_batch=n_batch,<br>    n_ctx=2048, # context window. By default 512<br>    f16_kv=True,  # MUST set to True, otherwise you will run into problem after a couple of calls<br>    callback_manager=callback_manager,<br>    verbose=True, # Verbose is required to pass to the callback manager<br>)</pre><p><strong>Helper Function to Run RAG Process</strong></p><p>Our main endeavor is to fetch relevant answers to any posed question. Hence, creating a function that encapsulates the entire RAG (Retrieval Augmented Generation) process is crucial.</p><pre>def question_answer(question: str, vectorstore: Weaviate):<br>    embedding = compute_embedding(question)<br>    similar_docs = vectorstore.max_marginal_relevance_search_by_vector(embedding)<br>    content = [x.page_content for x in similar_docs]<br>    prompt_template = PromptTemplate.from_template(<br>    &quot;&quot;&quot;\<br>    Given context about the subject, answer the question based on the context provided to the best of your ability.<br>    Context: {context}<br>    Question:<br>    {question}<br>    Answer:<br>    &quot;&quot;&quot;<br>    )<br>    prompt = prompt_template.format(context=content, question=question)<br>    answer = llm(prompt)<br>    return answer, similar_docs</pre><p>This function, question_answer, works in stages. Initially, it computes an embedding of the provided question using our earlier-defined method. With this embedding, the system scours the vector store to find documents most related to our query. These documents serve as the context for our LLM, helping it craft a well-informed response.</p><p><strong>Run RAG on a Question</strong></p><p>Time to put our system to the test.</p><pre>client = weaviate.Client(weaviate_url)<br>vectorstore = Weaviate(client, &quot;Doc&quot;, &quot;text&quot;)<br><br>question = &quot;Give a summary of NFL Draft 2020 Scouting Reports: RB Jonathan Taylor, Wisconsin?&quot;<br><br>answer, similar_docs = question_answer(question, vectorstore)<br><br>print(&quot;\n\n\n-------------------------&quot;)<br>print(f&quot;QUERY: {question}&quot;)<br>print(&quot;\n\n\n-------------------------&quot;)<br>print(f&quot;Answer: {answer}&quot;)<br>print(&quot;\n\n\n-------------------------&quot;)<br>for index, result in enumerate(similar_docs):<br>    print(f&quot;\n\n-- RESULT {index+1}:\n&quot;)<br>    print(result)</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/0*XvEiRcXXvWZPU-Ug" /></figure><p>This result does show an accurate retrieval.</p><p><strong>Conclusion:</strong></p><p>In an era dominated by digital innovation, building robust RAG systems is no longer a herculean task. With the step-by-step guide laid out in this blog, we’ve offered an approach that is both effective and accessible. While the presented workflow can undergo further refinements to meet the demands of a production-grade environment, our primary focus was to maintain its simplicity for clarity’s sake. Moreover, the emphasis on local setups underscores the significance of data privacy. By opting for in-house RAG implementations, organizations can strike a balance between harnessing the power of cutting-edge AI technology and ensuring that sensitive information remains safeguarded. As part of our ongoing commitment to enhancing data privacy, we’re thrilled to announce that support for RBAC (Role-Based Access Control) is on our roadmap and will be rolling out in the coming weeks. As AI continues to permeate various sectors, prioritizing data security becomes paramount. Through local RAG systems, we are not just stepping into the future of AI; we are doing so responsibly and ethically.</p><p>Encountered challenges following this tutorial or have questions? I encourage you to join our <a href="https://short.unstructured.io/vqfyc51">community Slack group</a>. There, you can connect with fellow users, exchange insights, and stay updated on the latest developments. We’re excited to see the innovations you come up with!</p><p>A link to the full code can be found in this <a href="https://github.com/jackretterer/local-rag">Github</a> repo.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d42f34692ca7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unstructured-io/setting-up-a-private-retrieval-augmented-generation-rag-system-with-local-vector-database-d42f34692ca7">Setting up a Private Retrieval Augmented Generation (RAG) System with Local Llama 2 model and…</a> was originally published in <a href="https://medium.com/unstructured-io">Unstructured</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning GPT 3.5 with Unstructured: A Comprehensive Guide]]></title>
            <link>https://medium.com/unstructured-io/fine-tuning-gpt-3-5-with-unstructured-a-comprehensive-guide-d0a22fe47150?source=rss-5869fa8f0623------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a22fe47150</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <dc:creator><![CDATA[Jack]]></dc:creator>
            <pubDate>Tue, 19 Sep 2023 16:57:32 GMT</pubDate>
            <atom:updated>2023-11-02T17:33:49.008Z</atom:updated>
            <content:encoded><![CDATA[<p>The world is not static; it’s always changing. And while Large Language Models (LLMs) like GPT-4 are immensely powerful, to truly unlock their potential they need exposure to domain-specific data. This is where fine-tuning enters the picture. But before we dive into the “how,” it’s essential to understand the “why.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*9GqghlUASslpAdMD9mSe_g.png" /></figure><p>The advancements in LLMs have brought about a technological disruption. OpenAI’s GPT-3, 3.5, and 4 have democratized access to high-power language models, enabling a wide array of users to interact with them. Still, while LLMs are brimming with knowledge, they are bound by the confines of time — their knowledge, as vast as it is, remains general and frozen at a specific point in time.</p><p>Take GPT’s cut-off in September 2021, for instance. In a fast-paced world, where new research, cultural phenomena, and global events emerge almost daily, how does one ensure the relevance of an LLM post-2021?</p><p>While GPT’s knowledge is broad, ensuring it possesses knowledge specific to your organization or area of expertise is a critical challenge to address. Imagine trying to discuss the intricacies of a specific domain — while GPT provides a general overview, it may lack the detailed knowledge and updated insights necessary to be useful.</p><p>These inherent limitations require organizations to utilize techniques such as fine-tuning and Retrieval Augmented Generation (RAG). While RAG provides a mechanism for models to access novel information, fine-tuning ensures that specialized and domain-specific knowledge is directly encoded into the memory of the model itself.</p><p>In this way, pre-training could be compared to a general purpose high school degree, fine tuning to a highly specialized vocational program, and RAG as a mechanism to allow access to new information after graduating with a high school degree or vocational program.</p><p>This is where Unstructured steps in as a game-changer. If you’re aiming to power GPT with the most up-to-date data or niche-specific insights, look no further. Want to learn how? Dive into this guide and try it for yourself.</p><h3>How to use Unstructured to add New Data to ChatGPT:</h3><p><strong>Prerequisites:</strong></p><ul><li>Install Unstructured from <a href="https://pypi.org/project/unstructured/">PyPI</a> or <a href="https://github.com/Unstructured-IO/unstructured">GitHub repo</a></li><li>Install Unstructured Google Cloud connectors <a href="https://unstructured-io.github.io/unstructured/upstream_connectors/google_drive.html#">here</a></li><li>Obtain Unstructured API Key <a href="https://unstructured.io/#get-api-key">here</a></li><li>Obtain OpenAI API Key <a href="https://platform.openai.com/account/api-keys">here</a></li><li>A Google Drive folder full of documents you want to process</li><li>Basic knowledge of Python &amp; Jupyter Notebook</li></ul><h3><strong>Acquire Our Dataset</strong></h3><p>To kick off our fine-tuning adventure, we’ll want a specialized dataset. For demonstration, we’ll be training GPT-3.5 on the Federal Open Market Committee’s (FOMC) meeting notes, focusing on topics like inflation and the federal funds rate.</p><p>I’ve gone ahead and downloaded the meeting notes from the federal reserves website (<a href="https://www.federalreserve.gov/monetarypolicy/fomccalendars.htm">link</a>) and uploaded them to a google drive folder.</p><p><strong>Enable Google Drive Access:</strong></p><p>Before GPT-3.5 can learn from these notes, we need to ingest and preprocess. If you’re new to Google Drive integrations, follow along. If you’re a seasoned pro, you can leapfrog to the subsequent section.</p><ol><li>Navigate to your Google Cloud Console (<a href="https://console.cloud.google.com/welcome?project=unstructured-ingest-test">link</a>)</li><li>Click on the project that you plan on grabbing data from</li><li>On the left side of the screen click IAM &amp; Admin -&gt; Service Accounts</li><li>Click “+ Create Service Account” and fill in the fields</li><li>Once you’ve created the new service account click on it and go to “KEYS”</li><li>Click “ADD KEY”-&gt;”Create new key”-&gt;JSON</li><li>That will download a JSON file with your keys. Please ensure that these keys are properly stored as they pose a security risk.</li><li>Place the JSON file somewhere safe and in a path you can access later on</li></ol><p><strong>Run Unstructured API with Google Drive Connector:</strong></p><p>The journey to tapping into the wealth of information within your documents starts with setting up the Unstructured API. Think of this as bridging the world of your stored knowledge (on Google Drive) with the potential of GPT’s fine-tuning capabilities.</p><p>To harness this, you’ll first need the Unstructured open-source library, available on <a href="https://github.com/Unstructured-IO/unstructured">GitHub</a>. Navigate to the repository, follow the setup instructions, and arm yourself with the essential tools.</p><p>To streamline this process, ensure you’ve installed the necessary dependencies. Let’s fetch them:</p><pre>!pip install &quot;unstructured[gdrive,all-docs]&quot; openai</pre><p>Once you installed the additional dependencies, you are good to run the API. For this tutorial, I’ve elected to run the API in a python notebook.</p><pre># Set up the environment with Google Drive service account credentials<br>os.environ[&#39;GOOGLE_APPLICATION_CREDENTIALS&#39;] = &#39;/Path/To/Your/Google/Credentials/keyfile.json&#39;<br><br># https://drive.google.com/drive/folders/{folder-id}<br><br>command = [<br>    &quot;unstructured-ingest&quot;,<br>    &quot;gdrive&quot;,<br>    &quot;--drive-id&quot;, &quot;folder-id&quot;,<br>    &quot;--service-account-key&quot;, &quot;Path/To/Your/Google/Credentials/keyfile.json&quot;,<br>    &quot;--structured-output-dir&quot;, &quot;/Path/To/Your/Output/Folder&quot;,<br>    &quot;--num-processes&quot;, &quot;2&quot;,<br>    &quot;--recursive&quot;,<br>    &quot;--verbose&quot;,<br>    &quot;--partition-pdf-infer-table-structure&quot;, &quot;true&quot;, # Only necessary if tables are present in the PDFs<br>    &quot;--partition-strategy&quot;, &quot;hi_res&quot;, # Only necessary if images/tables are in the PDFs<br>    &quot;--api-key&quot;, &quot;&lt;YOUR-API-KEY&gt;&quot;,<br>    &quot;--partition-by-api&quot;<br>]<br><br># Let&#39;s break this down:<br># 1. Set Google Drive&#39;s service account credentials<br># 2. Specify &#39;gdrive&#39; as our data connector <br># 3. Set the output directory with `--structured-output-dir`<br># 4. Optimize processing speed using `--num-processes`<br># 5. Insert your `--api-key` obtained earlier<br># 6. The `--partition-by-api` flag allows the API (not just the library) to handle data partitioning<br><br># Now, let&#39;s execute and collect our data<br>process = subprocess.Popen(command, stdout=subprocess.PIPE)<br>output, error = process.communicate()<br><br># Print output<br>if process.returncode == 0:<br>    print(&#39;Command executed successfully. Output:&#39;)<br>    print(output.decode())<br>else:<br>    print(&#39;Command failed. Error:&#39;)<br>    print(error.decode())</pre><h3><strong>Form Our Fine-Tuning Dataset:</strong></h3><p>Now that we have all the preprocessed files into JSON we can now extract all the text and put it into a dictionary for each of the files:</p><pre># FUNCTION USED FOR GRABBING ALL THE TEXT ELEMENTS FROM THE JSON FILES<br>def extract_text_from_json(directory):<br>    # Ensure the provided path exists and is a directory<br>    if not os.path.exists(directory) or not os.path.isdir(directory):<br>        print(f&quot;&#39;{directory}&#39; is not a valid directory path.&quot;)<br>        return<br><br>    extracted_texts = {}<br><br>    # Iterate over each file in the directory<br>    for filename in os.listdir(directory):<br>        # Check if the file is a JSON file<br>        if filename.endswith(&#39;.json&#39;):<br>            filepath = os.path.join(directory, filename)<br>            <br>            with open(filepath, &#39;r&#39;) as json_file:<br>                try:<br>                    data = json.load(json_file)<br>                    <br>                    # Initialize a list to collect texts from the current file<br>                    texts_list = []<br>                    <br>                    # If data is a list, iterate through its items<br>                    if isinstance(data, list):<br>                        for item in data:<br>                            if &quot;text&quot; in item:<br>                                texts_list.append(item[&quot;text&quot;])<br>                    # If data is a dictionary, just get the &quot;text&quot; key<br>                    elif isinstance(data, dict):<br>                        texts_list.append(data.get(&quot;text&quot;, &quot;&quot;))<br>                    <br>                    # Join the texts and store in the dictionary<br>                    extracted_texts[filename] = &quot;\n&quot;.join(texts_list)<br><br>                except json.JSONDecodeError:<br>                    print(f&quot;&#39;{filename}&#39; is not a valid JSON file.&quot;)<br><br>    return extracted_text</pre><p>Now we can run the helper function:</p><pre>texts_dict = extract_text_from_json(&#39;Path/To/Output/Folder&#39;)</pre><p><strong>Generate Fine-Tuning Data</strong></p><p>With our cleaned and extracted text from the PDFs we can create a fine-tuning dataset. I leveraged GPT4 to take the text from each dictionary and to create a series of questions and answers to fine-tune the model.</p><p>My prompt was:</p><pre>Given the following section, craft 15 insightful questions and their corresponding answers. Ensure the questions and answers encapsulate key details for an effective fine-tuning dataset:<br><br>The format of the question should be: <br><br>{&quot;messages&quot;: [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;Marv is a factual chatbot well-versed in recent Federal Reserve data.&quot;}, {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What did the Federal Reserve announce on July 26, 2023 regarding the federal funds rate?&quot;}, {&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;On July 26, 2023, the Federal Reserve announced that they decided to raise the target range for the federal funds rate to 5-1/4 to 5-1/2 percent.&quot;}]}<br><br>Section: [ENTER-YOUR-SECTION-COPIED-FROM-THE-DICTIONARY]</pre><p><strong>Assemble The Fine-Tuned Dataset</strong></p><p>OpenAI states that the minimum dataset required to fine tune a model is 10 examples. However OpenAI generally recommends at least 50 to 100 training examples with “gpt-3.5-turbo”.</p><p>I repeated the above code for each file (5 files in my case, so 75 examples) and put the results into a finetuning_gpt.jsonl file.</p><p>I also encountered a fair amount of errors while formatting the .jsonl format and created a helper function to find and identify the errors:</p><pre>dataset = []<br>data_path = &quot;/PATH/TO/finetuning_gpt.jsonl&quot;<br><br># Helper to diagnose JSON decoding errors<br>with open(data_path, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>    for idx, line in enumerate(f):<br>        try:<br>            dataset.append(json.loads(line))<br>        except json.JSONDecodeError:<br>            print(f&quot;Error decoding JSON on line {idx + 1}:&quot;)<br>            print(line)<br>            print(&quot;-&quot; * 40)</pre><p>For those keen on diving deeper into the fine-tuning intricacies, there’s this invaluable Python notebook I discovered: <a href="https://github.com/openai/openai-cookbook/blob/main/examples/Chat_finetuning_data_prep.ipynb">link</a></p><p>Here are some helpers to ensure your dataset structure is pristine:</p><pre># Load the dataset<br>with open(data_path, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>    dataset = [json.loads(line) for line in f]<br><br># Initial dataset stats<br>print(&quot;Num examples:&quot;, len(dataset))<br>print(&quot;First example:&quot;)<br>for message in dataset[0][&quot;messages&quot;]:<br>    print(message)<br><br># Format error checks<br>format_errors = defaultdict(int)<br><br>for ex in dataset:<br>    if not isinstance(ex, dict):<br>        format_errors[&quot;data_type&quot;] += 1<br>        continue<br>        <br>    messages = ex.get(&quot;messages&quot;, None)<br>    if not messages:<br>        format_errors[&quot;missing_messages_list&quot;] += 1<br>        continue<br>        <br>    for message in messages:<br>        if &quot;role&quot; not in message or &quot;content&quot; not in message:<br>            format_errors[&quot;message_missing_key&quot;] += 1<br>        <br>        if any(k not in (&quot;role&quot;, &quot;content&quot;, &quot;name&quot;) for k in message):<br>            format_errors[&quot;message_unrecognized_key&quot;] += 1<br>        <br>        if message.get(&quot;role&quot;, None) not in (&quot;system&quot;, &quot;user&quot;, &quot;assistant&quot;):<br>            format_errors[&quot;unrecognized_role&quot;] += 1<br>            <br>        content = message.get(&quot;content&quot;, None)<br>        if not content or not isinstance(content, str):<br>            format_errors[&quot;missing_content&quot;] += 1<br>    <br>    if not any(message.get(&quot;role&quot;, None) == &quot;assistant&quot; for message in messages):<br>        format_errors[&quot;example_missing_assistant_message&quot;] += 1<br><br>if format_errors:<br>    print(&quot;Found errors:&quot;)<br>    for k, v in format_errors.items():<br>        print(f&quot;{k}: {v}&quot;)<br>else:<br>    print(&quot;No errors found&quot;)</pre><p><strong>Token Limits</strong></p><p>Keep in mind, OpenAI has established limits: each training example can have a maximum of 4096 tokens. Any examples stretching beyond this will be trimmed to fit within this limit. Therefore, it’s a good strategy to keep individual examples well below this — aim for under 4,000 tokens for some breathing space. Also, keep an eye on the total tokens across your dataset, considering a max cap of 50 million tokens for the entirety of your training job (calculated as tokens_in_dataset * n_epochs).</p><p>To steer clear of these boundaries, use this helper to ensure each example stays within bounds:</p><pre>encoding = tiktoken.get_encoding(&quot;cl100k_base&quot;)<br><br># not exact!<br># simplified from https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb<br>def num_tokens_from_messages(messages, tokens_per_message=3, tokens_per_name=1):<br>    num_tokens = 0<br>    for message in messages:<br>        num_tokens += tokens_per_message<br>        for key, value in message.items():<br>            num_tokens += len(encoding.encode(value))<br>            if key == &quot;name&quot;:<br>                num_tokens += tokens_per_name<br>    num_tokens += 3<br>    return num_tokens<br><br>def num_assistant_tokens_from_messages(messages):<br>    num_tokens = 0<br>    for message in messages:<br>        if message[&quot;role&quot;] == &quot;assistant&quot;:<br>            num_tokens += len(encoding.encode(message[&quot;content&quot;]))<br>    return num_tokens<br><br>def print_distribution(values, name):<br>    print(f&quot;\n#### Distribution of {name}:&quot;)<br>    print(f&quot;min / max: {min(values)}, {max(values)}&quot;)<br>    print(f&quot;mean / median: {np.mean(values)}, {np.median(values)}&quot;)<br>    print(f&quot;p5 / p95: {np.quantile(values, 0.1)}, {np.quantile(values, 0.9)}&quot;)<br><br># Warnings and tokens counts<br>n_missing_system = 0<br>n_missing_user = 0<br>n_messages = []<br>convo_lens = []<br>assistant_message_lens = []<br><br>for ex in dataset:<br>    messages = ex[&quot;messages&quot;]<br>    if not any(message[&quot;role&quot;] == &quot;system&quot; for message in messages):<br>        n_missing_system += 1<br>    if not any(message[&quot;role&quot;] == &quot;user&quot; for message in messages):<br>        n_missing_user += 1<br>    n_messages.append(len(messages))<br>    convo_lens.append(num_tokens_from_messages(messages))<br>    assistant_message_lens.append(num_assistant_tokens_from_messages(messages))<br>    <br>print(&quot;Num examples missing system message:&quot;, n_missing_system)<br>print(&quot;Num examples missing user message:&quot;, n_missing_user)<br>print_distribution(n_messages, &quot;num_messages_per_example&quot;)<br>print_distribution(convo_lens, &quot;num_total_tokens_per_example&quot;)<br>print_distribution(assistant_message_lens, &quot;num_assistant_tokens_per_example&quot;)<br>n_too_long = sum(l &gt; 4096 for l in convo_lens)<br>print(f&quot;\n{n_too_long} examples may be over the 4096 token limit, they will be truncated during fine-tuning&quot;)</pre><p><strong>Estimate Costs</strong></p><p>OpenAI does impose charges when you fine-tune their models. For a detailed pricing breakdown, visit this <a href="https://openai.com/pricing">link</a>. To give you a ballpark figure of the potential costs, you can use OpenAI’s formula:</p><p>base cost per 1k tokens * number of tokens in the input file * number of epochs trained</p><p>To illustrate, if you have a training file consisting of 100,000 tokens and you train it over 3 epochs, you would incur an estimated cost of about $2.40 USD.</p><p>Here’s a Python code to help you estimate the cost:</p><pre># Pricing and default n_epochs estimate<br>MAX_TOKENS_PER_EXAMPLE = 4096<br><br>TARGET_EPOCHS = 3<br>MIN_TARGET_EXAMPLES = 100<br>MAX_TARGET_EXAMPLES = 25000<br>MIN_DEFAULT_EPOCHS = 1<br>MAX_DEFAULT_EPOCHS = 25<br><br>n_epochs = TARGET_EPOCHS<br>n_train_examples = len(dataset)<br>if n_train_examples * TARGET_EPOCHS &lt; MIN_TARGET_EXAMPLES:<br>    n_epochs = min(MAX_DEFAULT_EPOCHS, MIN_TARGET_EXAMPLES // n_train_examples)<br>elif n_train_examples * TARGET_EPOCHS &gt; MAX_TARGET_EXAMPLES:<br>    n_epochs = max(MIN_DEFAULT_EPOCHS, MAX_TARGET_EXAMPLES // n_train_examples)<br><br>n_billing_tokens_in_dataset = sum(min(MAX_TOKENS_PER_EXAMPLE, length) for length in convo_lens)<br>print(f&quot;Dataset has ~{n_billing_tokens_in_dataset} tokens that will be charged for during training&quot;)<br>print(f&quot;By default, you&#39;ll train for {n_epochs} epochs on this dataset&quot;)<br>print(f&quot;By default, you&#39;ll be charged for ~{n_epochs * n_billing_tokens_in_dataset} tokens&quot;)</pre><p>Once you’ve assembled your fine-tuning dataset, it’s time to upload it to OpenAI. Here’s how:</p><pre>os.environ[&#39;OPENAI_API_KEY&#39;] = &quot;YOUR-OPENAI-API-KEY&quot;<br>openai.api_key = os.getenv(&quot;OPENAI_API_KEY&quot;)<br>file_id = openai.File.create(<br>  file=open(&quot;/PATH/TO/finetuning_gpt.jsonl&quot;, &quot;rb&quot;),<br>  purpose=&#39;fine-tune&#39;<br>)</pre><h3>Creating the Fine-Tuned Model:</h3><p>After uploading your dataset, the next step is to initiate a fine-tuning job. However, it’s essential to have a grasp on what to expect in terms of training duration.</p><p><strong>Anticipating Training Duration</strong></p><p>With OpenAI, the training duration can be somewhat unpredictable. Your job might get queued behind other tasks, meaning you could wait anywhere from mere minutes to several hours. Rest assured, once the training concludes, you’ll receive an email notification.</p><p>For perspective, my dataset — which was relatively compact — took about 35 minutes to be processed. I launched the training at 4:30 pm PST, and by 5:05 pm PST, it was complete.</p><pre># Initiate a fine-tuned model<br>job = openai.FineTuningJob.create(<br>training_file=file_id.id, <br>model=&quot;gpt-3.5-turbo&quot;<br>)<br>job_id = job.id</pre><p>Side Note: While I was executing the process, I bumped into an error message that read, “AttributeError: module ‘openai’ has no attribute ‘FineTuneingJob’”. If you stumble upon this, a simple kernel restart of your notebook and an OpenAI library update did the trick for me.</p><p>Post-training, you’ll need to fetch the model name. This name is crucial when making calls to the OpenAI API:</p><pre>model_name_pre_object = openai.FineTuningJob.retrieve(job_id)<br>model_name = model_name_pre_object.fine_tuned_model<br>print(model_name)</pre><p>Once your job concludes, you can utilize the model for inference immediately. If, however, you notice timeouts or can’t locate the model name, it might be because the model is still in the loading phase. Give it a few minutes and try again.</p><pre>completion = openai.ChatCompletion.create(<br>    model=model_name,<br>    messages=[<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;When did the FED decide to raise federal funds to 5%?&quot;}<br>    ]<br>)<br><br>print(completion.choices[0].message[&#39;content&#39;])</pre><h3>Results:</h3><p>The outcomes from fine-tuning have been fairly positive. With the updated knowledge up to September 2023, the model can address a range of basic queries, although it has its limitations.</p><p>To demonstrate its capabilities, here are two instances where the fine-tuned model provided accurate responses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B3g7tRD1D05KVyJP" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ex2n0x8kQmO7riPF" /></figure><p>However, it’s not always flawless. There are moments when the model’s responses are off the mark, or it presents skewed information — sometimes referred to as “hallucinations.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L65AZ1nvP0xW7VVP" /></figure><p>Despite these occasional hiccups, a comparative analysis shows that the fine-tuned model outperforms the default GPT-3.5-turbo model in terms of accuracy and relevance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BGKe0iafsFQZRlsZ" /></figure><h3><strong>Conclusion:</strong></h3><p>Fine-tuning is an effective solution to the “frozen-in-time” aspect of LLMs and hones models’ expertise on specific domains. That said, we would recommend utilizing a fine tuned model in conjunction with a RAG (Retriever-Augmented Generation) architecture for the best results.</p><p>Now, while the concepts of fine-tuning and RAG might seem daunting to many, Unstructured is simplifying the process. Our platform integrates your favorite or up-to-date data sources, making fine-tuning not just effective, but also effortless and intuitive.</p><p>I’m currently in the midst of crafting a comprehensive blog post that will juxtapose the two techniques of fine-tuning and RAG. This piece will delve into their unique strengths and limitations. So, keep an eye out for this insightful exploration in the upcoming weeks!</p><p>Encountered challenges following this tutorial or have questions? I encourage you to join <a href="https://short.unstructured.io/vqfyc51">our community Slack group</a>. There, you can connect with fellow users, exchange insights, and stay updated on the latest developments. We’re excited to see the innovations you come up with!</p><p>A link to the full code can be found in this <a href="https://colab.research.google.com/drive/1RvigtSBmhk66lDpDNq2CS_d-sWZFVAJF?authuser=0#scrollTo=Sq5ntrOxAQyT">Google Colab</a> notebook.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a22fe47150" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unstructured-io/fine-tuning-gpt-3-5-with-unstructured-a-comprehensive-guide-d0a22fe47150">Fine-Tuning GPT 3.5 with Unstructured: A Comprehensive Guide</a> was originally published in <a href="https://medium.com/unstructured-io">Unstructured</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Table Extraction: Revolutionize Your Earnings Reports Analysis with AI]]></title>
            <link>https://medium.com/unstructured-io/mastering-table-extraction-revolutionize-your-earnings-reports-analysis-with-ai-1bc32c22720e?source=rss-5869fa8f0623------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bc32c22720e</guid>
            <category><![CDATA[financial-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Jack]]></dc:creator>
            <pubDate>Tue, 29 Aug 2023 16:34:18 GMT</pubDate>
            <atom:updated>2023-11-02T17:35:33.192Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine sifting through hundreds of pages of earnings reports, looking for specific data. Now, imagine a tool doing all that work in mere seconds. Welcome to the future of financial analysis.</p><p>We all recognize the power of quarterly earnings reports. They’re a goldmine for investors keen on evaluating a company’s performance. Yet, the tedious task of poring over these documents often feels like a never-ending maze. There’s a hope that technology might be our guiding light, but not all solutions are up to the task. Take, for instance, traditional AI models like ChatGPT. While impressive, they struggle with lengthy, unstructured documents. Even the latest large language models (LLM) hit roadblocks; they have constraints in processing vast volumes of data, can’t interpret visual content, and their knowledge is “frozen in time”, anchored only to their last training update.</p><p>To address these shortcomings, one might turn to a Retrieval Augmented Generation (RAG) pipeline, supercharged by Unstructured’s library.</p><p>In the following blog I am going to go through a step by step guide on how to analyze long financial documents with a RAG pipeline that leverages the computational strength of a LLM and the memory of a vector database while showcasing the table extraction capabilities of Unstructured’s library. In this blog we will be using LangChain, ChromaDB and OpenAI’s embedding “ada” model and their LLM “davinci”. However, feel free to use whatever vector db or LLM you’d like!</p><h3>Why Unstructured?</h3><p>Unfortunately, when you extract just the text from the earnings report, the LLM cannot process or understand the table structures. However, with the aid of Unstructured’s library, these structures can be rendered in an HTML format. Notably, when presented in this format, LLMs are adept at discerning and analyzing the row and column layout of tables.</p><h3>Getting Started</h3><p>Prerequisites:</p><ul><li>Install Unstructured from <a href="https://pypi.org/project/unstructured/">PyPI</a> or <a href="https://github.com/Unstructured-IO/unstructured">GitHub repo</a></li><li>Obtain OpenAI API Key <a href="https://platform.openai.com/account/api-keys">here</a></li><li>Basic technical skills</li></ul><p>For the purpose of this tutorial I am analyzing Nvidia’s earning reports. They can be found <a href="https://investor.nvidia.com/financial-info/quarterly-results/default.aspx">here</a>. I am going to be storing the downloaded PDFs into a local folder for access. The first page of the earnings report looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ReRS-rDrPDwYWCpz" /></figure><p>Data Ingestion</p><p>Follow these <a href="https://unstructured-io.github.io/unstructured/introduction/getting_started.html">instructions</a> in the repository to install the library and please make sure to install the additional dependencies below:</p><pre>!pip install unstructured unstructured-inference langchain openai chromadb</pre><p>Once you’ve obtained your earnings reports now we can begin to extract their tables for analysis.</p><pre>import os<br>from unstructured.partition.pdf import partition_pdf<br>from unstructured.staging.base import elements_to_json<br><br>filename = &quot;/Path/To/Your/File&quot; # For this notebook I uploaded Nvidia&#39;s earnings into the Files directory called &quot;/content/&quot;<br>output_dir = &quot;/Path/To/Your/Desired/Output&quot;<br><br># Define parameters for Unstructured&#39;s library<br>strategy = &quot;hi_res&quot; # Strategy for analyzing PDFs and extracting table structure<br>model_name = &quot;yolox&quot; # Best model for table extraction. Other options are detectron2_onnx and chipper depending on file layout<br><br># Extracts the elements from the PDF<br>elements = partition_pdf(<br>filename=filename, <br>strategy=strategy, <br>infer_table_structure=True, <br>model_name=model_name<br>)<br><br># Store results in json<br>elements_to_json(elements, filename=f&quot;{filename}.json&quot;) # Takes a while for file to show up on the Google Colab</pre><h3>Data Preprocessing</h3><p>Having extracted the elements, we can now filter and retain only the table elements. For this example we will ignore all the paragraphs and titles and solely focus on the tables for our analysis.</p><p>In order to extract only the table elements I’ve written a helper function to do so:</p><pre>def process_json_file(input_filename):<br>    # Read the JSON file<br>    with open(input_filename, &#39;r&#39;) as file:<br>        data = json.load(file)<br>    <br>    # Iterate over the JSON data and extract required table elements<br>    extracted_elements = []<br>    for entry in data:<br>        if entry[&quot;type&quot;] == &quot;Table&quot;:<br>            extracted_elements.append(entry[&quot;metadata&quot;][&quot;text_as_html&quot;])<br>    <br>    # Write the extracted elements to the output file<br>    with open(&quot;/Path/To/Your/Desired/Output/nvidia-yolox.txt&quot;, &#39;w&#39;) as output_file:<br>        for element in extracted_elements:<br>            output_file.write(element + &quot;\n\n&quot;)  # Adding two newlines for separation</pre><p>We can now run the helper function using our JSON file.</p><pre>process_json_file(f&quot;{filename}.json&quot;) # Takes a while for the .txt file to show up in Colab</pre><p>With the tables’ HTML now stored in a .txt file, we can utilize LangChain’s document loader. This tool greatly simplifies the subsequent steps.</p><pre>from langchain.document_loaders import TextLoader<br><br>loader = TextLoader(text_file)<br>documents = loader.load()</pre><p>The next step is to chunk our information into smaller more digestible chunks for our LLMs. We need to do so for a few reasons. The first is that LLMs have limited context windows and what that means is that they can only process a certain amount of words before they return an error. The second is that there is some research showing that when larger contexts are provided to LLMs they struggle to process information in the middle of the chunks. So reducing the chunk size might be a solution to it. If you are curious about the research feel free to read more about it <a href="https://arxiv.org/abs/2307.03172">here</a>.</p><pre>from langchain.text_splitter import CharacterTextSplitter<br><br># split it into chunks<br>text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=200)<br>docs = text_splitter.split_documents(documents)</pre><p>Now that we have our chunks we can embed them using OpenAI’s ada model. There are other options such as Sentence Transformers and Cohere but for consistency purposes I stuck with OpenAI.</p><pre>from langchain.embeddings.openai import OpenAIEmbeddings<br><br>os.environ[&#39;OPENAI_API_KEY&#39;] = &quot;&lt;YOUR-API-KEY&gt;&quot;<br>embeddings = OpenAIEmbeddings()</pre><h3>Vector Database</h3><p>For this tutorial I have also elected to use ChromaDB but there are many options out there each with their own strengths and weaknesses. But Chroma makes it very easy to get started.</p><pre>import chromadb<br>from chromadb.utils import embedding_functions<br><br>db = Chroma.from_documents(docs, embeddings)</pre><h3>Generation</h3><p>With everything in place, we can now directly query the financial document’s tables.</p><pre>from langchain.chat_models import ChatOpenAI<br>from langchain.chains import RetrievalQA<br><br># Initialize your model and retriever<br>llm = ChatOpenAI(model_name=&quot;gpt-3.5-turbo&quot;, temperature=0)<br>qa_chain = RetrievalQA.from_chain_type(llm, retriever=db.as_retriever())<br><br># List of questions<br>questions = [<br>    &quot;How much revenue did Nvidia make in Q2 FY24?&quot;,<br>    &quot;What was the operating income for Q2 FY24?&quot;,<br>    &quot;How much was the net income in Q1 FY24?&quot;,<br>    &quot;What was the Q/Q revenue growth between Q1 and Q2 of FY24?&quot;,<br>    &quot;How much revenue did Nvidia&#39;s Data Center produce in Q2 FY24?&quot;,<br>    &quot;How much revenue did Nvidia&#39;s Gaming sector produce in Q2 FY24?&quot;,<br>    &quot;What percent of the total revenue in Q2 FY24 did Data Centers represent?&quot;<br>]<br><br># Store responses in output_list<br>output_list = []<br><br>for query in questions:<br>    response = qa_chain({&quot;query&quot;: query})<br>    output_list.append(response)</pre><p>Now we can print the results. I chose to use pprint for legibility reasons.</p><pre>import pprint<br><br># Use pprint to pretty print the output list<br>pp = pprint.PrettyPrinter(indent=4)<br>pp.pprint(output_list)</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*iy1hI54FVpBI4Ck0" /></figure><h3>Closing Thoughts</h3><p>Throughout this notebook, we’ve highlighted the capability of Unstructured’s library in rendering accurate HTML representations for tables, and how LLMs adeptly interpret these table structures to furnish comprehensive analysis and summaries. Yet, there remains room for refinement.</p><p>One notable observation is the omission of table titles in the generated HTML. This can sometimes mislead our generation process. For instance, querying “operating income for Q2 FY24” may yield Non-GAAP figures in certain instances and GAAP in others. By integrating the table’s HTML with the core text of the document, we believe that table titles could be better incorporated, potentially enhancing the results. However, to maintain clarity and brevity for this post, we’ve opted for a simpler presentation.</p><p>Furthermore, we’ve pinpointed a challenge with our table extraction models, particularly when faced with tables displaying varied row background colors — a common trait in quarterly earnings reports. In response, we’re optimizing an image-to-text model with aspirations of elevating our table extraction performance. Stay tuned for more updates on this in the upcoming weeks!</p><p>If you have any questions or encounter any issues please join <a href="https://short.unstructured.io/vqfyc51">our community Slack channel</a>.</p><p>The full code for this blog can be found in this Google Colab <a href="https://colab.research.google.com/drive/1bdx61CWtXoC1GHD6hPN9pM0EwrfVPssH#scrollTo=iyc_rMfQhpY9">notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bc32c22720e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unstructured-io/mastering-table-extraction-revolutionize-your-earnings-reports-analysis-with-ai-1bc32c22720e">Mastering Table Extraction: Revolutionize Your Earnings Reports Analysis with AI</a> was originally published in <a href="https://medium.com/unstructured-io">Unstructured</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build an End-to-End RAG Pipeline with Unstructured’s API]]></title>
            <link>https://medium.com/unstructured-io/how-to-build-an-end-to-end-rag-pipeline-with-unstructureds-api-8c291b3a7f67?source=rss-5869fa8f0623------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c291b3a7f67</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[unstructured-data]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Jack]]></dc:creator>
            <pubDate>Mon, 14 Aug 2023 19:01:44 GMT</pubDate>
            <atom:updated>2023-11-02T17:36:13.582Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s say you have a lot of PDFs in your Google Cloud Storage (GCS) and you want to leverage a vector database to give your large language model (LLM) more context for more accurate and up to date answers, you first need to extract, clean, and transform those PDFs into a format that an LLM can understand (e.g. JSON). Unstructured’s library can help.</p><p><a href="https://www.unstructured.io/">Unstructured.io</a> offers a powerful toolkit that handles the ingestion and data preprocessing step, allowing you to focus on the more exciting downstream steps in your machine learning pipeline. Unstructured has over a dozen <a href="https://unstructured-io.github.io/unstructured/connectors.html">data connectors</a> that easily integrate with various data sources, including AWS S3, Discord, Slack, Wikipedia, and more.</p><p>In this guide we will go through a step-by-step guide on how to grab your data from GCS, and preprocess that data and upload it to a vector database for Retrieval Augmented Generation (RAG).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*FosYAVAcSpqg4PP2L4_vIw.png" /></figure><h3>Getting Started with Unstructured</h3><p><strong>Prerequisites:</strong></p><ul><li>Install Unstructured from <a href="https://pypi.org/project/unstructured/">PyPI</a> or <a href="https://github.com/Unstructured-IO/unstructured">GitHub repo</a></li><li>Install Unstructured Google Cloud connectors <a href="https://unstructured-io.github.io/unstructured/connectors.html#google-cloud-storage-connector">here</a></li><li>Obtain Unstructured API Key <a href="https://unstructured.io/#get-api-key">here</a></li><li>Obtain OpenAI API Key <a href="https://platform.openai.com/account/api-keys">here</a></li><li>Obtain Pinecone API key <a href="https://www.pinecone.io/">here</a></li><li>A Google Cloud Storage (GCS) bucket full of documents you want to process</li><li>Basic knowledge of command line operations</li></ul><h3>Data Ingestion</h3><p><strong>Enable GCS Access:</strong></p><p>In order to access your files from GCS you need to grant access to reading the buckets within your GCS account. If you are unfamiliar with this process I will go over a basic example but if you are already familiar with the steps feel free to skip to the next section.</p><ol><li>Navigate to your Google Cloud Console (<a href="https://console.cloud.google.com/">link</a>)</li><li>Click on the project that you plan on grabbing data from</li><li>On the left side of the screen click IAM &amp; Admin -&gt; Service Accounts</li><li>Click “+ Create Service Account” and fill in the fields</li><li>Once you’ve created the new service account click on it and go to “KEYS”</li><li>Click “ADD KEY”-&gt;”Create new key”-&gt;JSON</li><li>That will download a JSON file with your keys. Please ensure that these keys are properly stored as they pose a security risk.</li><li>Place the JSON file somewhere safe and in a path you can access later on</li></ol><p><strong>Run Unstructured API with GCS Connector:</strong></p><p>With your Unstructured API key and GCS bucket ready, it’s time to run the Unstructured API. To run the `unstructured-ingest` command, you need to install the unstructured open-source library that can be easily obtained from this <a href="https://github.com/Unstructured-IO/unstructured">GitHub repository</a>.</p><p>Follow the instructions in the repository to install the library and please make sure to install the additional dependencies below:</p><pre>!pip install &quot;unstructured[gcs,all-docs]&quot; langchain openai pinecone-client</pre><p>Once you installed the additional dependencies, you are good to run the API. For this tutorial, I’ve elected to run the API in a python notebook.</p><pre><br># Add the environment variable for the GCP service account credentials<br>os.environ[&#39;GOOGLE_APPLICATION_CREDENTIALS&#39;] = &#39;/Path/to/your/keyfile.json&#39;<br><br>command = [<br>    &quot;unstructured-ingest&quot;,<br>    &quot;gcs&quot;,<br>    &quot;--remote-url&quot;, &quot;gs://&lt;YOUR-BUCKET&gt;/&quot;,<br>    &quot;--structured-output-dir&quot;, &quot;/YOUR/OUTPUT/PATH&quot;,<br>    &quot;--num-processes&quot;, &quot;2&quot;,<br>    &quot;--api-key&quot;, &quot;&lt;UNSTRUCTURED-API-KEY&gt;&quot;,<br>    &quot;--partition-by-api&quot;<br>]</pre><p>Explanation:</p><ul><li>os.environ[‘GOOGLE_APPLICATION_CREDENTIALS’] =/path/to/your/keyfile.json — Add the environment variable for the GCP service account credentials</li><li>“gcs” — specifies the data connector</li><li>Set the output directory using ` — structured-output-dir` parameter</li><li>Specify ` — num-processes` to distribute the workload across multiple processes</li><li>Include ` — api-key` from the earlier step in your API call</li><li>Use the ` — partition-by-api` flag to indicate running the partition through API rather than the library</li></ul><p>We are proud to introduce our free-to-use API, tailor-made for enthusiasts and developers keen on prototyping. We’ve allocated a specific number of dedicated workers to ensure seamless operation and to cater to our user community. However, we understand the needs of larger corporations and heavy-duty tasks. With this in mind, we are excited to announce that an enterprise-grade API is on the horizon.</p><p><strong>Review the API output:</strong></p><p>Once the command completes, you’ll find the structured outputs in your specified output directory. These files contain the extracted information from your unstructured data, ready for analysis or further processing.</p><h3>Data Preprocessing</h3><p>When we build RAG systems, our LLMs model only performs as well as the context that is provided to them. For that reason, we want to remove unnecessary textual elements found in our files. Fortunately Unstructured classifies text by <a href="https://unstructured-io.github.io/unstructured/getting_started.html#document-elements">element types</a> and when Unstructured finds elements that it can’t classify it labels them as “UncategorizedText”. Often that “UncategorizedText” is unhelpful and distracting to our LLM. In this tutorial we will remove those “UncategorizedText” elements with the following helper function.</p><pre>def remove_element_from_json(filepath, item_to_remove):<br>    # Read the file and load its contents as JSON<br>    with open(filepath, &#39;r&#39;) as file:<br>        data = json.load(file)<br><br>    # Filter out the elements with the specified &#39;type&#39;<br>    updated_data = [item for item in data if item[&#39;type&#39;] != item_to_remove]<br><br>    # Write the updated data back to the file<br>    with open(filepath, &#39;w&#39;) as file:<br>        json.dump(updated_data, file, indent=4)</pre><p>Now you can call the above helper function by specifying the file path to the document and the element type you’d like to remove.</p><pre>filepath = &#39;/YOUR/OUTPUT/PATH&#39;<br>remove_element_from_json(filepath, &quot;UncategorizedText&quot;)</pre><p><strong>Load Json Files Into Langchain:</strong></p><p>The next step is to load in your cleaned and processed structured data into LangChain’s document loaders. In this case we will use the UnstructuredFileLoader by LangChain.</p><pre>from langchain.document_loaders import UnstructuredFileLoader<br><br># Now you will load to files outputted from the Unstructured API. You&#39;ll find the files in the output directory. <br>loader = UnstructuredFileLoader(<br>&quot;/YOUR/OUTPUT/PATH&quot;, <br>post_processors=[clean_extra_whitespace],<br>)<br>docs = loader.load()</pre><p><strong>Chunk the Text Files:</strong></p><p>Next we will need to chunk the original documents so that our Large Language Models (LLMs) can process the potentially long documents. A current limitation of LLMs is that they can only process a certain amount of context. We chunk data to make the pieces of text digestible for them.</p><p>For the purpose of this tutorial I have chosen to go with LangChain’s RecursiveCharacterTextSplitter however they have a variety of different chunking methods found <a href="https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/character_text_splitter">here</a>.</p><pre>text_splitter = RecursiveCharacterTextSplitter(<br>chunk_size=1000, <br>chunk_overlap=200<br>)<br>docs = text_splitter.split_documents(docs)</pre><h3>Initializing Vector Database</h3><p><strong>Setting up Pinecone:</strong></p><p>Once we’ve chunked our data now we can embed it and store it in a vector database. In this demo I am going to use Pinecone however there are a variety of different options each with their own strengths.</p><pre># Storing API keys like this poses a security risk. Only do this in prototyping<br>os.environ[&#39;PINECONE_API_KEY&#39;] = &quot;&lt;PINECONE-API-KEY&gt;&quot;<br><br># Pinecone Environment Variable<br>os.environ[&#39;PINECONE_ENV&#39;] = &quot;&lt;PINECONE-LOCATION&gt;&quot;<br><br>pinecone.init(<br>    api_key=os.getenv(&quot;PINECONE_API_KEY&quot;),  # find at app.pinecone.io<br>    environment=os.getenv(&quot;PINECONE_ENV&quot;),  # next to api key in console<br>)<br><br>index_name = &quot;gcs-demo&quot;<br># First, check if our index already exists. If it doesn&#39;t, we create it<br>if index_name not in pinecone.list_indexes():<br>    # we create a new index<br>    pinecone.create_index(<br>      name=index_name,<br>      metric=&#39;cosine&#39;,<br>      dimension=1536  <br>)</pre><p><strong>Create Helper Functions:</strong></p><p>The next step is once we’ve initialized our Pinecone database we need to create a helper function to grab all the relevant chunks of information from our vector database to be later used by our LLM (<a href="https://www.pinecone.io/learn/openai-gen-qa/">inspiration</a>).</p><pre>limit = 3600<br>embed_model = &quot;text-embedding-ada-002&quot;<br><br>def retrieve(query):<br>    res = openai.Embedding.create(<br>        input=[query],<br>        engine=embed_model<br>    )<br><br>    # retrieve from Pinecone<br>    xq = res[&#39;data&#39;][0][&#39;embedding&#39;]<br><br>    # get relevant contexts<br>    res = index.query(xq, top_k=3, include_metadata=True)<br>    contexts = [<br>        x[&#39;metadata&#39;][&#39;text&#39;] for x in res[&#39;matches&#39;]<br>    ]<br><br>    # build our prompt with the retrieved contexts included<br>    prompt_start = (<br>        &quot;Given the contextual information and not prior knowledge, answer the question. If the answer is not in the context, inform the user that you can&#39;t answer the question.\n\n&quot;+<br>        &quot;Context:\n&quot;<br>    )<br>    prompt_end = (<br>        f&quot;\n\nQuestion: {query}\nAnswer:&quot;<br>    )<br>    # append contexts until hitting limit<br>    for i in range(1, len(contexts)):<br>        if len(&quot;\n\n---\n\n&quot;.join(contexts[:i])) &gt;= limit:<br>            prompt = (<br>                prompt_start +<br>                &quot;\n\n---\n\n&quot;.join(contexts[:i-1]) +<br>                prompt_end<br>            )<br>            break<br>        elif i == len(contexts)-1:<br>            prompt = (<br>                prompt_start +<br>                &quot;\n\n---\n\n&quot;.join(contexts) +<br>                prompt_end<br>            )<br>    return prompt</pre><p>Now that we have a helper function to retrieve the relevant chunks of information we need a helper function to take in our augmented prompt and feed it into our LLM.</p><pre>def complete(prompt):<br>    # query text-davinci-003<br>    res = openai.Completion.create(<br>        engine=&#39;text-davinci-003&#39;,<br>        prompt=prompt,<br>        temperature=0,<br>        max_tokens=400,<br>        top_p=1,<br>        frequency_penalty=0,<br>        presence_penalty=0,<br>        stop=None<br>    )<br>    return res[&#39;choices&#39;][0][&#39;text&#39;].strip()</pre><h3>Putting it All Together</h3><p><strong>Query Vector Database:</strong></p><p>With our two helper functions, we can now leverage our vector database to retrieve relevant pieces of information to help our LLM answer domain specific questions.</p><pre>query = &quot;A question you&#39;d like to ask about your PDF&quot;<br><br># first we retrieve relevant items from Pinecone<br>query_with_contexts = retrieve(query)<br>query_with_contexts</pre><p>Once we’ve retrieved the necessary context we can feed it into our LLM to answer our question.</p><pre># then we complete the context-infused query<br>complete(query_with_contexts)</pre><p><strong>Conclusion:</strong></p><p>In this blog we’ve gone from unstructured raw PDFs stored in a GCS bucket to pre-processing them, and uploading them to a vector database for our LLM to query from. There are a lot of potential improvements in this process including chunking techniques but I hope you now can take any file format and build a RAG pipeline around it. The complete code can be found in <a href="https://colab.research.google.com/drive/11QM5DzaWl36MoeOs8QCQL-NFDKvYwCmO?authuser=0#scrollTo=SgebnVcsxDEj">this Google Colab notebook</a>.</p><p>If you have questions, join our <a href="https://short.unstructured.io/vqfyc51">community Slack group</a> to connect with other users, ask questions, share your experiences, and get the latest updates. We can’t wait to see what you’ll build!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c291b3a7f67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/unstructured-io/how-to-build-an-end-to-end-rag-pipeline-with-unstructureds-api-8c291b3a7f67">How to build an End-to-End RAG Pipeline with Unstructured’s API</a> was originally published in <a href="https://medium.com/unstructured-io">Unstructured</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>