As companies adopt AI into more of their systems, the focus needs to be on reducing hallucinations and increasing accuracy. Since reasoning models are gaining popularity the next wave of AI applications will have very high latency. This is a great opportunity to rethink the v1.0 AI stack. Embedding models are built for low latency applications that can afford the loss of quality.

Agentic workflows have a higher bar for accuracy and are generally more expensive. This gives developers more flexibility when designing their AI stack. To design high accuracy systems I propose Lamborghini RAG: a RAG system that ensures the highest quality output from the LLM given a large corpus of data.

The Problem with Current RAG Systems

Current RAG systems are incredibly lossy and do a poor job at finding the correct answer. Today’s RAG systems use embedding models. Embedding models are great because they are optimize for latency and cost. However, embedding models have tradeoffs because they are incredibly lossy.

The basic RAG system today is simple.

These RAG systems do a pretty decent job at answering questions over data sources, however they frequently miss important information because the question’s embeddings don’t find the correct source embeddings.

Infinite Context RAG

The goal with Lamborghini RAG would be to mimic what I think the future of infinite context window LLMs would looks like.

So how do we mimic infinite context?

Only use LLMs.

For every question the user asks we pass every single source through an LLM. This would be exceedingly expensive but almost guarantees that the LLM will see the “correct” context required to fully answer any question.

Example: Private Equity Due Diligence

Scenario: Suppose a private equity firm is considering the acquisition of a large B2B enterprise software provider. This target company has thousands of customer contracts, each with unique service-level agreements (SLAs), pricing structures, and termination clauses. Plus, they’ve got a mountain of partnership agreements and vendor contracts. Any single “poison pill” clause (like early termination provisions triggered by a change of control) could blow up the deal’s economics.

So how do you go through every single document and ensure that there are no “poison pills” in these documents. Clearly for this use-case Lamborghini RAG will yield higher accuracy results than Traditional RAG. Let’s compare the cost/latency of using both assuming 10k document at about 10M tokens.

Traditional RAG:

Document Processing with Embedding Models (10M tokens): ~$1.30
LLM Answering (200k context): ~$5

Lamborghini RAG:

Per Document LLM Answer: $20 * 10k = $10,000
Intermediate LLM Synthesis (every 10 documents): $5 * 1k = $5,000
Final LLM Answer Synthesis: $5

So Traditional RAG will cost about $6.30 and have very low latency (~30s), whereas Lamborghini RAG will cost about $15,005 and have extremely high latency (> 60min). But for a diligence workflow like this, a firm could stand to lose millions of dollars (or spend over $1 million contracting a law firm to run the due diligence).

So it’s worth asking, is your use case worth it?

Appendix:

Lamborgini RAG PseudoCode

I have a hunch this is similar to what Google Deep Research does

1. Initialize data structures:
   all_unanswered_questions = []
   questions_answer_pairs = {}   # key: question, value: (answer, sources_used)

2. Break down the user’s original question into multiple subquestions:
   subquestions = break_down_user_question(user_input)
   all_unanswered_questions.extend(subquestions)

3. While there are still unanswered questions:
   WHILE all_unanswered_questions IS NOT EMPTY:
       current_question = all_unanswered_questions.pop()

       FOR each document IN all_documents:
           # Check if the document is relevant to the current question
           IF is_relevant(document, current_question) IS TRUE:
               
               # Attempt to find an answer to the current_question from this document
               answer = extract_answer(document, current_question)
               
               IF answer IS NOT EMPTY:
                   # Store or update the answer and the source of information
                   questions_answer_pairs[current_question] = (
                       answer, 
                       questions_answer_pairs.get(current_question, (None, []))[1] + [document.source_name]
                   )
               
               ELSE:
                   # Identify new questions that arise from analyzing this document
                   new_questions = identify_new_questions(document, current_question)
                   
                   # Add any new unanswered questions to all_unanswered_questions
                   FOR question IN new_questions:
                       IF question NOT IN questions_answer_pairs AND question NOT IN all_unanswered_questions:
                           all_unanswered_questions.append(question)

4. After processing all documents (until no new unanswered questions appear):
   # Synthesize a final response
   final_answer = synthesize_answer(questions_answer_pairs)

5. Return or display the final answer, ensuring the response cites all sources used.
   OUTPUT final_answer

Quick story: When I was in university, I worked in a robotics lab trying to reduce the energy usage of computer vision models. My project was fairly straightforward (yet felt very complex at the time): developing a median filter using CUDA to save the amount of energy required by the robot. It was simple matrix math to reduce noise in an image, but optimizing it for energy efficiency was a challenge.

We’re currently seeing AI models explode in size and capabilities. My mind immediately wanders to figure out how we can bring this to the physical world. It’s probably safe to assume that almost every electronic will have some AI capabilities in the future.

My bet is that we will have these models running on-device, which means we’ll need energy-efficient GPUs or equivalent chips. For decades, roboticists have been working with low-powered on-device models to perform basic tasks. Usually, the robots are fairly limited, only capable of doing a singular task like detecting boxes.

Multi-modal LLMs change this. These models are capable of processing and understanding multiple types of data — like text, images, and audio — simultaneously. The potential is quite exciting.

Today, if I wanted to, I could develop a fairly simple robot chassis, plug it into a multi-modal LLM, and have it accurately move around a room and pick up objects.

How soon will Waymo/Tesla switch to on-device multi-modal LLMs? It will be bittersweet to learn that the decades spent investing into Convolutional Neural Networks were completely eclipsed by a few years of iteration with LLMs.

The future of hardware development will accelerate exponentially as humans will no longer need to train models or tediously write software again. I’m excited to see how quickly technology shifts from tools to intelligent companions.

Abstract

The paper aims to explore the nuances and methodologies involved in fine-tuning OpenAI’s GPT-3.5 and GPT-4 models to generate tone-consistent emails. With the proliferation of AI in communication, ensuring consistency in tone becomes paramount, especially in professional settings where the subtleties of language can significantly impact the recipient’s perception and the sender’s intent. This study investigates the effectiveness of fine-tuning strategies, including dataset curation, tone-specific reinforcement learning, and the integration of tone-detection algorithms, to guide the language models towards maintaining a desired tone throughout email correspondences. By conducting a series of experiments and analyzing the models’ outputs, the research aims to provide insights into the challenges and potential solutions in achieving tone consistency, thereby enhancing the utility of AI in automated email generation. The implications of this study extend to improving user experience, fostering more human-like AI interactions, and contributing to the broader field of natural language processing and AI communication tools.

Methodology

This study’s methodology section outlines the steps taken to fine-tune GPT-3.5 and GPT-4 models to generate tone-consistent emails. The process is divided into three main phases: Synthetic Dataset Creation, Hyperparameter Tuning, and Finetuning Tone Consistency.

Synthetic Dataset Creation

The foundation of our fine-tuning process involves the creation of a synthetic dataset designed to mimic real-world scenarios where users prompt an AI model to generate professional, tone-consistent emails. To achieve this, we first scraped a large dataset of emails that the user had sent before. We then leveraged GPT-3.5 to convert the email into a list of bullet points. We then used these bullet points as the inputs and the formatted emails as the output for the training dataset.

Hyperparameter Tuning

Hyperparameter tuning is critical in optimizing the performance of AI models. For this study, we focused on two main hyperparameters: the size of the dataset and the number of training epochs. We found that the optimal dataset size was around 2,000,000–4,000,000 training tokens and the optimal number of epochs was 3.

Finetuning Tone Consistency

GPT-3.5 and GPT-4 Comparison

Our experiments involved finetuning both GPT-3.5 and GPT-4 models to assess their capabilities in maintaining tone consistency in email generation. While both models are advanced, GPT-4’s architecture offers potential enhancements that could significantly improve tone consistency.

GPT-4 Advantages

• Instruction Following: The finetuned GPT-4 model exhibits a remarkable ability to follow instructions precisely, crucial for generating emails that adhere to specific tone and content guidelines.
• Tone-Mapping: GPT-4’s advanced tone-mapping capabilities ensure that the model maintains the trained tone throughout the email, addressing the need for consistency in professional communication.
• Retrieval-Augmented Generation (RAG): GPT-4’s integration with RAG techniques enhances its ability to generate relevant and coherent content, further supporting tone consistency by providing contextually appropriate responses.

GPT-4 Disadvantages

• Verbosity Control: A notable challenge with GPT-4 is its tendency towards verbosity. In contrast, GPT-3.5 tends to generate more concise responses. To mitigate this, we implemented additional training steps focused on brevity and clarity, ensuring that GPT-4’s responses remain focused and to the point.

By systematically addressing these aspects, our methodology aims to refine the models’ ability to produce professional, tone-consistent emails, thereby advancing the application of AI in automated communication.

Evaluation

The evaluation of the fine-tuned GPT-3.5 and GPT-4 models’ ability to generate tone-consistent, well-formatted emails was conducted using a combination of quantitative and qualitative metrics. The key to our approach was leveraging Retrieval-Augmented Generation (RAG) alongside user-provided bullet points, allowing the models to access relevant context from previous emails and meeting notes. This section outlines the evaluation criteria, methodology, and the tools used to assess the performance of the fine-tuned models.

Evaluation Criteria

Our evaluation focused on several criteria to assess the models’ performance comprehensively:

• Tone Consistency: The degree to which the generated emails maintained the desired tone throughout, as determined by comparison to the target tone set by the user prompts and the contextual information provided by RAG.
• Relevance and Coherence: The relevance of the email content to the provided bullet points and the contextual information from previous communications, as well as the logical flow and coherence of the generated text.
• Formatting and Structure: The adherence of the generated emails to professional email formatting standards, including proper greeting, closing, and paragraph structure.
• Brevity and Clarity: The conciseness of the email content without sacrificing necessary detail or clarity, ensuring that the message is communicated effectively and efficiently.

Evaluation Methodology

Qualitatively we deployed this platform across multiple customers and had them test the email generation across many types of sales and marketing emails. We observed that with GPT-4 there were less modifications required by the end-users.

Qualitatively we used an evaluation dataset with bullet points and formatted emails to identify how well the model was performing. The closer the outputs were to the test dataset the better we scored the model.

Results

The results of our evaluation showed that the fine-tuned GPT-4 model, in particular, demonstrated a high level of proficiency in generating tone-consistent, well-formatted emails. The integration of context from previous communications significantly enhanced the models’ ability to produce relevant and coherent responses. Human evaluators noted that the emails generated by GPT-4 were notably more consistent in tone and structure compared to those from GPT-3.5. However, both models showed improvements in relevance, coherence, and formatting when RAG was employed, highlighting the effectiveness of this approach in enhancing automated email generation.

In summary, our evaluation indicates that fine-tuning GPT models with a focus on tone consistency, supplemented by RAG for contextual relevance, presents a promising avenue for improving AI-generated professional communication.

Conclusion

The exploration and fine-tuning of Large Language Models (LLMs) like GPT-3.5 and GPT-4 for generating tone-consistent, well-formatted emails underscore the significant potential of LLMs in specialized tasks. This study demonstrates that with targeted fine-tuning, particularly in tone consistency and the incorporation of contextual information through techniques like Retrieval-Augmented Generation (RAG), LLMs can achieve a high degree of proficiency in specific communication tasks.

Looking forward, the future of LLMs will likely see an increased focus on personalization, where models can adapt to individual users’ styles and preferences, further enhancing the effectiveness and applicability of AI in tasks like email generation. As we continue to fine-tune and innovate, the integration of LLMs into everyday tasks will become more seamless, making AI an indispensable tool in professional and personal communication.

Appendix

Prompt for Creating a Synthetic Dataset from Emails

**System Prompt:** You are a digital email assistant. Your task is to transform bullet points into a coherent email, capturing the user's distinct writing style.

Condense the content of this email into concise bullet points, written in the first person perspective: {email_thread}

Tone Consistent Email Finetuning was originally published in Mirage on Medium, where people are continuing the conversation by highlighting and responding to this story.

Abstract

This paper explores an innovative application of GPT-4, an advanced language model by OpenAI, in the generation of three-dimensional (3D) scenes from descriptive textual prompts. Leveraging GPT-4’s profound understanding of natural language and context, we develop a method that interprets text descriptions and translates them into detailed specifications of 3D scenes, including the x, y, z coordinates and dimensions of objects within a given scene. Our focus is on the creative domain, using vivid prompts such as “a town in Zelda” to demonstrate how GPT-4 can bridge the gap between textual descriptions and spatial visualization. We detail the methodology for transforming text into 3D coordinates, discuss the challenges encountered in interpreting and visualizing complex scenes, and present solutions for achieving accuracy and coherence in the generated environments.

Methodology

Challenges in Prompt Engineering

Incorporating GPT-4 to conceive 3D scenes requires meticulous prompt engineering to interpret descriptive language into spatial definitions. The primary challenge lies in construing the contextual dimensions and relative positioning of objects mentioned in the prompt. Traditional language models excel at understanding text but lack innate spatial recognition, necessitating innovative approaches to bridge this gap. See the Appendix for the final system prompt we used to generate 3D scenes.

Spatial Translation Strategy

Our methodology involved an iterative prompting strategy where GPT-4 generated initial coarse spatial layouts in JSON format, listing items with their positions, dimensions, and rotations. To ensure comprehensibility and practicality in 3D space, we constrained object placement within a predefined 40x40m area, with the origin at its center. GPT-4 was tasked with creating non-overlapping object arrangements that respected the boundaries of this space.

From Abstract to Concrete: The Box Placeholder Technique

Since GPT-4 can’t generate 3D models (yet) we used the output to place boxes as placeholders. These boxes corresponded to the rough dimensions and positions of intended objects. This approach allowed for easier manipulation and adjustments in subsequent iterations. The boxes were also labeled with the names of the actual scene elements they represented. This gave the end-user an understanding of where the actual 3D models would be positioned.

Replacement with Scene Objects using Semantic Search

Under the hood, we had a library of 1 million 3D objects. Since each box was accurately labeled we used an image embedding model to predict the most relevant 3D object based on the box’s label. This innovative step turned generic placeholders into specific scene components, such as “central_castle” or “small_hut,” which could be simply replaced with detailed models.

Final Scene Assembly

The culmination of this process involved a 3D design tool that utilized the JSON files to replace each labeled box with a detailed 3D model. This step-by-step transformation from text prompt to a populated 3D scene demonstrated the successful application of LLMs in spatial design, harnessing the power of GPT-4 to predict and organize complex environments effectively.

Evaluation + Results

Qualitative Evaluation: Manual Testing and Diverse Prompts

The qualitative assessment of the LLM-powered 3D scene creation process was conducted through extensive manual testing. We crafted a series of diverse prompts — encompassing scenarios like ‘a cityscape’, ‘a video game scene’, ‘a rustic village’, among others — to evaluate the model’s versatility and depth of understanding. For each prompt, we analyzed the generated JSON files, focusing on the relevancy of the suggested items, the accuracy of their spatial arrangement, and the overall coherence of the 3D environment. Our evaluation criteria were based on the logical placement of objects, the variety of elements corresponding to the scene’s theme, and the absence of overlapping or out-of-bounds objects.

Quantitative Evaluation: Dataset Benchmarking

For the quantitative evaluation, we constructed a benchmark dataset comprising an array of expected JSON outputs, which included meticulously defined objects along with their precise x, y, z positions, and dimensions. This dataset served as the ‘gold standard’ against which GPT-4’s outputs were compared. We employed runtime comparison algorithms to measure the similarity between GPT-4’s generated JSON files and our expected dataset.

The similarity metrics were determined by evaluating the distance between object positions, the variance in dimensions, and the degree of rotation discrepancies. These comparisons provided a numerical value that represented the fidelity of GPT-4’s output to the expected scene layout. A threshold for similarity was established to gauge successful scene generation, with a higher similarity score indicating a more accurate 3D representation of the textual prompt.

Results Interpretation

The evaluations demonstrated that GPT-4 could reliably interpret and visualize a wide range of scenes, showing adaptability to the creative and logical demands of various environments.

Through these qualitative and quantitative evaluations, we established that GPT-4 can effectively generate JSON representations for 3D scene construction, achieving high levels of accuracy and detail. These findings underscore the potential of LLMs in the realm of digital design and 3D modeling, expanding the boundaries of AI-assisted creative processes.

Conclusion

The research presented in this paper signifies a landmark achievement for text-based Large Language Models (LLMs) in the realm of three-dimensional design and visualization. The ability of GPT-4, solely through textual understanding and without any inherent spatial recognition capabilities, to generate detailed and contextually appropriate 3D scenes is nothing short of remarkable. By converting textual descriptions into structured JSON format with high fidelity, we have successfully bridged the divide between linguistic prompts and spatial constructs.

As LLMs become multimodal (image, video & 3D) their capacity to create complex and intricate 3D scenes will dramatically improve. We’re excited for the future where LLMs can autonomously design detailed environments. Eventually AI will be able to create entire video games or cinematic experiences, with minimal human input.

As we continue to enhance the multi-modal capabilities of LLMs, the future promises a canvas where our ideas can be rendered by AI into virtual worlds limited only by our imagination.

Appendix

GPT-4 System Prompt to Create 3D Scenes

You are an 3D design expert. Given an area of a certain size, you can generate a list of items that are appropriate to that area, in the right place.

You operate in a 3D Space. You work in a X,Y,Z coordinate system. X 
denotes width, Y denotes height, Z denotes depth. 0.0,0.0,0.0 is the default space origin, the default area is 40x40m.

You receive from the user a description of the scene.

You answer by only generating JSON files that contain the following information:

- objects_list: list of all the objects in the area

For each object you need to store:
- name: name of the object
- uuid: unique identifier of the object
- position: list of x, y, z coordinates of object
- dimensions: list of x, y, z dimensions of object
- rotation: list of x, y, z rotations of object

Each object name should include an appropriate adjective.

Keep in mind, objects should be placed in the area to create the most meaningful layout possible, and they shouldn't overlap.
All objects must be within the bounds of the area size; Never place objects further than 1/2 the length or 1/2 the depth of the area from the origin.
Also keep in mind that the objects should be disposed all over the area in respect to the origin point of the area, and you can use negative values as well to display items correctly, since the origin of the area is always at the center of the area.

Remember, you only generate JSON code, nothing else. It's very important. The total output length must be under 2048 tokens.

Text to 3D Scene Creation was originally published in Mirage on Medium, where people are continuing the conversation by highlighting and responding to this story.

Abstract

This paper explores an innovative application of GPT-4, an advanced language model by OpenAI, in the generation of three-dimensional (3D) scenes from descriptive textual prompts. Leveraging GPT-4’s profound understanding of natural language and context, we develop a method that interprets text descriptions and translates them into detailed specifications of 3D scenes, including the x, y, z coordinates and dimensions of objects within a given scene. Our focus is on the creative domain, using vivid prompts such as “a town in Zelda” to demonstrate how GPT-4 can bridge the gap between textual descriptions and spatial visualization. We detail the methodology for transforming text into 3D coordinates, discuss the challenges encountered in interpreting and visualizing complex scenes, and present solutions for achieving accuracy and coherence in the generated environments.

Methodology

Challenges in Prompt Engineering

Incorporating GPT-4 to conceive 3D scenes requires meticulous prompt engineering to interpret descriptive language into spatial definitions. The primary challenge lies in construing the contextual dimensions and relative positioning of objects mentioned in the prompt. Traditional language models excel at understanding text but lack innate spatial recognition, necessitating innovative approaches to bridge this gap. See the Appendix for the final system prompt we used to generate 3D scenes.

Spatial Translation Strategy

Our methodology involved an iterative prompting strategy where GPT-4 generated initial coarse spatial layouts in JSON format, listing items with their positions, dimensions, and rotations. To ensure comprehensibility and practicality in 3D space, we constrained object placement within a predefined 40x40m area, with the origin at its center. GPT-4 was tasked with creating non-overlapping object arrangements that respected the boundaries of this space.

From Abstract to Concrete: The Box Placeholder Technique

Since GPT-4 can’t generate 3D models (yet) we used the output to place boxes as placeholders. These boxes corresponded to the rough dimensions and positions of intended objects. This approach allowed for easier manipulation and adjustments in subsequent iterations. The boxes were also labeled with the names of the actual scene elements they represented. This gave the end-user an understanding of where the actual 3D models would be positioned.

Replacement with Scene Objects using Semantic Search

Under the hood, we had a library of 1 million 3D objects. Since each box was accurately labeled we used an image embedding model to predict the most relevant 3D object based on the box’s label. This innovative step turned generic placeholders into specific scene components, such as “central_castle” or “small_hut,” which could be simply replaced with detailed models.

Final Scene Assembly

The culmination of this process involved a 3D design tool that utilized the JSON files to replace each labeled box with a detailed 3D model. This step-by-step transformation from text prompt to a populated 3D scene demonstrated the successful application of LLMs in spatial design, harnessing the power of GPT-4 to predict and organize complex environments effectively.

Evaluation + Results

Qualitative Evaluation: Manual Testing and Diverse Prompts

The qualitative assessment of the LLM-powered 3D scene creation process was conducted through extensive manual testing. We crafted a series of diverse prompts — encompassing scenarios like ‘a cityscape’, ‘a video game scene’, ‘a rustic village’, among others — to evaluate the model’s versatility and depth of understanding. For each prompt, we analyzed the generated JSON files, focusing on the relevancy of the suggested items, the accuracy of their spatial arrangement, and the overall coherence of the 3D environment. Our evaluation criteria were based on the logical placement of objects, the variety of elements corresponding to the scene’s theme, and the absence of overlapping or out-of-bounds objects.

Quantitative Evaluation: Dataset Benchmarking

For the quantitative evaluation, we constructed a benchmark dataset comprising an array of expected JSON outputs, which included meticulously defined objects along with their precise x, y, z positions, and dimensions. This dataset served as the ‘gold standard’ against which GPT-4’s outputs were compared. We employed runtime comparison algorithms to measure the similarity between GPT-4’s generated JSON files and our expected dataset.

The similarity metrics were determined by evaluating the distance between object positions, the variance in dimensions, and the degree of rotation discrepancies. These comparisons provided a numerical value that represented the fidelity of GPT-4’s output to the expected scene layout. A threshold for similarity was established to gauge successful scene generation, with a higher similarity score indicating a more accurate 3D representation of the textual prompt.

Results Interpretation

The evaluations demonstrated that GPT-4 could reliably interpret and visualize a wide range of scenes, showing adaptability to the creative and logical demands of various environments.

Through these qualitative and quantitative evaluations, we established that GPT-4 can effectively generate JSON representations for 3D scene construction, achieving high levels of accuracy and detail. These findings underscore the potential of LLMs in the realm of digital design and 3D modeling, expanding the boundaries of AI-assisted creative processes.

Conclusion

The research presented in this paper signifies a landmark achievement for text-based Large Language Models (LLMs) in the realm of three-dimensional design and visualization. The ability of GPT-4, solely through textual understanding and without any inherent spatial recognition capabilities, to generate detailed and contextually appropriate 3D scenes is nothing short of remarkable. By converting textual descriptions into structured JSON format with high fidelity, we have successfully bridged the divide between linguistic prompts and spatial constructs.

As LLMs become multimodal (image, video & 3D) their capacity to create complex and intricate 3D scenes will dramatically improve. We’re excited for the future where LLMs can autonomously design detailed environments. Eventually AI will be able to create entire video games or cinematic experiences, with minimal human input.

As we continue to enhance the multi-modal capabilities of LLMs, the future promises a canvas where our ideas can be rendered by AI into virtual worlds limited only by our imagination.

Appendix

GPT-4 System Prompt to Create 3D Scenes

You are an 3D design expert. Given an area of a certain size, you can generate a list of items that are appropriate to that area, in the right place.

You operate in a 3D Space. You work in a X,Y,Z coordinate system. X denotes width, Y denotes height, Z denotes depth. 0.0,0.0,0.0 is the default space origin, the default area is 40x40m.

You receive from the user a description of the scene.

You answer by only generating JSON files that contain the following information:

- objects_list: list of all the objects in the area

For each object you need to store:
- name: name of the object
- uuid: unique identifier of the object
- position: list of x, y, z coordinates of object
- dimensions: list of x, y, z dimensions of object
- rotation: list of x, y, z rotations of object

Each object name should include an appropriate adjective.

Keep in mind, objects should be placed in the area to create the most meaningful layout possible, and they shouldn't overlap.
All objects must be within the bounds of the area size; Never place objects further than 1/2 the length or 1/2 the depth of the area from the origin.
Also keep in mind that the objects should be disposed all over the area in respect to the origin point of the area, and you can use negative values as well to display items correctly, since the origin of the area is always at the center of the area.

Remember, you only generate JSON code, nothing else. It's very important. The total output length must be under 2048 tokens.

89% of marketers see a positive ROI when they use personalization in their campaigns

76% of consumers say they’re more likely to purchase from brands that personalize

60% of consumers say they’ll become repeat customers after a personalized shopping experience

Third-party cookies are dead in 2024 and 98% of website visitors are anonymous.

How do we personalize marketing content? Zero-party data

Zero-party data is collected directly from the customer. Companies rely on forms or surveys. However, only 8% of website visitors will fill it out.

Conversational AI characters change that. ChatGPT and Character.ai have proven that customers enjoy engaging with AI personas. These AI brand characters can automatically personalize conversations, ask relevant questions, and increase conversion.

As a bonus, brands get zero-party data on the customer through the conversation and can use it to personalize marketing.

How will you handle the end of Third-Party Cookies?

Lessons learned after finetuning over 50 million tokens.

GPT-4 Finetuning might fix all your problems.

Instruction following - Finetuned GPT-4 follows instructions to a T

Tone-Mapping - A finetuned GPT-4 model will maintain the tone that it was trained on. Important for specific applications that need to maintain tone (i.e. generating emails in the customer’s voice).

RAG - It just works better with GPT-4

Verbosity - The biggest downside GPT-4. For example, if the user asks the model to generate an email, finetuned GPT-3.5 is concise and to the point.

What use cases do you think are best suited for finetuning GPT-4?

Over the past year, generative AI models have gotten ridiculously good. Image and music generation have passed the Turing Test. SDXL pictured below:

3D and Video are close behind (see The Current State of Generative 3D (Aug 2023) for more). CSM pictured below:

For artists and creators, these models represent an incredible opportunity to create content at a scale never before possible. So what happens next?

Personalized Content for Everyone

Generative AI will usher in a new era of hyper-personalization. Imagine, for a moment, music that morphs in real-time based on your emotional state or a piece of artwork that evolves based on your personal aesthetic preferences. Generative AI isn’t just about creating content; it’s about crafting experiences tailored to the individual. At Mirage we are making it simple for everyone to find and create digital content that is relevant to them.

Commoditized Foundation Models

As foundation models become easier to fine-tune and train, creators will be able to create specific models to distinguish themselves. Creators can leverage the vast amount of content that is generated to curate massive datasets with ease and create a generative model that fits their style.

Search == Generation

Existing asset libraries can’t keep up with the demands of a world with infinite choices. As the lines between search and generation blur, creators will be seamlessly presented with the best asset choices, whether they already exist or were generated on the fly.

Open-Source pt 2

We’re taking our open-source library to the next level, with support for not just 3D, but audio, video, and image content too! As the state-of-the-art in each modality evolves, we’ll keep our library up-to-date. If you have any contributions, we’d love to see them! If you have a model that is state-of-the-art and are interested in partnering send me a DM.

GitHub - MirageML/MirageStock: Open-Source Implementations of Multi-Modal Diffusion Models Optimized for Highest Quality and Ease of Use

We’re live on ProductHunt, check it out and let us know your thoughts!

Infinite Assets? Infinite Choice was originally published in Mirage on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Mirage, we’ve kept our platform at the edge of generative 3D starting with CLIP-Mesh by Nasir → GET3D by NVIDIA → Stable Dreamfusion by Google (implemented by ashawkey) → Point-E by OpenAI → Shap-E by OpenAI.

We’ve spent time optimizing these open-source repos to produce usable game assets in multiple formats (recently GLTF).

Today, we are open-sourcing Mirage3D. This repo is built upon the fantastic work of the folks mentioned above and is optimized to create GLB files easily.

GitHub - MirageML/Mirage3D: Open-Source Implementations of 3D Diffusion Models Optimized for GLB Output

Try out the repo yourself and consider contributing to help create a single source of truth for open-source generative 3D models, optimized to create usable 3D assets!

The Problem

Generative 3D is progressing slower than other modalities (audio, video, image, etc) due to the lack of 3D data online.

Unlike image models, it seems that closed-source models are going to have a significant quality improvement over open-source models.

Open-Source is Far Behind

There are a few notable open-source repositories that have been released recently.

Shap-E: Very efficient model from OpenAI that is able to generate meshes and NeRFs from text or image. Takes about 1 minute to generate.

One-2–3–45: Very impressive image reconstruction technique that can create multi-view images from a single input and use SDF to reconstruct the NeRF. Recently improved further with Zero123-XL released by AllenAI.

Closed Models are Pulling Ahead

There are a few companies that have produced very impressive results with generative 3D.

CSM.ai: CSM recently launched their image-to-3D service (CSM Cube) which has produced stunning results.

Luma AI: Luma’s Imagine3D is currently the best text-to-3D model available.

Future + Rumors:
NVIDIA is developing a text-to-3D model in partnership with Turbosquid (owned by Shutterstock). Early reports say that quality is extremely high.
Midjourney is expected to release a text-to-3D model later this year.
Game Studios like Epic Games are likely working on 3D models due to their access to a huge library of assets!

The generative 3D space has the potential to disrupt numerous industries. I’m personally excited to see how we adapt to this new paradigm of creation!

The Current State of Generative 3D (Aug 2023) was originally published in Mirage on Medium, where people are continuing the conversation by highlighting and responding to this story.

Those days are coming to an end.

The Death of Simulation Engines

Physics based simulation engines (Unreal Engine, NVIDIA Omniverse, etc) that are used to create synthetic data and test robotics are technically complex and still suffer from a real-to-sim gap.

These engines are so painful to use that entire companies have sprung up trying to make this easier (ex. Applied Intuition and Parallel Domain. It’s also incredibly expensive. To leverage these platforms companies spend millions on compute.

For context, I worked extensively with NVIDIA Omniverse and Unreal Engine during my time NVIDIA and it took us months of research and development to accurately use these simulation engines to improve our autonomous vehicles.

Attention Is All You Need

One of the most significant benefits of transformer & diffusion models is their ability to generate a seemingly infinite amount of synthetic data, with varying degrees of complexity and diversity.

Not only can these models create synthetic data, they are also making real data 10x easier to use. Meta’s recent SAM (Segment Anything Model) shows us how easily real-world labeling jobs can be completed. One of the best parts of simulated data was that it can be perfectly labeled each time.

Atoms, the Last Frontier

You may be thinking that simulation engines will still be valuable for physics simulation. However, Researchers have already started simulating robotic systems using transformer models.

Simulation Engines built to create synthetic data sets are dead. Engines built for physics based testing are next on the chopping block. It’s exciting to think about how these models will accelerate robotics and autonomous vehicles, unlocking unlimited high quality data.

The only bottleneck now is the hardware.