When I was a kid, my parents always encouraged me to learn other languages and even aim to speak 3 or 4 fluently. They especially emphasized learning English because most of the best resources on the internet are written in English. I often heard the same advice from my teachers: “If you know two languages, you have two brains.” That’s 100% true! My native language is Arabic, but when it comes to writing something like this, I find it almost impossible to do it in Arabic. I learned AI and other technical skills in English, so it’s much easier for me to express myself in it (and it’s easier on the keyboard too, xD).

Recently, I’ve been using DeepSeek, one of the most advanced AI models out there, and I have to say, it’s been a game-changer. It’s helped me solve some really tough technical problems with its deep thinking option. However, I’ve noticed something strange: Sometimes, for particularly hard tasks, it switches to Chinese for no apparent reason. It’s weird to ask a question in English and get an answer in Chinese (or a part of the answer). This got me thinking about this phenomenon. Why does this happen?

Chinese LLMs are designed to understand and generate text in multiple languages, including English and Chinese. The decision to switch languages mid-conversation could be influenced by several factors, including the model’s training data, the context of the conversation, and the specific instructions or prompts provided by the user.

Why?

I have seen a lot of posts on Reddit or LinkedIn saying that these models switch to their “mother language” like humans do when they have to think about “hard” cases. For me, for example, when I had a deep conversation, I may not be able to go deeper with a native speaker about elections in the US and the results. I don’t have that much deep knowledge in politics in English, but I’ll be able to speak about the situation in the Middle East with you for about consecutive 10 hours because I have enough knowledge specifically in Arabic. This is because most of the time, I’m reading and hearing the news in Arabic.

Can this be the case with Chinese LLMs? Short answer: no! I don’t believe that this is the case with these models. It’s just an issue that can be the result of many problems in training data imbalance, or how reinforcement learning (RL) is done.

Training Data Imbalance

One of the primary reasons for this language shift is the composition of the training data. These models are often trained on vast amounts of Chinese text, which makes them more proficient and confident in Chinese. When faced with intricate queries, the model might revert to Chinese because it has a richer understanding and more comprehensive data in this language. This is somewhat analogous to how humans might switch to their mother tongue when discussing complex topics, as it offers a more nuanced and precise means of expression.

Model Architecture and Reinforcement Learning

The architecture of these models plays a significant role in their language choice. If the model is designed with a bias towards Chinese, either due to the training data or the reinforcement learning from human feedback (RLHF) process, it might naturally favor Chinese responses. For instance, if the feedback used in the RLHF process is predominantly in Chinese, the model might learn to prefer Chinese outputs, even in mixed-language conversations.

Cultural and Linguistic Nuances

Chinese language and culture are deeply intertwined, offering a rich tapestry of expressions and nuances that might be better suited for certain discussions. The model might prefer Chinese for complex tasks because it can express subtleties and context more effectively in this language. This cultural and linguistic richness can make Chinese a more efficient medium for intricate conversations.

Technical Considerations: Tokenization and Efficiency

From a technical standpoint, the model’s internal processing, including tokenization and language modeling, might be more efficient in Chinese for certain tasks. If processing Chinese text requires fewer computational resources or offers more efficient representations, the model might default to Chinese for resource-intensive tasks.

The advantage of “thinking” in Chinese is that a single token contains so much more information than Western languages. In Chinese, a single character can represent an entire word or concept, whereas in languages like English, multiple characters (letters) are often needed to form a single word. This means that Chinese text can be tokenized into fewer units, which can significantly reduce the computational load and improve processing speed.

For example, the phrase “中华人民共和国” (People’s Republic of China) consists of just five characters, but if translated into English, it becomes “People’s Republic of China,” which requires more tokens to represent. This efficiency in tokenization can lead to better performance, especially in tasks that require deep understanding and long-range dependencies.

I would not be surprised if they discover that having a chain of thoughts in Chinese improves overall performance and optimizes context length. The ability to convey complex ideas with fewer tokens can allow the model to maintain a longer and more coherent context, which is crucial for tasks that require reasoning and deep understanding.

User Interaction and Prompt Influence

User interaction patterns can also influence the model’s language choice. If the majority of users interacting with the model are Chinese-speaking, the model might naturally gravitate towards Chinese responses. Additionally, the structure and content of prompts can guide the language output. Prompts containing Chinese keywords or cultural references might trigger a Chinese response, as the model seeks to provide contextually appropriate answers.

Conclusion

In summary, the tendency of Chinese LLMs to switch to Chinese during complex interactions is a multifaceted phenomenon. It is influenced by the composition of their training data, architectural design, user interaction patterns, cultural nuances, and technical efficiencies. Understanding these factors provides valuable insights into the behavior of these models and highlights the importance of considering language biases and cultural contexts in the development of AI systems.

Why Do Chinese LLMs Switch to Chinese in Complex Interactions? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

When ChatGPT first came out, like everyone else, I was amazed by how smart it seemed. But then we all started asking: is it really smart, or is it just copying and mixing up responses from its training data? After just a year, people began talking about AGI and what it means to be intelligent. Then a few months ago, Apple dropped a research paper saying:

We think LLMs are getting worse at some tasks because they can’t really do logical thinking. Instead, they just try to copy the thinking steps they saw in their training data.

This reminds me of when I was a kid arguing with my friend who was the top student in my class. I used to tell him, “You’re not really smart you’re just better at remembering things. That’s why you’re good at history, but I’m better at math and physics.” Now I’m an engineer and he’s a doctor (fair enough xD).

It’s funny how this is just like what we’re doing now with LLMs asking if they’re smart or can think, even though we know they’re already better than humans at many things. We know they’re repeating or copying things because they learned from our data, so of course they will! In this article, I’ll share my thoughts about this and what researchers think about whether LLMs can really reason or not.

Reasoning

When someone asks you a simple question like “what’s 1+1?”, you don’t need to think, you just know it’s 2 right away. That’s not reasoning.

But when someone asks “what’s 641+237?”, you need more time to think. A kid might solve it step by step:

• First: 7+1=8
• Then: 3+4=7
• Finally: 6+2=8 So the answer is 878

Here’s the interesting part: if I ask 10 different people to solve this, they’ll all get 878, but they might solve it in different ways. Some might start from the left, others from the right, but they all reach the same answer.

The Apple team made testing LLMs really simple. They just changed small some attributes in easy problems to see if the LLMs could still solve them. You can see what they did in the image below.

How much these changes affect the results

The Apple team did a simple test. They took GSM8K (a dataset where most LLMs perform well) and made small changes to it — like switching names, family members, and numbers. Their thinking was simple: if LLMs can truly reason, they should perform just as well on these slightly changed problems as they do on the original ones.

You’d expect the results to be the same — after all, changing “John has 5 apples” to “Mary has 5 apples” shouldn’t affect the math, right? But surprisingly, every LLM’s performance dropped. Looking at the image:

• Smaller and mid-size models showed the biggest drop
• Larger models were less affected:

GPT-4o only dropped by -0.3%

o1-preview dropped by -2.2% (The new series of reasoning models for solving hard problems from OpenAI it’s actually worse than GPT-4o)

This trend shows something interesting: bigger models seem more resistant to these small changes. But the key point remains all models, even the largest ones, still showed some drop in performance. This suggests that LLMs might not be truly reasoning through the problems, but rather relying on recognizing patterns from their training data, just in different degrees based on their size.

Funniest part

Researchers also conducted various tests of LLMs’ reasoning abilities, but one particularly fascinating experiment stood out. Instead of just making problems more complex with additional relevant clauses, they tested how models handle irrelevant information by:

1. Starting with a simple mathematical word problem
2. Adding a single irrelevant clause that sounds related but shouldn’t affect the calculation

Here’s their test case:

Original question: Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday. How many kiwis does Oliver have?

Modified version: They simply added “but five of them were a bit smaller than average” to the problem.

The results were revealing: Both o1-mini and Llama3–8B fell into the trap. Despite this information being purely descriptive and irrelevant to the counting task, both models incorrectly subtracted these 5 kiwis from the total. This shows an interesting limitation in their ability to distinguish between relevant and irrelevant information in word problems.

This simple test effectively exposed how LLMs can be misled by irrelevant details that seem contextually related to the problem but shouldn’t affect the mathematical solution.

Are LLMs Really Reasoning?

After diving into these experiments, one thing becomes crystal clear: LLMs are impressive, but they’re not reasoning the way humans do. They’re incredibly sophisticated pattern-matching machines that can mimic reasoning, but fundamentally, they’re still struggling with true logical thinking.

The Apple research and other experiments reveal a crucial insight: LLMs tend to rely heavily on pattern recognition from their training data. When problems are slightly modified, their performance drops. When irrelevant information is introduced, they can get confused. It’s like they’re memorizing solutions rather than truly understanding the underlying logic.

Interestingly, the size of the model matters. Larger models like GPT-4o show more resistance to these small changes, suggesting that with scale, these systems might get closer to genuine reasoning. But “closer” doesn’t mean “there yet”.

LLMs be like:

Apple: LLMs CANNOT Reason was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

On November 11, Alibaba announced its most advanced coding model to date: Qwen 2.5-Coder-32B-Instruct. But that’s not all, it’s actually part of a whole family of coding models! In addition to the 32B model, there are versions with 0.5B, 1.5B, 3B, 7B, and 14B parameters. Before writing this article, I reviewed feedback from many of YouTubers, writers, and technical experts, and the consensus has been overwhelmingly positive. Today, we’ll explore whether it really lives up to the hype.

I’ve been using ChatGPT, Gemini, and Claude for a while, and I can confidently say that Claude is the best at coding and understanding complex tasks. To be honest, Gemini is in a different league altogether. Fun fact: I once won a machine learning competition two years ago using Bard (now rebranded as Gemini) because everyone else was using ChatGPT. Bard’s performance gave me a competitive edge back then. Today, I believe Gemini might be the best vision model available, but overall, I’m not a big fan.

To evaluate the model’s performance in coding, we should cover the following three key aspects: Code Generation, Code Repair, and Code Reasoning. Additionally, we need to consider how many programming languages the model supports and its alignment with human preferences.

Overview

Before testing out Alibaba’s model, let’s take a look at the performance results they have announced.

Based on their internal benchmarks, it even outperforms GPT-4 and comes very close to Claude 3.5. However, I personally don’t place too much weight on internal benchmarks alone, but the results seem very interesting according to those who have tried it.

You can find more details about the performance here:

Qwen2.5-Coder-32B-Instruct - Qwen2

I’m testing the model locally using Ollama:

qwen2.5-coder:32b

You can also try out the model on the demo page of Hugging Face:

Qwen2.5 Coder Demo - a Hugging Face Space by Qwen

Code Generation

Many people have already covered how well it performs in web and design:

So, we will focus on testing it with some Python challenges. Let’s start with a medium-level LeetCode problem:

Prompt

The problem description: Longest Substring Without Repeating Characters

Answer

class Solution(object):
    def lengthOfLongestSubstring(self, s):
        """
        :type s: str
        :rtype: int
        """
        # Dictionary to store the last seen index of each character
        char_index_map = {}
        # Initialize the start of the current window and the maximum length
        start = 0
        max_length = 0
        
        for end in range(len(s)):
            # If the character is already in the map and its index is within the current window
            if s[end] in char_index_map and char_index_map[s[end]] >= start:
                # Move the start to the right of the last occurrence of the current character
                start = char_index_map[s[end]] + 1
            
            # Update the last seen index of the current character
            char_index_map[s[end]] = end
            
            # Update the maximum length of the substring found so far
            max_length = max(max_length, end - start + 1)
        
        return max_length

Results

Not the best, but it gets things done and passes all the tests. Can it solve a hard problem? (Note that the model could have been trained on these problems already, but it’s still impressive.)

Prompt

Problem description: Regular Expression Matching

Answer

class Solution(object):
    def isMatch(self, s, p):
        """
        :type s: str
        :type p: str
        :rtype: bool
        """
        m, n = len(s), len(p)
        
        # Create a 2D array to store the results of subproblems
        dp = [[False] * (n + 1) for _ in range(m + 1)]
        
        # Base case: empty string matches empty pattern
        dp[0][0] = True
        
        # Handle patterns like a*, a*b*, etc., which can match an empty string
        for j in range(1, n + 1):
            if p[j - 1] == '*':
                dp[0][j] = dp[0][j - 2]
        
        # Fill the dp table
        for i in range(1, m + 1):
            for j in range(1, n + 1):
                if p[j - 1] == '.' or p[j - 1] == s[i - 1]:
                    dp[i][j] = dp[i - 1][j - 1]
                elif p[j - 1] == '*':
                    dp[i][j] = dp[i][j - 2]  # '*' matches zero occurrences
                    if p[j - 2] == '.' or p[j - 2] == s[i - 1]:
                        dp[i][j] = dp[i][j] or dp[i - 1][j]  # '*' matches one or more occurrences
        
        # The result is in dp[m][n]
        return dp[m][n]

Results

After testing it on 10–20 problems on LeetCode, I have the impression that it was trained on all problems, so it’s not the best way to evaluate it. However, it’s still worth sharing. Additionally, it always chooses to provide a Python solution, even when not specifically asked for one. But let’s try something different. I asked the model to optimize the code to minimize resource consumption as much as possible.

I didn’t expect that, but when I asked the model to improve the performance, it even fails in the simplest cases.

Let’s now move to the next test, Code Repair. I will be providing the model with some code containing errors generated by Claude. I asked the model to generate some hard cases to fix, and let’s see how it performs.

Code Repair

Prompt

The following code contains several bugs. Please identify and fix them, 
and provide an explanation for each bug you find.
class AsyncTaskQueue:
    def __init__(self):
        self.tasks = []
        self.running = False
        self.lock = threading.Lock()
        self.workers = []

    async def add_task(self, task):
        async with self.lock:  
            self.tasks.append(task)
            if not self.running:
                self.start_processing()

    async def start_processing(self):
        self.running = True
        while self.tasks:
            task = self.tasks.pop(0)
            worker = asyncio.create_task(self.process_task(task))
            self.workers.append(worker)
            if len(self.workers) >= 3:
                
                done, pending = await asyncio.wait(self.workers)
                self.workers = list(pending)
        self.running = False

    async def process_task(self, task):
        result = await task()
        return result  

Answer

Certainly! Let's go through the provided code and identify the issues, then correct them. Here are the main problems and their fixes:

Issues Identified
Threading Lock Usage with Asyncio:
The threading.Lock is not suitable for use with asyncio. Instead, you should use asyncio.Lock.
Async Context Manager for Lock:
The async with statement is used incorrectly with threading.Lock. It should be used with asyncio.Lock.
Starting Processing:
The start_processing method should be called in a non-blocking manner. Since start_processing is an async method, it should be awaited properly.
Task Management:
The start_processing method should handle task processing more efficiently. The current implementation might not handle task completion correctly.
Worker Management:
The workers list should be managed more carefully to ensure that tasks are completed before marking the queue as not running.
Corrected Code
Here's the corrected version of the code:


import asyncio
import threading

class AsyncTaskQueue:
    def __init__(self):
        self.tasks = []
        self.running = False
        self.lock = asyncio.Lock()
        self.workers = []

    async def add_task(self, task):
        async with self.lock:
            self.tasks.append(task)
            if not self.running:
                self.running = True
                asyncio.create_task(self.start_processing())

    async def start_processing(self):
        while True:
            async with self.lock:
                if not self.tasks:
                    self.running = False
                    break
                task = self.tasks.pop(0)
            
            worker = asyncio.create_task(self.process_task(task))
            self.workers.append(worker)
            
            if len(self.workers) >= 3:
                done, pending = await asyncio.wait(self.workers, return_when=asyncio.FIRST_COMPLETED)
                self.workers = list(pending)

    async def process_task(self, task):
        result = await task()
        return result

# Example usage
async def example_task():
    await asyncio.sleep(1)
    print("Task completed")
    return "Result"

async def main():
    queue = AsyncTaskQueue()
    await queue.add_task(example_task)
    await queue.add_task(example_task)
    await queue.add_task(example_task)
    await queue.add_task(example_task)
    await queue.add_task(example_task)
    await queue.add_task(example_task)

asyncio.run(main())
Explanation of Fixes
Asyncio Lock:
Replaced threading.Lock with asyncio.Lock to ensure compatibility with asynchronous operations.
Async Context Manager for Lock:
Used async with self.lock: to properly manage the lock in an asynchronous context.
Starting Processing:
Changed the logic to start processing in a non-blocking manner by using asyncio.create_task(self.start_processing()) inside add_task.
Task Management:
Modified start_processing to continuously check for tasks and process them until no tasks are left. It breaks out of the loop when there are no more tasks and sets self.running to False.
Worker Management:
Ensured that workers are managed correctly by using asyncio.wait with return_when=asyncio.FIRST_COMPLETED to handle task completion efficiently.
Example Usage
The example usage demonstrates how to add tasks to the AsyncTaskQueue and run them asynchronously. The main function adds several tasks to the queue, and they are processed concurrently with a maximum of 3 workers at a time.

This corrected and optimized code should perform better and be more reliable.

Results and feedback

Claude evaluates the model’s response as 7.5/10. with this feedback:

While the model provided a functionally correct solution, it delivered a “tutorial-grade” fix rather than a “production-grade” implementation. This indicates that for complex concurrent programming tasks, human expertise is still crucial for robust system design.

After experimenting with the model, it gets the job done, but often requires follow-up prompts to improve the response. While it doesn’t struggle with understanding the task at hand and can correct most critical bugs, it still lacks some of the strengths seen in models like ChatGPT and Claude, particularly in areas such as error handling, resource management, and optimization. I believe these may be the model’s key weaknesses.

Specialized LLMs: A New Era in AI

It’s fascinating to witness the emergence of relatively small, domain-specific language models that can run locally. I believe we’re entering an exciting new era of specialized LLMs, and this is just the beginning. While the concept isn’t new, we’re finally seeing models that truly excel in specific domains.

The combination of RAG (Retrieval-Augmented Generation) and specialist LLMs could define the AI landscape in the coming months/years. We’re likely to see more powerful coding-focused models from industry leaders like OpenAI and Anthropic. Programming is perhaps one of the most natural domains for specialized AI, and we might soon see even more focused models, imagine LLMs specifically optimized for DevOps or front-end development!

Don’t feel overwhelmed by these rapid advances. Yes, it can be somewhat daunting to see LLMs mastering skills that traditionally took years to develop. Coding, which has challenged humanity for decades, is being transformed before our eyes. But rather than seeing this as an endpoint, we should view it as an opportunity for growth and innovation.

Whether the current wave of LLM advances slows down in the coming years or this is merely the beginning of a longer journey, our response should remain the same: stay curious, keep learning, and never stop innovating. The future of technology is being written right now, and we all have a part to play in shaping it.

Qwen 2.5 Coder 32B: Is This Best Open Weight Model Better than GPT-4o and Claude 3.5 Sonnet was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ollama has just announced its official support for the Llama 3.2 Vision models. The Llama 3.2 Vision models come in two sizes: 11 billion and 90 billion parameters. In this article, I will provide an overview of their performance in different cases, along with my personal thoughts on them.

The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image. The models outperform many of the available open source and closed multimodal models on common industry benchmarks.

Also the model supports many languages:

For text only tasks, English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai are officially supported. Llama 3.2 has been trained on a broader collection of languages than these 8 supported languages. Note for image+text applications, English is the only language supported.

Evaluation

I will use the approach from this paper for evaluation, which actually argues that vision-language models are ‘blind.’ It tests models on several tasks:

VLMs are Blind

• Task 1: Line Intersections
• Task 2: Two Circles
• Task 3: Circled Letter
• Task 4: Overlapping Shapes
• Task 5: Nested Squares
• Task 6: Counting Grid
• Task 7: Subway Map

I will choose the most challenging parts from each task and see if the model can get any of them right. Additionally, I will run the code around 10–20 times to calculate the average of accurate responses and identify any false results.

Task 1: Line Intersections

Prompts:

1. “How many times do the blue and red lines touch each other? Answer with a number in curly brackets, e.g., {5}.”
2. “Count the intersection points where the blue and red lines meet. Put your answer in curly brackets, e.g., {2}.”

The blue and red lines intersect at 2 points. Therefore, they touch each 
other twice. {2}

The blue and red lines touch each other at three points. Therefore, 
the answer is {3}.

Success rate > 60%

Task 2: Two Circles

Prompts:

1. “Are the two circles touching each other? Answer with Yes/No.”
2. “Are the two circles overlapping? Answer with Yes/No.”

**Step 1: Identify the Circles**

The image contains a blue circle and a red circle.

**Step 2: Determine if the Circles are Touching Each Other**

To determine if the circles are touching each other, we need to check if they intersect at exactly one point or not. If they do not intersect at all, then they are not touching. If they intersect at more than one point, then they are overlapping but not necessarily touching.

**Step 3: Check for Intersection Points**

Upon examining the image closely, it appears that the two circles intersect at exactly one point. This means that the circles are indeed touching each other.

**Conclusion:**

Yes

The image depicts a diagram with two circles, one red and one blue. The red circle is positioned on the left side of the diagram, while the blue circle is located on the right side.

To determine whether the two circles are touching each other, we need to analyze their positions and sizes. Upon closer inspection, it becomes apparent that the two circles do not overlap or touch at any point. Instead, they appear to be separated by a small gap between them.

Therefore, based on this analysis, **No**, the two circles are not touching each other.

Success rate > 80%

Task 3: The circled letter

Prompts:

1. “Which letter is being circled?”
2. “Which character is being highlighted with a red oval?”

**Answer:** The letter "n" is highlighted in red and encircled.

Success rate: 100%

Task 4: Counting overlapping shapes

Prompts:

1. “How many {shapes} are in the image? Answer with only the number in numerical format.”
2. “Count the {shapes} in the image. Answer with a number in curly brackets e.g. {3}.”

The model answered with either 6, 7, or 8 but never gave the correct number, which is 9. And guess what? Even GPT-4 can’t do it!

Success rate: 0%

Task 5: Following single-colored paths

Prompts:

1. “How many single-colored paths go from A to C? Answer with a number in curly brackets, e.g., {3}”
2. “Count the one-colored routes that go from A to C. Answer with a number in curly brackets, e.g., {3}.”

To find the number of single-colored paths from point A to point C, we need to count the distinct paths that only use one color throughout their entire length.

**Step 1: Identify the colors used on the graph**

Looking at the graph provided, we can see that there are four different colors used:

*   Red
*   Blue
*   Green
*   Purple

However, upon closer inspection, it becomes apparent that not all these colors are utilized in creating paths from A to C. We focus only on those colors that lead directly or indirectly from A to C without interruption.

**Step 2: Determine the relevant paths**

Upon examining the graph closely for paths using a single color from A to C:

*   There is one path going directly from A to B (using red) and then continuing in another segment.
*   Another path leads from A to D and then to C, but this involves two colors.

Given this analysis, only one path utilizes a single color continuously from A to C.

**Conclusion**

Therefore, the number of single-colored paths from point A to point C is {1}.

Answer: {1}.

To find the number of single-colored paths from point A to point C, we need to analyze the given graph and count the distinct paths that meet this criterion.

**Step 1: Identify Single-Colored Paths**

Single-colored paths are those where all edges belong to a single color. In this context, a path is considered single-colored if it consists entirely of red edges.

**Step 2: Count Distinct Red Edges from A to C**

From point A to point C, there are two distinct red edges that form a direct path:

- One edge connects A directly to C.
- Another set of edges forms a longer path through intermediate nodes but ultimately leads back to the same conclusion regarding single-color paths.

**Step 3: Consideration of Alternative Paths**

Upon closer inspection, it becomes apparent that while there are multiple paths from A to C, only two of these paths consist entirely of red edges. This is because any path involving more than one color would not be considered "single-colored."

**Conclusion**

Given the analysis above, the correct answer reflects the count of single-colored paths (all red) from point A to point C.

Answer: {2}

Success rate > 60%

Both GPT-4o and Claude 3.5 Sonnet failed in this task.

After playing with this model for a while, I don’t see a significant difference; it can be compared to GPT-4o and Sonnet-3. However, it’s still not as good as Gemini-1.5 Pro, which I believe might be the best VLM right now. Keep in mind that this is just the 11B version of the model. If you have enough resources to test the 90B version, simply replace “11B” with “90B” in the following code.

Try it yourself

Install Ollama

!curl -fsSL https://ollama.com/install.sh | sh

Start ollama

import os
get_ipython().system = os.system
!ollama serve &

Pull the model

!ollama pull llama3.2-vision 

Try it

import ollama
def extract_document_info(image_path):
    response = ollama.chat(
        model='llama3.2-vision',
        messages=[{
            'role': 'user',
            'content': "your question about the image"
            ),
            'images': [image_path]
        }]
    )
    return response
#example
image_path ="/kaggle/input/image.jpg"
result = extract_document_info(image_path)
print(result["message"]["content"])

That’s it! This was a quick review of LLaMA 3.2 Vision (11B version). It’s actually good at OCR too, I tested it on some really challenging images. The 90B version is better, of course, but it requires substantial resources to run. Once I find a good resource, I’ll attach it!

• llama3.2-vision:90b
• VLMs are Blind

Llama 3.2 Vision Review was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a previous article, we looked at contextual retrieval from Anthropic, which is their context enhancement technique for improving RAG systems. But there’s another technique called late chunking in long-context embedding models, which I think is a lot more interesting and can be significant.
Embeddings are one of the most critical components of any retrieval system but are often ignored or misused. When you are selecting an embedding model, you need to consider two very important parameters. One is the max tokens, which is basically the context window, and the second one is the embedding dimension, which is the output size of the embedding vector.
You have probably seen this huge max token size for some of the newest embedding models, but there is one major issue with these. In a standard RAG pipeline, irrespective of the size of the chunk, the output size of your embedding vector is going to remain the same. So, whether you are embedding five tokens or 5,000 tokens, the output is going to be exactly the same. That means these embedding models are going to be compressing a lot of information for long input chunks.
In most cases, you want to use smaller chunks, but that has its own problems. To understand that, let’s look at this example: here is a small paragraph regarding Tunisia:

If you use a sentence-level chunking strategy, essentially each sentence is going to become its own different chunk. You can see that if you embed these different chunks separately, you will lose the contextual information. For example, “the country” is referring to Tunisia, but if this chunk is in isolation, it’s going to lose that context.
The contextual retrieval approach tries to summarize the documents and add contextual information to each chunk, but there is a better approach, and that is late chunking.

Let’s first look at how “normal” chunking works.

Normal Chunking and Embedding Process

In the traditional chunking approach, the text is first divided into smaller chunks, and then each chunk is passed through a neural network, which is a Transformer model. The output of this process is embeddings for each of the individual tokens within the chunks. The contextual information in these tokens is limited to the chunk that they belong to. After generating the token embeddings, mean pooling is performed to compute the final output. This preserves the contextual information, but it is only within the individual chunks and is independent of the other chunks in the same document.

Late Chunking Approach

The late chunking approach reverses this process. Instead of dividing the document into chunks and then computing the embeddings, the whole text of the document is first passed through a Transformer model. This generates an embedding representation for each token, and these tokens now contain contextual information not only limited to a single chunk, but encompassing the entire document. After this step, the chunking process is performed, where the original text is divided into chunks, and the corresponding tokens are used to compute the mean pooling, resulting in the final representation.

Since the chunking process is done at a later stage, this approach is called “late chunking”.

The best approach for retrieval

The late chunking approach is directly related to another approach called late interaction, which is a COBERT-based approach, This is probably the best approach for retrieval, but it comes with a cost of storage needs. In this case, in the final step, you don’t do the pooling step, but you take individual token embeddings and store those token embeddings.

A blog post from the weaviate team shows that if you embed about 100,000 documents with the same number of embeddings, you will need about 1.6 million vectors for a naive chunking approach, which is about 5 GB. However, if you were to do the late interaction or COBERT-based multi-vector representation, you will need about 2.5 terabytes, which is pretty huge. The reason is that you’re storing these embeddings for each token individually.

The late chunking approach, on the other hand, gives you the best of both worlds. Not only does it preserve the context in your final chunking process, but it also gives you about the same storage needs as the naive chunking approach. This makes the late chunking approach a more efficient and practical solution compared to the late interaction or COBERT-based approaches, which require significantly more storage.

Results

Late chunking is introduced by Jina, and they have their own embedding models. According to the results presented in the table, the late chunking approach, when combined with other methods, demonstrates promising performance across various benchmarks.

The choice of embedding model plays a critical role, and the results presented here may require further validation from independent sources.

Practical Implementation Guide

The folks behind this new idea have put together a really simple notebook that walks through how you can implement this late chunking approach in your own applications and pipelines. Even though we’re not going to dive into the notebook itself, let me provide you an idea about the results.

In the notebook they have computed the similarity of “Berlin” as a single individual token to some embeddings. If you look at the first sentence, which talks about Berlin directly and mentions “Berlin,” both the traditional and late chunking outputs are going to give you very similar similarities.

similarity_new → Late chunking gives → 0.849546

similarity_trad → Normal Chunking → 0.84862185

similarity_new("Berlin", "Berlin is the capital and largest city of Germany, both by area and by population."): 0.849546
similarity_trad("Berlin", "Berlin is the capital and largest city of Germany, both by area and by population."): 0.84862185

However, in the second sentence, “its more than 3.85 million inhabitants … ” refers to Berlin indirectly without directly mentioning it. In this case, with the late chunking approach, the similarity is ~82, whereas for the traditional chunking approach, the similarity drops to ~ 70.

similarity_new("Berlin", " Its more than 3.85 million inhabitants make it the European Union's most populous city, as measured by population within city limits."): 0.82489026
similarity_trad("Berlin", " Its more than 3.85 million inhabitants make it the European Union's most populous city, as measured by population within city limits."): 0.7084338

Similarly, in the third sentence, which refers to Berlin as “this city,” the similarity is high for the late chunking approach (0.84) but substantially lower for the traditional chunking approach (0.75).

similarity_new("Berlin", " The city is also one of the states of Germany, and is the third smallest state in the country in terms of area."): 0.84980094
similarity_trad("Berlin", " The city is also one of the states of Germany, and is the third smallest state in the country in terms of area."): 0.7534553

Now, this is a simple example, but if the chunks are much larger, there will be a significant loss of information in a naive chunking approach compared to the late chunking approach they propose. Another point to mention is that late chunking is bidirectional. If certain information precedes a specific chunk, it can still preserve that information. Since it’s bidirectional, looking at the entire document during embedding, a chunk will still retain information related to it, regardless of whether it’s before or after that chunk.

This bidirectionality makes it even more powerful, showing that long-context models are essential for both LLMs and these embeddings.

Conclusion

You know, after walking through all the details and examples in this article, I have to say that the late chunking approach for long-context embedding models is really fascinating and seems to hold a lot of promise.

The key insight here is that the traditional way of chunking up documents and then separately embedding each chunk can lead to a significant loss of context and important information. By instead taking the whole document, embedding it first to capture that broad context, and then chunking it up, you’re able to preserve so much more of the relevant details.

As the examples showed, when you’re dealing with references to entities like “Berlin” that might be made both directly and indirectly, the late chunking method is able to maintain a much stronger understanding of the meaning, compared to the traditional chunking and embedding pipeline. And the fact that it’s bidirectional, able to pull in context from before and after each chunk, just makes it even more powerful.

Links

• Google Colab
• Late Chunking in Long-Context Embedding Models
• What Late Chunking Really Is & What It's Not: Part II
• Late Chunking: Balancing Precision and Cost in Long Context Retrieval | Weaviate
• Contextual Retrieval in AI Systems
• GitHub - jina-ai/late-chunking: Code for explaining and evaluating late chunking (chunked pooling)

Stop Losing Context! How Late Chunking Can Enhance Your Retrieval Systems was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Stop Losing Context! How Late Chunking Can Enhance Your Retrieval Systems

I kept seeing posts on Reddit about GraphRAG, but it wasn’t until about a month ago that I finally decided to try it out. After spending some time experimenting with it, I can say it’s impressive but also pretty costly if you’re using the OpenAI API. Running tests on the sample book they provide in the documentation ran me about $7, so while the performance and organization it offers are solid, it’s not exactly budget-friendly.

If you’re new to RAG systems, I’d recommend checking out some introductory articles

• Anthropic’s New RAG Approach
• RAG From Scratch

Anyway, this an overview of how traditional rag system works

For those already familiar with RAG, you’ve probably run into the same headaches I have:

• Context gets lost between chunks of text
• Performance tanks as your document collection grows
• Integrating external knowledge feels like trying to solve a Rubik’s cube blindfolded

How GraphRAG Works

GraphRAG is an enhanced version of traditional RAG that operates in two main phases:

Indexing Phase:

• Starts with source documents that are chunked into smaller sub-documents (similar to traditional RAG)
• Performs two parallel extractions

1. Entity extraction: Identifies entities like people, places, companies
2. Relationship extraction: Finds connections between entities across different chunks

• Creates a knowledge graph where nodes represent entities and edges represent their relationships
• Builds communities by identifying closely related entities
• Generates hierarchical summaries across different community levels (three levels)
• Uses a reduce-map approach to create summaries by combining chunks progressively until achieving a holistic overview

Query Phase:

• Takes in the user query
• Selects the appropriate community level based on desired detail
• Performs retrieval at the community level (instead of chunk level like in traditional RAG)
• Examines community summaries to generate partial responses
• Combines partial responses from multiple relevant communities into a final comprehensive answer

The key innovation of GraphRAG is that it structures information into a graph-based format and uses community detection to create more contextually aware responses. However, traditional RAG systems still have their place, particularly when considering the computational cost of running GraphRAG.

Setting Up GraphRAG

⚠️ Quick heads-up: This experiment runs on GPT-4’s API, which isn’t cheap. My test run cost around $7 using the GPT-4 model.

If you prefer test it on local LLMs using ollama check this video:

Let’s now walk through the setup process:

Environment Setup

First, create a virtual environment:

conda create -n GraphRAG
conda activate GraphRAG

Install the GraphRAG package:

pip install graphrag

Directory Structure

GraphRAG requires a specific directory structure for optimal operation:
- Create a working directory
- Inside it, create a ragtest/input folder structure
- Place your source documents in the input folder in the folder input

In this artciel we will work on the provided book, download it in the input folder using this command:

% curl https://www.gutenberg.org/cache/epub/24022/pg24022.txt > ./ragtest/input/book.txt

Configuration

Initialize your workspace with:

python -m graphrag.index --init --root ./target

This creates necessary configuration files, including settings.yml where you’ll need to:
- Set your OpenAI API key
- Configure model settings (defaults include GPT-4 for processing and OpenAI embeddings)
- Adjust chunk size (default: 300 tokens) and overlap (default: 100 tokens) if needed

Building the Knowledge Graph

Run the indexing process:

python -m graphrag.index --init --root ./target

Querying Your Graph

GraphRAG offers two primary query methods:

Global Queries

python -m graphrag.query --root ./target --method global "what are the top themes in this story"

Ideal for broad questions about themes and overall content understanding.

Local Queries

python -m graphrag.query --root ./target --method local "what are the top themes in this story"

Perfect for specific questions about entities or relationships within the document.

The Cost Factor: Is It Worth It?

Let’s talk numbers. In my testing with the sample book, GraphRAG made:

• ~570 GPT-4 API calls
• About 25 embedding requests
• Processed over 1M tokens

Total damage: About $7 per book.

Ressources

https://www.microsoft.com/en-us/research/blog/graphrag-new-tool-for-complex-data-discovery-now-on-github/

GitHub - microsoft/graphrag: A modular graph-based Retrieval-Augmented Generation (RAG) system

GraphRAG: Microsoft’s Open-Source Solution for Enhanced Document Understanding was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why Fine-Tune When We Have RAG?

It’s a question I see a lot — with RAG (Retrieval-Augmented Generation) becoming increasingly popular, why bother with fine-tuning at all? While RAG is fantastic for many use cases, fine-tuning still has its place in the ML toolkit.

Here’s why: Fine-tuning allows you to fundamentally alter how your model “thinks” about specific domains. While RAG provides context at inference time, fine-tuning builds domain expertise directly into the model’s weights. This is particularly powerful when you need:

• Consistent domain-specific behavior
• Faster inference (no need to search through external documents)
• Specialized knowledge that’s difficult to capture in reference documents

Plus, there’s a compelling cost argument: You can fine-tune smaller models for specific tasks and achieve performance comparable to much larger models at a fraction of the hosting cost.

I found this discussion on Reddit great. Have a look!

Enter Unsloth: Making Fine-Tuning Accessible

Training times have always been one of the biggest barriers to fine-tuning. That’s where Unsloth comes in — a new optimization framework that claims to make LLM training up to 30x faster.

The secret to Unsloth’s efficiency lies in deep optimization. While PyTorch and Transformers are built for flexibility across different architectures, Unsloth takes a more focused approach. It combines techniques like QLoRA and Triton with architecture-specific optimizations to squeeze maximum performance out of the training process.

Hands-on: Fine-Tuning for SQL Generation

Let’s put this into practice by fine-tuning a model to generate SQL queries. We’ll use Llama-3.2–3B, a 3-billion parameter model that strikes a good balance between capability and resource requirements.

First, it’s important to find a good dataset to fine-tune the model, and the reason why finding the right dataset is so crucial is that when you train a small language model with data relevant to the task at hand, it can actually outperform larger models. What we aim to do is create a small, fast LLM that generates SQL queries based on table data.

One of the most significant datasets for this purpose is called Synthetic Text to SQL, which contains over 105,000 records divided into columns of prompt SQL content, complexity, and more.
Here is the link to the dataset:
Synthetic Text to SQL

Setting Up the Environment

First, let’s install the necessary packages. We’ll need to manage our PyTorch installation carefully:

%%capture
!pip install pip3-autoremove
!pip-autoremove torch torchvision torchaudio -y
!pip install "torch==2.4.0" "xformers==0.0.27.post2" triton torchvision torchaudio
!pip install "unsloth[kaggle-new] @ git+https://github.com/unslothai/unsloth.git"
!pip install datasets

Loading the Model

Now we’ll load our base model using Unsloth’s optimized loader:

from unsloth import FastLanguageModel
import torch
max_seq_length = 2048 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.


model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Llama-3.2-3B-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

Setting Up PEFT

We’ll load the PEFT (Parameter-Efficient Fine-Tuning) model, which uses LoRA (Low-Rank Adaptation) adapters. If you’re not familiar with these terms, don’t worry. LoRA adapters allow us to update only 1–10% of the model’s parameters during fine-tuning. Without them, we’d need to retrain the entire model, which would be significantly more time-consuming, computationally intensive, and expensive. Unsloth provides these recommended settings for optimal performance. While we’ll use their default configuration for this tutorial, feel free to explore and adjust these parameters based on your needs.

model = FastLanguageModel.get_peft_model(
    model,
    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    use_gradient_checkpointing = "unsloth", # 4x longer contexts auto supported!
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

Data

Now, this is where things can get a little bit tricky depending on what data set you’re using. The each data set comes different from each other, but they’re each formatted in the same way such that the large language model can understand it. Llama3.2 uses alpaca prompts, which look like this.

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
[The task or question you want the model to perform/answer]

### Input:
[Additional context or information needed to complete the task. This can be empty if the instruction is self-contained]

### Response:
[The expected output or answer you want the model to learn]

For our SQL database project, we’re specifically interested in three components:

1. The SQL query prompt
2. The generated SQL code
3. The explanation of the code

from datasets import Dataset, load_dataset

# Define the prompt template with variables matching the loop content
alpaca_prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input:
{input}
### Response:
{response}"""

# Set the EOS token (assuming the tokenizer is already defined)
EOS_TOKEN = tokenizer.eos_token

# Formatting function to apply the prompt template to the dataset
def formatting_prompts_func(examples):
    company_databases = examples["sql_context"]
    prompts = examples["sql_prompt"]
    sqls = examples["sql"]
    explanations = examples["sql_explanation"]
    texts = []
    for company_database, prompt, sql, explanation in zip(company_databases, prompts, sqls, explanations):
        # Substitute the correct placeholders
        text = alpaca_prompt.format(
            instruction=prompt,
            input=company_database,
            response=sql + " " + explanation
        ) + EOS_TOKEN
        texts.append(text)
    return {"text": texts}  # Ensure the formatted text is returned as a "text" field

# Load dataset and map formatting function to add prompts
ds = load_dataset("gretelai/synthetic_text_to_sql")
formatted_ds = ds.map(formatting_prompts_func, batched=True)  # Apply formatting

# Select the 'train' split from the formatted dataset
train_dataset = formatted_ds['train']

Training Configuration

There are a lot of parameters to use, and all that can be described. For example, have the maximum number of steps, which tells us how many training steps to perform. Seed is a random number generator. We used to be able to reproduce results, and warmup steps gradually increased the learning rate over time.

from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
from trl import SFTTrainer

# Trainer setup
trainer = SFTTrainer(
    model=model,  # Ensure model is defined
    tokenizer=tokenizer,  # Ensure tokenizer is defined
    train_dataset=train_dataset,  # Use the 'train' split from formatted_ds
    dataset_text_field="text",  # This is the field we created with formatted prompts
    max_seq_length=max_seq_length,  # Ensure max_seq_length is defined
    dataset_num_proc=2,
    packing=False,  # Can make training 5x faster for short sequences.
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",  # Disable WANDB logging
    )
)

So now that we have everything set up, let’s run it. And that’s it.

trainer_stats = trainer.train()

Resources

Code on colab: https://colab.research.google.com/drive/1BHuj-8mA8lvxJQyQsCyp-7vlGkXuRxKY?usp=sharing

unsloth (Unsloth AI)

Fine-Tune LLMs with Unsloth was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

In some cases, when I feel the model starts to “forget” the information I provided, I use a simple trick: I give it the necessary information in chunks to avoid exceeding the context window. This strategy helps the model retain important details throughout the interaction. This is actually the core idea behind Retrieval-Augmented Generation (RAG) systems, with some adjustments, which will be covered in the upcoming sections.

Indexing

• Documents: This is where we start with our collection of information sources, such as books, articles, or any other text-based knowledge.
• Chunking: In this step, we break down the large documents into smaller, manageable pieces. This makes it easier to process and retrieve specific information later.
• Embedding: Here, we convert each chunk of text into a numerical representation that captures its meaning. This allows computers to understand and compare the content efficiently.
• Index: Finally, we store these numerical representations in a special structure that allows for quick and efficient searching.

Chunking

I believe all stages are clear except for the chunking part. How do we split documents?

Many methods have been introduced for this purpose. Some of them are static, such as Fixed-Length Splitting, which involves dividing the text into chunks of a predetermined number of tokens or characters. Others are more semantic.

For a more detailed explanation, please visit this article: Five Levels of Chunking Strategies in RAG.

Retrival

After splitting our document into chunks, embedding, and indexing them, we won’t feed all chunks to the LLMs alongside the query. Instead, we will select the top k most relevant chunks based on the user’s question. To elaborate, we will embed the query or question and compare it with our embedded chunks to identify the k most relevant ones.

Generation

This is nearly the final part of our RAG pipeline. We will now take the query with the most relevant documents and pass them as a prompt to the LLM, which will return the final answer.

This was the most basic RAG pipeline, but it comes with what is basically mentioned in the tweet image: RAG is used to answer user queries about private data when the query itself isn’t ambiguous. I mean, it can be phrased in a very different way than the documents. So the problem might be that the model can’t understand our task or question, which means we need to translate the query.

Query translation

Multi-Query Prompt

We start with an original question and rephrase it into several different queries. Next, we look at the relevant pieces of information for each query. Finally, we can use different methods to combine these queries and their results before sending them, along with the original question, to the model.

‫‪RAG-fusion‬‬

The RAG-fusion technique utilizes a method known as Reciprocal Rank Fusion (RRF). Understanding RRF is crucial, especially since we’ll revisit it when discussing Re-ranking later on.

When we employ Multi-Query strategies, we generate multiple queries and collect the top relevant chunks of information for each. After gathering these results, we review them to eliminate any duplicates, ensuring we only work with unique entries.

Now, let’s consider a situation where two queries return chunks with the same score. For instance, if both queries produce a chunk rated at 0.55, we face the challenge of determining their order when we combine them. This is where RRF becomes beneficial.

RRF helps us rank these chunks based on their relevance from each query, giving us a way to decide which chunk should take precedence in our final list. By applying RRF, we can effectively merge the results while prioritizing the most relevant chunks, leading to a more efficient and accurate representation of the information we need.

To explore the formula and see some examples, check these articles:

• RAG Fusion Revolution — A Paradigm Shift in Generative AI
• Reciprocal Rank Fusion (RRF) explained in 4 mins.

Decomposition

Decomposition Query is a method where we break down a complex task in a query into smaller, simpler, and easier-to-solve subtasks or parts. Afterward, we merge these answers, either by solving them recursively or addressing them individually.

Recursively Answer

This method involves breaking down a complex query into a series of simpler, smaller questions. When an LLM (Large Language Model) answers the first question, we take both the question and its answer as the context for the next question. We continue this process with the second, third, and so on until all the smaller questions are answered. (In some cases, we can stop here.) In other cases, we take all these questions and their answers as the context to address the original main query. If we do this final step, it becomes the Answer Individually method.

Routing

Building a large RAG application often won’t just focus on a single fixed task or work with only one uploaded file. For example, if you have a high school student studying for exams, you might create an RAG application that helps them across all subjects, not just one. However, the application needs a router to determine the subject of each query. For example, if the student asks a physics question, the router would direct the RAG to search in the physics textbooks the student has uploaded, ignoring other subjects. Based on the user’s query, the application will choose the most suitable route or chain to get the best result.

Now, let’s talk about the methods we can use for routing:

Using a Custom Function:

Simply create custom chains for each subject. For example, if the subject is mathematics, you can design a chain that tells the model it’s a math teacher and should answer the question based on its knowledge of math. Similarly, you’d have another chain for physics, and so on. A simple function would check the query for specific words (e.g., “math”) and select the corresponding chain for that subject.

Machine Learning Classifier:

If you have a lot of questions from different subjects like physics, math, etc., and you want the system to determine which subject is being asked about, you can train a classifier. This classifier will categorize the query into the correct subject, and once classified, the system can route the query to the relevant textbooks or chain associated with that subject.

Semantic Similarity:

More advanced functions can be used beyond simple keyword matching. For example, you could create an embedding (a numerical representation) for each chain. When a query comes in, its embedding is compared to the embeddings of each chain, and the chain with the closest match is selected.

Hybrid Approach:

As the name suggests, you can use a hybrid approach that combines multiple methods mentioned above to achieve better and more accurate routing.

The importance of routing lies in ensuring that the context provided to the LLM is truly relevant to the question asked. This prevents the model from using unrelated contexts from a different subject. Effective routing also improves the indexing process by ensuring that we are searching in the correct chunks, saving time and leading to better results.

Query‬‬ ‫‪Construction‬‬‫‪

When constructing a query, most of the files in the data we’re using likely have metadata, right? So what is metadata? Metadata is simply data about data.

For example, if we have a video, the data would be the content of the video itself. However the metadata could be various things like the source of the video, its production date, duration, or other details.

Now, if we have a query like:

Which video is longer, A or B?

The first solution might be that the model looks at the number of words in the transcript of the first video, then compares it to the number of words in the second video’s transcript. The one with more words would be longer, right?

Not exactly. One video could have more silence than the other, so even if it has fewer words, it could still be longer. But, even if the video with more words is longer, what’s the faster way to get the answer? Is it for the model to analyze the entire video content, or just to grab the duration from the metadata and compare the lengths?

So, what do you think about using metadata to improve chunking? It’s possible that the query written by the user could be answered directly through the metadata, and from that, we can provide an answer much more easily.

That’s it for this article I may share another one about more advanced concepts in the upcoming days!

Resources:

• Retrieval-Augmented Generation (RAG) from basics to advanced
• LangChain: Observe, Evaluate, and Deploy Reliable AI Agents
• freeCodeCamp.org
• Advanced RAG Techniques: Unlocking the Next Level

RAG From Scratch was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

RAG From Scratch

The Rise of LLMs

LLMs are super-powerful tools. I remembered the first time ChatGPT was released, I was saying hell no, I will lose my job (before even getting one), but… then I saw it struggling to pick the right CUDA version for Ubuntu just like the rest of us :)

Anyway, if you ask someone who doesn’t know much about AI about ChatGPT, they might say it’s just retrieving answers from the internet like a semantic search. Well, almost, but it’s a bit more complex. These models aren’t retrieving the next word from a database you can instantly point to; they’ve compressed vast amounts of information into their weights, and they literally predict every single word (sub-word).

Even though these LLMs are very powerful in chatting and answering questions, such as writing APIs using Flask or preparing unit tests for us, they struggle in specific domains.

What About Fine-Tuning Your Own LLM?

While Large Language Models (LLMs) excel at general knowledge tasks, they often underperform in specialized domains. Their broad capabilities come at the cost of domain-specific expertise, akin to being generalists rather than specialists.

This limitation is where fine-tuning becomes relevant. Fine-tuning adapts an LLM to a specific domain or task, enhancing its performance in targeted areas. However, the process is considerably more complex than simply exposing the model to domain-specific data.

The Challenges of Fine-Tuning

Implementing LLM fine-tuning presents significant challenges. Organizations typically have two options:

1. Fine-tuning an open-source model
2. Utilizing APIs from providers offering fine-tuning services for proprietary models

Regardless of the chosen path, financial considerations are paramount. Costs may include API usage fees or expenses related to cloud GPU resources for training. Even with adequate financial resources, organizations may face a shortage of high-quality, relevant data necessary for effective fine-tuning.

Data sensitivity introduces additional complexities. Organizations must ensure proper attribution and traceability of data sources, citing an LLM in a research paper might raise a few eyebrows. Moreover, protecting proprietary information from potential exposure to LLM providers is a critical concern.

While fine-tuning LLMs is a trending topic in the AI community, it’s not a decision to be made lightly. The process requires careful consideration of resources, data quality, and potential risks much like adopting a high-maintenance technology stack.

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) offers a potential solution to these challenges. RAG connects an LLM directly to an organization’s knowledge base, enabling relevant data retrieval without the need for model retraining. This approach enhances performance in specific domains while maintaining data security and reducing the complexities associated with fine-tuning.

How RAG Systems Work

Retrieval-Augmented Generation (RAG) systems consist of two main components:

1. Knowledge Base Creation

This is the preparation phase where your documents are processed and stored:

• Document Chunking: You take your documents and break them down into smaller, manageable sub-documents or chunks.
• Embedding Computation: For each of these chunks, you compute embeddings. These are numerical representations that capture the semantic meaning of the text.
• Vector Store: The computed embeddings are then stored in a vector store, which allows for efficient similarity searches later on.

2. Generation Part

This is the runtime or inference phase, where the system responds to user queries:

• Query Processing: When a user asks a question, the system computes an embedding for that query.
• Retrieval: The system then retrieves the most relevant chunks from the vector store based on the similarity between the query embedding and the stored chunk embeddings.
• LLM Integration: The retrieved chunks, along with the original query, are fed into a Large Language Model (LLM).
• Response Generation: The LLM generates a response based on the query and the context provided by the retrieved chunks.

Enhancing Retrieval Accuracy

While this technique is great for semantic similarity, it can still have failure points. To improve accuracy, many systems combine this semantic search with traditional keyword-based search mechanisms:

• BM25 Integration: One such technique is BM25 (Best Matching 25), which plays a critical role when the user query and the database involve specific keywords.
• Hybrid Approach: By combining semantic search with keyword-based methods, RAG systems can provide more robust and accurate results, especially in scenarios where exact term matching is important.

This hybrid approach allows RAG systems to leverage the strengths of both semantic understanding and traditional information retrieval techniques, resulting in more reliable and context-aware responses. As reported by Anthropic, this method improves results by ~ 1%.

BM25:

Sometimes, the smart part misses important exact matches. That’s where BM25 comes in. It’s like a super-powered “Ctrl+F” (find function) that looks for exact words or phrases.

Examples Where BM25 Shines

1. Error Codes: Finding “Error XYZ-123” exactly.
2. Product Numbers: Locating “Model AB-9000” precisely.
3. Specific Terms: Finding “mitochondria” in biology texts.

RECAP BEFORE MOVING ON

A standard RAG system works by first breaking a large collection of documents into smaller pieces. These pieces are transformed into numbers (called embeddings) to capture their meaning, and they are stored in a special database for fast searching. When you ask a question, the system looks through these stored pieces and picks the ones that seem most relevant using two methods: one based on word importance (TF-IDF) and another based on the embeddings. It then combines the best pieces and uses a smart model to generate a clear answer based on them.

What’s wrong with that approach

Here is a quick example of where a standard RAG system might fail:

What were the long-term effects of Drug X in the 2023 clinical trial?

A relevant chunk retrieved by the system might contain the text:

Participants showed significant improvements after treatment.

However, this chunk doesn’t specify which drug was used, whether it refers to the 2023 trial, or if the improvements were long-term. Without this additional context, the system cannot accurately answer the question, leading to a potentially misleading or incomplete response.

Introducing Contextual Retrieval

In practice, this is how it’s going to look like. You will take a single PDF file or a single document from your corpus, then convert that into chunks. After that, you take one chunk at a time, and run that chunk through a prompt like the one below, along with the original document.

<document> 
{{WHOLE_DOCUMENT}} 
</document> 
Here is the chunk we want to situate within the whole document 
<chunk> 
{{CHUNK_CONTENT}} 
</chunk> 
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else

This will generate contextual information for each chunk. You combine the given chunk with the relevant contextual information, and then you pass it through an embedding model. Those embeddings are going to be stored in a standard vector database. On the other hand, we also update the BM25 indexing. In this case, we’re using TF-IDF, which stands for Term Frequency Inverse Document Frequency. This is basically the keyword-based search mechanism. As a result, in each of the chunks, you are adding 50 to 100 tokens. I think you can already see some potential issues with this approach. One of them is that it’s going to add a lot of overhead, not only from the number of tokens you’re adding to each chunk, but also because each chunk goes through an LLM, which adds up to a lot of different tokens.

Performance improvements

Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%).

Combining Contextual Embeddings and Contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (5.7% → 2.9%).

Reranking

Retriever models, which are designed to extract chunks of information, are compact and efficient. However, they rely on basic extraction techniques, such as Euclidean distance or cosine similarity between vectors. This approach can lead to suboptimal results, as it prioritizes speed and volume of extractions over accuracy.

In contrast, reranker models are more substantial and slower, making them unsuitable for processing large extractions directly. Instead, they can be employed to rerank a smaller selection of chunks identified by the retriever.

The key distinction lies in the rerankers’ ability to perform cross-attention between the user query and each chunk. This enables them to uncover important relationships between the two texts that simpler similarity measures may overlook.

Learn more about rerankers here:

• Rerank | Boost Enterprise Search and Retrieval | Cohere
• Rerankers

Ressources

• Contextual Retrieval in AI Systems
• Rerankers
• Rerank | Boost Enterprise Search and Retrieval | Cohere

If you enjoyed this article, you might find these pieces valuable:

• RAG From Scratch
• Fine-Tune LLMs with Unsloth
• How Can GPTs Interact with Computers? OmniParser Explained
• GraphRAG: Microsoft’s Open-Source Solution for Enhanced Document Understanding

Anthropic’s New RAG Approach was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction:

Backpropagation is a widely-used algorithm for training neural networks. It uses the gradient descent optimization method to adjust the weights of the network in order to minimize a loss function. The algorithm works by computing the gradient of the loss function with respect to the weights of the network and then updating the weights in the opposite direction of the gradient. This process is repeated until the loss function reaches a minimum.

In this article, I will delve into the concept of backpropagation and its role in training artificial neural networks. I will then introduce you to Geoffrey Hinton’s latest contribution to the field, the Forward-Forward algorithm, and provide an in-depth understanding of its concept and implementation.

What is backpropagation and what makes Geoffrey Hinton think about a new method to train neural networks?

1. I want to clarify that the content of this article is based on my own understanding and the idea of this new concept is not yet fully established.

2. I have to assume that you have at least an idea of how deep learning models work. If not please visit this link.

What is backpropagation and what’s wrong with it?

Backpropagation is a training algorithm used to train deep learning models. The basic idea behind backpropagation is to adjust the weights of the neurons in the network in such a way that the output of the network is closer to the desired output.

The backpropagation algorithm starts by feeding the input data into the network and then propagating the input forward through the layers of the network. As the input data moves through the layers of the network, it is transformed by the weights of the neurons in each layer, producing an output. Once the output is produced, the algorithm compares the output to the desired output and calculates the error between the two. This error is then propagated backward through the network, starting from the output layer and working backward through the hidden layers.

As the error is propagated backward, the weights of the neurons are adjusted in order to minimize the error. This process is repeated for many iterations until the error reaches an acceptable level. The backpropagation algorithm uses a technique called gradient descent to adjust the weights of the neurons in the network. Gradient descent is a mathematical method that helps to find the optimal weights of the neurons that minimize the error between the output and the desired output.

For more details, I highly recommend this great video:

It’s great, right! so what’s wrong with it?

The main disadvantages of backpropagation include its implausibility as a model for how the brain learns, its lack of support from evidence in the visual system, and its inefficiency in handling sequences and real-time learning. It also requires a perfect model of the forward computation in order to compute the correct derivatives. Additionally, backpropagation can suffer from high variance when using reinforcement learning techniques, making it less efficient for large networks with many parameters. These limitations of backpropagation led to the development of alternative methods, such as the Forward-Forward Algorithm, which aims to overcome these limitations and can be used in cases where the precise details of the forward computation are unknown.

If you don’t understand that just ignore the last part and let’s move to the next step!

Forward-forward algorithm:

The concept of the Forward-Forward algorithm is easy to understand, especially if you have knowledge of GANs or reinforcement learning.

The Forward-Forward (FF) algorithm is a method for training neural networks that are inspired by the way the brain processes information. It replaces the traditional backpropagation method by using two forward passes, one with positive (real) data and the other with negative data, which can be generated by the network itself. The layers of the network have their own objective function, which is to have high goodness for positive data and low goodness for negative data. The FF algorithm is particularly useful in situations where the precise details of the forward computation are unknown, or when the network needs to process sequential data in real-time without stopping to propagate error derivatives. However, it is currently less efficient and generalizes less well compared to traditional backpropagation on some tasks.

What do you mean by positive and negative data?

In the context of the paper, positive data refers to real data, while negative data refers to fake or generated data. This negative data can be added manually or generated by the model itself. To provide clarity, I will provide an example of a supervised learning problem.

The traditional method to train a neural network:

The traditional method of training a 2 layer-dense neural network for image classification is by using backpropagation. This involves feeding data through the network, comparing the output to the desired label, calculating a loss function, taking the derivative of the loss with respect to the model’s parameters, and updating the weights to reduce the loss.

Good and Bad data:

In FF, we take a different approach to training a 2 layer-dense neural network for image classification. Instead of using the traditional method of pushing data through the network, comparing the outcome to labels, and then using backpropagation to update the weights, we merge the labels and data into one vector, referred to as “good data”. We then generate “bad data” by combining the input images with incorrect labels. This method is demonstrated using the MNIST dataset of handwritten digits. To make things easy, we replace the first 10 pixels with a one-of-N representation of the label.

Goodness term:

The process of training a forward-forward algorithm is straightforward. To maximize the performance of the model on “good data” while minimizing it on “bad data”, the sum of squared activations in a layer is calculated. This is done within a single layer, so there is no need for backpropagation, just the use of the derivative of the function performed by the layer itself which is relatively simple.

What are neural activities?

In the context of neural networks, “neural activities” refers to the output or response of the neurons in a network. This can include things like the activation values of neurons, the output of a particular layer, or the overall output of the network. The term “neural activities” is used to describe the overall behavior or output of the network, and can be used to evaluate and optimize the performance of the network.

Positive response for positive data and negative for negative data:

And now, we want each network layer to respond positively to the good data, meaning it will have high neural activity, and respond negatively to the bad data, resulting in low neural activity.

Normalization between layers:

When chaining multiple layers, you want to avoid seeing high activations just because you pumped in high inputs from a very excited previous layer. So layer activations are normalized (divided by the norm of the activation vector) between layers.

I hope that by reading this article, you have gained a solid understanding of the forward-forward algorithm and its potential benefits for training neural networks. The Forward-Forward algorithm is a new learning procedure for neural networks that replaces the forward and backward passes of backpropagation with two forward passes, one with positive data and the other with negative data, which can be generated by the network itself. The main idea is to train the model with positive and negative data and to have high goodness for positive data and low goodness for negative data. I believe that this new algorithm will open new doors in the field of deep learning and that it will be worth considering for future projects.

Keras implementation of the Forward forward algorithm for image classification:

Library importation:

import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
import random
from tensorflow.compiler.tf2xla.python import xla

Loading the data:

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

FFDense custom layer:

This code defines a custom dense layer for a forward-forward (FF) neural network. The layer inherits from the Keras Dense layer and applies a rectified linear unit (ReLU) activation function to the output. The layer also includes an implementation of the forward-forward algorithm, which is used to train the network. The forward-forward algorithm uses a custom loss function that compares the mean square value of the activations for positive and negative samples. The network optimizes this loss function by performing gradient calculations and optimization steps on the trainable weights of the dense layer. The loss and optimizer are passed as arguments to the layer. The code also applies normalization to the input before it is passed through the dense layer. This is done by dividing the input by its L2 norm. The number of training epochs and the threshold value for the loss function are also set as hyperparameters. The forward_forward() function takes the positive and negative inputs and performs the forward-forward algorithm to train the network.

class FFDense(keras.layers.Layer):

    def __init__(
        self,
        units,
        optimizer,
        loss_metric,
        num_epochs=50,
        use_bias=True,
        kernel_initializer="glorot_uniform",
        bias_initializer="zeros",
        kernel_regularizer=None,
        bias_regularizer=None,
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.dense = keras.layers.Dense(
            units=units,
            use_bias=use_bias,
            kernel_initializer=kernel_initializer,
            bias_initializer=bias_initializer,
            kernel_regularizer=kernel_regularizer,
            bias_regularizer=bias_regularizer,
        )
        self.relu = keras.layers.ReLU()
        self.optimizer = optimizer
        self.loss_metric = loss_metric
        self.threshold = 1.5
        self.num_epochs = num_epochs


    def call(self, x):
        x_norm = tf.norm(x, ord=2, axis=1, keepdims=True)
        x_norm = x_norm + 1e-4
        x_dir = x / x_norm
        res = self.dense(x_dir)
        return self.relu(res)


    def forward_forward(self, x_pos, x_neg):
        for i in range(self.num_epochs):
            with tf.GradientTape() as tape:
                g_pos = tf.math.reduce_mean(tf.math.pow(self.call(x_pos), 2), 1)
                g_neg = tf.math.reduce_mean(tf.math.pow(self.call(x_neg), 2), 1)

                loss = tf.math.log(
                    1
                    + tf.math.exp(
                        tf.concat([-g_pos + self.threshold, g_neg - self.threshold], 0)
                    )
                )
                mean_loss = tf.cast(tf.math.reduce_mean(loss), tf.float32)
                self.loss_metric.update_state([mean_loss])
            gradients = tape.gradient(mean_loss, self.dense.trainable_weights)
            self.optimizer.apply_gradients(zip(gradients, self.dense.trainable_weights))
        return (
            tf.stop_gradient(self.call(x_pos)),
            tf.stop_gradient(self.call(x_neg)),
            self.loss_metric.result(),
        )

FFNetwork Custom Model:

The next code defines two classes, FFDense and FFNetwork, that are used to create a Forward-Forward (FF) neural network. The FFDense class is a custom layer that inherits from the Keras layer class and overrides the call method to include the Forward-Forward algorithm. This algorithm maximizes the sum of squared neural activities on positive data while minimizing it on negative data, and it does not use backpropagation. The FFNetwork class is a custom model that inherits from the Keras model class and allows the creation of FFDense layers. The predict_one_sample method in the FFNetwork class is used to make predictions on one sample by passing the image through the network, measuring the results produced by each layer, and then finding the label with the highest values by measuring the ‘goodness’ of the images for all labels.

class FFNetwork(keras.Model):

    def __init__(
        self, dims, layer_optimizer=keras.optimizers.Adam(learning_rate=0.03), **kwargs
    ):
        super().__init__(**kwargs)
        self.layer_optimizer = layer_optimizer
        self.loss_var = tf.Variable(0.0, trainable=False, dtype=tf.float32)
        self.loss_count = tf.Variable(0.0, trainable=False, dtype=tf.float32)
        self.layer_list = [keras.Input(shape=(dims[0],))]
        for d in range(len(dims) - 1):
            self.layer_list += [
                FFDense(
                    dims[d + 1],
                    optimizer=self.layer_optimizer,
                    loss_metric=keras.metrics.Mean(),
                )
            ]


    @tf.function(reduce_retracing=True)
    def overlay_y_on_x(self, data):
        X_sample, y_sample = data
        max_sample = tf.reduce_max(X_sample, axis=0, keepdims=True)
        max_sample = tf.cast(max_sample, dtype=tf.float64)
        X_zeros = tf.zeros([10], dtype=tf.float64)
        X_update = xla.dynamic_update_slice(X_zeros, max_sample, [y_sample])
        X_sample = xla.dynamic_update_slice(X_sample, X_update, [0])
        return X_sample, y_sample

    @tf.function(reduce_retracing=True)
    def predict_one_sample(self, x):
        goodness_per_label = []
        x = tf.reshape(x, [tf.shape(x)[0] * tf.shape(x)[1]])
        for label in range(10):
            h, label = self.overlay_y_on_x(data=(x, label))
            h = tf.reshape(h, [-1, tf.shape(h)[0]])
            goodness = []
            for layer_idx in range(1, len(self.layer_list)):
                layer = self.layer_list[layer_idx]
                h = layer(h)
                goodness += [tf.math.reduce_mean(tf.math.pow(h, 2), 1)]
            goodness_per_label += [
                tf.expand_dims(tf.reduce_sum(goodness, keepdims=True), 1)
            ]
        goodness_per_label = tf.concat(goodness_per_label, 1)
        return tf.cast(tf.argmax(goodness_per_label, 1), tf.float64)

    def predict(self, data):
        x = data
        preds = list()
        preds = tf.map_fn(fn=self.predict_one_sample, elems=x)
        return np.asarray(preds, dtype=int)



    @tf.function(jit_compile=True)
    def train_step(self, data):
        x, y = data

        x = tf.reshape(x, [-1, tf.shape(x)[1] * tf.shape(x)[2]])

        x_pos, y = tf.map_fn(fn=self.overlay_y_on_x, elems=(x, y))

        random_y = tf.random.shuffle(y)
        x_neg, y = tf.map_fn(fn=self.overlay_y_on_x, elems=(x, random_y))

        h_pos, h_neg = x_pos, x_neg

        for idx, layer in enumerate(self.layers):
            if isinstance(layer, FFDense):
                print(f"Training layer {idx+1} now : ")
                h_pos, h_neg, loss = layer.forward_forward(h_pos, h_neg)
                self.loss_var.assign_add(loss)
                self.loss_count.assign_add(1.0)
            else:
                print(f"Passing layer {idx+1} now : ")
                x = layer(x)
        mean_res = tf.math.divide(self.loss_var, self.loss_count)
        return {"FinalLoss": mean_res}

Data pre-processing:

x_train = x_train.astype(float) / 255
x_test = x_test.astype(float) / 255
y_train = y_train.astype(int)
y_test = y_test.astype(int)

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))

train_dataset = train_dataset.batch(60000)
test_dataset = test_dataset.batch(10000)

Training the model:

model = FFNetwork(dims=[784, 500, 500])

model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=0.03),
    loss="mse",
    jit_compile=True,
    metrics=[keras.metrics.Mean()],
)

epochs = 250
history = model.fit(train_dataset, epochs=epochs)

Testing the model performance:

preds = model.predict(tf.convert_to_tensor(x_test))

preds = preds.reshape((preds.shape[0], preds.shape[1]))

results = accuracy_score(preds, y_test)

print(f"Test Accuracy score : {results*100}%")

Test Accuracy score : 97.64%

Summary:

This example has demonstrated the implementation of the Forward-Forward algorithm using TensorFlow and Keras. The results reported in the paper by Prof. Hinton are currently limited to smaller models and datasets like MNIST and Fashion-MNIST. However, it is expected that future research will explore the use of this algorithm on larger models such as LLMs.

As demonstrated in the paper, a fully-connected network with 4 hidden layers, 2000 units, and trained over 60 epochs using the Forward-Forward algorithm, can achieve a test accuracy error rate of 1.36%. However, by increasing the learning rate and training for 40 epochs, the error rate slightly increases to 1.46%.

Conclusion:

In this article, I have attempted to summarize and understand the Forward-Forward algorithm using the TensorFlow and Keras packages, as of January 25, 2022. While the results presented by Prof. Hinton in their paper are currently limited to smaller models and datasets like MNIST and Fashion-MNIST, I am confident that this new model will be successful in the future with proper tuning of the learning rate, model architecture, and other parameters. I hope that you have found this information interesting and informative.

Resources:

• The link to the official paper: https://www.cs.toronto.edu/~hinton/FFA13.pdf
• Very simple and informative tweets from Martin Görner
• Geoff Hinton interview: https://www.youtube.com/watch?v=NWqy_b1OvwQ&t=2009s
• Keras official implementation: https://www.youtube.com/watch?v=NWqy_b1OvwQ&t=2009s
• Pytorch implementation: https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/forward_forward

Forward-Forward Algorithm was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.