Markdown source

Transcript: Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Source Video

Local Cache

Transcript

Hello everyone. I'm Abid Matini. I'm senior backend developer at OGOV. Thanks to AI Engineer World Fair 2026 for giving me a time slot for online track. And I'm going to walk you through bypassing the multimodal text today. How we can have a have a framework free hybrid rag. Withdraw SQL RF and live telemetry. As you can see, I have got my live demo in the right side of the screen. So, [snorts] any slide that I pass and if there is anything related related to my presentation, I can show you. I have few tabs also ready here for you to show and walk you through uh the slides and the demos. Okay, let's begin.

So, we have two problems for every document we have two problems that we try to solve today. One is when we start to chat with any LLM and if we have to upload a document, we usually going to just drag and drop a PDF or a Word doc or an image and ask the chatbot to get ready for the questions that going to come up after we upload those documents. The issue with that is as soon as we upload these documents, we going to basically lose some of our tokens that we supposed to have only for uh processing these documents without even asking a question.

So, we already uh spend some tokens there without even asking any questions. That would be the first issue. And then, if we're building a chatbot and it's uh we want to have vector database there, we want to have keyword search and semantic search there, and we want to just add more tools. This is going to be too many tools uh uh combined all together. So, in the production uh in the production basically chatbot, we're going to have uh too many tools, and it's going to be complicated for us to manage uh our production chatbot properly. So, today I'm going to uh walk you through in uh three main uh sections.

One is how we can uh upload our documents and get our documents and make it ready bef- uh for our chatbot before uh users going to chat. So, on the on the right side, I have I do have a FAQ assistant chatbot that's running locally now for me. And um I'm going to show you the dashboard, the dashboard that I've created for it, and how we can chat, and uh how we get the results. So, this is going to be a uh supposedly FAQ um assistant for a sample employee handbook.

So, imagine that you have HR company, and this HR has uploaded the the the the handbook for uh for uh for the employees to ask their question, any question they have about leave, about uh uh like uh different kinds of leaves, and sickness, or uh parent like uh parental leave, or any other questions related to our, uh uh, possibly an HR would have as a common question that they get every day. So, um, that would be one sample that we're going to use to upload. So, all documents can get can get in the different formats or we can upload in different formats of PDF, uh, like PowerPoint documents and Word documents or even image.

So, we're going to talk about that. We're going to talk about how we have to chunk these because we're talking about drag here. We need to know how we can chunk this information uh, properly based on our the documents. So, we're going to talk about uh, different type of chunking here as well. We can, uh, we show you in the admin how we can choose different uh, different strategies to uh, to chunk the documents. Then we're going to see how the search is going to work, how the embedding is going to work, how did our database is going to be, how the search going to be based on the keyword search or the semantic search.

And we're going to see about the how our F in the Python going to work. And then we're going to talk about the observability, how we going to check the safety, how we going to observe, uh, all the process with LangFuse and how, uh, we can the prevent, uh, prompt injection pattern or, um, any risky question that might come to LLM.

I'm planning this talk for about under an hour and this, uh, the whole thing, the plan is the entire code base can be easily will be running on the GitHub code code space which you can just pull from the repo and one command run and it's going to be ready to test on GitHub code spaces. Um so, the stack we're using here are Python, FastAPI for the back end, React for the front end, uh PostgreSQL for our database.

Uh Docker for just easily being able to have the containers and we can just re- re- produce it anywhere and rerun it anywhere, for example, in code spaces or code spaces or any other uh environment we want. Then, we have Ollama and local language models and local language models and embedding models that running only locally that can be run on any uh server as well. It doesn't need a GPU. The CPU is going to be good enough for it. So, uh any staging server, any code space uh uh going going to be fine for it.

And LangFuse for uh tracking what's going on and how we can see the chat and the latency uh so, we can improve our chats better. Um This is the end-to-end blueprint of uh what we're going to talk about. So, we're going to have a document few documents as a source of truth for our chatbot. So, we're going to upload some documents. So, what we can do with Python is we're using Doc Link to convert those uh raw documents uh to markdown files. And then, we're going to chunk those markdown files to uh in our into our uh PostgreSQL. And then, we can also do some um scan as well here.

So, uh that would happen as well. Then, we would have some queries from the user. So, the employee is asking questions, HR-related questions, and it's going to be retrieved from the database. Is the the the result relevant results with the top rank would be passed through the LLM. And LLM would uh return the in the in our chatbots that we have here. Uh in the format of chat, so we can see and get our questions answered here, and we can observe in the length views what's going on in our system. So, um optimizing data ingest. So, we have two different ways of optimizing.

So, if we first uh going to just throw it into the We have a document, and if we just uploading it on our um uh cloud chatbot anywhere, the issue we just uh having of the one issue we have is we're spending tokens before we doing anything. Another issue is you don't know how that document looks li- looks like there and that chunking uh how it's being chunked and how uh basically that LLM is seeing it. So, that would be a risk quality. If if there is a table that we don't know how it's being read there, so we kind of don't see.

We're trying to uh have more clarity about how the how the document's being stored in our database and how it's being retrieved. So, another way would be a structure first, so we use the local CPU. If you're running it here, we we can uh upload a document or doc link would convert it to uh markdown file, and we would know the better chunk uh how the chunking is going to work. I'm going to show you a few examples. And uh of course, it's going to be a cheaper prompt.

If after that you want to talk to a lo- um an online cloud any- but if you want to If you want to have it locally, then it would be actually no cost for you. Uh, so the DocLink pipeline would be uh, we would be running it It's like a local locally running it on our server. The The The Python is going to get that document. It's going to export it to the markdown and save that markdown file. Then the get this going to chunk it based on different strategy and uh, we're using embedding models to embed it to our database and insert that on to our vector database.

Uh, so if we save while local because uh, imagine if if one or two pages the cost is next to nothing, but if you have 200 pages uh, then the cost going to be too much. And if you have a big company with lots of employees and they want to just us keep asking the same questions or uh, if they want to upload documents then it's going to be uh, going to be a big cost. So, um, there are four ways of chunking a document. Let me first before I show you the four ways, let me uh, show you the admin page that what I'm talking about.

So, imagine here you have We are We're having our FAQ admin page. So, uh, imagine you have Let me delete few of the documents that I have chunked before here. So, uh, you want to upload a new FAQ uh, file for your HR as for for your HR policy. I want to show uh, give your chatbot uh, data updated with our AI system. So, imagine you have um you have some question answers here. Uh before I do that, let me show you here what I mean. So, [snorts] um this is the heading FAQ, yeah.

So, um one way that you can feed our chatbot with information is if we actually create a question answer uh file. So, that would be easier to digest for our uh system to understand what's going on. So, part of that is uh imagine we're having a this the entire FAQ file. So, the entire uh basically handbook. If we want to chunk this uh handbook and just upload it, imagine uh we're uploading the entire PDF. This is going to be the way our PDF is going to look like after chunking. So, it started by acknowledgement, then they see a few chunk that doesn't have any meaning. And then something starts here.

There are some chunks that doesn't have any meaning. Uh for example, here signature date, that's basically kind of useless. So, this going to slow down and reduce the accuracy of our chat. It's going to increase hallucination. So, um instead of uploading uh straight uh 28 pages, we can if we can divide our documents to different chunks and just to different files, relevant files, and clean up that data a bit as much as we can, uh and then upload it, then we will have a better result. So, uh one way of uh uploading in the FAQ case is to actually have your your turning your data to question and answers.

So, uh So, talking about working hours, talking about different situations here, so you can have it based on questions and answers. Then this way you know when you reference your documents, you're having the uh correct reference, and it won't be any confusing. And there are uh different way to uh upload your documents. One way is you have the question and answer, and then you would like to upload it. This is my FAQ PDF file. Uh same PDF file, and I'd like to upload it based on uh heading-based chunking strategy. That means well, while I'm explaining the the parsing where Docling is happening in the background.

So, that means each uh chunking would be based on the heading that Docling would find. And the answer that comes based on that. So, each heading and answer is going to heading as a question and answer is going to become a chunk. So, we know our chunking is clean, and it's easy to uh reference. So, when this chunking finishes in a seconds, we would see, okay, this is the first FAQ, it could chunk successfully to uh based on the headings. So, it recognized the heading question, uh and then the relevant answer. So, each chunk has its own uh question and answer here.

So, if we say uh if we say, for example, uh >> Um as our questions here Uh do we do we have or something like this. So, we should expect a kind of answer with the reference that's coming to that chunk. So, we talking about heading chance going to understand the documents based on uh headings that you have. That's how it's going to chunk each document heading by heading. So, as you can see here, um our question was about how much This was the question we prepared, and this was the answer we prepared. Uh sample Coca-Cola 90% of the individual employee premium and 75% dependent premium.

What we see here as a reference, we getting that answer, and not only that, you getting the correct answer, you also getting the Uh you see that it's been this answer has been generated based on this document, based on this chunk. So, it's easy to reference. It's easy, you know, where the issue is if there is anything that's not working, you know why. Then you can go track it why this right chunk hasn't been retrieved. So, in this way you have more uh more uh clear path towards how you can debug it.

Because I was uh in our system, I was asking for um return top two chunks, it would uh suggesting me I'll talk about all these models here. Um it would bring top two response. So, then it was also checking that sample employee handbook that big file here, sample employee handbook. As you can see here, that's a big chunk. And if actually it's a good example because if you compare these two, uh because we just drag uh uploaded the entire chunk here, you can see although it's returned some information, but it's not that accurate and you don't know.

Of course, it went through that file and found something, but uh you don't know how accurate that information could be and you can't follow because the chunk is is just uh uh untrackable that easily. So, one way of chunking is chunking based on the headings. Second way of chunking is based on the paragraph. So, technically when you upload your documents, you would say uh just chunk each paragraph you catch, have it as a separate chunk. Doesn't matter there is a heading or not heading or anything. So, uh if I want to go back to the to our to our documents and say, let me first choose I want to do a paragraph chunking.

I want to say, okay, uh for paragraph um Let me find the paragraph general policy for paragraph. I want to upload that. Um So, for general policy, it's uploading here as you would see the whole file is just been paragraph by paragraph being chunked. So, there is no heading, there is no no separate information. And if you look at the uh if you look at that information as well, if I can find it here quickly, um general general paragraph uh policy paragraph as you can see the information is just paragraph by paragraph. So, you kind of cleaned up your data again, but you're dividing it based on the different system.

So, instead of heading you're talking about paragraphs. That's another way of uh basically feeding your rag for your information. Another way would be a fixed 512 of character. You can say, "Okay, one of the best practices for rag is you just divide based on 512 character, and you have 64% overlap between the chunks, so you won't lose the track." So, if your data is just too is not organized, if there's just random data that you have and you can't clean that up, that's another way you can upload your document. So, that would be instead of this, you can choose another file, and you can say basically there's another paragraph for global travel uh file here.

So, that's kind of when it's kind of overlap here when it's going to see here that the about this chunking strategy, you would see okay this started for chunking and then it ended in the 512, and then it's again uh using that uh I think it was 64 uh I was 64 overlap, yeah. So, it's using the 64 overlap here again. So, you would see it's basically uh chopped it from here. And so, from here to here is 512, and from here to here is 32 then another 32 from here to here. So, this is a 32 32 64 in total. So, that's how it's relating each chunks together. It's happening same for here.

As you can see it's happening same for here. It's the way that But you see the issue with that is it's coming here to assist the next system. So, it's breaking. It's not the best but in some cases it works for you. Another way that we can do is sentence base. So, it can be based on first counting based on the number of sentences and it chop it there. There are different use case for each ones. So, another interesting one is imagine that you There's some maintenance coming.

You're receiving some email and you just want to quickly upload the document in your chat without without cuz you do you don't have the it's just a screenshot and you can't do anything. Imagine you just receiving an email and says, "Okay, there is a maintenance plan for this day from this hour and you just wants to upload this in your chat for the for the user. So, if me as a employee ask, "Okay, what is is there any maintenance plan for this weekend?" Then I would know immediately. So, another way that we can easily upload that screenshot as well. Let me find easily upload that. Oh, sorry. That wasn't our screenshots.

So, the screenshots uh you can upload then we're using another um uh model that just turns that uh image to text and text to .md and then we're using this uh uh basically uh what's the name? Uh the sentence model for the emails that can convert uh that image to a text and then it can feed. So, we can see we're using the chunking strategy a sentence group is turned that image to text and it's already uh good for the uh for our chatbot. So, we can uh we can index it into our knowledge base now. So, then it's going to be a next one that's been uploaded here screenshot.

We can see how it's been chunked because this is just the email. It doesn't have heading. It doesn't necessarily have all the paragraph chunk properly. So, and if it's a shortage of time it's a small message temporary change. You don't need to sit and clean up a data for here. You just want to drag and drop updated for a weekend and everybody knows okay uh this weekend we're having a maintenance plan. So, uh when they will come here and ask is there any maintenance plan for this weekend? Then technically we should get the result of okay, this is the maintenance plan that's been happening and the reference documents.

So, uh the maintenance uh this is the exact maintenance uh window that we got and actually the reference you can see that's the screenshot that uh we uploaded just now. So, if we do if we had I done the screen the file name better, there would have been even a more relevant and somebody says, "Okay, according to this information that I got from here, we have a maintenance plan tomorrow from 6:00 p.m. onwards till midnight." So, and I can easily archive and delete any of these and then it won't be visible as well. So, these are basically different ways that you can chunk.

That's one of the that's basically the one of the most important part of the whole system because most of the LLMs would just work. The but the way you want to feed that data and you want to clean that data and not not having just drag and drop the entire handbook here and have a messy data, that would help you a lot to have a more successful in this case FAQ chatbot. Uh Yeah, so these different strategies would help you as well. And um Yeah, let's move on to the next slide. Um So, we're using we're using Postgres and um we are also having a few agents that they are just Python codes.

They are not really any We're not calling another LLM as an agent to do something for us. The real one reason for that is the speed cuz I'm running this locally. I'm having running this on Ollama. I don't have I don't need to three four agents to loop three times four times because then I'll have to wait I have to wait 20 30 seconds for it and user would lose the interest. You just want one natural language to run. The rest if something can be done by a Python function, let's a Python function then run it for you. Then you have also the full control over the function.

Won't be any hallucination because you need to you're going to write your test suite so to just cover most of the situations. So that would help you. The agents uh I just want to show you something here in the settings what can it be. So I can I can uh off and have it agents off and on as well. So I can say if somebody wants the current date then this function is it can be called and retrieve. You don't need the LLM to bring you back the current date.

You don't need the LLM to calculate for you something cuz also another plan for expanding this expanding this chatbot is having it for customer service for that's going to talk about the products and the prices and calculations. So that's another thing product info or even having another deeper search on the agent mode. But the point with that is then cuz it's going to be another search then you would lose that referencing on that. That would be important. So while I'm here, I'll show you what do I have as a LLM actually I'm using here. I'm using the smallest Qwen 2.5.

So it's 0.5 billion is parameters instruct model and it's I think it's 400 megabytes only. So it's that small and I'm using this model also for for the agent mode if I want to run so I would run and two more I would run another one and I have to run the agent mode is going to make it slightly slower. Um and then Um yeah, let's go back to the slides here. Um So Yeah, this uh way of working you have a full visibility on your poison function. That's what's going on. So, you only pass the in needed needed information clear information to LLM and receive a clear information.

So, it would reduce the chance of hallucination dramatically. Um also showing you here about uh vectors and things on the in the database. So, um So, when we're going to chunk, we're going to chunk the information. This is based on the ID and the section um and the what kind of vector embedding are we doing? And then um we have our chunk, we have we can see our basically how it's going to find the nearest neighbor here and um Also, the keyword match at the same time because if if you uh having a customer service product chatbot, you don't have no need the nearest information. You need the exact information.

If it comes to the number as well, if it's come to the product as well, you need exact uh if it's a medical chatbot, you need the exact medication. You need don't need something similar or close to it. So, we need to have both both basically uh keyword search and um and the semantic search together. So, we'll have a hybrid search that giving you a better result. And we're talking about the number number that we're going to have. I have it in the system limited to two because in this case you don't need more than that.

But, if you want to retrieve some information, if it's a product, you might have you don't want to just uh bring top two products that's being there for the user because you might have the same brand like a 20 product or 50 product of the same brand. If you don't let the window your window to pull those information, those products are never going to be sold and never going to be shown in the chatbot. So, it's important how many you can fetch. Uh and then there the more about the retrieval that when we're using the BM25 to get the exact keywords.

So, the previous slide was about to get the most closest with the cosine distance that we have uh because when we upload the information with the in our vector in our in our vector database, the information with the similar meanings are going to sit next to each other. And then when we user search for something and ask for something, then uh our information and our our search is going to convert that uh question that query uh in that database and go find a similar closest information to that question and will return it. So, that's the reason we can't just one.

We need to return more than those so we have multiple answers and more closer answers. But, here in the smart retrieval, we need the exact answer. So, we would need to we would need to filter it based on the language, for example, we need to based on the ID or exact product name, SKUs, or uh brand name or something. So, that would be the the keyword uh retrieval uh for us.

And then how we do the ranking based on which is very important because when we have the both vector and the BM25, we want to mix these and still bring the top uh top two or top five relevant answers that we need uh for us. So, more accurate if we if we bring too many if we bring top 20 here and uh bring shows top 20 answers is going to be quite confusing for the user. So, you just want to have make sure you bringing the right amount based on your use case based based on your use case. So, if you're having products, you would bring more. You retrieve more.

If it's something medical, you retrieve less. Uh you don't want to information to be all over the place. You want to accurate more accurate information because you would have more liability obviously on that. Um another point here is um So, this is uh this is more about reranker. So, when we have our reranker here, uh that would also um bring the information bring that uh And so, after that score uh how many of those information that shows by default. So, I'm only showing two there. Um so, I've I've briefly talked about the agent mode vs. direct rag. The direct rag is is just uh following the it's a fixed pipeline.

It's not going to go calling anything. You know, the the path is quite clear. You have embed embedded query, you have a hybrid retriever. And you just bring up the answer. It's good if compliance is quite important for you. But agent mode, you have a few extra tools that going to be called. So, a search is a compare product. These can bring more information back. It's less in your control. So, uh Oh, that's quite important and it takes longer because you have another agent that going to work. So, basically it's getting this information and then you have agent mode based on your request is going to call.

Um So, that would that would take slightly longer slightly longer. Um About that about the telemetry and application guidelines. So, um can use LangChain Fuse for we integrated LangChain Fuse uh in our functions. So, we can uh see what kind of answers uh we have. It's quite easy to everything is local. It's no outside calling or anything. So, it's easy to track the information that was chatting with the widget anonymous. I wasn't logged in. It's a FAQ FAQ chat. That's FAQ agent is name of the the whole projects that I'm running there. So, each conversation has a ID.

Uh you can track if there was any maintenance for this weekend and you can track the answer for it. You can see what model I've been using. You can see that I've returned the top two answers and the cuz it was top two is referenced twice on that image. Um page at most was off. Uh it was a chance. Two chunk of information was returned and it took this long milliseconds and uh yeah. That's basically the information we have. If you have external models, you can even estimate the cost that you're spending. That's another thing you can uh you can use that for.

And uh if you have different sessions, different users, you can track the logins. So, um that's give you a good insight of how users are treating you uh with your chat. So, um Yeah. So, as I said about the user sessions and IDs, um and then you would have the guardrails um as well. So, something because of you using functions for it, we can uh have a better control over it. So, if I ask is chatbot, how do I uh treat flu?

I have created a section uh to say without going to chatbot, the point of lots of these guardrails is that the issue should stop before we um the prompt injection or anything that we have uh or a question that is we're not supposed to respond. We shouldn't even send it to the LLM to generate anything. So, before even in the code, I can have So, we have created a section here uh to say if there is a medical escalation, bring this medical message back.

Uh and you can see how the system prompt is quite small because I'm basically prompting everything in the code, so uh, although the system prompt looks quite small and just few sentences, uh, I'm actually prompting way more in the uh, code. Instead of prompt, we use I'm using the code to say what do's and don'ts. And uh, another example is the medical escalation and you can add more based on your uh, concerns you would have for your product. So, um, yeah. So, this is the example for that. Um, can be uh, can needs to be expanded with different keywords, with different kind of sentences.

Should be cured and see how many times somebody could pass this or not. But technically, that would be example of how we should block uh, I have our guard there should block any query that's not related. Um, can do the same with the injection and so on. Um, what else do we have here? We talked about the length fuse and uh, we just need to create a generate a key and you have it in your uh, system. I would I would work is just uh, a little bit about the observability that I've talked about. Um, and then um, then the injection as well.

So, the tools for prompt injection again, it shouldn't go to the LLM and be stopped there. It should be stopped before before uh, it's being sent there. And yeah, we're using intent rejects and term dictionaries and LLM classifier to stop any of these issues. And of course, we can expand it more. But the good thing about that, because it's just a code, it doesn't it It need the long prompt. It doesn't need anything uh Uh outcome would be obvious. You know what you uh blocking and what you're not blocking.

So, it's not like uh uh LM that sometimes likes to listen to you as you're uh as the instruction and sometimes just escape and does its own thing. So, part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, "Okay, this is what I'm blocking and this is why." And it passes all the tests. So, you can be more assured uh about that. Um So, yeah. It's just a small slide about uh We can just create your the and I'm having it as a um I'm having it here as a as a ba- uh as a widget.

It could be also done here um instead of um in a bigger chat as well. So, another thing is you have to in your chat as a part of the rules, you should say also if somebody doesn't understand the terms that they're using. So, if not, the person wouldn't allow me to start the chat and not and uh Also, I gave the the widget capability if I want. So, user, you can reset the consent as well. It's just for my demo purposes. So, and then the the widget would show here. Uh if I say no, this wouldn't work. If I reset the consent, uh if I say yes, then it will start. Uh So, yeah.

So, that's also about the widgets. And is cuz I'm using React, it can be used anything else um um any other uh framework as well. I just use it for simplicity. Um here saying about Okay, the models I used. So, I've used two models. Uh I pulled the quant 2.5. Um Actually, first I've had the 7 billion, but it's um took longer. So, for a smaller server, I've changed it to uh Let me show you the admin here. I've changed it to uh uh 0.5. Otherwise, I have different options. So, I was trying with different models. If you have a bigger model, it's just going to take longer. And it's might be too wordy.

Interesting finding from my side was you don't need the biggest model. You if you can vet your data before sending it to your LLM, a smaller smallest model can sort you out and get a good uh uh answer for you. And uh it would give you less hallucination. It would uh it's safer because it's not going to make up information. If it doesn't have information, it's simply it's going to say no, I don't have that information. So, that's another way of guard railing um to not uh to not have that issue of your LLM randomly generating something. Um I have BGM embedding models.

So, when I upload anything here that uh upload anything here that embedding models basically that embedding models do the embedding for me. Uh so, technically you need two models only. Smallest chat model and uh one one of the smallest uh uh one of the smallest model for uh embedding. And then, if you have it on Docker, one click, everything's It's to be running. You need your uh Python requirements to be installed and then you have your React working on the front end. And then that's all you need. Uh one more thing that I should be adding here. I haven't the Langfuse also uh should be added here.

That's going to be running in a different port. That's missing from here. So, the key takeaway for um this is the structure markdown first. So, we use Docling for uh to do the heavy job. And then we use the Postgres vector database to just store everything. Uh didn't use any specific framework. Uh we just used uh Llama for our chats. And no other uh paid framework anywhere else. We used Lang Langfuse for our telemetry. That was also free to use. And this can be uh escalated to different products as well. It can be document catalog or from any database. Um Thank you very much uh for your time.

I would like to get connected with you and uh if you have any questions, if you have uh any kind of concern or something that you figure that would be uh better to integrate, please share your idea with me either in the comments or message me on LinkedIn. Um Yeah. Thank you so much. Uh thank you AI Engineer for the opportunity. And uh I'll see you next time. Cheers.