Better Chatbot Query Results with RAGFusion?
I’ve been looking to improve the responses coming back from my chatbot application, which takes a podcast and generates a list of quiz questions to test your memory retention.
The problem is, the LLM loves to generate questions from the start and end of the podcast, which is where all the promotional content tends to live — sponsors, host information, recaps of previous episodes, promotions for other podcasts, etc. No matter how much prompt engineering I threw at it, it wouldn’t grab the good stuff at the middle of the podcast. This is a known issue with large language models.
I’d given up on this feature of my app for a while, until I came across an article on something called RAGFusion — a different approach to generating LLM queries using an approach called Reciprocal Rank Fusion, referenced in a decade-old paper detailing the algorithm .
The author provides some sample code for implementation in Python, and I’ve adapted this for langchain and put into a colab notebook here that you can run to actually see the results in action (based on langchain’s own cookbook sample code ).
The code doesn’t actually conduct a query though, it just retrieves documents, and I wanted to see how well it works.
My code was built in nodejs and langchain, not python, so I had to rework things a bit to be able to actually perform comparison queries and test the results on real documents.
You can watch a video walkthrough of this implementation here on Youtube, (skip to the comparison here):
and I also made a gist of the frontend / backend code you’d need to implement this for yourself.
I’ll be continuing to test this approach and post results, but so far it does seem to be an improvement over standard out-of-the-box RAG. If you have similar luck with RAG Fusion, I’d love to hear about it!
