MCP is not all you need to achieve Agentic RAG

The AI Innovation Paradox

Last week, a Fortune 500 CTO asked me: “Should we abandon our search platform investment and focus on building MCP servers?” This question perfectly captures the confusion surrounding these technologies.

The AI Innovation Paradox

And I don’t blame him. Every week brings new models, breakthrough techniques, and revolutionary architectures. While this rapid pace of innovation is exciting, it creates what I call the “curse of optionality” – an overwhelming array of choices that can paralyze decision-making, especially for organizations outside the open-source AI community.

This lack of perspective, combined with the scarcity of skilled practitioners, makes it particularly challenging for enterprises to launch ambitious AI programs. Even seasoned professionals in the field find themselves confused. Based on countless conversations with colleagues, prospects, and industry friends, I’ve noticed significant confusion around Model Context Protocol (MCP) and Agentic RAG – two concepts that are often misunderstood in relation to each other.

Agentic RAG has emerged as the most popular AI agent use case, delivering proven, tangible value across organizations. Meanwhile, MCP is the new promising technology that many believe will solve most retrieval challenges in Agentic RAG implementations. Spoiler: It won’t!

Drawing from over 10 years of experience implementing AI-powered search applications across various departments and personas in Fortune 500 companies, I want to address the challenges of accessing large, diverse data sources while maintaining security, scalability, safety, and controllability in complex enterprise environments.

Understanding Agentic RAG

With rapid innovation comes new terminology that can muddy the waters. Let’s start with traditional RAG (Retrieval-Augmented Generation) – a powerful but limited approach where we retrieve relevant documents and use them to generate answers to user questions.

Agentic RAG takes this concept several steps further by introducing planning, reasoning, and tool usage capabilities. Instead of a simple retrieve-and-generate workflow, an agentic system can:

Plan strategically: Understanding whether the goal is to find a direct answer, locate a specific set of documents, summarize findings, or compare information across sources.

Use diverse tools: rewriting and expanding queries for better results, search for specific objects (documents, persons, etc.), reading documents in detail, and extracting specific passages from document collections.

Provide interactive experiences: Moving beyond linear question-answer patterns to allow users to guide and refine the agent’s approach throughout the process.

However, this enhanced capability comes with classic enterprise challenges: accessing vast arrays of data sources, handling different data formats and structures, unifying information for accurate filtering, and most critically – maintaining proper security and access

The Promise of MCP

Model Context Protocol represents a significant shift in how we approach system integrations. This powerful new concept fundamentally changes the responsibility model – instead of AI application development teams bearing the integration burden, MCP shifts this responsibility to the source system owners.

In a nutshell, MCP is a Client-Server protocol that provides an integration framework between a server exposing tools, resources and prompts templates (the MCP Server) and an AI-based application, it can be ChatGPT, Claude.ai or an internal AI agent, through the MCP Client. The beauty of it is that, whenever the MCP Server changes (like adding new tools, and new prompts to do more things), the Client can use it instantly. For more details, I really liked this simple introduction to MCP

The excitement around MCP is so substantial that there’s now genuine incentive for vendors to develop native MCP servers, often with customization options. This represents a paradigm shift that could dramatically simplify how AI applications connect to enterprise data sources.

It is so trendy that MCP Marketplaces or MCP search engines have emerged to help people find the MCP server they need, MCP.so being one of them.

MCP + Agentic RAG = Game Over? Not really…

At first glance, it seems like the combination of MCP and Agentic RAG might solve most enterprise AI challenges. With vendors developing native MCP servers and the promise of simplified integrations, many assume we’re approaching a complete solution.

Bad news, MCP isn’t magical, and it doesn’t resolve all Agentic RAG challenges, if at all.

I see a clear parallel here with the evolution from centralized search (indexing all content into one search engine) versus federated search (using different search APIs to unify the search experience). The reason is that these are almost the same things. Below is a non-exhaustive list of the challenges you will likely face if you are trying to build an Agentic RAG system using MCP only:

Performance Bottlenecks

When you need to interrogate 10 different systems simultaneously, your overall performance becomes limited by the slowest system. Real-time user experiences suffer when dependent on the least optimized data source. What if one of these system is down?

Security Complexity

Different systems often use varying authentication flows, or worse, completely different authentication methods. This impacts user experience and significantly complicates implementation. OAuth implementation isn’t even finalized in the current MCP standard – it’s still under active discussion. There are ways around it, but the user experience remains one of the biggest roadblock at the moment.

Data Model Inconsistencies

Applying filters or generating full-text queries becomes problematic when systems are designed with different data models and semantics. For instance, one might use or require CUSTOMER_NAME, while another only has CUSTOMER_ID. Complex data models with different identifiers for the same object, resulting from disjointed systems within organizations, create significant challenges. At scale, it becomes very hard to design an AI system that will consistently match all the requirements, resulting in partial retrieval.

Retrieval Feature Gaps

This challenge can be particularly frustrating. Some systems offer rich retrieval mechanisms with advanced filters, comprehensive metadata, matching passages retrieval, relevance scoring, and visual elements like thumbnails. Others provide only basic titles with minimal metadata and short, uncontrollable snippets. These inconsistencies severely impact RAG system capabilities as context definition and comprehensiveness is one of the key factor to achieve good results with RAG.

The Complete Solution Architecture

Don’t get me wrong, MCP is an awesome tool and it MCP will absolutely enhance Agentic RAG systems, making them more capable and powerful. However, we should view it as an additional tool rather than a replacement for a robust search platform.

Here’s my vision for a comprehensive and successful Agentic RAG implementation:

A. Centralized Search Platform: Handling complex content search with unified, scalable, and secure retrieval. This includes early-binding security with near real-time document and record permissions updates, consistent and unified data models (using NLP and AI at Indexing time), and auditable, controllable retrieval processes. Some great platforms like Sinequa should be a good start.

B. MCP Clients: Addressing needs for real-time data, analytics, transactional information, and system-specific operations that benefit from direct integration. Think of them as additional tools in the Agentic RAG diagram above.

A Real-World Example

Consider this complex enterprise query from a field engineer:

Do we have an approved replacement part for X in Product Y that we can ship by tomorrow morning?

This question requires both approaches:

• The centralized search platform handles the complex product catalog or PLM search, applying proper security filters and leveraging rich metadata about part specifications and approvals. Sometimes, this information will be somewhere in an Excel spreadsheet or a PDF document. No way to find it without a solid search platform.
• MCP clients access real-time inventory systems, shipping logistics, and supply chain data to determine availability and delivery timelines.

Neither approach alone could effectively handle this multi-faceted enterprise query.

Conclusion

While MCP represents an exciting advancement in how AI systems connect to enterprise data, it’s not a silver bullet for Agentic RAG challenges. The most effective enterprise implementations will thoughtfully combine centralized search platforms with strategic MCP integrations, each handling what they do best.

Success in enterprise Agentic RAG requires understanding both the capabilities and limitations of each approach, then architecting solutions that leverage their complementary strengths rather than expecting any single technology to solve all challenges.