The new upgrades that make AI useful
From chatbots to Skynet (hopefully without the apocalypse)
By now, we’ve all grown tired of yet another bullshit AI demo that doesn’t reflect the real-world challenges we’re trying to solve, whilst ChatGPT confidently tells us that glue would make a delicious pizza topping.
The problem is that Large language models (LLMs) today are incredibly smart in a vacuum, but we struggle to find value in them beyond the confines of their training data.
In this article, we’ll explore the technology advancements that promise to overcome the two challenges AI faces to be of any use in the real world:
Context. An AI Agent must understand the specific environment it’s operating in. It needs to tailor is responses to reflect the specifics of your company policies; runbooks; and knowledge bases.
Integration. We need AI Agents to be able to take action. It must be able to integrate and interact with other tools within our environment.
RAG
The capability and extent of AI tooling we have today is impressive, but those models have no idea about how your specific CI pipeline works, what standards your company expects, or even what “prod-east-legacy” means in your org.
“Even the most sophisticated models are constrained by their isolation from data – trapped behind information silos and legacy systems.” - Anthropic
Retrieval-Augmented Generation (RAG) is a way to change that. You can provide the system with knowledge about your environment, that it can use as a reference later. When asked a question, a RAG system uses a vector search to find relevant chunks of text from documents you’ve supplied — things like run-books, platform docs, or even Slack threads — and passes those to the model as extra context. Vector search doesn’t rely on exact matches; it works by comparing meaning. If an engineer asks, “How do I deploy to staging?”, RAG might surface a doc titled “Promoting services to the integration cluster”—even if the word “staging” isn’t in the file.
The nice thing is, you don’t need to train a new model to make this work. With RAG, all you need is a model that knows how to read, and a way to feed it the right information when the time comes. That combination of retrieval + generation is where the magic happens.
For platform teams, that opens up some really practical doors:
Engineers can ask environment specific questions and get answers that reflect the policies, best practices, and operating practices of your organisation.
Onboarding new employees become much easier by making it much easier to find answers, without having to wade through reams of documentation that sits across different platforms or teams.
MCP
If RAG enables AI to be contextually aware, then Model Context Protocol (MCP) is what helps it interact with the world around it in order do something useful.
MCP is a protocol to enable AI agents to discover, connect to, and interact with external tools - whether that’s querying a database or calling an API. It gives models a way to operate beyond their static training data, making them more adaptable and useful in real-world applications.
MCP is a new open standard that’s gaining traction, fast! Originally created by Anthropic in Nov 2024, it lets AI tools connect to real-world systems in a predictable, secure, and structured way.
Most AI tools today are still pretty passive. They respond to questions, but they can’t take action without custom code written for each use case. MCP flips that by giving LLMs a standardised, “plug and play” way to understand and interact with APIs, tools, and data sources.
Under the hood, MCP defines a few key concepts:
Resources – the stuff your AI can read from or write to (like a Git repo, a Kubernetes cluster, or an internal API)
Tools – the actions your AI is allowed to take (like “restart a pod” or “create a ticket”)
Prompts – the instructions that help the AI understand what’s going on and what it can do
MCP doesn’t require special model training. Instead, it gives the model access to structured metadata that describes how things work—what tools are available, what inputs they take, what responses they return. The AI doesn’t magically “know” how to use your CLI—it learns in real time, because MCP tells it everything it needs.
There is already a ton of MCP implementations available, including GitHub, GitLab, Slack, Kubernetes, Terraform, PostgreSQL, and many more. As you might expect, we’re yet to standardise on single marketplace, but in the meantime, check out https://mcpmarket.com/, https://mcp.so/, and https://cline.bot/mcp-marketplace.
This is a HUGE step forward for AI, allowing it to go far beyond search and summarisation. It can take action—safely and predictably—within the boundaries you define.
It’s still early days, but this is the direction things are headed. If RAG makes AI helpful, MCP makes it capable.
This article is my contribution towards the Content Club initiative where some of my fellow CNCF working group colleagues sharing their thoughts on a given topic each month. If you’d like to join us, pop by the #content-club CNCF Slack channel.
Some of the other contributions include:
Colin Lacy shared a short-form video to describe his vision of a platform enabled LLM.
Kalle Sirkesalo created a long-form video where explores how autonomous agents are becoming more capable through retrieval-based memory, open source frameworks, and improved reasoning.
Graziano Casto explains how GenAI can help developers navigate complex internal platforms by reducing cognitive load and easing tool fatigue.