The field of finding information relevant to a query from a large collection of documents, encompassing classical keyword search, semantic search based on vector embeddings, and the retrieval components of retrieval-augmented generation systems. Information retrieval is the backbone of search engines, enterprise knowledge bases, and the RAG architectures that give language models access to up-to-date knowledge beyond their training data.
Also known as IR, document retrieval, search and retrieval
Information retrieval systems index a collection of documents and respond to queries by returning the documents most relevant to the query, ranked by estimated relevance. Classical keyword-based retrieval uses the TF-IDF representation, which weights terms by how frequently they appear in a document relative to how frequently they appear across the full collection, and ranks documents by the overlap between their TF-IDF representations and the query terms. BM25 extends TF-IDF with document length normalization and tunable term saturation parameters, producing better retrieval quality than raw TF-IDF on most benchmark collections. These lexical retrieval methods match exact or stemmed query terms against document terms and fail to retrieve documents that discuss the query topic using different vocabulary.
Dense retrieval, the current state of the art for most retrieval tasks, represents queries and documents as dense vector embeddings from a dual-encoder neural network trained specifically for retrieval quality. The query and document encoders are trained together such that relevant query-document pairs have high embedding similarity and irrelevant pairs have low similarity. At retrieval time, the query is encoded to a dense vector, and documents are retrieved by approximate nearest-neighbor search over the pre-computed document embeddings. Dense retrieval captures semantic similarity rather than lexical overlap, retrieving documents that discuss the query topic even when they use different vocabulary. Hybrid retrieval, combining lexical and dense methods with a learned re-ranker, produces state-of-the-art retrieval quality on most benchmarks.
Retrieval-augmented generation combines information retrieval with language model generation to address the knowledge limitations of language models. Given a user query, a RAG system retrieves relevant documents from a knowledge base, includes the retrieved documents in the language model’s context window, and prompts the model to generate a response grounded in the retrieved content. This architecture enables the language model to produce accurate, up-to-date responses based on retrieved source material rather than relying on potentially outdated or hallucinated knowledge from its training data. The quality of RAG system outputs depends critically on retrieval quality: if the wrong documents are retrieved, the language model will either hallucinate because the retrieved content is irrelevant, or produce confidently wrong answers if the retrieved content is misleadingly relevant.
Agencies manage large volumes of research, strategy documents, competitive intelligence, creative assets, and client history. Making this knowledge findable and usable in AI-assisted workflows requires information retrieval that goes beyond file search to genuine semantic understanding of document content. A working ad agency that has built or integrated effective retrieval into its knowledge management and AI workflows compounds its institutional knowledge rather than losing it to the endless file folder.
RAG systems enable AI tools to work with current, client-specific knowledge. A language model used without retrieval augmentation can only draw on its training data, which has a cutoff date and no knowledge of specific client strategies, product details, or competitive intelligence. A RAG system that retrieves from the agency’s internal knowledge base grounds language model outputs in actual, current, organization-specific content. This is the architecture that transforms a general-purpose language model into an agency-specific intelligent assistant that knows about clients, campaigns, and strategies.
Retrieval quality is the binding constraint on RAG system quality. Many agencies that have deployed RAG-based systems report that the language model generation quality is satisfactory but that the system sometimes produces irrelevant or incorrect answers. The root cause is almost always retrieval failure: the right documents were not retrieved, so the language model had no useful context to draw on. Improving retrieval quality, through better embedding models, hybrid retrieval, or query expansion, is the highest-leverage intervention for RAG system quality improvement.
Semantic search over creative asset libraries changes asset discoverability. Searching a creative asset library by filename or manually applied tags depends on perfect tagging and naming discipline, which degrades rapidly at scale. Semantic search over asset embeddings enables finding images, copy, and design elements by descriptive query: “outdoor lifestyle photograph with warm lighting” retrieves the relevant assets regardless of what they are named or how they are tagged. Agencies that build semantic search into their asset management systems make their asset libraries substantially more useful as organizational knowledge resources.
An agency builds an internal knowledge assistant for its strategy team using a RAG architecture. The knowledge base indexes 3 years of client strategy decks, competitive research reports, and post-campaign analyses, totaling approximately 8,000 documents. An initial deployment using keyword-based retrieval produces poor performance on strategy questions that use conceptual vocabulary different from the documents: a query about “emotional resonance in automotive advertising” fails to retrieve highly relevant documents that discuss “brand affinity” and “aspirational positioning” in auto category campaigns. The agency replaces keyword retrieval with a dense retrieval system using a sentence transformer fine-tuned on marketing domain text, supplemented by BM25 for exact phrase matching in a hybrid configuration. Retrieval quality, measured by the fraction of relevant documents appearing in the top-5 retrieved results on a human-curated evaluation set, improves from 41% with keyword retrieval to 78% with hybrid dense retrieval. The strategy team reports that the assistant now produces useful answers to research questions that were previously answered by manual document search, and that the hybrid retrieval handles both conceptual queries and exact lookup queries such as specific campaign names or client brand guidelines effectively.
The automations and agents module covers how to build retrieval-augmented AI systems that ground language model outputs in accurate, current, organization-specific knowledge, including the retrieval quality practices that determine whether the system produces useful answers or confident hallucinations.