the pitch is always the same. we have too many docs. people cannot find things. let us build a knowledge base with retrieval, and the company will finally know what it knows. then someone buys a vector database, syncs three wikis and a shared drive, and discovers that the company does not know what it knows. it has opinions in twelve formats, half of them stale, and nobody owns the conflict.
i like retrieval. i have spent years on search systems. that is why i am suspicious of knowledge bases that arrive before the boring prerequisites. embeddings are good at finding similar text. they are not a substitute for deciding which text is allowed to be true.
if you are about to stand up a rag stack because search feels embarrassing, pause. some of the time you need better authorship and ownership, not a smarter index over the same landfill.
you do not have questions, you have vibes
a knowledge base needs a query distribution. not a fantasy list from a workshop. actual questions people ask when money or time is on the line. support tickets, sales objections, onboarding blockers, incident pages, "where is the runbook" messages in slack.
if you cannot produce fifty real questions with rough expected answers, you are not ready to evaluate retrieval. without evaluation, you will demo on friendly prompts and call it a launch. then the first angry customer question will retrieve a blog post from 2021 and a deck slide that says "tbd."
collect questions before you collect infrastructure. write them down. note who asked and what "good" looks like. this feels slow. it is faster than rebuilding your chunker three times because you never knew what success was.
nobody owns the source of truth
retrieval surfaces conflicts. it does not resolve them. if pricing lives in a notion page, a pdf, and a salesperson's google doc, your system will retrieve all three with confidence scores that look scientific. the model will reconcile them the way models reconcile things: by sounding sure.
before you index, name an owner for each domain you care about. not "the docs team" in the abstract. a person who can say which document wins when two disagree. if that person does not exist, your knowledge base will become a conflict amplifier with a chat ui.
ownership also means deprecation. a page without an owner never dies. it just ranks forever.
your corpus is a staging area
shared drives accumulate drafts, duplicates, and meeting notes that were never meant to answer customers. wikis accumulate abandoned spaces. confluence is where information goes to form committees. if you sync all of that into an embedding index, you have built a rumor engine.
the move is not "ingest everything and let reranking sort it out." reranking cannot invent editorial standards. start with a small allowlist: the docs that already have readers, reviewers, and a reason to stay current. leave the rest out on purpose. searchable junk is still junk. it is just faster junk.
if leadership wants "all company knowledge" in v1, translate that into a phased allowlist and make the politics visible. otherwise the politics show up later as wrong answers with citations.
process is missing, so answers cannot be stable
some questions are not document questions. they are process questions. "can we give this customer a discount" may depend on a slack approval thread, a crm field, and whoever is on duty. stuffing old discount emails into a vector store does not create a policy. it creates a highlight reel of exceptions.
if the real answer changes by account tier, region, or who you ask, a static knowledge base will lie with perfect grammar. fix the process, or route those questions to a human workflow. do not pretend retrieval is a decision system.
this is also true for internal engineering knowledge. if the runbook is "ask jordan," your first job is to get jordan's steps into a page jordan will maintain. indexing jordan's old incident notes is a temporary costume.
you cannot measure wrongness yet
teams skip eval because eval is unglamorous. then they argue from screenshots. a knowledge base you cannot score will be tuned toward whoever complained last.
you do not need a research harness on day one. you need a living set of questions, labeled sources when possible, and a habit of checking retrieval before you check prose. if you are unwilling to maintain that set, you are unwilling to operate the system. build a better site search or improve navigation instead. those fail more honestly.
also decide what "wrong" means. incomplete, outdated, unsafe, off-policy, and off-brand are different failures. if you only track thumbs-up, you will optimize for answers that feel helpful while being slightly false.
the product is a search box with no place to land
sometimes the right interface is not chat over chunks. it is a clearer IA, better titles, and a docs homepage that matches how people think. if users cannot browse to the right area, retrieval will also struggle, because the corpus itself is unlabeled mush.
i have seen companies spend a quarter on rag and then fix the issue with headings and a single curated FAQ. that outcome is not a failure of ambition. it is a correct diagnosis arriving late.
if your docs already work for humans who know the URL patterns, and the pain is only discovery, a plain keyword search with decent ranking may beat a half-built generation layer. generation adds failure modes. earn them.
compliance will not wait for your roadmap
if you are in a regulated space, or even a mildly careful enterprise space, indexing everything can create retention and access problems. content that was sitting in a restricted folder becomes answerable through a bot with broader read permissions. that is not an embedding issue. that is an access control issue you introduced because the demo looked good.
do not build a knowledge base that bypasses permissions you already have. if you cannot enforce document acl at retrieval time, do not ingest the restricted material. "we will add authz later" is how later becomes an incident.
a smaller thing that is actually ready
so when should you not build yet? when questions are undefined, ownership is undefined, the corpus is an unfiltered dump, process answers are being treated like document answers, you have no way to score failures, or access rules cannot be enforced.
what to do instead is less fun to announce. pick one domain. name an owner. curate thirty to fifty source pages. collect real questions. define which sources are allowed. ship search or retrieval against that island. watch what breaks. expand the allowlist when the island stops being embarrassing.
you can still use embeddings on that smaller set. the point is not to avoid rag forever. the point is to avoid building a company-wide memory before the company has a memory worth trusting.
a knowledge base is a distribution channel for decisions you have already made about truth. if those decisions are missing, the stack will still run. it will just run on sludge and call it knowledge. i have been that person shipping the index anyway. it feels like progress until the first confident wrong answer gets forwarded to a customer. then it feels like what it was: search over unfinished work.