关键词检索与向量检索结合Hybrid Search混合检索实战系列从零构建企业 RAG 知识库第 9 篇1. 为什么两种检索互补向量检索擅长同义表达和语义相关关键词检索擅长精确编号、产品名、错误码和罕见术语。“无法登录系统” ↔ “认证失败” 向量检索可能更强 “ERR-1042” ↔ 文档中的 ERR-1042 关键词检索通常更可靠Hybrid Search 同时获取两路候选再融合排名。它不是把两个分数直接相加因为 BM25 与余弦分数的量纲不同。2. 一个简化 BM25 实现fromcollectionsimportCounterfromdataclassesimportdataclassfrommathimportlogimportredeftokenize(text:str)-list[str]:教学分词生产中文检索应使用经过评测的分析器。returnre.findall(r[a-z0-9_-]|[\u4e00-\u9fff],text.lower())dataclass(frozenTrue)classSearchDocument:document_id:strtenant_id:strtext:strvector:tuple[float,...]defbm25_scores(query:str,documents:list[SearchDocument],k1:float1.5,b:float0.75,)-dict[str,float]:ifnotdocuments:return{}tokenized{doc.document_id:tokenize(doc.text)fordocindocuments}avg_lengthsum(map(len,tokenized.values()))/len(documents)query_termsset(tokenize(query))scores{doc.document_id:0.0fordocindocuments}forterminquery_terms:containingsum(termintokensfortokensintokenized.values())idflog(1(len(documents)-containing0.5)/(containing0.5))fordocumentindocuments:tokenstokenized[document.document_id]frequencyCounter(tokens)[term]iffrequency0:continuelength_factor1-bb*len(tokens)/max(avg_length,1)scores[document.document_id]idf*(frequency*(k11)/(frequencyk1*length_factor))returnscores这是教学实现未覆盖生产搜索引擎的分词、字段权重和优化。3. 向量排名frommathimportsqrtdefcosine(left:tuple[float,...],right:tuple[float,...])-float:ifnotleftorlen(left)!len(right):raiseValueError(向量维度不一致)denominatorsqrt(sum(x*xforxinleft))*sqrt(sum(x*xforxinright))return0.0ifdenominator0elsesum(x*yforx,yinzip(left,right))/denominatordefvector_ranking(query_vector:tuple[float,...],documents:list[SearchDocument],)-list[str]:return[item.document_idforiteminsorted(documents,keylambdadoc:(-cosine(query_vector,doc.vector),doc.document_id,),)]4. 使用 RRF 融合排名Reciprocal Rank Fusion倒数排名融合只使用名次不要求两路分数同尺度defreciprocal_rank_fusion(rankings:list[list[str]],rank_constant:int60,)-list[tuple[str,float]]:ifrank_constant0:raiseValueError(rank_constant 必须大于 0)scores:dict[str,float]{}forrankinginrankings:forrank,document_idinenumerate(ranking,start1):scores[document_id]scores.get(document_id,0.0)(1.0/(rank_constantrank))returnsorted(scores.items(),keylambdaitem:(-item[1],item[0]))60是常见示例参数不是必须值应通过查询集调优。5. 完整的权限内混合检索defhybrid_search(query:str,query_vector:tuple[float,...],documents:list[SearchDocument],tenant_id:str,allowed_documents:frozenset[str],top_k:int5,)-list[SearchDocument]:# 两路检索使用完全相同的授权候选集合candidates[docfordocindocumentsifdoc.tenant_idtenant_idanddoc.document_idinallowed_documents]ifnotcandidates:return[]lexical_scoresbm25_scores(query,candidates)lexical_rank[keyforkey,scoreinsorted(lexical_scores.items(),keylambdaitem:(-item[1],item[0]),)ifscore0]semantic_rankvector_ranking(query_vector,candidates)fused_ids[document_idfordocument_id,_inreciprocal_rank_fusion([lexical_rank,semantic_rank])[:top_k]]document_map{doc.document_id:docfordocincandidates}return[document_map[document_id]fordocument_idinfused_ids]6. 可复验测试deftest_rare_error_code_is_recovered()-None:documents[SearchDocument(semantic,t1,登录认证失败处理,(1.0,0.0)),SearchDocument(exact,t1,错误码 ERR-1042 修复步骤,(0.0,1.0)),SearchDocument(other,t2,ERR-1042 内部秘密,(0.0,1.0)),]resulthybrid_search(ERR-1042,query_vector(1.0,0.0),documentsdocuments,tenant_idt1,allowed_documentsfrozenset({semantic,exact,other}),top_k2,)ids[item.document_idforiteminresult]assertexactinidsassertothernotinids7. 如何评测混合检索分别报告关键词、向量和混合三组结果RecallK正确证据是否进入前 KMRR第一个正确结果排在多前NDCG多个相关等级的排序质量零结果率、P95 延迟和检索成本按编号查询、自然语言、错别字和多语言切片。如果混合结果没有优于单路就不应仅因架构更复杂而上线。8. 对抗性审查两路检索必须使用相同权限过滤不把两种原始分数直接相加关键词索引和向量索引更新应保持版本一致对重复 Chunk 去重恶意文档可堆砌关键词需要来源质量和重排查询日志不默认保存敏感原文。9. 总结混合检索用关键词守住精确匹配用向量扩展语义召回再用稳定融合算法组合候选。它提升的是召回候选质量最终仍需要重排和生成校验。