Building an In-Process Semantic Search Pipeline in Go with ONNX, pgvector, and RRF

Relying on external SaaS APIs for text embeddings introduces non-trivial network latency, rate limits, and recurring cloud costs. For high-throughput services written in Go, running model inference directly in-process eliminates outbound HTTP overhead and yields predictable sub-millisecond tokenization and vector generation. By combining an embedded ONNX runtime, PostgreSQL's pgvector extension, and Reciprocal Rank Fusion (RRF), you can build an enterprise-grade hybrid search engine entirely within your Go application ecosystem.

In-Process Embeddings with Go and ONNX Runtime

Executing transformer models in Go requires bridging Go with the C-based ONNX Runtime using CGo. Using pre-quantized models such as all-MiniLM-L6-v2 or bge-small-en-v1.5 in ONNX format allows a single Go binary to process raw strings into 384-dimensional dense vectors within memory. A dedicated tokenizer step maps text into input IDs and attention masks, feeds them to the session, and performs mean pooling over the final hidden states. Because inference runs in the same address space, you bypass network serialization overhead and avoid paying per-token API charges.

Efficient Vector Search with pgvector and HNSW

Once generated, dense embeddings are persisted in PostgreSQL using the pgvector extension. While flat searches or IVFFlat indexes suffice for small datasets, production semantic search demands Hierarchical Navigable Small World (HNSW) indexing. HNSW constructs a multi-layer graph that trades a modest amount of memory and build time for blazingly fast approximate nearest neighbor (ANN) lookups with high recall. By tuning the m (bidirectional links per node) and ef_construction (size of dynamic candidate list) parameters, Go services can query millions of vectors using cosine distance in under 5 milliseconds.

Combining Lexical and Semantic Signals with Reciprocal Rank Fusion

Dense vector search excels at conceptual matching, but it often struggles with exact domain terms, SKU codes, and acronyms where traditional BM25 keyword search thrives. Hybrid search addresses this by querying both inverted text indexes (via PostgreSQL's native tsvector) and vector embeddings simultaneously. Rather than attempting to calibrate radically different raw scores, we employ Reciprocal Rank Fusion (RRF) to merge both result sets:

• Run the lexical full-text query and the HNSW vector query concurrently in a single database transaction or parallel Go goroutines.
• Assign an RRF score to each candidate using the formula 1 / (k + rank), where k is typically set to 60 to soften the penalty for lower-ranked items.
• Sum the scores across both lists to generate a balanced final ranking that respects both semantic relevance and exact keyword overlap.

Adopting this hybrid pipeline in Go delivers a resilient search infrastructure with zero third-party API dependencies. Memory management remains critical: always pool your ONNX inference sessions and monitor CGo allocations carefully. The end result is a self-contained, highly performant search service capable of serving rich, context-aware queries at scale with exceptional latency profiles.