Field notes on applied machine learning, data tooling and the occasional paper review.
Most teams ship retrieval-augmented systems long before anyone writes a proper evaluation set. A few cheap proxies — answer-span overlap, citation coverage and disagreement between two rerankers — get you surprisingly far while the real labels are being collected.
For invoice and form extraction, a well-prompted small model with a strict output schema often beats a larger one used loosely. The gains come mostly from constrained decoding and a validation step, not from parameter count.
Re-embedding a whole corpus every time the model changes is expensive. A versioned index with lazy, query-driven re-embedding keeps costs flat and lets you compare old and new vectors side by side.