RAG or fine-tuning when data changes daily

Separate the facts from the task

Retrieval-augmented generation, or RAG, supplies selected source material as context for an answer. Fine-tuning changes a model through additional training, and neither removes the need to test the final answer.

A useful question is whether failures come from missing evidence or from how the model handles evidence it already has. Training it to write a better summary will not fix a policy that never reached it.

What the StockTick system needed

For Waymark’s StockTick work, AURA LOGIC LABS used retrieval over an approved financial-report corpus in the client’s AWS environment. The system ingested reports daily, used Qdrant as its retrieval store and kept calculations outside the language model.

The design kept responsibilities separate: bring the relevant report into the answer, retain an inspectable reference, and calculate figures with deterministic logic. A citation alone does not prove correctness.

Choose the next experiment from the failure

• If answers omit newly published material, inspect ingestion delay, indexing and retrieval before changing the model.
• If answers use the wrong document version, define which source is authoritative and how superseded records are handled.
• If the evidence is correct but the output breaks format, try clearer instructions or task-specific training where your platform supports it.
• If different people should see different records, design access enforcement into the retrieval flow. Fine-tuning is not a substitute for that control.

Test freshness separately from answer quality

Build an evaluation set from real questions and expected sources: one whose answer changes after a document update, one with conflicting versions, and one the corpus cannot answer.

Then check retrieval and generation separately: did the right evidence arrive, and did the answer use it accurately? This distinguishes a stale index from a model that misreads a current report.

A practical starting point

Start with a small approved corpus, a clear output requirement and an evaluation baseline. Add retrieval when current or private evidence is needed, and consider fine-tuning only when a measured task failure justifies the work.

For an enterprise deployment, see the permission design guide.