Demo 2 · Impact on retrieval and caching
Ten pretrained encoders (text, code, general LM, SPARQL), compared before and after results-aware fine-tuning on SQuARE.
1 Query retrieval
Given a new query, find earlier queries that return (nearly) the same answers. Real test queries, ranked against all other test queries.
Explorer: where does text similarity go wrong?
| TF-IDF rank | candidate query | answers | text cos | answer J |
|---|
Ranking: TF-IDF (the strongest non-neural baseline in the paper), computed from the released files by demo/scripts/build_demo_data.py. J from the complete test labels; "partial" means 0.1 < J < 0.8. ★ = the highest-overlap query in the collection, the one answer-overlap supervision rewards ranking first.
Hits@1 with complete labels
Share of test queries whose top-ranked query has J ≥ 0.8 (473 / 968 test queries). Encoder rows are means over ten encoders.
What the numbers say
- Fine-tuning improves Hits@1 for all 10 encoders on both graphs.
- Fine-tuned encoders beat TF-IDF, the strongest non-neural baseline: 0.70 vs 0.58 (DBpedia), 0.43 vs 0.37 (Wikidata).
- The gain comes from the answer labels: the same encoders trained on syntax labels reach only 0.52 and 0.36.
2 Approximate result caching
Reuse the cached answers of the most similar cached query if cosine ≥ θ, otherwise execute. A reuse is correct if J ≥ 0.8.
Hit accuracy before and after fine-tuning (θ = 0.8)
Accuracy = correct hits / all hits, pooled over five seeds and five stream orders. Fine-tuning raises accuracy for all ten encoders on both graphs.
Coverage vs. reliability: move θ
Takeaway
Results-aware fine-tuning makes reuse decisions much more accurate (DBpedia 0.15–0.33 → 0.59–0.79). A single similarity threshold is still not enough for safe reuse everywhere: ranking and calibration are separate problems.