vltruong01/TechQA-Adaptive-Hybrid-RAG

TechQA Adaptive Hybrid RAG

A reproducible technical-support Retrieval-Augmented Generation (RAG) system built on the reduced nvidia/TechQA-RAG-Eval corpus.

This repository packages the Adaptive Hybrid RRF system selected by the controlled five-system comparison published in:

vltruong01/TechQA-Hybrid-RAG-System-Comparison

Selection result: Adaptive Hybrid RRF achieved the strongest overall held-out document-ranking performance in the controlled comparison and was selected as the primary retrieval strategy.

System behavior: the generation stage is deliberately conservative. When the provided evidence does not directly support the requested technical answer, the system is instructed to abstain rather than invent unsupported troubleshooting guidance.

Important distinction: this repository is the reusable implementation of the selected system. The companion comparison repository remains the source of the controlled five-system experimental comparison.

---

Quick overview

Item	Setting
Upstream dataset	nvidia/TechQA-RAG-Eval
Experimental corpus	496 unique Technotes
QA rows	910
TRAIN	600 questions: 450 answerable + 150 impossible
DEV	310 questions: 160 answerable + 150 impossible
Primary retriever	Adaptive Hybrid RRF
Lexical retriever	BM25
Dense retriever	BAAI/bge-base-en-v1.5
Router	StandardScaler + LogisticRegression(class_weight="balanced")
Adaptive specialist boost	3.0
Chunk size / overlap	384 / 64 tokens
Generator	Qwen/Qwen2.5-7B-Instruct
Generator precision	4-bit NF4
Evidence budget	5 chunks, max 2 per document
Maximum prompt length	4096 tokens
Maximum new tokens	256
Evidence-sufficiency threshold	0.535
Fine-tuning	None
Seed	42
Reference / usage notebook	TechQA_Adaptive_Hybrid_RAG_Reference_Guide.ipynb
Comparison repository	TechQA-Hybrid-RAG-System-Comparison

---

1. What this repository is

This is a complete RAG system repository, not a standalone fine-tuned language-model checkpoint.

It packages:

• the executed end-to-end Adaptive Hybrid RAG notebook;
• a separate reference / usage notebook that downloads the released artifacts and demonstrates inference without retraining the complete experiment;
• the trained lightweight Adaptive query router;
• the trained evidence-sufficiency / answerability classifier;
• processed TechQA corpus and chunk artifacts;
• BGE embeddings and FAISS index;
• retrieval evaluation outputs;
• router and abstention diagnostics;
• generation evaluation outputs;
• example outputs;
• exact system configuration and artifact checksums.

What this repository is not

• It is not a fine-tuned Qwen checkpoint.
• It is not a fine-tuned BGE checkpoint.
• It does not contain copied Qwen or BGE pretrained weights.
• It is not a replacement for the upstream TechQA-RAG-Eval dataset.
• It does not claim that Adaptive Hybrid RRF is best on every metric.
• It does not claim that automated answer-similarity metrics are equivalent to technical correctness.

The pretrained models are loaded from their original Hugging Face repositories when the notebook runs.

---

2. Relationship to the comparison repository

The project is split into two repositories with different roles:

Repository	Purpose
TechQA-Hybrid-RAG-System-Comparison	Controlled comparison of BM25, Dense BGE, Static Hybrid RRF, Static RRF + Neural Reranker, and Adaptive Hybrid RRF
TechQA-Adaptive-Hybrid-RAG	Complete implementation and reusable artifacts for the selected Adaptive Hybrid RRF RAG system

The research flow is:

Reduced TechQA-RAG-Eval corpus
          |
          v
Controlled five-system retrieval comparison
          |
          v
Adaptive Hybrid RRF selected
          |
          v
This repository:
Adaptive retrieval + evidence selection + grounded Qwen generation

The selection of Adaptive Hybrid RRF is based on the controlled comparison, not on a claim that it dominates every downstream generation metric.

---

3. System architecture

Question
   |
   +----------------------+
   |                      |
   v                      v
BM25 ranking         Dense BGE ranking
   |                      |
   +----------+-----------+
              |
              v
       Query feature extraction
              |
              v
      TRAIN-learned router
              |
              v
 Query-specific BM25/Dense weights
              |
              v
    Weighted Reciprocal Rank Fusion
              |
              v
       Adaptive Hybrid RRF
              |
              v
   Diverse evidence selection
    top 5 chunks, max 2/doc
              |
              v
  Evidence-sufficiency classifier
              |
        +-----+------+
        |            |
        v            v
     abstain       answer path
                     |
                     v
        Qwen2.5-7B-Instruct
             4-bit NF4
                     |
                     v
        Grounded answer + citations

The controlled 30-question answerable generation evaluation bypasses the calibrated evidence-sufficiency gate so it remains comparable with the generation experiment in the companion comparison repository. The user-facing ask_techqa() helper applies the calibrated gate by default.

---

4. Data and preprocessing

The system loads:

nvidia/TechQA-RAG-Eval

from its published train split and recovers the benchmark partitions from question IDs:

Split	Total	Answerable	Impossible
TRAIN	600	450	150
DEV	310	160	150
Total	910	610	300

The reduced corpus contains 496 unique Technotes.

Text processing

• document identity is based on Technote filename;
• duplicate filename variants are consolidated;
• title and body are separated when a Title: line is present;
• chunking uses the BGE tokenizer with offset mappings;
• original text is sliced by character offsets rather than reconstructed by decoding.

Chunking

chunk size      = 384 tokens
chunk overlap   = 64 tokens
minimum tail    = 40 tokens

Chunk IDs follow:

<filename>::chunk_0000
<filename>::chunk_0001
...

---

5. Retrieval components

BM25

The lexical branch uses technical-token-aware tokenization and BM25 retrieval.

BM25_TOP_K = 50

Dense BGE

Dense retrieval uses:

BAAI/bge-base-en-v1.5

Queries are encoded with:

Represent this sentence for searching relevant passages:

Embeddings are normalized and searched with FAISS inner-product similarity.

DENSE_TOP_K = 50

Weighted Reciprocal Rank Fusion

The two ranked lists are combined using weighted RRF:

score(d) =
    w_bm25  / (60 + rank_bm25(d))
  + w_dense / (60 + rank_dense(d))

Static Hybrid RRF uses equal weights. Adaptive Hybrid RRF changes the weights for each query.

---

6. Adaptive query router

The router is trained only on 450 answerable TRAIN questions.

It predicts one of:

BM25-heavy
Balanced
Dense-heavy

TRAIN target distribution:

Router class	Samples
BM25-heavy	17
Balanced	371
Dense-heavy	62
Total	450

The router uses 14 lightweight features:

query_terms
query_avg_token_length
technical_token_ratio
digit_token_ratio
uppercase_token_ratio
bm25_top1
bm25_margin
bm25_relative_margin
dense_top1
dense_margin
dense_relative_margin
bm25_dense_top1_agree
top5_document_overlap
top5_document_jaccard

Model:

StandardScaler
    +
LogisticRegression(
    class_weight="balanced",
    max_iter=3000,
    random_state=42
)

TRAIN out-of-fold router diagnostics

Metric	Value
Accuracy	0.7311
Balanced accuracy	0.5182
Macro F1	0.4762

The Adaptive specialist boost is selected using TRAIN out-of-fold retrieval evaluation:

ADAPTIVE_HEAVY_WEIGHT = 3.0

DEV labels are not used to select this value.

---

7. Held-out retrieval performance

Adaptive Hybrid RRF is evaluated on all 160 answerable DEV questions.

Metric	Adaptive Hybrid RRF
Recall@1	0.8500
Recall@5	0.94375
Recall@10	0.96875
MRR@10	0.889105
nDCG@10	0.908251

The standalone notebook reproduces the selected retrieval metrics from the controlled comparison.

Latency is hardware- and runtime-dependent and should not be compared across independent executions without controlling the environment.

Why Adaptive was selected

In the companion five-system comparison:

System	R@1	R@5	R@10	MRR@10	nDCG@10
Adaptive Hybrid RRF	0.8500	0.9438	0.9688	0.8891	0.9083
Static Hybrid RRF	0.8500	0.9313	0.9688	0.8884	0.9076
Dense BGE	0.7938	0.9625	0.9750	0.8617	0.8899
BM25	0.8000	0.9063	0.9250	0.8473	0.8664
Static RRF + Neural Reranker	0.7438	0.9313	0.9688	0.8213	0.8572

Adaptive was selected for its strongest overall top-rank document-ranking performance, while Dense BGE remained strongest at deeper recall and Static Hybrid RRF remained extremely competitive.

For the complete controlled comparison and its interpretation, see the companion repository.

---

8. Evidence-sufficiency / abstention gate

A separate lightweight classifier estimates whether the retrieved evidence is sufficient to answer the question.

Model:

StandardScaler
    +
LogisticRegression(
    class_weight="balanced",
    max_iter=3000,
    random_state=42
)

The classifier uses 14 features derived from the question and retrieval evidence.

The threshold is selected on TRAIN out-of-fold predictions:

threshold = 0.535
TRAIN OOF ROC-AUC = 0.7806

Held-out DEV evaluation

The gate is evaluated on all 310 DEV questions:

Metric	Value
Answerable answer rate	0.6750
Impossible abstention rate	0.8133
False-answer rate on impossible questions	0.1867
Balanced accuracy	0.7442
Overall accuracy	0.7419
ROC-AUC	0.8095

This classifier is an auxiliary safety / evidence-sufficiency component. It is separate from the primary Adaptive retrieval contribution.

---

9. Grounded generation

The generator is:

Qwen/Qwen2.5-7B-Instruct

loaded with 4-bit NF4 quantization.

max prompt tokens = 4096
max new tokens    = 256
sampling          = disabled
evidence chunks   = 5
max chunks/doc    = 2

The prompt requires Qwen to:

• answer only from the numbered evidence sources;
• identify and answer the core technical question;
• preserve exact versions, error codes, commands, paths, and parameter values;
• avoid unsupported troubleshooting steps or recommendations;
• cite factual prose inline using [1], [2], etc.;
• abstain when the evidence supports only related topics rather than the requested answer.

Exact abstention text:

The available documentation does not contain enough information to answer this question.

Citation retry is enabled when the first answer does not satisfy the notebook's citation-format checker.

---

10. Standalone generation evaluation

The model repository contains a standalone Adaptive-only generation run on the same seeded 30 answerable DEV questions used by the comparison experiment.

The calibrated evidence-sufficiency gate is not applied to this 30-question evaluation because the controlled comparison evaluates the generator directly on answerable questions.

Metric	Value
Questions	30
Answer rate	0.6333
Exact false-abstention rate	0.3667
Gold-document retrieval rate	1.0000
Mean gold chunks in prompt	1.6333
Mean unique documents in prompt	3.5667
Mean best reference-chunk cosine	0.7634
Mean top-3 reference-chunk cosine	0.7105
ROUGE-L F1	0.2313
BGE answer cosine	0.6921
Citation presence among detected answers	1.0000
Citation-format validity among detected answers	1.0000

Interpreting these values

The standalone run should not replace the controlled cross-system generation table in the comparison repository.

Independent executions can differ modestly because of environment, package, model-revision, and low-level kernel differences. Cross-system claims should therefore be taken from the companion comparison repository, where all five systems were evaluated in the same experiment.

The standalone run is useful for verifying that the packaged Adaptive system behaves consistently and that its retrieval results reproduce exactly.

Conservative answer behavior

The system intentionally prefers abstention to unsupported technical guidance.

On the 30-question answerable sample, the notebook's exact abstention detector marks 11/30 outputs as abstentions. This metric is based on normalization of the configured exact abstention sentence; longer refusal-like answers may not always be counted as abstentions. Manual inspection is therefore recommended when studying answer coverage.

Citation-format validity also does not prove factual entailment. It verifies citation presence and source-index formatting, not whether every cited claim is technically correct.

---

11. Repository contents

TechQA-Adaptive-Hybrid-RAG/
|
|-- README.md
|-- TechQA_Adaptive_Hybrid_RAG.ipynb
|-- TechQA_Adaptive_Hybrid_RAG_Reference_Guide.ipynb
|-- system_config.json
|-- requirements.txt
|-- artifact_manifest.json
|
`-- artifacts/
    |-- models/
    |   |-- router_pipeline.joblib
    |   `-- abstention_pipeline.joblib
    |
    |-- index/
    |   |-- techqa_bge_embeddings.npy
    |   |-- techqa_bge_faiss.index
    |   |-- techqa_chunks.parquet
    |   |-- techqa_unique_documents.parquet
    |   `-- techqa_qa.parquet
    |
    |-- retrieval/
    |   |-- retrieval_metrics.csv
    |   `-- retrieval_detail.csv
    |
    |-- router/
    |   |-- router_oof_metrics.csv
    |   |-- router_train_oof.csv
    |   `-- adaptive_weight_search.csv
    |
    |-- abstention/
    |   |-- abstention_metrics.csv
    |   |-- abstention_train_oof_scores.csv
    |   |-- abstention_threshold_tuning.csv
    |   |-- abstention_dev_outputs.csv
    |   |-- abstention_feature_weights.csv
    |   `-- abstention_errors.csv
    |
    |-- generation/
    |   |-- generation_metrics.csv
    |   |-- generation_eval_questions.csv
    |   |-- generation_eval_outputs.csv
    |   `-- manual_generation_review.csv
    |
    `-- examples/
        |-- answerable_example_output.json
        |-- impossible_example_output.json
        `-- custom_question_output.json

artifact_manifest.json records the released files with their byte sizes and SHA-256 hashes.

Which notebook should I use?

Notebook	Best for
TechQA_Adaptive_Hybrid_RAG_Reference_Guide.ipynb	Recommended starting point for new users. Downloads the released artifacts, verifies checksums, restores the Adaptive retriever and evidence-sufficiency gate, and optionally runs Qwen generation without retraining the full experiment.
TechQA_Adaptive_Hybrid_RAG.ipynb	Full end-to-end reproduction: rebuild corpus/indexes, train/select the router, calibrate the gate, evaluate DEV, run generation, and export artifacts.

---

12. Quick start

This repository is notebook-first rather than a conventional transformers checkpoint.

Do not expect the following to load the complete RAG system:

AutoModelForCausalLM.from_pretrained(
    "vltruong01/TechQA-Adaptive-Hybrid-RAG"
)

The repository does not contain a replacement Qwen checkpoint. Instead, it combines released retrieval artifacts, lightweight scikit-learn models, frozen BGE embeddings/indexes, and the original Qwen generator.

Option A — Use the released system artifacts

This is the recommended path for most users.

Open:

TechQA_Adaptive_Hybrid_RAG_Reference_Guide.ipynb

The reference guide:

1. downloads this Hugging Face repository with snapshot_download();
2. verifies the released files using artifact_manifest.json;
3. loads the processed corpus, BGE embeddings, and FAISS index;
4. loads router_pipeline.joblib and abstention_pipeline.joblib;
5. rebuilds the lightweight BM25 index;
6. restores Adaptive Hybrid RRF inference;
7. demonstrates retrieval and query-specific BM25/Dense weights;
8. restores the evidence-sufficiency gate;
9. optionally loads Qwen/Qwen2.5-7B-Instruct in 4-bit NF4;
10. exposes a user-facing ask_techqa() helper.

A T4-class GPU or better is recommended if you want to run Qwen generation. Repository inspection and retrieval can also be performed without loading Qwen.

Example after initialization:

output = ask_techqa(
    "Your technical-support question here",
    use_calibrated_gate=True,
    show_sources=True,
)

Option B — Reproduce the complete pipeline

If you want to rebuild and study the entire experiment, open:

TechQA_Adaptive_Hybrid_RAG.ipynb

This notebook reconstructs the reduced corpus, regenerates chunks and embeddings, rebuilds the FAISS index, trains/selects the Adaptive router from TRAIN, calibrates the evidence-sufficiency model, evaluates DEV, loads Qwen, and exports the released artifacts.

Local clone

You can also clone the repository:

git clone https://huggingface.co/vltruong01/TechQA-Adaptive-Hybrid-RAG
cd TechQA-Adaptive-Hybrid-RAG
pip install -r requirements.txt

Then choose either the Reference Guide for artifact-based usage or the full notebook for complete reproduction.

---

13. Full reproduction from scratch

The following sequence refers to TechQA_Adaptive_Hybrid_RAG.ipynb, not the lightweight Reference Guide.

The full notebook rebuilds the pipeline in the following order:

1. load nvidia/TechQA-RAG-Eval;
2. recover TRAIN and DEV;
3. reconstruct the 496-Technote reduced corpus;
4. create text-preserving 384-token chunks;
5. build BM25;
6. encode chunks with frozen BGE;
7. build the FAISS index;
8. train/select the Adaptive router using TRAIN only;
9. evaluate Adaptive retrieval on all 160 answerable DEV questions;
10. train and evaluate the evidence-sufficiency gate;
11. load Qwen2.5-7B-Instruct in 4-bit NF4;
12. run grounded generation;
13. evaluate the fixed 30-question generation subset;
14. export model, index, configuration, and evaluation artifacts.

Main dependencies

• datasets
• transformers
• accelerate
• bitsandbytes
• sentence-transformers
• faiss-cpu
• rank-bm25
• rouge-score
• pandas
• scikit-learn
• joblib

See requirements.txt for the released environment specification.

---

14. Reproducibility safeguards

The selected system follows the same leakage controls as the comparison experiment:

• router targets are derived only from TRAIN;
• router model selection uses TRAIN out-of-fold predictions;
• the Adaptive fusion strength is selected on TRAIN;
• the evidence-sufficiency threshold is selected on TRAIN out-of-fold predictions;
• DEV labels are used only for held-out evaluation;
• the generator and embedding model remain frozen;
• no LoRA or QLoRA is used;
• the fixed generation question subset uses the same seed and configuration as the controlled comparison.

The notebook includes integrity checks for the expected router target distribution and selected Adaptive specialist boost.

---

15. Intended use

This repository is intended for:

• learning how to load and reuse a released RAG system from Hugging Face artifacts;
• technical-support RAG research;
• reproducible Adaptive Hybrid RRF experiments;
• hybrid lexical + dense retrieval;
• query-adaptive fusion research;
• retrieval-vs-generation analysis;
• abstention / evidence-sufficiency research;
• educational demonstrations of end-to-end RAG system construction.

It can also serve as a reference implementation for a lightweight query-adaptive retriever that does not require fine-tuning the embedding model or generator.

---

16. Limitations

1. Reduced corpus
   The system uses the reduced TechQA-RAG-Eval corpus, not the original full TechQA / full IBM Technotes collection.

2. Generation evaluation size
   End-to-end generation is evaluated on a seeded subset of 30 answerable DEV questions rather than all 160 answerable DEV questions.

3. Conservative generation can abstain on answerable questions
   Successful document retrieval does not guarantee that the generator will use the evidence correctly. The grounded prompt intentionally abstains when the evidence appears insufficiently explicit.

4. Gold-document retrieval is not answer-span retrieval
   A gold Technote appearing in the evidence set does not guarantee that the exact answer-bearing passage is optimally positioned in the limited 5-chunk prompt.

5. Automated answer metrics are incomplete
   ROUGE-L and embedding similarity measure similarity to references but do not guarantee technical correctness, completeness, or absence of misleading details.

6. Citation checks are format checks
   A valid [n] citation means the source index is syntactically valid. It does not prove that the cited source entails every claim.

7. Exact abstention detection can undercount semantic refusals
   Evaluation normalization is centered on the configured abstention sentence. Longer refusal-style generations can require manual inspection.

8. Frozen generator
   Qwen2.5-7B-Instruct is not fine-tuned for TechQA. Some failures occur after successful retrieval because evidence utilization remains a downstream bottleneck.

9. Router class imbalance
   Most TRAIN router targets are Balanced; BM25-heavy examples are rare. Balanced accuracy and macro F1 should be considered alongside raw router accuracy.

10. Not authoritative support guidance
    Generated outputs are research artifacts. Technical-support answers should be checked against the cited documentation before operational use.

11. Latency is environment-dependent
    Retrieval and generation times vary with GPU, CPU, package versions, model revisions, and runtime configuration.

---

17. Dataset provenance and released artifacts

The upstream dataset is:

nvidia/TechQA-RAG-Eval

The companion comparison repository is:

vltruong01/TechQA-Hybrid-RAG-System-Comparison

Processed QA, document, chunk, embedding, and FAISS artifacts in this repository are derived from the reduced TechQA-RAG-Eval corpus and are included for system reproducibility.

This repository does not redistribute the pretrained Qwen or BGE model weights.

Users should also follow the license and attribution requirements of the upstream dataset and model repositories.

Original TechQA resource:

• https://github.com/IBM/techqa

---

18. Main conclusion

For users who want to try the packaged system rather than reproduce training and evaluation, start with TechQA_Adaptive_Hybrid_RAG_Reference_Guide.ipynb.

The selected system preserves the main finding of the controlled comparison:

Adaptive Hybrid RRF achieved the strongest overall held-out document-ranking performance and was therefore selected as the primary retrieval strategy.

The standalone model reproduction exactly matches the selected full-DEV retrieval metrics while retaining a conservative grounded-generation policy.

The broader project also shows that:

Strong document retrieval does not automatically guarantee strong downstream generation. Evidence selection, limited context budgets, abstention behavior, and generator evidence utilization remain important bottlenecks.

For cross-system claims, use the controlled comparison repository rather than the standalone reproduction in this model repository.

---

Citation

If you use the packaged selected system, please cite this repository:

@misc{vltruong01_techqa_adaptive_hybrid_rag_2026,
  author       = {vltruong01},
  title        = {TechQA Adaptive Hybrid RAG},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/vltruong01/TechQA-Adaptive-Hybrid-RAG}
}

For the controlled five-system experiment, please also cite:

@misc{vltruong01_techqa_hybrid_rag_2026,
  author       = {vltruong01},
  title        = {TechQA Hybrid RAG System Comparison},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/datasets/vltruong01/TechQA-Hybrid-RAG-System-Comparison}
}

Please also follow the citation / attribution guidance of the upstream TechQA and TechQA-RAG-Eval resources.

---

Upstream models and resources

• Dataset: nvidia/TechQA-RAG-Eval
• Comparison repository: vltruong01/TechQA-Hybrid-RAG-System-Comparison
• Original TechQA: IBM/techqa
• Dense retriever: BAAI/bge-base-en-v1.5
• Generator: Qwen/Qwen2.5-7B-Instruct

---

Repository

Hugging Face Model:
https://huggingface.co/vltruong01/TechQA-Adaptive-Hybrid-RAG