title: Truthfulness Checker
emoji: 📰
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 5.4.0
app_file: app.py
pinned: false
license: apache-2.0
Implementation Steps: Validating Information with Context
Validating the accuracy or degree of truthfulness of a given piece of information requires context—factual and relevant details surrounding the claim. Here’s how we approach this process step-by-step:
Step 1: Retrieving Context from Knowledge Graph Substitute - FAISS with Semantic Search
Instead of relying on a traditional Knowledge Graph (KG), we use FAISS (Facebook AI Similarity Search), a faster, scalable, and flexible alternative for semantic search.
Why FAISS is Better than a Traditional KG
- Sentence-Level Retrieval: Unlike traditional KGs that often rely on pre-defined entities and relationships, FAISS uses dense embeddings to directly match the semantic meaning of entire sentences.
- Scalable and High-Speed Retrieval: FAISS efficiently handles millions of embeddings, making it highly scalable for real-world applications.
- Flexibility: It works with unstructured text, removing the need to pre-process information into entities and relations, which is often time-consuming.
- Generalization: FAISS enables approximate nearest neighbor (ANN) search, allowing retrieval of contextually related results, even if they are not exact matches.
Dataset Used
We leverage the News Category Dataset (Kaggle Link), which contains news headlines and short descriptions across various categories.
- Why This Dataset?
It covers a wide range of topics, making it useful for general-purpose context building.- Headlines and descriptions provide rich semantic embeddings for similarity searches.
- Categories allow filtering relevant results if required (e.g., "science" or "technology").
Process:
- We use SentenceTransformer (all-MiniLM-L6-v2) to generate embeddings for the query (the input news).
- We search against pre-computed embeddings stored in a FAISS index to retrieve the top-K most relevant entries.
- These results form the initial context, capturing related information already present in the dataset.
Step 2: Online Search for Real-Time Context
To augment the context retrieved from FAISS, we incorporate real-time online search using an API.
Why Online Search is Critical?
- Fresh Information: News and facts evolve, especially in areas like science, technology, or politics. Online search ensures access to the latest updates that may not exist in the static dataset.
- Diverse Sources: It broadens the scope by pulling information from multiple credible sources, reducing bias and enhancing reliability.
- Fact-Checking: Search engines often index trusted fact-checking websites that we can incorporate into the context.
Process:
- Use an API with a search query derived from the input news.
- Retrieve relevant snippets, headlines, or summaries.
- Append these results to the context built using FAISS.
Step 3: Building Context from Combined Sources
Both FAISS-based retrieval and online search results are combined into a single context string. This provides a comprehensive knowledge base around the input information.
- Why Combine Both?
- FAISS offers pre-indexed knowledge—ideal for static facts or concepts.
- Online search complements it with dynamic and up-to-date insights—perfect for verifying recent developments.
This layered context improves the model’s ability to assess the truthfulness of the given information.
Step 4: Truthfulness Prediction with Zero-Shot Classification Model
We use the Facebook/BART-Large-MNLI model, a zero-shot classification model, for evaluation.
Why BART-Large-MNLI?
- Zero-Shot Capability: It can handle claims and hypotheses without needing task-specific training—perfect for this flexible, multi-domain use case.
- Contextual Matching: It compares the input claim (news) with the constructed context to assess semantic consistency.
- High Accuracy: Pre-trained on natural language inference tasks, making it adept at understanding relationships like entailment and contradiction.
- Multi-Label Support: Can evaluate multiple labels simultaneously, ideal for degrees of truthfulness.
Process:
- Input the news as the claim and the context as the hypothesis.
- Compute a truthfulness score between 0 and 1, where:
- 0: Completely false.
- 1: Completely true.
- Generate explanations based on the score and suggest actions (e.g., further verification if uncertain).
End-to-End Example
Input News:
"Scientists Demonstrate 'Negative Time' In Groundbreaking Quantum Experiment."
Context Built:
- FAISS Search: Finds prior research on quantum time reversal and entanglement theories.
- Online Search: Retrieves recent articles discussing quantum breakthroughs and expert views.
Model Evaluation:
- Model compares the news with the combined context and outputs:
Score: 0.72 (Likely True).
Result Explanation:
News: "Scientists Demonstrate 'Negative Time' In Groundbreaking Quantum Experiment."
Truthfulness Score: 0.72 (Likely true)
Analysis: You can reasonably trust this information, but further verification is always recommended for critical decisions.
Why This Approach Works?
- Balanced Context: Combines static knowledge (KG substitute) and dynamic knowledge (real-time search).
- Model Flexibility: Zero-shot model adapts to diverse topics without retraining.
- Scalable and Cost-Effective: Uses pre-trained models, FAISS indexing, and simple APIs for implementation.
- Interpretability: Outputs include confidence scores and explanations for transparency.
This modular approach ensures that the truthfulness assessment is scalable, explainable, and adaptable to new domains.