Post
55
sipa_signal: Rule-Based AI Filler-Stripper
A deterministic utility designed to separate real claims ("signal") from AI-generated conversational bloat ("filler") using a fixed pattern table instead of relying on a model's subjective opinion of its own cleanliness.
Core Architecture
sipa_signal evaluates walls of AI text, splitting sentences into two distinct categories:
* Signal: Sentences carrying substantive claims or core information.
* Filler: Throat-clearing, hedges, meta-commentary, self-reference, and apologies.
The execution model is strictly deterministic: identical inputs yield identical splits every single time, with zero API keys required.
Bug Fixes & Edge Cases
* The Orphan Period Bug:
* The Issue: Phrases like "Sure, I'd be happy to help you with that." survived as valid content because stripping the matched filler phrase left behind a lone period. That single leftover punctuation mark was counted as a word, clearing the minimum threshold for "valid content."
* The Fix: Updated the word counter so a valid word must contain at least one alphanumeric character/digit. Additionally, filler patterns now match longest-phrase-first, preventing short matches from eating parts of longer phrases and leaving orphan fragments behind.
* Noise Ratio vs. Compression Nuance:
* The Issue: On a sample run, the text showed 32% noise by word count, but only 2% actual text deletion.
* The Mechanics: A sentence containing both a hedge word and a legitimate claim is kept intact (filler and all). noise_ratio accounts for every individual filler word wherever it sits, while compression strictly measures sentences that are fully excised. Two distinct metrics tracking two different things.
Project Specs & Access
* Test Suite: 12 tests green
* Execution: Includes run_demo.py for local testing
* Hackathon Track: Built for the "eliminate cognitive noise" track of the WeAreDevelopers Hackathon (Team SIPA_OS)
* Repository: github.com/soulinpsyabstract/sipa-signal(
A deterministic utility designed to separate real claims ("signal") from AI-generated conversational bloat ("filler") using a fixed pattern table instead of relying on a model's subjective opinion of its own cleanliness.
Core Architecture
sipa_signal evaluates walls of AI text, splitting sentences into two distinct categories:
* Signal: Sentences carrying substantive claims or core information.
* Filler: Throat-clearing, hedges, meta-commentary, self-reference, and apologies.
The execution model is strictly deterministic: identical inputs yield identical splits every single time, with zero API keys required.
Bug Fixes & Edge Cases
* The Orphan Period Bug:
* The Issue: Phrases like "Sure, I'd be happy to help you with that." survived as valid content because stripping the matched filler phrase left behind a lone period. That single leftover punctuation mark was counted as a word, clearing the minimum threshold for "valid content."
* The Fix: Updated the word counter so a valid word must contain at least one alphanumeric character/digit. Additionally, filler patterns now match longest-phrase-first, preventing short matches from eating parts of longer phrases and leaving orphan fragments behind.
* Noise Ratio vs. Compression Nuance:
* The Issue: On a sample run, the text showed 32% noise by word count, but only 2% actual text deletion.
* The Mechanics: A sentence containing both a hedge word and a legitimate claim is kept intact (filler and all). noise_ratio accounts for every individual filler word wherever it sits, while compression strictly measures sentences that are fully excised. Two distinct metrics tracking two different things.
Project Specs & Access
* Test Suite: 12 tests green
* Execution: Includes run_demo.py for local testing
* Hackathon Track: Built for the "eliminate cognitive noise" track of the WeAreDevelopers Hackathon (Team SIPA_OS)
* Repository: github.com/soulinpsyabstract/sipa-signal(