Repository: yusufkaraaslan/Skill_Seekers (python)

Issue:
feat: Add 15 missing languages to PDF scraper

## 📋 Description

The PDF scraper currently supports **19 languages** while the documentation scraper supports **34 languages**. This creates an inconsistency where PDFs containing certain languages won't have proper code highlighting.

## 🔍 Problem

**Missing 15 languages in PDF scraper:**
- `bash` (has `shell` but not `bash` specifically)
- `typescript` ⭐ (very popular)
- `jsx` (React)
- `tsx` (React + TypeScript)
- `vue` (Vue.js)
- `dart` (Flutter)
- `elixir` (Elixir/Phoenix)
- `lua` (Game development)
- `perl`
- `scala` (Big data)
- `powershell` (Windows automation)
- `r` (Data science)
- `markdown` (Documentation)
- `scss` (CSS preprocessor)
- `sass` (CSS preprocessor)

## 📊 Current State

| Scraper | Languages | Coverage |
|---------|-----------|----------|
| Documentation | 34 | ✅ Full |
| PDF | 19 | ⚠️ 56% of doc scraper |
| GitHub | 600+ | ✅ Unlimited (API) |

## 🎯 Goal

Add regex patterns to `cli/pdf_extractor_poc.py` for the 15 missing languages to achieve parity with the documentation scraper.

## 🔧 Technical Details

**Location:** `cli/pdf_extractor_poc.py` - `detect_language_from_code()` method (line 211)

**Pattern format:**
```python
'language_name': [
    (r'regex_pattern', weight),  # Higher weight = stronger indicator
    (r'another_pattern', weight),
    # ...
],
```

**Example implementation (TypeScript):**
```python
'typescript': [
    (r'\binterface\s+\w+', 3),           # interface keyword
    (r'\btype\s+\w+\s*=', 2),            # type aliases
    (r':\s*\w+(\[\])?', 1),              # type annotations
    (r'\bas\s+\w+', 1),                  # type assertions
    (r'<\w+>(\(.*?\))?', 1),             # generics
],
```

## ✅ Acceptance Criteria

- [ ] Add regex patterns for all 15 missing languages
- [ ] Each language should have 3-5 distinctive patterns
- [ ] Patterns should have appropriate confidence weights (1-5)
- [ ] Add test cases in `tests/test_pdf_extractor.py` for each language
- [ ] Verify no false positives with similar languages (e.g., TypeScript vs JavaScript)
- [ ] Update documentation with new supported languages

## 🧪 Testing

For each language, test with real-world code samples:
```python
def test_detect_typescript():
    code = """
    interface User {
        name: string;
        age: number;
    }
    
    const getUser = (): User => {
        return { name: "John", age: 30 };
    }
    """
    lang, confidence = extractor.detect_language_from_code(code)
    assert lang == 'typescript'
    assert confidence > 0.5
```

## 📝 Implementation Notes

**Priority order (by popularity):**
1. **High priority:** `typescript`, `jsx`, `tsx`, `bash`, `dart`
2. **Medium priority:** `vue`, `powershell`, `r`, `scala`, `markdown`
3. **Low priority:** `elixir`, `lua`, `perl`, `scss`, `sass`

**Pattern design tips:**
- Use `\b` for word boundaries to avoid partial matches
- Include language-specific keywords (e.g., `interface` for TypeScript)
- Look for unique syntax (e.g., `=>` is common in modern JS/TS)
- Test against similar languages to avoid false positives

## 🔗 Related

- PR #154 added these languages to documentation scraper
- `cli/doc_scraper.py` has reference implementations (lines 276-308)
- GitHub scraper supports all languages via API (no action needed)

## 📌 Labels

`enhancement`, `good first issue`, `pdf-scraper`, `language-support`

---

**Note:** This issue is tracked but not currently being worked on. Feel free to pick it up if you want to contribute!


Patch:
diff --git a/src/skill_seekers/cli/language_detector.py b/src/skill_seekers/cli/language_detector.py
index ff1b1cfe..5694d358 100644
--- a/src/skill_seekers/cli/language_detector.py
+++ b/src/skill_seekers/cli/language_detector.py
@@ -310,6 +310,67 @@
         (r"\b_ready\s*\(", 4),
         (r"\b_process\s*\(", 4),
     ],
+    "dart": [
+        (r"\bimport\s+['\"]package:", 5),
+        (r"\bclass\s+\w+\s+extends\s+StatelessWidget", 5),
+        (r"\bclass\s+\w+\s+extends\s+StatefulWidget", 5),
+        (r"@override\b", 4),
+        (r"\bWidget\s+build\s*\(", 5),
+        (r"\bimport\s+['\"]dart:", 5),
+        (r"\bfinal\s+\w+\s+\w+;", 4),
+        (r"=>\s*\w+\(", 4),
+        (r"\basync\s*\{", 3),
+        (r"\bawait\s+", 3),
+        (r"\bsetState\s*\(", 4),
+        (r"\bvoid\s+main\s*\(", 3),
+    ],
+    "scala": [
+        (r"\bcase\s+class\s+\w+", 5),
+        (r"\btrait\s+\w+", 5),
+        (r"\bdef\s+\w+[^:]*:\s*\w+\s*=", 5),
+        (r"\bimport\s+scala\.", 4),
+        (r"\bmatch\s*\{", 4),
+        (r"\bval\s+\w+\s*:\s*\w+\s*=", 4),
+        (r"\bobject\s+\w+", 5),
+        (r"=>", 3),
+        (r"\bdef\s+\w+\[\w+\]", 4),
+        (r"\bextends\s+\w+", 2),
+    ],
+    "elixir": [
+        (r"\bdefmodule\s+[A-Z]", 5),
+        (r"\bdef\s+\w+\s+do\b", 5),
+        (r"\bdefp\s+\w+", 5),
+        (r"\|>", 5),
+        (r"\buse\s+[A-Z]", 4),
+        (r"\balias\s+[A-Z]", 4),
+        (r"#\{", 4),
+        (r"@[\w_]+", 3),
+        (r"\bcase\s+\w+\s+do\b", 3),
+    ],
+    "lua": [
+        (r"\blocal\s+\w+\s*=", 5),
+        (r"\.\.\.(?!\.)", 5),
+        (r"\brepeat\b.*\buntil\b", 5),
+        (r"~=", 4),
+        (r"\belseif\b", 4),
+        (r"\bthen\b", 3),
+        (r"\bfunction\s+\w+\s*\(", 3),
+        (r"\bend\b", 2),
+    ],
+    "perl": [
+        (r"\bmy\s+\$\w+", 5),
+        (r"\buse\s+strict", 5),
+        (r"\buse\s+warnings", 5),
+        (r"\bsub\s+\w+\s*\{", 5),
+        (r"\bchomp\s*\(", 5),
+        (r"@\w+\s*=", 5),
+        (r"%\w+\s*=", 5),
+        (r"\$\w+\s*=~\s*/", 4),
+        (r"\$[0-9]+", 4),
+        (r"->", 3),
+    ],
+
+
     # ===== Markup/Config Languages =====
     "html": [
         (r"<!DOCTYPE\s+html>", 5),
@@ -327,6 +388,28 @@
         (r"#[\w-]+\s*\{", 2),
         (r"@import", 2),
     ],
+    "scss": [
+        (r"\$[\w-]+\s*:", 5),
+        (r"@mixin\s+[\w-]+", 5),
+        (r"@include\s+[\w-]+", 5),
+        (r"@extend\s+", 4),
+        (r"@function\s+[\w-]+", 4),
+        (r"&[:\.]", 4),
+        (r"#\{", 4),
+        (r"@import\s+['\"]", 3),
+        (r"@if\s+", 5),
+        (r"@for\s+", 5),
+        (r"@each\s+", 5),
+    ],
+    "sass": [
+        (r"\$[\w-]+\s*:", 5),
+        (r"=[\w-]+", 5),
+        (r"\+[\w-]+", 5),
+        (r"@for\s+.+\s+through\s+", 5),
+        (r"@mixin\s+[\w-]+", 4),
+        (r"@if\s+", 4),
+        (r"^\s{2,}[\w-]+:", 3),
+    ],
     "json": [
         (r"^\s*\{", 3),
         (r"^\s*\[", 3),


Test patch:
diff --git a/tests/test_pdf_extractor.py b/tests/test_pdf_extractor.py
index c0e321e7..3b240e92 100644
--- a/tests/test_pdf_extractor.py
+++ b/tests/test_pdf_extractor.py
@@ -119,6 +119,195 @@ def test_confidence_range(self):
             self.assertGreaterEqual(confidence, 0.0)
             self.assertLessEqual(confidence, 1.0)
 
+    def test_detect_scss_with_confidence(self):
+        """Test SCSS detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        $primary-color: #3498db;
+
+        @mixin border-radius($radius) {
+          border-radius: $radius;
+        }
+
+        .button {
+          color: $primary-color;
+          @include border-radius(5px);
+
+          &:hover {
+            background: darken($primary-color, 10%);
+          }
+        }
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "scss")
+        self.assertGreater(confidence, 0.8)
+
+    def test_detect_dart_with_confidence(self):
+        """Test Dart detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        import 'package:flutter/material.dart';
+
+        class MyApp extends StatelessWidget {
+          @override
+          Widget build(BuildContext context) {
+            return MaterialApp(
+              home: Text('Hello'),
+            );
+          }
+        }
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "dart")
+        self.assertGreater(confidence, 0.6)
+
+    def test_detect_scala_with_confidence(self):
+        """Test Scala detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        case class Person(name: String, age: Int)
+
+        object Main extends App {
+          val person = Person("Alice", 30)
+          person match {
+            case Person(n, a) if a >= 18 => println(s"Adult: $n")
+            case _ => println("Minor")
+          }
+        }
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "scala")
+        self.assertGreater(confidence, 0.7)
+
+    def test_detect_sass_with_confidence(self):
+        """Test SASS detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        $primary-color: #3498db
+
+        =border-radius($radius)
+          border-radius: $radius
+
+        .button
+          color: $primary-color
+          +border-radius(5px)
+
+          &:hover
+            background: darken($primary-color, 10%)
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "sass")
+        self.assertGreater(confidence, 0.8)
+
+    def test_detect_elixir_with_confidence(self):
+        """Test Elixir detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        defmodule MyApp.User do
+          def greet(name) do
+            "Hello, #{name}"
+          end
+
+          defp calculate_age(birth_year) do
+            2024 - birth_year
+          end
+
+          def process(data) do
+            data
+            |> String.trim()
+            |> String.downcase()
+            |> String.split(",")
+          end
+        end
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "elixir")
+        self.assertGreater(confidence, 0.8)
+
+    def test_detect_lua_with_confidence(self):
+        """Test Lua detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = """
+        local function calculate_sum(numbers)
+          local total = 0
+          for i = 1, #numbers do
+            total = total + numbers[i]
+          end
+          return total
+        end
+
+        local items = {1, 2, 3, 4, 5}
+        local result = calculate_sum(items)
+        print("Sum: " .. result)
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "lua")
+        self.assertGreater(confidence, 0.7)
+
+    def test_detect_perl_with_confidence(self):
+        """Test Perl detection"""
+        extractor = self.PDFExtractor.__new__(self.PDFExtractor)
+        from skill_seekers.cli.language_detector import LanguageDetector
+
+        extractor.language_detector = LanguageDetector(min_confidence=0.15)
+
+        code = r"""
+        #!/usr/bin/perl
+        use strict;
+        use warnings;
+
+        sub process_line {
+          my $line = shift;
+          chomp($line);
+
+          if ($line =~ /^(\w+)=(\w+)$/) {
+            my ($name, $value) = ($1, $2);
+            return "$name has value $value";
+          }
+          return undef;
+        }
+
+        my @lines = ("foo=10", "bar=20");
+        foreach my $line (@lines) {
+          my $result = process_line($line);
+          print $result if defined $result;
+        }
+        """
+
+        language, confidence = extractor.detect_language_from_code(code)
+        self.assertEqual(language, "perl")
+        self.assertGreater(confidence, 0.8)
+
 
 class TestSyntaxValidation(unittest.TestCase):
     """Test syntax validation for different languages"""
@@ -315,7 +504,11 @@ def test_merge_continued_blocks(self):
             {
                 "page_number": 1,
                 "code_samples": [
-                    {"code": "def hello():", "language": "python", "detection_method": "pattern"}
+                    {
+                        "code": "def hello():",
+                        "language": "python",
+                        "detection_method": "pattern",
+                    }
                 ],
                 "code_blocks_count": 1,
             },
@@ -346,7 +539,11 @@ def test_no_merge_different_languages(self):
             {
                 "page_number": 1,
                 "code_samples": [
-                    {"code": "def foo():", "language": "python", "detection_method": "pattern"}
+                    {
+                        "code": "def foo():",
+                        "language": "python",
+                        "detection_method": "pattern",
+                    }
                 ],
                 "code_blocks_count": 1,
             },