Guidebook.md

AI Recruiting Assistant — Guide Book (Updated)

0) Overview

What this tool does

This AI Recruiting Assistant is a decision-support system that helps recruiters and hiring managers:

• Extract job requirements from a job description (JD)
• Evaluate resumes against verified requirements using evidence-based matching
• Assess job-relevant culture/working-style signals using retrieved company documents
• Run factuality checks to detect ungrounded claims
• Run a bias & fairness audit across the JD, analyses, and the model’s final recommendation

The problem it addresses

Recruiting teams often face three recurring issues when using AI:

1. Hallucinated requirements: LLMs may “invent” skills that are not explicitly required.
2. Opaque scoring: Many tools produce fit scores without clearly showing evidence.
3. Bias risks: Hiring language and reasoning can leak pedigree/class proxies or subjective criteria.

This tool addresses those issues by enforcing:

• Deterministic verification gates (requirements are verified before scoring)
• Evidence-backed scoring (only verified requirements are scored; each match includes a quote)
• Self-verification and self-correction (factuality checks can trigger automatic revision)
• Bias auditing (flags risky language and inconsistent standards)

How it differentiates from typical recruiting tools

Compared with “black-box” resume screeners or generic LLM chatbots, this system emphasizes:

• Transparency: Outputs include what was required, what was verified, what was dropped, and why.
• Auditability: The scoring math is deterministic and traceable to inputs.
• Self-verifying behavior: Claims are checked against source text; unverified claims can be removed.
• Bias checks by design: Bias-sensitive content is audited explicitly instead of implicitly influencing scores.
• Culture check that’s job-performance aligned: Culture attributes are framed as job-relevant behaviors, not background proxies.

---

1) Inputs and Document Handling

1.1 What the user uploads

The tool operates on three inputs:

1. Company culture / values documents (PDF/DOCX)
2. Resumes (PDF/DOCX)
3. Job description (pasted text)

1.2 Resume anonymization

Before resumes are stored or analyzed, the tool applies heuristic redaction:

• Emails, phone numbers, URLs
• Addresses / location identifiers
• Explicit demographic fields
• Likely name header (first line)

This reduces exposure of personal identifiers and keeps analysis focused on job evidence.

1.3 Vector stores (retrieval)

The tool maintains two separate Chroma collections:

• Resumes (anonymized + chunked)
• Culture docs (chunked)

Chunking uses a recursive splitter with overlap to preserve context.

---

2) End-to-End Logic Flow (Step-by-Step)

Below is the stepwise flow executed when a recruiter clicks Analyze Candidates.

Step 0 — Prerequisite: Documents exist in storage

• Culture docs and resumes must be stored first.
• If not stored, retrieval will be empty or low-signal.

Step 1 — Extract required skills from the Job Description (LLM-driven)

Goal: Identify only skills that are explicitly required.

• The tool prompts the LLM to return JSON only:

  • required_skills: [{skill, evidence_quote}]

• The LLM is instructed to:

  • include only MUST HAVE / explicitly required skills
  • exclude “nice-to-haves” and implied skills
  • copy a short verbatim quote as evidence

LLM role: structured extraction.

Failure behavior: If JSON parsing fails, the tool stops and prints the raw output.

Step 2 — Verify extracted skills against the JD (deterministic, Python)

Goal: Block hallucinated requirements from entering scoring.

Each extracted item is classified:

• Quote-verified (strong): the evidence quote appears verbatim in the JD
• Name-only (weak): the skill name appears in the JD, but the quote doesn’t match
• Unverified (dropped): neither quote nor name appears

Deterministic gate:

• Only quote-verified skills are used as the final required list for scoring.
• Name-only and dropped skills are reported for transparency.

Output: “Requirements Verification” section shows:

• extracted count
• quote-verified vs name-only vs dropped
• list of skills used for scoring
• list of retracted/dropped items (with reason)

Step 3 — Retrieve the most relevant culture chunks (deterministic retrieval)

Goal: Ground culture evaluation in actual company documents.

• The tool runs similarity search over culture docs using the JD as query.
• It selects the top k chunks (e.g., k=3).

Deterministic component: vector retrieval parameters.

Output artifact: culture_context is the concatenated text of retrieved culture chunks.

Step 4 — Generate job-performance culture attributes (LLM-driven)

Goal: Create a small set of job-relevant behavioral attributes to evaluate consistently.

• The tool prompts the LLM to return JSON:

  • cultural_attributes: ["...", "..."] (4–6 items)

Attribute rules:

• Must be job-performance aligned behaviors (e.g., “evidence-based decision making”).
• Must avoid pedigree / class / prestige language.
• Must avoid non-performance preferences (e.g., remote-first, time zone).

LLM role: label generation from retrieved culture context.

Step 5 — Retrieve top resume chunks for the JD (deterministic retrieval)

Goal: Identify the most relevant candidates and their relevant resume text.

• The tool runs similarity search over resumes using the JD.
• It retrieves top k chunks (e.g., k=10) and groups them by resume_id.

Note: Only retrieved chunks are analyzed. If relevant evidence isn’t retrieved, it may be missed.

Step 6 — Culture evidence matching per candidate (LLM + deterministic cleanup + deterministic scoring)

Goal: Determine which culture attributes are supported by resume evidence.

LLM-driven matching:

• For each attribute, the LLM may return a match with:

  • evidence_type: direct or inferred
  • evidence_quotes: 1–2 verbatim resume quotes
  • inference: required for inferred
  • confidence: 1–5

Deterministic cleanup rules (Python): A match is kept only if:

• attribute is present
• evidence_type is direct or inferred
• at least one non-trivial quote exists
• confidence is an integer 1–5
• inferred matches include an inference sentence
• inferred matches can be required to meet a minimum confidence

Deterministic culture scoring (Python):

• Direct evidence weight: 1.0
• Inferred evidence weight: 0.5

Culture score is computed as:

• (sum(weights for matched attributes) / number_of_required_attributes) * 100

Step 7 — Skills matching per candidate (LLM + deterministic scoring)

Goal: Match only the verified required skills to resume evidence.

Inputs:

• Candidate resume text (retrieved chunks)
• Verified required skills list (quote-only)

LLM output (JSON):

• matched: [{skill, evidence_snippet}]
• missing: [skill] (treated as advisory; missing is recomputed deterministically)

Deterministic missing calculation (Python):

• Missing = required_set − matched_set

Deterministic skills scoring (Python):

• (number_of_matched_required_skills / number_of_required_skills) * 100

Step 8 — Implied competencies (NOT SCORED) for phone-screen guidance (LLM-driven, advisory)

Goal: When a required skill is missing explicitly, suggest whether it may be implied by adjacent evidence.

• This step is not scored and does not affect proceed/do-not-proceed.

• The LLM may suggest implied competencies only if it:

  • uses conservative language (“may be implied”)
  • includes verbatim resume quotes
  • provides a phone-screen validation question

Hard guardrail: Tool-specific skills (e.g., R/SAS/MATLAB) must be explicitly present in the resume to be suggested.

Step 9 — Factuality verification (LLM-driven verifier)

Goal: Detect ungrounded evidence claims.

• The verifier checks evidence-backed match lines (e.g., - Skill: snippet).

• It ignores:

  • numeric score lines
  • missing lists
  • policy text

Outputs:

• verified claims (✓)
• unverified claims (✗)
• factuality score

Step 10 — Final recommendation (LLM, policy-constrained)

Goal: Produce a structured recommendation without changing scores.

• The model is given:

  • skills analysis
  • culture analysis
  • fixed computed scores
  • deterministic decision policy

Decision policy:

• If skills_score ≥ 70 → PROCEED
• If skills_score < 60 → DO NOT PROCEED
• If 60 ≤ skills_score < 70 → PROCEED only if culture_score ≥ 70 else DO NOT PROCEED

Non-negotiables:

• LLM must not re-score.
• LLM must not introduce new claims.

Step 11 — Self-correction (triggered by verification issues)

Goal: Remove/correct any unverified claims while preserving scores/policy.

• If any unverified claims exist:

  • The tool asks the LLM to revise the recommendation
  • Only the flagged claims may be removed/corrected
  • Scores and policy must remain unchanged

Step 12 — Bias audit (LLM-driven audit across docs + reasoning)

Goal: Flag biased reasoning, biased JD language, or inconsistent standards.

Audit scope includes:

• Job description
• Skills analysis
• Culture analysis
• Final recommendation text
• Culture context

What it flags (examples):

• Prestige/pedigree signals (elite employers/education as proxy)
• Vague “polish/executive presence” language not tied to job requirements
• Non-job-related culture screening
• Inconsistent standards (penalizing requirements not in JD)
• Overclaiming certainty

Outputs:

• structured list of bias indicators (category, severity, trigger text, why it matters, recommended fix)
• recruiter guidance

---

3) Scoring and Decision Rules (Deterministic)

3.1 Skills score

• Only quote-verified required skills count.
• Score = matches / required.

3.2 Culture score

• Score = weighted matches / attributes.
• Direct = 1.0; inferred = 0.5.

3.3 Labels

• ≥70: Strong fit
• 50–69: Moderate fit
• <50: Not a fit

3.4 Recommendation

Recommendation follows the fixed policy described in Step 10.

---

4) System Flow Diagram (Textual)

Below is a simplified, end-to-end flow of how data moves through the system.

[User Uploads]
   |
   v
+-------------------+
| Culture Documents |
+-------------------+        +-----------+
           |                 | Job Desc  |
           v                 +-----------+
+-------------------+               |
| Culture Vector DB |<--------------+
+-------------------+               |
           |                        v
           |               +---------------------+
           |               | Skill Extraction    |
           |               | (LLM, JSON Output)  |
           |               +---------------------+
           |                        |
           |                        v
           |               +---------------------+
           |               | Requirement         |
           |               | Verification        |
           |               | (Deterministic)     |
           |               +---------------------+
           |                        |
           |                        v
           |               Verified Required Skills
           |                        |
           |                        v
+-------------------+        +---------------------+
| Resume Documents  |------->| Resume Vector DB    |
+-------------------+        +---------------------+
                                   |
                                   v
                           Similarity Search (k=10)
                                   |
                                   v
                           Resume Chunks (Grouped)
                                   |
                                   v
                     +-----------------------------+
                     | Culture Attribute Generator |
                     | (LLM, JSON Output)          |
                     +-----------------------------+
                                   |
                                   v
                     +-----------------------------+
                     | Culture Evidence Matching   |
                     | (LLM + Rules + Weights)     |
                     +-----------------------------+
                                   |
                                   v
                     Culture Score (Deterministic)
                                   |
                                   v
                     +-----------------------------+
                     | Technical Skill Matching    |
                     | (LLM + Deterministic Scoring)|
                     +-----------------------------+
                                   |
                                   v
                     Skills Score (Deterministic)
                                   |
                                   v
                     +-----------------------------+
                     | Implied Competencies (LLM)  |
                     | (Not Scored, Advisory)      |
                     +-----------------------------+
                                   |
                                   v
                     +-----------------------------+
                     | Factuality Verification     |
                     | (LLM Verifier)              |
                     +-----------------------------+
                                   |
                                   v
                     +-----------------------------+
                     | Recommendation Generator    |
                     | (Policy-Constrained LLM)    |
                     +-----------------------------+
                                   |
                                   v
                     +-----------------------------+
                     | Bias & Fairness Audit        |
                     | (LLM Audit)                 |
                     +-----------------------------+
                                   |
                                   v
                           Final Recruiter Report

---

5) Audit Artifacts and Traceability

For every analysis run, the system produces and retains multiple audit artifacts that enable post-hoc review, regulatory defensibility, and debugging.

5.1 Input Artifacts

1. Original Job Description

   • Full pasted JD text

2. Sanitized Resume Text

   • Redacted resume content
   • Redaction summary (internal)

3. Retrieved Culture Chunks

   • Top-k (default: 3) culture document segments
   • Vector similarity scores (internal)

4. Retrieved Resume Chunks

   • Top-k (default: 10) resume segments
   • Resume ID metadata

---

5.2 Requirement Verification Artifacts

1. Raw LLM Skill Extraction Output

2. Parsed Required Skills JSON

3. Verification Classification Table

   • Quote-verified
   • Name-only
   • Dropped

4. Dropped-Skill Justifications

---

5.3 Culture Analysis Artifacts

1. Generated Culture Attribute List

2. LLM Raw Matching Output

3. Cleaned Match Records

   • Evidence type
   • Quotes
   • Inference
   • Confidence

4. Weighted Match Table

5. Computed Culture Score

---

5.4 Skills Analysis Artifacts

1. Verified Required Skill List
2. LLM Raw Matching Output
3. Accepted Matched Skills
4. Deterministic Missing-Skill Set
5. Computed Skills Score

---

5.5 Implied Competency Artifacts (Advisory)

1. Missing Skill List

2. LLM Implied Output (JSON)

3. Accepted Implied Records

   • Resume quotes
   • Explanation
   • Phone-screen questions

4. Rejected Inferences (internal)

---

5.6 Verification and Correction Artifacts

1. Verifier Prompt and Output
2. Verified / Unverified Claim Lists
3. Factuality Scores
4. Self-Correction Prompts and Revisions (if triggered)

---

5.7 Recommendation and Policy Artifacts

1. Final Recommendation Prompt
2. Policy Threshold Snapshot
3. Immutable Score Values
4. Generated Recommendation Text

---

5.8 Bias Audit Artifacts

1. Bias Audit Prompt
2. Audit Input Bundle (JD + Analyses + Recommendation)
3. Structured Bias Indicator List
4. Severity and Mitigation Suggestions
5. Recruiter Guidance Text

---

5.9 System Metadata

1. Timestamp of run
2. Model version
3. Prompt versions
4. Chunking parameters
5. Retrieval k-values
6. Scoring parameters

---

6) Known Limitations

1. Retrieval scope: evaluation depends on retrieved chunks; some evidence may be missed.
2. Attribute generation variance: culture attributes can vary per run unless cached or cataloged.
3. LLM evidence overreach: mitigated by verification and cleanup, but not eliminated.
4. Bias audit is advisory: it flags issues; it does not enforce policy changes unless you add an auto-rewrite step.

---

6) Governance and Change Control

• Prompt changes must preserve JSON contracts.
• Any change that affects scoring or policy should be versioned.
• Audit outputs should be retained for traceability.

---

7) Intended Use

This tool is built for:

• faster, evidence-based screening
• transparent reasoning
• safer use of LLMs via verification and audits

It is not a substitute for:

• human judgment
• legal review
• formal HR policy compliance

---

High-level pipeline (inputs → outputs)

Inputs uploaded by recruiter

1. Company culture/values docs (PDF/DOCX)
2. Resumes (PDF/DOCX)
3. Job description (text)

⬇️

Indexing (deterministic, Python)

• Culture docs → chunk + embed → culture_store
• Resumes → anonymize → chunk + embed → resume_store

⬇️

Candidate assessment (per JD run)

1. Extract required skills (LLM) → JSON required_skills[{skill,evidence_quote}]

2. Verify extracted skills (Python) → quote-verified / name-only / dropped → quote-only list used for scoring

3. Retrieve relevant culture context (deterministic retrieval)

• Query: JD
• Retrieve: top-k culture chunks (current: k=3)
• Output: culture_context

4. Generate job-relevant culture attributes (LLM) → JSON cultural_attributes[4–6]

5. Retrieve relevant resume chunks (deterministic retrieval)

• Query: JD
• Retrieve: top-k resume chunks (current: k=10)
• Group by resume_id

6. Per candidate: culture matching (LLM → cleanup → deterministic score)

• LLM proposes matches (direct/inferred) + quotes
• Python enforces validity gates
• Deterministic weighted culture score (direct=1.0, inferred=0.5)

7. Per candidate: skills matching (LLM → deterministic score)

• LLM proposes matched skills + evidence snippets
• Python recomputes missing list deterministically
• Deterministic skills score using quote-verified requirements only

8. Per candidate: implied competencies (LLM, NOT SCORED)

• Inputs: missing skills + matched skills + resume + JD
• Output: implied items with quotes + phone-screen questions
• Guardrail: tool-like skills (R/SAS/MATLAB) require explicit mention

9. Factuality verification (LLM verifier) → ✓/✗ for evidence-backed match lines + factuality score

10. Recommendation (LLM, policy constrained) → uses fixed scores + fixed decision policy

11. Self-correction (conditional) → triggered if any unverified claims exist

12. Bias audit (LLM) → audits JD + analyses + recommendation → structured bias indicators + guidance

⬇️

Outputs per candidate

• Requirements verification summary (global)
• Culture analysis + score
• Skills analysis + score
• Implied (not scored) follow-ups
• Fact-check results
• Final recommendation (+ revision note if corrected)
• Bias audit

---

Component map (LLM vs deterministic)

LLM-driven components

• Required skill extraction (JSON)
• Culture attribute generation (JSON)
• Culture match proposals (JSON)
• Skills match proposals (JSON)
• Implied (not scored) follow-ups (JSON)
• Factuality verification (✓/✗)
• Final recommendation (policy constrained)
• Bias audit (structured)

Deterministic / Python-enforced components

• Resume anonymization
• Chunking + embedding + storage
• Retrieval parameters (top-k)
• Required-skill verification (quote/name-only/dropped)
• Deduplication of requirements
• Culture match cleanup rules (validity gates)
• Skills missing list recomputation
• Skills score computation
• Culture score computation with weights
• Decision thresholds (proceed / do not proceed)
• Self-correction trigger (presence of unverified claims)

---

Audit Artifacts

This section lists the primary artifacts produced (or recommended to persist) to make runs reviewable and defensible.

Inputs (source-of-truth)

• Job description text (as provided)
• Culture documents (original files)
• Resumes (original files)

Pre-processing

• Sanitized resume text (post-anonymization)
• Redaction notes (what was removed/masked)
• Chunking configuration (chunk_size, chunk_overlap)
• Embedding configuration (embedding model + settings)

Retrieval

• Culture retrieval query: JD text
• Culture retrieved chunks: top-k (current: k=3)
• Resume retrieval query: JD text
• Resume retrieved chunks: top-k (current: k=10)
• Candidate grouping: chunks grouped by resume_id

Requirements verification

• LLM required_skills JSON (raw)

• Normalized required skill list (deduped)

• Verification output:

  • quote-verified list
  • name-only list
  • dropped/unverified list
  • counts and factuality score

• Final scoring-required list: quote-verified only

Per-candidate analyses

Culture analysis

• Raw LLM culture-match JSON
• Post-cleanup matched culture list
• Missing culture attributes list
• Culture score + label
• Culture evidence lines shown to recruiters

Skills analysis

• Raw LLM skills-match JSON
• Matched skills list (with evidence snippets)
• Deterministically computed missing skills list
• Skills score + label

Implied (NOT SCORED)

• Raw LLM implied JSON
• Filtered implied list (must include resume quotes + phone-screen questions)

Verification & correction

• Verifier raw output (✓/✗ lines)
• Verified claims list
• Unverified claims list
• Factuality score
• Self-correction trigger status (yes/no)
• Corrected recommendation (if triggered) + revision note

Bias audit

• Bias audit raw output (structured)
• Bias indicators list (category, severity, trigger_text, why_it_matters, recommended_fix)
• Overall assessment
• Recruiter guidance

Run-level trace (recommended)

For reproducibility/governance, also persist:

• Timestamp, model name, temperature, seed
• Prompt versions (hash or version ID)
• Retrieval parameters (k values)
• Score thresholds and policy version
• Any configuration overrides used during the run

End-to-End Pipeline (Swim-Lane View)

Step	Recruiter / Input	Python / Deterministic Logic	LLM (Groq)	Storage / Output
1	Upload culture documents	Chunk + embed	—	culture_store (indexed)
2	Upload resumes	Anonymize → chunk → embed	—	resume_store (indexed)
3	Paste JD + Run	Send JD to LLM	Extract required skills + evidence quotes	required_skills JSON
4	—	Verify requirements (quote / name-only / dropped)	—	Verified list + debug report
5	—	Retrieve culture context (k=3)	—	culture_context
6	—	—	Generate culture attributes (job-performance aligned)	cultural_attributes JSON
7	—	Retrieve resume chunks (k=10), group by resume_id	—	Candidate chunks
8	—	—	Propose culture matches (direct/inferred + quotes)	Raw culture-match JSON
9	—	Cleanup + weighted scoring (direct=1.0, inferred=0.5)	—	Culture score + evidence
10	—	—	Propose skill matches + evidence snippets	Raw skills-match JSON
11	—	Compute missing list + skills score (verified reqs only)	—	Skills score + missing list
12	—	—	Infer implied skills (NOT SCORED) + phone questions	Implied follow-ups
13	—	—	Verify evidence (✓/✗)	Factuality report
14	—	—	Generate recommendation (policy constrained)	Final recommendation
15	—	Trigger self-correction (if needed)	Revise flagged claims only	Corrected recommendation
16	—	—	Run bias audit (JD + analyses + decision)	Bias indicators + guidance
17	Review output	Assemble final report	—	Full candidate report

Current Retrieval Parameters

• Culture store: k = 3 chunks (JD query)
• Resume store: k = 10 chunks (JD query)