leandersen/ny-solar-siting-rag

NY Solar Siting RAG

A retrieval-augmented regulatory assistant for early-stage utility-scale solar site evaluation in New York State. Ask it a developer's question about a candidate site (permitting track, deal-killers, statutory deadlines, agricultural mitigation) and it answers only from a curated corpus of current NY statutes, regulations, and agency guidance, citing every claim in [Document, Section] form. When the corpus can't support an answer, it declines and points to the right source instead.

Introduction

Early-stage solar developers triage many candidate sites with incomplete information, deciding which ones deserve real diligence dollars. Getting a New York regulatory question wrong at this stage is expensive in both directions: missing a fatal flaw (a preempted permit assumption, an agricultural-mitigation payment, an interconnection dead-end) wastes months, while a hallucinated obstacle kills a viable site. General-purpose LLMs struggle here for a structural reason. New York reorganized its renewable siting law with the 2024 RAPID Act, which repealed the prior siting statute, so any model trained before then answers from repealed law, and the problem repeats with every revision cycle (some documents were even revised mid-build). I built a RAG pipeline rather than fine-tuning because the task is knowledge retrieval over changing documents. The corpus can be patched the same day a rule changes, while a fine-tuned model's knowledge is frozen at its training cutoff. The system pairs a bge-base / FAISS retriever with Qwen3-4B-Instruct-2507 under a strict grounding prompt. On my 75-item test set, retrieval surfaces a gold source document for 87.7% of items, and grounding collapses the fabrication rate on unanswerable questions from 94.4% (the same model answering without retrieval) to 5.6%. RAG also lifts Housing Statute QA accuracy from a no-context coin flip (0.500) to 0.632. Against two comparison models of similar size run through the identical retriever, this configuration has both the lowest fabrication rate and the lowest rate of refusing answerable questions; the comparison models fabricate at twice the rate and decline more often.

Data

Corpus. The knowledge base is 18 documents (~216,000 words), all official New York State sources, organized into six document families that serve as both retrieval metadata and evaluation strata: siting (SIT: PSL Article 8, 16 NYCRR Parts 1100/1101, ORES FAQ), interconnection (INT: the Standardized Interconnection Requirements), incentives (INC: NY-Sun Program Manual, VDER summary), agricultural (AGR: NYSAGM construction-mitigation guidelines), environmental (ENV: 6 NYCRR Part 617, SEQR), and local (see Limitations). PDFs were extracted with pypdf; the ORES FAQ was scraped to one chunk per Q&A pair; each statute section was fetched individually via the OpenLegislation API so statute chunks align exactly with citable provisions. Three documents are kept on disk but excluded from the index as recorded decisions in the manifest (sources.csv, which tracks title, family, effective date, access date, and source URL for every document). 16 NYCRR Part 1102 and PSL §§ 141/143 were excluded because they govern electric transmission siting in language that mirrors the solar provisions and would compete with them in retrieval. The 131k-word NYSERDA Solar Guidebook was excluded because at 38% of the corpus by volume its model-law template chapters crowded out the governing regulations (its agricultural chapter is retained as a standalone document).

Test cases. No public benchmark covers NY solar permitting, so I built a 75-item evaluation set by reverse generation: sample a real corpus chunk, generate a realistic developer question that chunk answers plus an ideal cited answer, and verify every gold quote word-for-word against the source. Items are stratified three ways. Answerable items (45) are ones the corpus states the answer to. Unanswerable items (18) are ones the corpus knowably cannot answer, such as town-specific zoning, where the correct behavior is an explicit decline naming the authoritative source. Derivable items (12) are ones where the answer follows from applying a rule in the corpus but is never stated in words, for example what a 25 MW statutory threshold implies for a 40 MW project. The three strata measure different failures: wrong answers, fabrication (over-answering), and refusal of supported inference (over-declining). A practicing solar-development professional reviewed the set for face validity and realism; item-level adjudication was not performed.

Methodology

The pipeline is kept simple so failures are easy to attribute. Documents are split with recursive character splitting (1,500 characters, 150 overlap, with a separator hierarchy of \n## headers, then paragraphs, then sentences), which gives the FAQ true per-Q&A chunks and keeps statute sections intact but splits long regulations on character count rather than section boundaries. This is a disclosed deviation from the original design, and citations are therefore document-level plus quoted text rather than clause-level. Chunks are embedded with BAAI/bge-base-en-v1.5 (normalized, cosine similarity via FAISS, k=5, the BGE query prefix on the query side). I tested bge-large and Qwen3-Embedding-0.6B in a combination grid: added capacity within the same training recipe actually hurt source-level retrieval, and the Qwen embedder traded away source-level hits that the citation scoring depends on, so bge-base was retained. The generator, Qwen3-4B-Instruct-2507, was selected in a four-model bake-off (0.5B/1B/4B/7B) on held-out items. The 4B was the only model that both answered correctly from context and declined cleanly on the unsupported half of a mixed question, and its instruction-following (IFEval 83.4) is the capability this task's cite-and-decline format most depends on. Generation is greedy (do_sample=False, 400 max new tokens) under a strict containment preamble: answer only from the retrieved passages, cite every claim, decline explicitly otherwise. I tested relaxing that rule to permit reasoning from stated thresholds. Five other interventions failed identically first, which isolated the preamble as the cause, and the relaxed version then produced a directionally wrong preemption answer on some runs, trading a safe failure (an unnecessary decline) for an unsafe one (a confident wrong answer about who issues permits). So the strict preamble stayed. No fine-tuning was used: the failures observed in smaller models (verbatim example-copying, fabricated checklists) got worse with few-shot examples rather than better, which is not the failure profile fine-tuning fixes.

Evaluation

Model	Own test cases: over-answer / over-decline / gold-citation	Housing Statute QA (NY): accuracy	MTRAG (govt): decline on unans. / ROUGE-L ans.	LegalBench-RAG (mini): snippet recall@5
This pipeline (Qwen3-4B + RAG)	0.056 / 0.175 / 0.474	0.632	0.750 / 0.181	0.226*
Base model (Qwen3-4B, no retrieval)	0.944 / 0.018 / 0.368	0.500	0.000 / 0.149	—
Llama-3.2-3B-Instruct + same retriever	0.111 / 0.263 / 0.439	0.584	0.750 / 0.185	0.226*
Qwen2.5-7B-Instruct + same retriever	0.111 / 0.281 / 0.351	0.574	0.667 / 0.195	0.226*

Lower is better for over-answer and over-decline; higher is better for every other column.

* Shared across all RAG rows: LegalBench-RAG's metrics score the retrieval stage, which is identical when only the generator is swapped. The same holds for the other retrieval-side figures for every RAG row: gold-source hit rate on my test cases is 0.877, MTRAG passage recall@5 is 0.333, and Housing Statute QA citation-normalized retrieval hit is 0.284.

I chose the three external benchmarks to triangulate the two capabilities this system needs. LegalBench-RAG (mini protocol: 194 queries per sub-benchmark, seed 0, per the paper's own subsampling) tests whether retrieval finds the precise governing clause in legal text rather than something nearby, which is the core difficulty of statutory QA. Housing Statute QA (filtered to New York questions and statutes, the task-relevant population, run complete within that filter) is structurally this project's job, statutory yes/no questions against a state's law, and shows the pipeline generalizes beyond its own corpus. MTRAG (govt domain) contains labeled unanswerable questions, directly measuring the willingness to say "I don't know" that my unanswerable stratum is built around. As comparison models I chose Llama-3.2-3B-Instruct (the nearest similar-size instruct model from a different model family, to check whether results are Qwen-specific) and Qwen2.5-7B-Instruct (the 4B's larger sibling, isolating scale within a family, and the model whose fabrication behavior in the bake-off motivated the choice of the 4B). Both run through the identical retriever and prompt, so every difference in the table is attributable to the generator.

The headline result is that grounding works: retrieval collapses the base model's fabrication rate on unanswerable questions from 0.944 to 0.056 and lifts Housing Statute QA from 0.500 to 0.632. The selected 4B is also the best generator on the task's own error axes. It fabricates at half the comparison models' rate (0.056 vs 0.111), refuses answerable questions least (0.175 vs 0.263/0.281), and cites the gold source most often (0.474), while the external benchmarks show the comparison models are competent rather than broken (the 7B even edges ahead on MTRAG ROUGE-L). The gap on my test set therefore reflects grounding discipline more than general capability. Finally, in the family-controlled comparison, the larger model turned out more cautious rather than more capable: on the derivable stratum (answers that follow from a stated rule but are never stated in words), the Qwen 7B refuses half of the supported inferences its 4B sibling completes, with over-decline of 50% vs 25% and the cross-family Llama-3B between them at 33%. This replicates at benchmark scale what the bake-off first showed on two items.

Usage and Intended Uses

This is a pipeline release, not a fine-tuned checkpoint: the repo documents the retriever + generator configuration, the corpus manifest, and the evaluation set. The scripts in scripts/ read their paths and model settings from a .env file — copy example.env to .env and point it at your own data, cache, and benchmark locations. Both models load directly from the Hub:

import numpy as np, torch
from sentence_transformers import SentenceTransformer
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import glob, os
import pandas as pd
from pypdf import PdfReader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# --- corpus -> chunks ------------------------------------------------------
read = lambda p: ("\n".join(pg.extract_text() or "" for pg in PdfReader(p).pages)
                  if p.endswith(".pdf") else open(p).read())

m = pd.read_csv("sources.csv")
files = []
for f in m[m.ingest.astype(str).str.lower() == "true"].filename:
    files += sorted(glob.glob(f"RAG Data/{f}*.txt")) if f.endswith("/") else [f"RAG Data/{f}"]
files = [f for f in files if not f.endswith(("s141.txt", "s143.txt")) and "Guidebook" not in f]

splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150,
                                          separators=["\n## ", "\n\n", "\n", ". ", " "])
chunks = [{"text": c, "source": os.path.basename(f)}
          for f in files for c in splitter.split_text(read(f))]

# --- retriever ------------------------------------------------------------
embedder = SentenceTransformer("BAAI/bge-base-en-v1.5")
PREFIX = "Represent this sentence for searching relevant passages: "

mat = embedder.encode([c["text"] for c in chunks], normalize_embeddings=True)

def retrieve(query, k=5):
    q = embedder.encode([PREFIX + query], normalize_embeddings=True)[0]
    idx = np.argsort(-(mat @ q))[:k]
    return [chunks[i] for i in idx]

# --- generator ------------------------------------------------------------
name = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(name)
mdl = AutoModelForCausalLM.from_pretrained(name, device_map="auto",
                                           dtype=torch.bfloat16)
llm = pipeline("text-generation", model=mdl, tokenizer=tok)

SYSTEM_PREAMBLE = (
    "You are a regulatory assistant for early-stage solar development in "
    "New York State. Answer the developer's question using ONLY the context "
    "passages below. Cite the supporting document and section for every "
    "claim in the form [Document, Section]. If the context does not contain "
    'the answer, reply: "The provided sources do not answer this question" '
    "and briefly say what source likely would."
)

def answer(question, k=5):
    hits = retrieve(question, k)
    context = "\n\n".join(f'[{i+1}] ({h["source"]}): "{h["text"]}"'
                          for i, h in enumerate(hits))
    prompt = f"{SYSTEM_PREAMBLE}\n\nContext:\n{context}\n\nQuestion: {question}"
    out = llm([{"role": "user", "content": prompt}],
              max_new_tokens=400, do_sample=False, return_full_text=False)
    return out[0]["generated_text"]

Intended uses: early-stage screening and triage of New York utility-scale solar sites: identifying the applicable permitting track, surfacing likely deal-killers, and pointing at the governing document to read next. It is a research/repository project and a diligence starting point. It is not legal advice, not a substitute for counsel or for reading the cited sources, and not suitable for compliance decisions, filings, or any use where an uncaught error is costly. Out of scope by design: states other than New York, local zoning specifics, NYISO interconnection procedures, and timeline prediction.

Prompt Format

The prompt is a strict-grounding system preamble, followed by the retrieved passages as numbered context blocks, followed by the question, all sent as a single user message through the model's chat template:

You are a regulatory assistant for early-stage solar development in New York
State. Answer the developer's question using ONLY the context passages below.
Cite the supporting document and section for every claim in the form
[Document, Section]. If the context does not contain the answer, reply: "The
provided sources do not answer this question" and briefly say what source
likely would.

Context:
[1] (psl_a8_s142.txt): "§ 142. Application and review of siting permits..."
[2] (ores_faq.md): "## Q: How long does the Office have to review..."

Question: ORES has had my completed application for 70 days with no
completeness determination. Where does that leave me?

Expected Output Format

Answers are short prose with an inline [Document, Section] citation for every claim. When the corpus cannot support an answer, the model declines in a fixed phrase and names the source that would have it. The two canonical shapes:

Under PSL § 142(1), the office must make a completeness determination within
sixty days of filing [PSL Article 8, § 142]. If the office does not act
within that period, the application is deemed complete by operation of law
[PSL Article 8, § 142]. Your application is therefore deemed complete, and
the review clock has begun.

The provided sources do not answer this question. Setback distances for
ground-mounted solar in a specific town are set by that town's local solar
law or zoning code, which is not part of my sources. The Town of Batavia's
zoning office or its published solar law would be the authoritative source.

Limitations

The most important limitation is designed in: local zoning is excluded from the corpus. For a sub-25 MW project the tool cannot answer the questions that most often decide the site (setbacks, moratoria, permitted-use status) and can only say that the town decides and point to the town. Hundreds of non-uniform municipal codes were not collectible within the project timeline, and the LOC family exists chiefly so unanswerable questions have a clean decline target.

Retrieval is also scope-blind: nothing in the pipeline knows a query's project size, so on a 40 MW question a chunk from an incentive program that caps far below 40 MW can outrank the governing preemption statute. This is the clearest argument for a size/track routing stage as future work.

On prohibition-with-exception statutes (e.g. PSL § 144(2) preemption), the strict grounding rule produces over-declines on derivable questions, 25% of that stratum even for the selected model, and my testing of a relaxed rule showed the alternative is worse: in earlier sessions the 4B's reading of such clauses flipped direction between runs, and an unstable preemption answer is more dangerous for a diligence tool than a stable refusal. A repeated-trial check (one greedy plus five sampled runs at temperature 0.7 per item) found the deployed configuration stable on both probes. The 40 MW preemption question answered with the correct direction, no town permit, in 6 of 6 runs, and an 8 MW "who approves it" question declined-with-pointer in 6 of 6. Two caveats apply. The stable correct answer arrives via the ORES FAQ's explicitly stated 25 MW jurisdiction threshold rather than by inference from § 144(2) itself (the statute did not surface in the top-5 for this phrasing), so the harder inference pathway where instability was originally observed went unexercised. And the 8 MW runs show the cost of the strict rule: the model quotes the retrieved sentence stating most projects "are reviewed and approved at the local level" while declining to assert it as the answer. That is a stable, predictable failure, and it is the trade this design accepts.

PDF extraction flattens fee schedules and threshold tables (exactly the content developers want), and a unit-vocabulary gap (queries say "MW", PSL § 145 says "one thousand dollars for each thousand kilowatts") can defeat retrieval on fee questions. Query-side unit normalization is identified but untested.

One staleness case is live in this very corpus. The ingested Part 617 is the January 2019 edition, and DEC amendments adopted April 24, 2026 took effect on June 12, 2026, weeks before this corpus was downloaded. Among other changes, the amendments codify the SEQR exclusion for ORES, which is directly relevant to this tool's domain. The superseded text remains in the v1 index because re-ingesting would invalidate the evaluation runs above, so it is disclosed here, flagged in the manifest, and first in line for corpus maintenance.

The VDER summary is a 2017 document, the oldest source in a corpus whose premise is currency. It is retained as background with its date recorded in the manifest, and replacing it is future work. Effective dates for all sources are tracked in sources.csv because guidance staleness is a real failure mode for this tool.

Several single-run observations from model selection (which model fabricates where) rest on n=1 and are reported as observations rather than results; the flip-rate protocol above is the start of fixing that. Finally, statutory clocks quoted from the corpus (60 days, one year) are legal deadlines, not predictions. The tool deliberately refuses real-world timeline estimation, which depends on queue dynamics no document captures.

---

Corpus manifest: sources.csv. Evaluation set: eval_set_v1.jsonl. Eval code: scripts/. Per-run metrics for every number in the table: results/. Corpus/scrape tooling: build_corpus.py. Author: Lucas Andersen, DS 5002 (How to Train Your LLM), University of Virginia, August 2026.