improve documentation

planning_ai/chains/prompts/extract.txt

@@ -1 +0,0 @@
-Extract the relevant text **verbatim** relating to the following themes:

planning_ai/chains/prompts/map.txt

@@ -1,20 +1,10 @@
-This is a list of themes proposed the South Cambridgeshire Council:
+Summarise the following response to a planning application, focusing on the themes and policies proposed by the council. Follow these steps:
 
-- Climate change: Help Greater Cambridge transition to net zero carbon by 2050, by ensuring that development is sited in places that help to limit carbon emissions, is designed to the highest achievable standards for energy and water use, and is resilient to current and future climate risks.  
-- Biodiversity and green spaces: Increase and improve our network of habitats for wildlife, and green spaces for people, ensuring that development leaves the natural environment better than it was before.  
-- Wellbeing and social inclusion: Help people in Greater Cambridge to lead healthier and happier lives, ensuring that everyone benefits from the development of new homes and jobs. 
-- Great places: Sustain the unique character of Cambridge and South Cambridgeshire, and complement it with beautiful and distinctive development, creating a place where people want to live, work and play. 
-- Jobs: Encourage a flourishing and mixed economy in Greater Cambridge which includes a wide range of jobs, while maintaining our area's global reputation for innovation.  
-- Homes: Plan for enough housing to meet our needs, including significant quantities of housing that is affordable to rent and buy, and different kinds of homes to suit our diverse communities. 
-- Infrastructure: Plan for transport, water, energy and digital networks; and health, education and cultural facilities; in the right places and built at the right times to serve our growing communities.  
-
-Summarise the following response to a planning application that was proposed with these themes in mind, following these steps:
-
-1. **Summary:** Provide a brief, neutral summary that captures the key points of the response with respect to the list of themes proposed by the council.
-2. **Stance:** Indicate the author’s overall stance as one of the following: 'SUPPORT', 'OPPOSE', 'MIXED' or 'NEUTRAL'.
-3. **Themes:** Identify what themes proposed by the council are discussed.
-4. **Places:** Identify the places the author has considered for discussion.
-4. **Constructiveness Rating:** Rate how constructive the response is on a scale from 1 to 10, with 1 being unconstructive and 10 being highly constructive.
+1. **Summary:** Provide a concise, neutral summary that captures the key points of the response, particularly in relation to the council's proposed themes.
+2. **Themes:** List the council's themes discussed in the response.
+3. **Policies:** Identify relevant policies associated with the extracted themes.
+4. **Places:** Mention any geographical locations considered by the author.
+5. **Constructiveness:** Indicate whether the response is constructive. A response is constructive if it provides any feedback or commentary on the plan, regardless of its depth or specificity.
 
 **Few-shot examples for reference:**
 
@@ -26,10 +16,9 @@ Response:
 "I am in favour of this new park development as it will provide much-needed green space for families. However, the parking situation needs to be reconsidered."
 
 - **Summary:** The author supports the park development for its benefit to families but expresses concern about parking.
-- **Stance:** SUPPORT
 - **Themes:** Biodiversity and green spaces, Infrastructure
 - **Places:** None
-- **Constructiveness Rating:** 8
+- **Constructiveness:** True
 
 ---
 
@@ -39,10 +28,9 @@ Response:
 "This development in Cambridge will destroy local wildlife and create traffic chaos. It should not go ahead."
 
 - **Summary:** The author opposes the development due to concerns about wildlife and traffic congestion.
-- **Stance:** OPPOSE
 - **Themes:** Biodiversity and green spaces, Infrastructure
 - **Places:** Cambridge
-- **Constructiveness Rating:** 3
+- **Constructiveness:** True
 
 ---
 

planning_ai/chains/prompts/ocr.txt

@@ -0,0 +1,10 @@
+The images provided are from a planning response form filled out by a member of the public, containing free-form responses related to a planning application. These responses may be handwritten or typed.
+
+Please follow these instructions to process the images:
+
+1. **Extract Free-Form Information Only**: Focus on extracting and outputting the free-form written content from the images. Do not include single-word answers, brief responses, or any extra content that is not part of the detailed responses.
+2. **Verbatim Output**: Ensure that the extracted information is output exactly as it appears in the images. Add a heading before each section of free-form text if it helps with organisation, but ensure the heading is not added by the model itself. Ignore blank sections entirely—do not generate or include any additional thoughts or content.
+3. **Sequential Processing**: The images are sequentially ordered. A response might continue from one image to the next, so capture the full context across multiple images if necessary.
+4. **Ignore Non-Relevant Content**: Exclude any content that does not fit the criteria of free-form, detailed responses. 
+
+Thank you for your attention to these details.

planning_ai/chains/prompts/themes.txt

@@ -0,0 +1,82 @@
+The following themes are proposed by the South Cambridgeshire Council with each of their associated policies.
+
+# Climate change
+
+Net zero carbon new buildings
+Water efficiency in new developments
+Designing for a changing climate
+Flooding and integrated water management
+Renewable energy projects and infrastructure
+Reducing waste and supporting the circular economy
+Supporting land-based carbon sequestration
+
+# Biodiversity and green spaces
+
+Biodiversity and geodiversity
+Green infrastructure
+Improving Tree Canopy Cover and the Tree Population
+River corridors
+Protecting open spaces
+Providing and enhancing open spaces
+
+# Wellbeing and social inclusion
+
+Creating healthy new developments
+Community, sports and leisure facilities
+Meanwhile uses during long term redevelopments
+Creating inclusive employment and business opportunities through new developments
+Pollution, health and safety
+
+# Great places
+
+People and place responsive design
+Protection and enhancement of landscape character
+Protection and enhancement of the Cambridge Green Belt
+Achieving high quality development
+Establishing high quality landscape and public realm
+Conservation and enhancement of heritage assets
+Adapting heritage assets to climate change
+Protection of public houses
+
+# Jobs
+
+New employment and development proposals
+Supporting the rural economy
+Protecting the best agricultural land
+Protecting existing business space
+Enabling remote working
+Affordable workspace and creative industries
+Supporting a range of facilities in employment parks
+Retail and centres
+Visitor accommodation, attractions and facilities
+Faculty development and specialist / language schools
+
+# Homes
+
+Affordable housing
+Exception sites for affordable housing
+Housing mix
+Housing density
+Garden land and subdivision of existing plots
+Residential space standards and accessible homes
+Specialist housing and homes for older people
+Self and custom build homes
+Build to rent homes
+Houses in multiple occupation (HMOs)
+Student accommodation
+Dwellings in the countryside
+Residential moorings
+Residential caravan sites
+Gypsy and Traveller and Travelling Showpeople sites
+Community-led housing
+
+# Infrastructure
+
+Sustainable transport and connectivity
+Parking and electric vehicles
+Freight and delivery consolidation
+Safeguarding important infrastructure
+Aviation development
+Energy infrastructure masterplanning
+Infrastructure and delivery
+Digital infrastructure

planning_ai/nodes/hallucination_node.py

@@ -9,6 +9,20 @@ from planning_ai.states import DocumentState, OverallState
 
 
 def check_hallucination(state: DocumentState):
+    """Checks for hallucinations in the summary of a document.
+
+    This function uses the `hallucination_chain` to evaluate the summary of a document.
+    If the hallucination score is 1, it indicates no hallucination, and the summary is
+    considered fixed. If the iteration count exceeds 5, the process is terminated.
+
+    Args:
+        state (DocumentState): The current state of the document, including its summary
+            and iteration count.
+
+    Returns:
+        dict: A dictionary containing either a list of fixed summaries or hallucinations
+        that need to be addressed.
+    """
     if state["iteration"] > 5:
         state["iteration"] = -99
         return {"summaries_fixed": [state]}
@@ -33,10 +47,36 @@ def check_hallucination(state: DocumentState):
 
 
 def map_hallucinations(state: OverallState):
+    """Maps summaries to the `check_hallucination` function.
+
+    This function prepares a list of summaries to be checked for hallucinations by
+    sending them to the `check_hallucination` function. Allows summaries to be checked
+    in parrallel.
+
+    Args:
+        state (OverallState): The overall state containing all summaries.
+
+    Returns:
+        list: A list of Send objects directing each summary to the check_hallucination
+        function.
+    """
     return [Send("check_hallucination", summary) for summary in state["summaries"]]
 
 
 def fix_hallucination(state: DocumentState):
+    """Attempts to fix hallucinations in a document's summary.
+
+    This function uses the `fix_chain` to correct hallucinations identified in a summary.
+    The corrected summary is then updated in the document state.
+
+    Args:
+        state (DocumentState): The current state of the document, including its summary
+            and hallucination details.
+
+    Returns:
+        dict: A dictionary containing the updated summaries after attempting to fix
+        hallucinations.
+    """
     response = fix_chain.invoke(
         {
             "context": state["document"],
@@ -58,6 +98,19 @@ def fix_hallucination(state: DocumentState):
 
 
 def map_fix_hallucinations(state: OverallState):
+    """Maps hallucinations to the `fix_hallucination` function.
+
+    This function filters out hallucinations that need fixing and prepares them to be
+    sent to the `fix_hallucination` function. Allows hallucinations to be fixed in
+    parrallel.
+
+    Args:
+        state (OverallState): The overall state containing all hallucinations.
+
+    Returns:
+        list: A list of Send objects directing each hallucination to the
+        fix_hallucination function.
+    """
     hallucinations = []
     if "hallucinations" in state:
         hallucinations = [

planning_ai/nodes/map_node.py

@@ -1,15 +1,30 @@
+import json
+from pathlib import Path
+
 from langgraph.constants import Send
 from presidio_analyzer import AnalyzerEngine
 from presidio_anonymizer import AnonymizerEngine
 
 from planning_ai.chains.map_chain import map_chain
+from planning_ai.common.utils import Paths
 from planning_ai.states import DocumentState, OverallState
 
 anonymizer = AnonymizerEngine()
 analyzer = AnalyzerEngine()
 
 
-def remove_pii(document: str):
+def remove_pii(document: str) -> str:
+    """Removes personally identifiable information (PII) from a document.
+
+    This function uses the Presidio Analyzer and Anonymizer to detect and anonymize
+    PII such as names, phone numbers, and email addresses in the given document.
+
+    Args:
+        document (str): The document text from which PII should be removed.
+
+    Returns:
+        str: The document text with PII anonymized.
+    """
     results = analyzer.analyze(
         text=document,
         entities=["PERSON", "PHONE_NUMBER", "EMAIL_ADDRESS"],
@@ -19,22 +34,71 @@ def remove_pii(document: str):
     return document
 
 
-def generate_summary(state: DocumentState):
+def generate_summary(state: DocumentState) -> dict:
+    """Generates a summary for a document after removing PII.
+
+    This function first anonymizes the document to remove PII, then generates a summary
+    using the `map_chain`. The summary is added to the document state.
+
+    Args:
+        state (DocumentState): The current state of the document, including its text
+            and filename.
+
+    Returns:
+        dict: A dictionary containing the generated summary and updated document state.
+    """
     state["document"] = remove_pii(state["document"])
     response = map_chain.invoke({"context": state["document"]})
-    return {
-        "summaries": [
-            {
-                "summary": response,
-                "document": state["document"],
-                "filename": state["filename"],
-                "iteration": 1,
-            }
-        ]
+    summary = response.summary
+    themes = [theme.value for theme in response.themes]
+    policies = [policy.dict() for policy in response.policies]
+
+    out_policies = []
+    for theme in policies:
+        name = theme["theme"].value
+        policy_list = theme["policies"]
+        out_policies.append({"theme": name, "policies": policy_list})
+
+    out_places = []
+    for place in response.places:
+        name = place.place
+        sentiment = place.sentiment.value
+        out_places.append({"place": name, "sentiment": sentiment})
+
+    save_output = {
+        "summary": summary,
+        "themes": themes,
+        "policies": out_policies,
+        "places": out_places,
+    }
+
+    outfile = f"{Path(state["filename"]).stem}_summary.json"
+    with open(Paths.SUMMARIES / outfile, "w") as file:
+        json.dump(save_output, file, indent=4)
+
+    output = {
+        "summary": response,
+        "document": state["document"],
+        "filename": str(state["filename"]),
+        "iteration": 1,
     }
 
+    return {"summaries": [output]}
+
+
+def map_summaries(state: OverallState) -> list[Send]:
+    """Maps documents to the `generate_summary` function for processing.
+
+    This function prepares a list of documents to be summarized by sending them to the
+    `generate_summary` function. It allows for parallel processing of document summaries.
+
+    Args:
+        state (OverallState): The overall state containing all documents and their filenames.
 
-def map_summaries(state: OverallState):
+    Returns:
+        list: A list of Send objects directing each document to the `generate_summary`
+        function.
+    """
     return [
         Send(
             "generate_summary",

planning_ai/nodes/reduce_node.py

@@ -3,6 +3,21 @@ from planning_ai.states import OverallState
 
 
 def generate_final_summary(state: OverallState):
+    """Generates a final summary from fixed summaries.
+
+    This function checks if the number of documents matches the number of fixed summaries.
+    It then filters the summaries to include only those with a non-neutral stance and a
+    rating of 5 or higher (constructiveness). These filtered summaries are then combined
+    into a final summary using the `reduce_chain`.
+
+    Args:
+        state (OverallState): The overall state containing documents, summaries, and
+            other related information.
+
+    Returns:
+        dict: A dictionary containing the final summary, along with the original
+        documents, summaries, fixed summaries, and hallucinations.
+    """
     if len(state["documents"]) == len(state["summaries_fixed"]):
         summaries = [
             str(summary["summary"])