## Source Code

This notebook covers how to load source code files using a special approach with language parsing: each top-level function and class in the code is loaded into separate documents. Any remaining code top-level code outside the already loaded functions and classes will be loaded into a separate document.

This approach can potentially improve the accuracy of QA models over source code.

The supported languages for code parsing are:

*   C (\*)
*   C++ (\*)
*   C# (\*)
*   COBOL
*   Go (\*)
*   Java (\*)
*   JavaScript (requires package `esprima`)
*   Kotlin (\*)
*   Lua (\*)
*   Perl (\*)
*   Python
*   Ruby (\*)
*   Rust (\*)
*   Scala (\*)
*   TypeScript (\*)

Items marked with (\*) require the packages `tree_sitter` and `tree_sitter_languages`. It is straightforward to add support for additional languages using `tree_sitter`, although this currently requires modifying LangChain.

The language used for parsing can be configured, along with the minimum number of lines required to activate the splitting based on syntax.

If a language is not explicitly specified, `LanguageParser` will infer one from filename extensions, if present.

```
%pip install -qU esprima esprima tree_sitter tree_sitter_languages
```

```
import warningswarnings.filterwarnings("ignore")from pprint import pprintfrom langchain_community.document_loaders.generic import GenericLoaderfrom langchain_community.document_loaders.parsers import LanguageParserfrom langchain_text_splitters import Language
```

```
loader = GenericLoader.from_filesystem(    "./example_data/source_code",    glob="*",    suffixes=[".py", ".js"],    parser=LanguageParser(),)docs = loader.load()
```

```
for document in docs:    pprint(document.metadata)
```

```
{'content_type': 'functions_classes', 'language': <Language.PYTHON: 'python'>, 'source': 'example_data/source_code/example.py'}{'content_type': 'functions_classes', 'language': <Language.PYTHON: 'python'>, 'source': 'example_data/source_code/example.py'}{'content_type': 'simplified_code', 'language': <Language.PYTHON: 'python'>, 'source': 'example_data/source_code/example.py'}{'content_type': 'functions_classes', 'language': <Language.JS: 'js'>, 'source': 'example_data/source_code/example.js'}{'content_type': 'functions_classes', 'language': <Language.JS: 'js'>, 'source': 'example_data/source_code/example.js'}{'content_type': 'simplified_code', 'language': <Language.JS: 'js'>, 'source': 'example_data/source_code/example.js'}
```

```
print("\n\n--8<--\n\n".join([document.page_content for document in docs]))
```

```
class MyClass:    def __init__(self, name):        self.name = name    def greet(self):        print(f"Hello, {self.name}!")--8<--def main():    name = input("Enter your name: ")    obj = MyClass(name)    obj.greet()--8<--# Code for: class MyClass:# Code for: def main():if __name__ == "__main__":    main()--8<--class MyClass {  constructor(name) {    this.name = name;  }  greet() {    console.log(`Hello, ${this.name}!`);  }}--8<--function main() {  const name = prompt("Enter your name:");  const obj = new MyClass(name);  obj.greet();}--8<--// Code for: class MyClass {// Code for: function main() {main();
```

The parser can be disabled for small files.

The parameter `parser_threshold` indicates the minimum number of lines that the source code file must have to be segmented using the parser.

```
loader = GenericLoader.from_filesystem(    "./example_data/source_code",    glob="*",    suffixes=[".py"],    parser=LanguageParser(language=Language.PYTHON, parser_threshold=1000),)docs = loader.load()
```

```
print(docs[0].page_content)
```

```
class MyClass:    def __init__(self, name):        self.name = name    def greet(self):        print(f"Hello, {self.name}!")def main():    name = input("Enter your name: ")    obj = MyClass(name)    obj.greet()if __name__ == "__main__":    main()
```

## Splitting[​](#splitting "Direct link to Splitting")

Additional splitting could be needed for those functions, classes, or scripts that are too big.

```
loader = GenericLoader.from_filesystem(    "./example_data/source_code",    glob="*",    suffixes=[".js"],    parser=LanguageParser(language=Language.JS),)docs = loader.load()
```

```
from langchain_text_splitters import (    Language,    RecursiveCharacterTextSplitter,)
```

```
js_splitter = RecursiveCharacterTextSplitter.from_language(    language=Language.JS, chunk_size=60, chunk_overlap=0)
```

```
result = js_splitter.split_documents(docs)
```

```
print("\n\n--8<--\n\n".join([document.page_content for document in result]))
```

```
class MyClass {  constructor(name) {    this.name = name;--8<--}--8<--greet() {    console.log(`Hello, ${this.name}!`);  }}--8<--function main() {  const name = prompt("Enter your name:");--8<--const obj = new MyClass(name);  obj.greet();}--8<--// Code for: class MyClass {// Code for: function main() {--8<--main();
```

## Adding Languages using Tree-sitter Template[​](#adding-languages-using-tree-sitter-template "Direct link to Adding Languages using Tree-sitter Template")

Expanding language support using the Tree-Sitter template involves a few essential steps:

1.  **Creating a New Language File**:
    *   Begin by creating a new file in the designated directory (langchain/libs/community/langchain\_community/document\_loaders/parsers/language).
    *   Model this file based on the structure and parsing logic of existing language files like **`cpp.py`**.
    *   You will also need to create a file in the langchain directory (langchain/libs/langchain/langchain/document\_loaders/parsers/language).
2.  **Parsing Language Specifics**:
    *   Mimic the structure used in the **`cpp.py`** file, adapting it to suit the language you are incorporating.
    *   The primary alteration involves adjusting the chunk query array to suit the syntax and structure of the language you are parsing.
3.  **Testing the Language Parser**:
    *   For thorough validation, generate a test file specific to the new language. Create **`test_language.py`** in the designated directory(langchain/libs/community/tests/unit\_tests/document\_loaders/parsers/language).
    *   Follow the example set by **`test_cpp.py`** to establish fundamental tests for the parsed elements in the new language.
4.  **Integration into the Parser and Text Splitter**:
    *   Incorporate your new language within the **`language_parser.py`** file. Ensure to update LANGUAGE\_EXTENSIONS and LANGUAGE\_SEGMENTERS along with the docstring for LanguageParser to recognize and handle the added language.
    *   Also, confirm that your language is included in **`text_splitter.py`** in class Language for proper parsing.

By following these steps and ensuring comprehensive testing and integration, you’ll successfully extend language support using the Tree-Sitter template.

Best of luck!