Spaces:
Running
Running
Simplify name to "Sage" (#35)
Browse files- .gitignore +1 -1
- MANIFEST.in +1 -1
- README.md +22 -21
- assets/storia-logo.png +0 -0
- {codesage β sage}/.sample-env +0 -0
- {codesage β sage}/__init__.py +0 -0
- {codesage β sage}/chat.py +2 -2
- {codesage β sage}/chunker.py +0 -0
- {codesage β sage}/data_manager.py +1 -1
- {codesage β sage}/embedder.py +2 -2
- {codesage β sage}/github.py +2 -2
- {codesage β sage}/index.py +5 -5
- {codesage β sage}/llm.py +0 -0
- {codesage β sage}/sample-exclude.txt +0 -0
- {codesage β sage}/vector_store.py +0 -0
- setup.py +5 -5
- tests/conftest.py +1 -1
- tests/test_chunker.py +8 -8
.gitignore
CHANGED
|
@@ -3,4 +3,4 @@ __pycache__
|
|
| 3 |
*.cpython.*
|
| 4 |
build/
|
| 5 |
repos/
|
| 6 |
-
|
|
|
|
| 3 |
*.cpython.*
|
| 4 |
build/
|
| 5 |
repos/
|
| 6 |
+
sage.egg-info/
|
MANIFEST.in
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
include
|
|
|
|
| 1 |
+
include sage/sample-exclude.txt
|
README.md
CHANGED
|
@@ -1,13 +1,14 @@
|
|
| 1 |
<div align="center">
|
| 2 |
-
<
|
| 3 |
-
<
|
| 4 |
<div>
|
| 5 |
-
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/Storia-AI/
|
| 6 |
-
<img alt="GitHub License" src="https://img.shields.io/github/license/Storia-AI/
|
| 7 |
<a href="https://x.com/StoriaAI">
|
| 8 |
<img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/StoriaAI"></img>
|
| 9 |
</a>
|
| 10 |
</div>
|
|
|
|
| 11 |
<figure>
|
| 12 |
<!-- The <kbd> and <sub> tags are work-arounds for styling, since GitHub doesn't take into account inline styles. Note it might display awkwardly on other Markdown editors. -->
|
| 13 |
<kbd><img src="assets/chat_screenshot2.png" alt="screenshot" /></kbd>
|
|
@@ -24,7 +25,7 @@
|
|
| 24 |
Make sure pipx is installed on your system (see <a href="https://pipx.pypa.io/stable/installation/">instructions</a>), then run:
|
| 25 |
|
| 26 |
```
|
| 27 |
-
pipx install git+https://github.com/Storia-AI/
|
| 28 |
```
|
| 29 |
|
| 30 |
</details>
|
|
@@ -34,16 +35,16 @@ pipx install git+https://github.com/Storia-AI/CodeSage.git@main
|
|
| 34 |
Alternatively, you can manually create a virtual environment and install Code Sage via pip:
|
| 35 |
|
| 36 |
```
|
| 37 |
-
python -m venv
|
| 38 |
-
source
|
| 39 |
-
pip install git+https://github.com/Storia-AI/
|
| 40 |
```
|
| 41 |
|
| 42 |
</details>
|
| 43 |
|
| 44 |
## Prerequisites
|
| 45 |
|
| 46 |
-
`
|
| 47 |
|
| 48 |
1. Indexes your codebase (requiring an embdder and a vector store)
|
| 49 |
2. Enables chatting via LLM + RAG (requiring access to an LLM)
|
|
@@ -110,12 +111,12 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
|
|
| 110 |
|
| 111 |
2. Index the repository. This might take a few minutes, depending on its size.
|
| 112 |
```
|
| 113 |
-
|
| 114 |
```
|
| 115 |
|
| 116 |
3. Chat with the repository, once it's indexed:
|
| 117 |
```
|
| 118 |
-
|
| 119 |
```
|
| 120 |
To get a public URL for your chat app, set `--share=true`.
|
| 121 |
|
|
@@ -131,7 +132,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
|
|
| 131 |
|
| 132 |
2. Index the repository. This might take a few minutes, depending on its size.
|
| 133 |
```
|
| 134 |
-
|
| 135 |
--embedder-type=openai \
|
| 136 |
--vector-store=pinecone \
|
| 137 |
--index-name=$PINECONE_INDEX_NAME
|
|
@@ -139,7 +140,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
|
|
| 139 |
|
| 140 |
3. Chat with the repository, once it's indexed:
|
| 141 |
```
|
| 142 |
-
|
| 143 |
--vector-store-type=pinecone \
|
| 144 |
--index-name=$PINECONE_INDEX_NAME \
|
| 145 |
--llm-provider=openai \
|
|
@@ -173,14 +174,14 @@ where:
|
|
| 173 |
|
| 174 |
To specify an inclusion file (i.e. only index the specified files):
|
| 175 |
```
|
| 176 |
-
|
| 177 |
```
|
| 178 |
|
| 179 |
To specify an exclusion file (i.e. index all files, except for the ones specified):
|
| 180 |
```
|
| 181 |
-
|
| 182 |
```
|
| 183 |
-
By default, we use the exclusion file [sample-exclude.txt](
|
| 184 |
</details>
|
| 185 |
|
| 186 |
<details>
|
|
@@ -192,17 +193,17 @@ export GITHUB_TOKEN=...
|
|
| 192 |
|
| 193 |
To index GitHub issues without comments:
|
| 194 |
```
|
| 195 |
-
|
| 196 |
```
|
| 197 |
|
| 198 |
To index GitHub issues with comments:
|
| 199 |
```
|
| 200 |
-
|
| 201 |
```
|
| 202 |
|
| 203 |
To index GitHub issues, but not the codebase:
|
| 204 |
```
|
| 205 |
-
|
| 206 |
```
|
| 207 |
</details>
|
| 208 |
|
|
@@ -211,7 +212,7 @@ cs-index $GITHUB_REPO --index-issues --no-index-repo
|
|
| 211 |
Sometimes you just want to learn how a codebase works and how to integrate it, without spending hours sifting through
|
| 212 |
the code itself.
|
| 213 |
|
| 214 |
-
`
|
| 215 |
|
| 216 |
Features:
|
| 217 |
|
|
@@ -222,7 +223,7 @@ Features:
|
|
| 222 |
|
| 223 |
# Changelog
|
| 224 |
|
| 225 |
-
- 2024-09-16: Renamed `repo2vec` to `
|
| 226 |
- 2024-09-03: Support for indexing GitHub issues.
|
| 227 |
- 2024-08-30: Support for running everything locally (Marqo for embeddings, Ollama for LLMs).
|
| 228 |
|
|
|
|
| 1 |
<div align="center">
|
| 2 |
+
<img src="assets/storia-logo.png" alt="Logo" width="50" style="border-radius: 15px;">
|
| 3 |
+
<h1 align="center">Sage: Chat with any codebase</h1>
|
| 4 |
<div>
|
| 5 |
+
<img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/Storia-AI/sage?logo=github" />
|
| 6 |
+
<img alt="GitHub License" src="https://img.shields.io/github/license/Storia-AI/sage"></img>
|
| 7 |
<a href="https://x.com/StoriaAI">
|
| 8 |
<img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/StoriaAI"></img>
|
| 9 |
</a>
|
| 10 |
</div>
|
| 11 |
+
<br />
|
| 12 |
<figure>
|
| 13 |
<!-- The <kbd> and <sub> tags are work-arounds for styling, since GitHub doesn't take into account inline styles. Note it might display awkwardly on other Markdown editors. -->
|
| 14 |
<kbd><img src="assets/chat_screenshot2.png" alt="screenshot" /></kbd>
|
|
|
|
| 25 |
Make sure pipx is installed on your system (see <a href="https://pipx.pypa.io/stable/installation/">instructions</a>), then run:
|
| 26 |
|
| 27 |
```
|
| 28 |
+
pipx install git+https://github.com/Storia-AI/sage.git@main
|
| 29 |
```
|
| 30 |
|
| 31 |
</details>
|
|
|
|
| 35 |
Alternatively, you can manually create a virtual environment and install Code Sage via pip:
|
| 36 |
|
| 37 |
```
|
| 38 |
+
python -m venv sage-venv
|
| 39 |
+
source sage-venv/bin/activate
|
| 40 |
+
pip install git+https://github.com/Storia-AI/sage.git@main
|
| 41 |
```
|
| 42 |
|
| 43 |
</details>
|
| 44 |
|
| 45 |
## Prerequisites
|
| 46 |
|
| 47 |
+
`sage` performs two steps:
|
| 48 |
|
| 49 |
1. Indexes your codebase (requiring an embdder and a vector store)
|
| 50 |
2. Enables chatting via LLM + RAG (requiring access to an LLM)
|
|
|
|
| 111 |
|
| 112 |
2. Index the repository. This might take a few minutes, depending on its size.
|
| 113 |
```
|
| 114 |
+
sage-index $GITHUB_REPO
|
| 115 |
```
|
| 116 |
|
| 117 |
3. Chat with the repository, once it's indexed:
|
| 118 |
```
|
| 119 |
+
sage-chat $GITHUB_REPO
|
| 120 |
```
|
| 121 |
To get a public URL for your chat app, set `--share=true`.
|
| 122 |
|
|
|
|
| 132 |
|
| 133 |
2. Index the repository. This might take a few minutes, depending on its size.
|
| 134 |
```
|
| 135 |
+
sage-index $GITHUB_REPO \
|
| 136 |
--embedder-type=openai \
|
| 137 |
--vector-store=pinecone \
|
| 138 |
--index-name=$PINECONE_INDEX_NAME
|
|
|
|
| 140 |
|
| 141 |
3. Chat with the repository, once it's indexed:
|
| 142 |
```
|
| 143 |
+
sage-chat $GITHUB_REPO \
|
| 144 |
--vector-store-type=pinecone \
|
| 145 |
--index-name=$PINECONE_INDEX_NAME \
|
| 146 |
--llm-provider=openai \
|
|
|
|
| 174 |
|
| 175 |
To specify an inclusion file (i.e. only index the specified files):
|
| 176 |
```
|
| 177 |
+
sage-index $GITHUB_REPO --include=/path/to/inclusion/file
|
| 178 |
```
|
| 179 |
|
| 180 |
To specify an exclusion file (i.e. index all files, except for the ones specified):
|
| 181 |
```
|
| 182 |
+
sage-index $GITHUB_REPO --exclude=/path/to/exclusion/file
|
| 183 |
```
|
| 184 |
+
By default, we use the exclusion file [sample-exclude.txt](sage/sample-exclude.txt).
|
| 185 |
</details>
|
| 186 |
|
| 187 |
<details>
|
|
|
|
| 193 |
|
| 194 |
To index GitHub issues without comments:
|
| 195 |
```
|
| 196 |
+
sage-index $GITHUB_REPO --index-issues
|
| 197 |
```
|
| 198 |
|
| 199 |
To index GitHub issues with comments:
|
| 200 |
```
|
| 201 |
+
sage-index $GITHUB_REPO --index-issues --index-issue-comments
|
| 202 |
```
|
| 203 |
|
| 204 |
To index GitHub issues, but not the codebase:
|
| 205 |
```
|
| 206 |
+
sage-index $GITHUB_REPO --index-issues --no-index-repo
|
| 207 |
```
|
| 208 |
</details>
|
| 209 |
|
|
|
|
| 212 |
Sometimes you just want to learn how a codebase works and how to integrate it, without spending hours sifting through
|
| 213 |
the code itself.
|
| 214 |
|
| 215 |
+
`sage` is like an open-source GitHub Copilot with the most up-to-date information about your repo.
|
| 216 |
|
| 217 |
Features:
|
| 218 |
|
|
|
|
| 223 |
|
| 224 |
# Changelog
|
| 225 |
|
| 226 |
+
- 2024-09-16: Renamed `repo2vec` to `sage`.
|
| 227 |
- 2024-09-03: Support for indexing GitHub issues.
|
| 228 |
- 2024-08-30: Support for running everything locally (Marqo for embeddings, Ollama for LLMs).
|
| 229 |
|
assets/storia-logo.png
ADDED
|
{codesage β sage}/.sample-env
RENAMED
|
File without changes
|
{codesage β sage}/__init__.py
RENAMED
|
File without changes
|
{codesage β sage}/chat.py
RENAMED
|
@@ -12,8 +12,8 @@ from langchain.chains.combine_documents import create_stuff_documents_chain
|
|
| 12 |
from langchain.schema import AIMessage, HumanMessage
|
| 13 |
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
|
| 14 |
|
| 15 |
-
import
|
| 16 |
-
from
|
| 17 |
|
| 18 |
load_dotenv()
|
| 19 |
|
|
|
|
| 12 |
from langchain.schema import AIMessage, HumanMessage
|
| 13 |
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
|
| 14 |
|
| 15 |
+
import sage.vector_store as vector_store
|
| 16 |
+
from sage.llm import build_llm_via_langchain
|
| 17 |
|
| 18 |
load_dotenv()
|
| 19 |
|
{codesage β sage}/chunker.py
RENAMED
|
File without changes
|
{codesage β sage}/data_manager.py
RENAMED
|
@@ -35,7 +35,7 @@ class GitHubRepoManager(DataManager):
|
|
| 35 |
):
|
| 36 |
"""
|
| 37 |
Args:
|
| 38 |
-
repo_id: The identifier of the repository in owner/repo format, e.g. "Storia-AI/
|
| 39 |
local_dir: The local directory where the repository will be cloned.
|
| 40 |
inclusion_file: A file with a lists of files/directories/extensions to include. Each line must be in one of
|
| 41 |
the following formats: "ext:.my-extension", "file:my-file.py", or "dir:my-directory".
|
|
|
|
| 35 |
):
|
| 36 |
"""
|
| 37 |
Args:
|
| 38 |
+
repo_id: The identifier of the repository in owner/repo format, e.g. "Storia-AI/sage".
|
| 39 |
local_dir: The local directory where the repository will be cloned.
|
| 40 |
inclusion_file: A file with a lists of files/directories/extensions to include. Each line must be in one of
|
| 41 |
the following formats: "ext:.my-extension", "file:my-file.py", or "dir:my-directory".
|
{codesage β sage}/embedder.py
RENAMED
|
@@ -11,8 +11,8 @@ from typing import Dict, Generator, List, Optional, Tuple
|
|
| 11 |
import marqo
|
| 12 |
from openai import OpenAI
|
| 13 |
|
| 14 |
-
from
|
| 15 |
-
from
|
| 16 |
|
| 17 |
Vector = Tuple[Dict, List[float]] # (metadata, embedding)
|
| 18 |
|
|
|
|
| 11 |
import marqo
|
| 12 |
from openai import OpenAI
|
| 13 |
|
| 14 |
+
from sage.chunker import Chunk, Chunker
|
| 15 |
+
from sage.data_manager import DataManager
|
| 16 |
|
| 17 |
Vector = Tuple[Dict, List[float]] # (metadata, embedding)
|
| 18 |
|
{codesage β sage}/github.py
RENAMED
|
@@ -8,8 +8,8 @@ from typing import Any, Dict, Generator, List, Tuple
|
|
| 8 |
import requests
|
| 9 |
import tiktoken
|
| 10 |
|
| 11 |
-
from
|
| 12 |
-
from
|
| 13 |
|
| 14 |
tokenizer = tiktoken.get_encoding("cl100k_base")
|
| 15 |
|
|
|
|
| 8 |
import requests
|
| 9 |
import tiktoken
|
| 10 |
|
| 11 |
+
from sage.chunker import Chunk, Chunker
|
| 12 |
+
from sage.data_manager import DataManager
|
| 13 |
|
| 14 |
tokenizer = tiktoken.get_encoding("cl100k_base")
|
| 15 |
|
{codesage β sage}/index.py
RENAMED
|
@@ -7,11 +7,11 @@ import time
|
|
| 7 |
|
| 8 |
import pkg_resources
|
| 9 |
|
| 10 |
-
from
|
| 11 |
-
from
|
| 12 |
-
from
|
| 13 |
-
from
|
| 14 |
-
from
|
| 15 |
|
| 16 |
logging.basicConfig(level=logging.INFO)
|
| 17 |
logger = logging.getLogger()
|
|
|
|
| 7 |
|
| 8 |
import pkg_resources
|
| 9 |
|
| 10 |
+
from sage.chunker import UniversalFileChunker
|
| 11 |
+
from sage.data_manager import GitHubRepoManager
|
| 12 |
+
from sage.embedder import build_batch_embedder_from_flags
|
| 13 |
+
from sage.github import GitHubIssuesChunker, GitHubIssuesManager
|
| 14 |
+
from sage.vector_store import build_from_args
|
| 15 |
|
| 16 |
logging.basicConfig(level=logging.INFO)
|
| 17 |
logger = logging.getLogger()
|
{codesage β sage}/llm.py
RENAMED
|
File without changes
|
{codesage β sage}/sample-exclude.txt
RENAMED
|
File without changes
|
{codesage β sage}/vector_store.py
RENAMED
|
File without changes
|
setup.py
CHANGED
|
@@ -7,18 +7,18 @@ def readfile(filename):
|
|
| 7 |
|
| 8 |
|
| 9 |
setup(
|
| 10 |
-
name="
|
| 11 |
version="0.1.0",
|
| 12 |
packages=find_packages(),
|
| 13 |
include_package_data=True,
|
| 14 |
package_data={
|
| 15 |
-
"
|
| 16 |
},
|
| 17 |
install_requires=open("requirements.txt").readlines() + ["setuptools"],
|
| 18 |
entry_points={
|
| 19 |
"console_scripts": [
|
| 20 |
-
"
|
| 21 |
-
"
|
| 22 |
],
|
| 23 |
},
|
| 24 |
author="Julia Turc & Mihail Eric / Storia AI",
|
|
@@ -26,7 +26,7 @@ setup(
|
|
| 26 |
description="A library to index a code repository and chat with it via LLMs.",
|
| 27 |
long_description=open("README.md").read(),
|
| 28 |
long_description_content_type="text/markdown",
|
| 29 |
-
url="https://github.com/Storia-AI/
|
| 30 |
classifiers=[
|
| 31 |
"Programming Language :: Python :: 3",
|
| 32 |
"License :: OSI Approved :: MIT License",
|
|
|
|
| 7 |
|
| 8 |
|
| 9 |
setup(
|
| 10 |
+
name="sage",
|
| 11 |
version="0.1.0",
|
| 12 |
packages=find_packages(),
|
| 13 |
include_package_data=True,
|
| 14 |
package_data={
|
| 15 |
+
"sage": ["sample-exclude.txt"],
|
| 16 |
},
|
| 17 |
install_requires=open("requirements.txt").readlines() + ["setuptools"],
|
| 18 |
entry_points={
|
| 19 |
"console_scripts": [
|
| 20 |
+
"sage-index=sage.index:main",
|
| 21 |
+
"sage-chat=sage.chat:main",
|
| 22 |
],
|
| 23 |
},
|
| 24 |
author="Julia Turc & Mihail Eric / Storia AI",
|
|
|
|
| 26 |
description="A library to index a code repository and chat with it via LLMs.",
|
| 27 |
long_description=open("README.md").read(),
|
| 28 |
long_description_content_type="text/markdown",
|
| 29 |
+
url="https://github.com/Storia-AI/sage",
|
| 30 |
classifiers=[
|
| 31 |
"Programming Language :: Python :: 3",
|
| 32 |
"License :: OSI Approved :: MIT License",
|
tests/conftest.py
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
import os
|
| 2 |
import sys
|
| 3 |
|
| 4 |
-
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../
|
|
|
|
| 1 |
import os
|
| 2 |
import sys
|
| 3 |
|
| 4 |
+
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../sage")))
|
tests/test_chunker.py
CHANGED
|
@@ -11,12 +11,12 @@ import os
|
|
| 11 |
|
| 12 |
from pytest import mark, param
|
| 13 |
|
| 14 |
-
import
|
| 15 |
|
| 16 |
|
| 17 |
def test_text_chunker_happy_path():
|
| 18 |
"""Tests the happy path for the TextFileChunker."""
|
| 19 |
-
chunker =
|
| 20 |
|
| 21 |
file_path = os.path.join(os.path.dirname(__file__), "../README.md")
|
| 22 |
with open(file_path, "r") as file:
|
|
@@ -29,9 +29,9 @@ def test_text_chunker_happy_path():
|
|
| 29 |
|
| 30 |
def test_code_chunker_happy_path():
|
| 31 |
"""Tests the happy path for the CodeFileChunker."""
|
| 32 |
-
chunker =
|
| 33 |
|
| 34 |
-
file_path = os.path.join(os.path.dirname(__file__), "../
|
| 35 |
with open(file_path, "r") as file:
|
| 36 |
content = file.read()
|
| 37 |
metadata = {"file_path": file_path}
|
|
@@ -48,22 +48,22 @@ def test_code_chunker_typescript(filename):
|
|
| 48 |
content = file.read()
|
| 49 |
metadata = {"file_path": file_path}
|
| 50 |
|
| 51 |
-
chunker =
|
| 52 |
chunks = chunker.chunk(content, metadata)
|
| 53 |
# There's a bug in the tree-sitter-language-pack library for TypeScript. Before it gets fixed, we expect this to
|
| 54 |
# return an empty list (instead of crashing).
|
| 55 |
assert len(chunks) == 0
|
| 56 |
|
| 57 |
# However, the UniversalFileChunker should fallback onto a regular text chunker, and return some chunks.
|
| 58 |
-
chunker =
|
| 59 |
chunks = chunker.chunk(content, metadata)
|
| 60 |
assert len(chunks) >= 1
|
| 61 |
|
| 62 |
|
| 63 |
def test_ipynb_chunker_happy_path():
|
| 64 |
"""Tests the happy path for the IPynbChunker."""
|
| 65 |
-
code_chunker =
|
| 66 |
-
chunker =
|
| 67 |
|
| 68 |
file_path = os.path.join(os.path.dirname(__file__), "assets/sample-notebook.ipynb")
|
| 69 |
with open(file_path, "r") as file:
|
|
|
|
| 11 |
|
| 12 |
from pytest import mark, param
|
| 13 |
|
| 14 |
+
import sage.chunker
|
| 15 |
|
| 16 |
|
| 17 |
def test_text_chunker_happy_path():
|
| 18 |
"""Tests the happy path for the TextFileChunker."""
|
| 19 |
+
chunker = sage.chunker.TextFileChunker(max_tokens=100)
|
| 20 |
|
| 21 |
file_path = os.path.join(os.path.dirname(__file__), "../README.md")
|
| 22 |
with open(file_path, "r") as file:
|
|
|
|
| 29 |
|
| 30 |
def test_code_chunker_happy_path():
|
| 31 |
"""Tests the happy path for the CodeFileChunker."""
|
| 32 |
+
chunker = sage.chunker.CodeFileChunker(max_tokens=100)
|
| 33 |
|
| 34 |
+
file_path = os.path.join(os.path.dirname(__file__), "../sage/chunker.py")
|
| 35 |
with open(file_path, "r") as file:
|
| 36 |
content = file.read()
|
| 37 |
metadata = {"file_path": file_path}
|
|
|
|
| 48 |
content = file.read()
|
| 49 |
metadata = {"file_path": file_path}
|
| 50 |
|
| 51 |
+
chunker = sage.chunker.CodeFileChunker(max_tokens=100)
|
| 52 |
chunks = chunker.chunk(content, metadata)
|
| 53 |
# There's a bug in the tree-sitter-language-pack library for TypeScript. Before it gets fixed, we expect this to
|
| 54 |
# return an empty list (instead of crashing).
|
| 55 |
assert len(chunks) == 0
|
| 56 |
|
| 57 |
# However, the UniversalFileChunker should fallback onto a regular text chunker, and return some chunks.
|
| 58 |
+
chunker = sage.chunker.UniversalFileChunker(max_tokens=100)
|
| 59 |
chunks = chunker.chunk(content, metadata)
|
| 60 |
assert len(chunks) >= 1
|
| 61 |
|
| 62 |
|
| 63 |
def test_ipynb_chunker_happy_path():
|
| 64 |
"""Tests the happy path for the IPynbChunker."""
|
| 65 |
+
code_chunker = sage.chunker.CodeFileChunker(max_tokens=100)
|
| 66 |
+
chunker = sage.chunker.IpynbFileChunker(code_chunker)
|
| 67 |
|
| 68 |
file_path = os.path.join(os.path.dirname(__file__), "assets/sample-notebook.ipynb")
|
| 69 |
with open(file_path, "r") as file:
|