juliaturc commited on
Commit
52c1352
Β·
1 Parent(s): b6a1c55

Simplify name to "Sage" (#35)

Browse files
.gitignore CHANGED
@@ -3,4 +3,4 @@ __pycache__
3
  *.cpython.*
4
  build/
5
  repos/
6
- codesage.egg-info/
 
3
  *.cpython.*
4
  build/
5
  repos/
6
+ sage.egg-info/
MANIFEST.in CHANGED
@@ -1 +1 @@
1
- include codesage/sample-exclude.txt
 
1
+ include sage/sample-exclude.txt
README.md CHANGED
@@ -1,13 +1,14 @@
1
  <div align="center">
2
- <h1 align="center">Code Sage</h1>
3
- <p align="center">An open-source pair programmer for chatting with any codebase.</p>
4
  <div>
5
- <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/Storia-AI/CodeSage?logo=github" />
6
- <img alt="GitHub License" src="https://img.shields.io/github/license/Storia-AI/CodeSage"></img>
7
  <a href="https://x.com/StoriaAI">
8
  <img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/StoriaAI"></img>
9
  </a>
10
  </div>
 
11
  <figure>
12
  <!-- The <kbd> and <sub> tags are work-arounds for styling, since GitHub doesn't take into account inline styles. Note it might display awkwardly on other Markdown editors. -->
13
  <kbd><img src="assets/chat_screenshot2.png" alt="screenshot" /></kbd>
@@ -24,7 +25,7 @@
24
  Make sure pipx is installed on your system (see <a href="https://pipx.pypa.io/stable/installation/">instructions</a>), then run:
25
 
26
  ```
27
- pipx install git+https://github.com/Storia-AI/CodeSage.git@main
28
  ```
29
 
30
  </details>
@@ -34,16 +35,16 @@ pipx install git+https://github.com/Storia-AI/CodeSage.git@main
34
  Alternatively, you can manually create a virtual environment and install Code Sage via pip:
35
 
36
  ```
37
- python -m venv codesage-venv
38
- source codesage-venv/bin/activate
39
- pip install git+https://github.com/Storia-AI/CodeSage.git@main
40
  ```
41
 
42
  </details>
43
 
44
  ## Prerequisites
45
 
46
- `codesage` performs two steps:
47
 
48
  1. Indexes your codebase (requiring an embdder and a vector store)
49
  2. Enables chatting via LLM + RAG (requiring access to an LLM)
@@ -110,12 +111,12 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
110
 
111
  2. Index the repository. This might take a few minutes, depending on its size.
112
  ```
113
- cs-index $GITHUB_REPO
114
  ```
115
 
116
  3. Chat with the repository, once it's indexed:
117
  ```
118
- cs-chat $GITHUB_REPO
119
  ```
120
  To get a public URL for your chat app, set `--share=true`.
121
 
@@ -131,7 +132,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
131
 
132
  2. Index the repository. This might take a few minutes, depending on its size.
133
  ```
134
- cs-index $GITHUB_REPO \
135
  --embedder-type=openai \
136
  --vector-store=pinecone \
137
  --index-name=$PINECONE_INDEX_NAME
@@ -139,7 +140,7 @@ If you are planning on indexing GitHub issues in addition to the codebase, you w
139
 
140
  3. Chat with the repository, once it's indexed:
141
  ```
142
- cs-chat $GITHUB_REPO \
143
  --vector-store-type=pinecone \
144
  --index-name=$PINECONE_INDEX_NAME \
145
  --llm-provider=openai \
@@ -173,14 +174,14 @@ where:
173
 
174
  To specify an inclusion file (i.e. only index the specified files):
175
  ```
176
- cs-index $GITHUB_REPO --include=/path/to/inclusion/file
177
  ```
178
 
179
  To specify an exclusion file (i.e. index all files, except for the ones specified):
180
  ```
181
- cs-index $GITHUB_REPO --exclude=/path/to/exclusion/file
182
  ```
183
- By default, we use the exclusion file [sample-exclude.txt](codesage/sample-exclude.txt).
184
  </details>
185
 
186
  <details>
@@ -192,17 +193,17 @@ export GITHUB_TOKEN=...
192
 
193
  To index GitHub issues without comments:
194
  ```
195
- cs-index $GITHUB_REPO --index-issues
196
  ```
197
 
198
  To index GitHub issues with comments:
199
  ```
200
- cs-index $GITHUB_REPO --index-issues --index-issue-comments
201
  ```
202
 
203
  To index GitHub issues, but not the codebase:
204
  ```
205
- cs-index $GITHUB_REPO --index-issues --no-index-repo
206
  ```
207
  </details>
208
 
@@ -211,7 +212,7 @@ cs-index $GITHUB_REPO --index-issues --no-index-repo
211
  Sometimes you just want to learn how a codebase works and how to integrate it, without spending hours sifting through
212
  the code itself.
213
 
214
- `codesage` is like an open-source GitHub Copilot with the most up-to-date information about your repo.
215
 
216
  Features:
217
 
@@ -222,7 +223,7 @@ Features:
222
 
223
  # Changelog
224
 
225
- - 2024-09-16: Renamed `repo2vec` to `codesage`.
226
  - 2024-09-03: Support for indexing GitHub issues.
227
  - 2024-08-30: Support for running everything locally (Marqo for embeddings, Ollama for LLMs).
228
 
 
1
  <div align="center">
2
+ <img src="assets/storia-logo.png" alt="Logo" width="50" style="border-radius: 15px;">
3
+ <h1 align="center">Sage: Chat with any codebase</h1>
4
  <div>
5
+ <img alt="GitHub Repo stars" src="https://img.shields.io/github/stars/Storia-AI/sage?logo=github" />
6
+ <img alt="GitHub License" src="https://img.shields.io/github/license/Storia-AI/sage"></img>
7
  <a href="https://x.com/StoriaAI">
8
  <img alt="X (formerly Twitter) Follow" src="https://img.shields.io/twitter/follow/StoriaAI"></img>
9
  </a>
10
  </div>
11
+ <br />
12
  <figure>
13
  <!-- The <kbd> and <sub> tags are work-arounds for styling, since GitHub doesn't take into account inline styles. Note it might display awkwardly on other Markdown editors. -->
14
  <kbd><img src="assets/chat_screenshot2.png" alt="screenshot" /></kbd>
 
25
  Make sure pipx is installed on your system (see <a href="https://pipx.pypa.io/stable/installation/">instructions</a>), then run:
26
 
27
  ```
28
+ pipx install git+https://github.com/Storia-AI/sage.git@main
29
  ```
30
 
31
  </details>
 
35
  Alternatively, you can manually create a virtual environment and install Code Sage via pip:
36
 
37
  ```
38
+ python -m venv sage-venv
39
+ source sage-venv/bin/activate
40
+ pip install git+https://github.com/Storia-AI/sage.git@main
41
  ```
42
 
43
  </details>
44
 
45
  ## Prerequisites
46
 
47
+ `sage` performs two steps:
48
 
49
  1. Indexes your codebase (requiring an embdder and a vector store)
50
  2. Enables chatting via LLM + RAG (requiring access to an LLM)
 
111
 
112
  2. Index the repository. This might take a few minutes, depending on its size.
113
  ```
114
+ sage-index $GITHUB_REPO
115
  ```
116
 
117
  3. Chat with the repository, once it's indexed:
118
  ```
119
+ sage-chat $GITHUB_REPO
120
  ```
121
  To get a public URL for your chat app, set `--share=true`.
122
 
 
132
 
133
  2. Index the repository. This might take a few minutes, depending on its size.
134
  ```
135
+ sage-index $GITHUB_REPO \
136
  --embedder-type=openai \
137
  --vector-store=pinecone \
138
  --index-name=$PINECONE_INDEX_NAME
 
140
 
141
  3. Chat with the repository, once it's indexed:
142
  ```
143
+ sage-chat $GITHUB_REPO \
144
  --vector-store-type=pinecone \
145
  --index-name=$PINECONE_INDEX_NAME \
146
  --llm-provider=openai \
 
174
 
175
  To specify an inclusion file (i.e. only index the specified files):
176
  ```
177
+ sage-index $GITHUB_REPO --include=/path/to/inclusion/file
178
  ```
179
 
180
  To specify an exclusion file (i.e. index all files, except for the ones specified):
181
  ```
182
+ sage-index $GITHUB_REPO --exclude=/path/to/exclusion/file
183
  ```
184
+ By default, we use the exclusion file [sample-exclude.txt](sage/sample-exclude.txt).
185
  </details>
186
 
187
  <details>
 
193
 
194
  To index GitHub issues without comments:
195
  ```
196
+ sage-index $GITHUB_REPO --index-issues
197
  ```
198
 
199
  To index GitHub issues with comments:
200
  ```
201
+ sage-index $GITHUB_REPO --index-issues --index-issue-comments
202
  ```
203
 
204
  To index GitHub issues, but not the codebase:
205
  ```
206
+ sage-index $GITHUB_REPO --index-issues --no-index-repo
207
  ```
208
  </details>
209
 
 
212
  Sometimes you just want to learn how a codebase works and how to integrate it, without spending hours sifting through
213
  the code itself.
214
 
215
+ `sage` is like an open-source GitHub Copilot with the most up-to-date information about your repo.
216
 
217
  Features:
218
 
 
223
 
224
  # Changelog
225
 
226
+ - 2024-09-16: Renamed `repo2vec` to `sage`.
227
  - 2024-09-03: Support for indexing GitHub issues.
228
  - 2024-08-30: Support for running everything locally (Marqo for embeddings, Ollama for LLMs).
229
 
assets/storia-logo.png ADDED
{codesage β†’ sage}/.sample-env RENAMED
File without changes
{codesage β†’ sage}/__init__.py RENAMED
File without changes
{codesage β†’ sage}/chat.py RENAMED
@@ -12,8 +12,8 @@ from langchain.chains.combine_documents import create_stuff_documents_chain
12
  from langchain.schema import AIMessage, HumanMessage
13
  from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
14
 
15
- import codesage.vector_store as vector_store
16
- from codesage.llm import build_llm_via_langchain
17
 
18
  load_dotenv()
19
 
 
12
  from langchain.schema import AIMessage, HumanMessage
13
  from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
14
 
15
+ import sage.vector_store as vector_store
16
+ from sage.llm import build_llm_via_langchain
17
 
18
  load_dotenv()
19
 
{codesage β†’ sage}/chunker.py RENAMED
File without changes
{codesage β†’ sage}/data_manager.py RENAMED
@@ -35,7 +35,7 @@ class GitHubRepoManager(DataManager):
35
  ):
36
  """
37
  Args:
38
- repo_id: The identifier of the repository in owner/repo format, e.g. "Storia-AI/codesage".
39
  local_dir: The local directory where the repository will be cloned.
40
  inclusion_file: A file with a lists of files/directories/extensions to include. Each line must be in one of
41
  the following formats: "ext:.my-extension", "file:my-file.py", or "dir:my-directory".
 
35
  ):
36
  """
37
  Args:
38
+ repo_id: The identifier of the repository in owner/repo format, e.g. "Storia-AI/sage".
39
  local_dir: The local directory where the repository will be cloned.
40
  inclusion_file: A file with a lists of files/directories/extensions to include. Each line must be in one of
41
  the following formats: "ext:.my-extension", "file:my-file.py", or "dir:my-directory".
{codesage β†’ sage}/embedder.py RENAMED
@@ -11,8 +11,8 @@ from typing import Dict, Generator, List, Optional, Tuple
11
  import marqo
12
  from openai import OpenAI
13
 
14
- from codesage.chunker import Chunk, Chunker
15
- from codesage.data_manager import DataManager
16
 
17
  Vector = Tuple[Dict, List[float]] # (metadata, embedding)
18
 
 
11
  import marqo
12
  from openai import OpenAI
13
 
14
+ from sage.chunker import Chunk, Chunker
15
+ from sage.data_manager import DataManager
16
 
17
  Vector = Tuple[Dict, List[float]] # (metadata, embedding)
18
 
{codesage β†’ sage}/github.py RENAMED
@@ -8,8 +8,8 @@ from typing import Any, Dict, Generator, List, Tuple
8
  import requests
9
  import tiktoken
10
 
11
- from codesage.chunker import Chunk, Chunker
12
- from codesage.data_manager import DataManager
13
 
14
  tokenizer = tiktoken.get_encoding("cl100k_base")
15
 
 
8
  import requests
9
  import tiktoken
10
 
11
+ from sage.chunker import Chunk, Chunker
12
+ from sage.data_manager import DataManager
13
 
14
  tokenizer = tiktoken.get_encoding("cl100k_base")
15
 
{codesage β†’ sage}/index.py RENAMED
@@ -7,11 +7,11 @@ import time
7
 
8
  import pkg_resources
9
 
10
- from codesage.chunker import UniversalFileChunker
11
- from codesage.data_manager import GitHubRepoManager
12
- from codesage.embedder import build_batch_embedder_from_flags
13
- from codesage.github import GitHubIssuesChunker, GitHubIssuesManager
14
- from codesage.vector_store import build_from_args
15
 
16
  logging.basicConfig(level=logging.INFO)
17
  logger = logging.getLogger()
 
7
 
8
  import pkg_resources
9
 
10
+ from sage.chunker import UniversalFileChunker
11
+ from sage.data_manager import GitHubRepoManager
12
+ from sage.embedder import build_batch_embedder_from_flags
13
+ from sage.github import GitHubIssuesChunker, GitHubIssuesManager
14
+ from sage.vector_store import build_from_args
15
 
16
  logging.basicConfig(level=logging.INFO)
17
  logger = logging.getLogger()
{codesage β†’ sage}/llm.py RENAMED
File without changes
{codesage β†’ sage}/sample-exclude.txt RENAMED
File without changes
{codesage β†’ sage}/vector_store.py RENAMED
File without changes
setup.py CHANGED
@@ -7,18 +7,18 @@ def readfile(filename):
7
 
8
 
9
  setup(
10
- name="codesage",
11
  version="0.1.0",
12
  packages=find_packages(),
13
  include_package_data=True,
14
  package_data={
15
- "codesage": ["sample-exclude.txt"],
16
  },
17
  install_requires=open("requirements.txt").readlines() + ["setuptools"],
18
  entry_points={
19
  "console_scripts": [
20
- "cs-index=codesage.index:main",
21
- "cs-chat=codesage.chat:main",
22
  ],
23
  },
24
  author="Julia Turc & Mihail Eric / Storia AI",
@@ -26,7 +26,7 @@ setup(
26
  description="A library to index a code repository and chat with it via LLMs.",
27
  long_description=open("README.md").read(),
28
  long_description_content_type="text/markdown",
29
- url="https://github.com/Storia-AI/codesage",
30
  classifiers=[
31
  "Programming Language :: Python :: 3",
32
  "License :: OSI Approved :: MIT License",
 
7
 
8
 
9
  setup(
10
+ name="sage",
11
  version="0.1.0",
12
  packages=find_packages(),
13
  include_package_data=True,
14
  package_data={
15
+ "sage": ["sample-exclude.txt"],
16
  },
17
  install_requires=open("requirements.txt").readlines() + ["setuptools"],
18
  entry_points={
19
  "console_scripts": [
20
+ "sage-index=sage.index:main",
21
+ "sage-chat=sage.chat:main",
22
  ],
23
  },
24
  author="Julia Turc & Mihail Eric / Storia AI",
 
26
  description="A library to index a code repository and chat with it via LLMs.",
27
  long_description=open("README.md").read(),
28
  long_description_content_type="text/markdown",
29
+ url="https://github.com/Storia-AI/sage",
30
  classifiers=[
31
  "Programming Language :: Python :: 3",
32
  "License :: OSI Approved :: MIT License",
tests/conftest.py CHANGED
@@ -1,4 +1,4 @@
1
  import os
2
  import sys
3
 
4
- sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../codesage")))
 
1
  import os
2
  import sys
3
 
4
+ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../sage")))
tests/test_chunker.py CHANGED
@@ -11,12 +11,12 @@ import os
11
 
12
  from pytest import mark, param
13
 
14
- import codesage.chunker
15
 
16
 
17
  def test_text_chunker_happy_path():
18
  """Tests the happy path for the TextFileChunker."""
19
- chunker = codesage.chunker.TextFileChunker(max_tokens=100)
20
 
21
  file_path = os.path.join(os.path.dirname(__file__), "../README.md")
22
  with open(file_path, "r") as file:
@@ -29,9 +29,9 @@ def test_text_chunker_happy_path():
29
 
30
  def test_code_chunker_happy_path():
31
  """Tests the happy path for the CodeFileChunker."""
32
- chunker = codesage.chunker.CodeFileChunker(max_tokens=100)
33
 
34
- file_path = os.path.join(os.path.dirname(__file__), "../codesage/chunker.py")
35
  with open(file_path, "r") as file:
36
  content = file.read()
37
  metadata = {"file_path": file_path}
@@ -48,22 +48,22 @@ def test_code_chunker_typescript(filename):
48
  content = file.read()
49
  metadata = {"file_path": file_path}
50
 
51
- chunker = codesage.chunker.CodeFileChunker(max_tokens=100)
52
  chunks = chunker.chunk(content, metadata)
53
  # There's a bug in the tree-sitter-language-pack library for TypeScript. Before it gets fixed, we expect this to
54
  # return an empty list (instead of crashing).
55
  assert len(chunks) == 0
56
 
57
  # However, the UniversalFileChunker should fallback onto a regular text chunker, and return some chunks.
58
- chunker = codesage.chunker.UniversalFileChunker(max_tokens=100)
59
  chunks = chunker.chunk(content, metadata)
60
  assert len(chunks) >= 1
61
 
62
 
63
  def test_ipynb_chunker_happy_path():
64
  """Tests the happy path for the IPynbChunker."""
65
- code_chunker = codesage.chunker.CodeFileChunker(max_tokens=100)
66
- chunker = codesage.chunker.IpynbFileChunker(code_chunker)
67
 
68
  file_path = os.path.join(os.path.dirname(__file__), "assets/sample-notebook.ipynb")
69
  with open(file_path, "r") as file:
 
11
 
12
  from pytest import mark, param
13
 
14
+ import sage.chunker
15
 
16
 
17
  def test_text_chunker_happy_path():
18
  """Tests the happy path for the TextFileChunker."""
19
+ chunker = sage.chunker.TextFileChunker(max_tokens=100)
20
 
21
  file_path = os.path.join(os.path.dirname(__file__), "../README.md")
22
  with open(file_path, "r") as file:
 
29
 
30
  def test_code_chunker_happy_path():
31
  """Tests the happy path for the CodeFileChunker."""
32
+ chunker = sage.chunker.CodeFileChunker(max_tokens=100)
33
 
34
+ file_path = os.path.join(os.path.dirname(__file__), "../sage/chunker.py")
35
  with open(file_path, "r") as file:
36
  content = file.read()
37
  metadata = {"file_path": file_path}
 
48
  content = file.read()
49
  metadata = {"file_path": file_path}
50
 
51
+ chunker = sage.chunker.CodeFileChunker(max_tokens=100)
52
  chunks = chunker.chunk(content, metadata)
53
  # There's a bug in the tree-sitter-language-pack library for TypeScript. Before it gets fixed, we expect this to
54
  # return an empty list (instead of crashing).
55
  assert len(chunks) == 0
56
 
57
  # However, the UniversalFileChunker should fallback onto a regular text chunker, and return some chunks.
58
+ chunker = sage.chunker.UniversalFileChunker(max_tokens=100)
59
  chunks = chunker.chunk(content, metadata)
60
  assert len(chunks) >= 1
61
 
62
 
63
  def test_ipynb_chunker_happy_path():
64
  """Tests the happy path for the IPynbChunker."""
65
+ code_chunker = sage.chunker.CodeFileChunker(max_tokens=100)
66
+ chunker = sage.chunker.IpynbFileChunker(code_chunker)
67
 
68
  file_path = os.path.join(os.path.dirname(__file__), "assets/sample-notebook.ipynb")
69
  with open(file_path, "r") as file: