ronantakizawa/codereview-bench huggingface dataset:
Repository: LazyVim/LazyVim
File: lua/lazyvim/plugins/extras/lang/typescript.lua (Lua)

Code before:
                "<node_internals>/**",
                "node_modules/**",
              },
              resolveSourceMapLocations = {
                "${workspaceFolder}/**",
                "!**/node_modules/**",
              },
            },
          }
        end
      end
    end,
  },

  -- Filetype icons
  {
    "nvim-mini/mini.icons",
    opts = {
      file = {
        [".eslintrc.js"] = { glyph = "󰱺", hl = "MiniIconsYellow" },
        [".node-version"] = { glyph = "", hl = "MiniIconsGreen" },
        [".prettierrc"] = { glyph = "", hl = "MiniIconsPurple" },
        [".yarnrc.yml"] = { glyph = "", hl = "MiniIconsBlue" },
        ["eslint.config.js"] = { glyph = "󰱺", hl = "MiniIconsYellow" },
        ["package.json"] = { glyph = "", hl = "MiniIconsGreen" },
        ["tsconfig.json"] = { glyph = "", hl = "MiniIconsAzure" },
        ["tsconfig.build.json"] = { glyph = "", hl = "MiniIconsAzure" },
        ["yarn.lock"] = { glyph = "", hl = "MiniIconsBlue" },
      },
    },
  },
}


Review comment (bug):
Why is this needed? we already have `js-debug-adapter` in mason

Code after:
              sourceMaps = true,
              runtimeExecutable = runtimeExecutable,
              skipFiles = {
                "<node_internals>/**",
                "node_modules/**",
              },
              resolveSourceMapLocations = {
                "${workspaceFolder}/**",
                "!**/node_modules/**",
              },
            },
          }
        end
      end
    end,
  },

  {
    "jay-babu/mason-nvim-dap.nvim",
    optional = true,
    opts = {
      -- chrome adapter is deprecated, use js-debug-adapter instead
      automatic_installation = { exclude = { "chrome" } },
    },
  },

  -- Filetype icons
  {
    "nvim-mini/mini.icons",
    opts = {
      file = {
        [".eslintrc.js"] = { glyph = "󰱺", hl = "MiniIconsYellow" },
        [".node-version"] = { glyph = "", hl = "MiniIconsGreen" },
        [".prettierrc"] = { glyph = "", hl = "MiniIconsPurple" },
        [".yarnrc.yml"] = { glyph = "", hl = "MiniIconsBlue" },
        ["eslint.config.js"] = { glyph = "󰱺", hl = "MiniIconsYellow" },
        ["package.json"] = { glyph = "", hl = "MiniIconsGreen" },
        ["tsconfig.json"] = { glyph = "", hl = "MiniIconsAzure" },
        ["tsconfig.build.json"] = { glyph = "", hl = "MiniIconsAzure" },
        ["yarn.lock"] = { glyph = "", hl = "MiniIconsBlue" },
      },
    },
  },
}
Repository: Mintplex-Labs/anything-llm
File: collector/scripts/sitemap.py (Python)

Code before:
import requests
import xml.etree.ElementTree as ET
from scripts.link import parse_links

def parse_sitemap(url):
    response = requests.get(url)
    root = ET.fromstring(response.content)
    
    urls = []
    for element in root.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}url'):
        for loc in element.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}loc'):
            urls.append(loc.text)
    
    return urls

def sitemap():
    sitemap_url = input("Enter the URL of the sitemap: ")
    
    if(len(sitemap_url) == 0):
        print("No valid sitemap provided!")
        exit(1)

    url_array = parse_sitemap(sitemap_url)

    #parse links from array
    parse_links(url_array)


Review comment (suggestion):
Can you add a comment above this line like there is in `links.py` to a URL of a sitemap (can be anywhere) as it helps people know what a good example target looks like.



Code after:
import requests
import xml.etree.ElementTree as ET
from scripts.link import parse_links

def parse_sitemap(url):
    response = requests.get(url)
    root = ET.fromstring(response.content)
    
    urls = []
    for element in root.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}url'):
        for loc in element.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}loc'):
            urls.append(loc.text)
    
    return urls

# Example sitemap URL https://www.nerdwallet.com/blog/wp-sitemap-news-articles-1.xml
def sitemap():
    sitemap_url = input("Enter the URL of the sitemap: ")
    
    if(len(sitemap_url) == 0):
        print("No valid sitemap provided!")
        exit(1)

    url_array = parse_sitemap(sitemap_url)

    #parse links from array
    parse_links(url_array)
Repository: Mintplex-Labs/anything-llm
File: collector/scripts/twitter.py (Python)

Code before:
            with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file:
                json.dump(meta_link, file, ensure_ascii=True, indent=4)

            with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file:
                json.dump(meta_link, file, ensure_ascii=True, indent=4)
            
            # print(f"{transaction_output_dir}/{transaction_output_filename}")

        print(f"{tokenCount} tokens written over {tweets_df.shape[0]} records.")

    except BaseException as e:
        print("Status Failed: ", str(e))
        time.sleep(3)


def twitter_meta(row, metadata_only = False):
  url = f"http://twitter.com/anyuser/status/{row['id']}"
  title = f"Tweet {row['id']}"
  meta = {
    'url': url,
    'title':  title,
    'description': 'Tweet from ' + row["Screen Name"],
    'published': row["Date Created"].strftime('%Y-%m-%d %H:%M:%S'),
    'wordCount': len(row["Tweet"]),
  }
  return "Tweet JSON Metadata:\n"+json.dumps(meta)+"\n\n\nText Content:\n" + row["Tweet"] if metadata_only == False else meta


Review comment (suggestion):
I am fairly sure this function does not work because `http://twitter.com/anyuser/status/{row['id']}` will try to resolve the `@anyuser` handle doing this - which won't work since the users handle can be anything.


Code after:
            meta_link['pageContent'] = full_text
            meta_link['token_count_estimate'] = tokenCount
            totalTokens += tokenCount

            with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file:
                json.dump(meta_link, file, ensure_ascii=True, indent=4)

            with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file:
                json.dump(meta_link, file, ensure_ascii=True, indent=4)
            
            # print(f"{transaction_output_dir}/{transaction_output_filename}")

        print(f"{tokenCount} tokens written over {tweets_df.shape[0]} records.")

    except BaseException as e:
        print("Status Failed: ", str(e))
        time.sleep(3)


def twitter_meta(row, metadata_only = False):
  # Note that /anyuser is a known twitter hack for not knowing the user's handle
  # https://stackoverflow.com/questions/897107/can-i-fetch-the-tweet-from-twitter-if-i-know-the-tweets-id
  url = f"http://twitter.com/anyuser/status/{row['id']}"
  title = f"Tweet {row['id']}"
  meta = {
    'url': url,
    'title':  title,
    'description': 'Tweet from ' + row["Screen Name"],
    'published': row["Date Created"].strftime('%Y-%m-%d %H:%M:%S'),
    'wordCount': len(row["Tweet"]),
  }
  return "Tweet JSON Metadata:\n"+json.dumps(meta)+"\n\n\nText Content:\n" + row["Tweet"] if metadata_only == False else meta
Repository: Mintplex-Labs/anything-llm
File: frontend/src/models/system.js (JavaScript)

Code before:
  chats: async (offset = 0) => {
    return await fetch(`${API_BASE}/system/workspace-chats`, {
      method: "POST",
      headers: baseHeaders(),
      body: JSON.stringify({ offset }),
    })
      .then((res) => res.json())
      .catch((e) => {
        console.error(e);
        return [];
      });
  },
  deleteChat: async (chatId) => {
    return await fetch(`${API_BASE}/system/workspace-chats/${chatId}`, {
      method: "DELETE",
      headers: baseHeaders(),
    })
      .then((res) => res.json())
      .catch((e) => {
        console.error(e);
        return { success: false, error: e.message };
      });
  },
  exportChats: async (type = "csv") => {
    const url = `${API_BASE}/system/export-chats?type=${type}`;
    return await fetch(url, {
      method: "GET",
      headers: baseHeaders(),
    })
      .then((res) => res.text())
      .catch((e) => {
        console.error(e);
        return null;
      });
  },
  updateUser: async (data) => {
    return await fetch(`${API_BASE}/system/user`, {
      method: "POST",
      headers: baseHeaders(),
      body: JSON.stringify(data),
    })
      .then((res) => res.json())
      .catch((e) => {
        console.error(e);
        return { success: false, error: e.message };
      });
  },
  dataConnectors: DataConnector,
};

export default System;

Review comment (suggestion):
Use search params constructor to encode + safely encode URL options
```
const url = new URL(`${fullApiUrl()}/system/export-chats`);
 url.searchParams.append("type", encodeURIComponent(type));
 ```

Code after:
    url.searchParams.append("type", encodeURIComponent(type));
    return await fetch(url, {
      method: "GET",
      headers: baseHeaders(),
    })
      .then((res) => res.text())
      .catch((e) => {
        console.error(e);
        return null;
      });
  },
  updateUser: async (data) => {
    return await fetch(`${API_BASE}/system/user`, {
      method: "POST",
      headers: baseHeaders(),
      body: JSON.stringify(data),
    })
      .then((res) => res.json())
      .catch((e) => {
        console.error(e);
        return { success: false, error: e.message };
      });
  },
  dataConnectors: DataConnector,
};

export default System;
Repository: Mintplex-Labs/anything-llm
File: server/utils/helpers/chat/convertTo.js (JavaScript)

Code before:
      .replace(/"/g, '""')
      .replace(/\n/g, " ")}"`;
    const escapedResponse = `"${item.response
      .replace(/"/g, '""')
      .replace(/\n/g, " ")}"`;
    const row = `"${item.id}","${item.username}","${item.workspace}",${escapedPrompt},${escapedResponse},"${item.sent_at}"`;
    rows.push(row);
  }
  return rows.join("\n");
}

async function convertToJSON(workspaceChatsMap) {
  const allMessages = [].concat.apply(
    [],
    Object.values(workspaceChatsMap).map((workspace) => workspace.messages)
  );
  return JSON.stringify(allMessages);
}

async function convertToJSONL(workspaceChatsMap) {
  return Object.values(workspaceChatsMap)
    .map((workspaceChats) => JSON.stringify(workspaceChats))
    .join("\n");
}

async function prepareWorkspaceChatsForExport(type = "jsonl") {
  const chats = await WorkspaceChats.whereWithData({}, null, null, {
    id: "asc",
  });

  if (type === "csv") {
    const preparedData = chats.map((chat) => {
      const responseJson = JSON.parse(chat.response);
      return {
        id: chat.id,
        username: chat.user ? chat.user.username : "unknown user",
        workspace: chat.workspace ? chat.workspace.name : "unknown workspace",
        prompt: chat.prompt,
        response: responseJson.text,
        sent_at: chat.createdAt,
      };
    });

    return preparedData;
  }

  const workspaceIds = [...new Set(chats.map((chat) => chat.workspaceId))];

  const workspacesWithPrompts = await Promise.all(
    workspaceIds.map((id) => Workspace.get({ id: Number(id) }))
  );

Review comment (suggestion):
Do a validation check here to ensure that the type is in the `exportMap` key set.

Code after:
      escapeCsv(item.username),
      escapeCsv(item.workspace),
      escapeCsv(item.prompt),
      escapeCsv(item.response),
      item.sent_at,
    ].join(",");
    rows.push(record);
  }
  return rows.join("\n");
}

async function convertToJSON(workspaceChatsMap) {
  const allMessages = [].concat.apply(
    [],
    Object.values(workspaceChatsMap).map((workspace) => workspace.messages)
  );
  return JSON.stringify(allMessages);
}

async function convertToJSONL(workspaceChatsMap) {
  return Object.values(workspaceChatsMap)
    .map((workspaceChats) => JSON.stringify(workspaceChats))
    .join("\n");
}

async function prepareWorkspaceChatsForExport(format = "jsonl") {
  if (!exportMap.hasOwnProperty(format))
    throw new Error("Invalid export type.");

  const chats = await WorkspaceChats.whereWithData({}, null, null, {
    id: "asc",
  });

  if (format === "csv") {
    const preparedData = chats.map((chat) => {
      const responseJson = JSON.parse(chat.response);
      return {
        id: chat.id,
        username: chat.user ? chat.user.username : "unknown user",
        workspace: chat.workspace ? chat.workspace.name : "unknown workspace",
        prompt: chat.prompt,
        response: responseJson.text,
        sent_at: chat.createdAt,
      };
    });

    return preparedData;
  }

  const workspaceIds = [...new Set(chats.map((chat) => chat.workspaceId))];
Repository: Mintplex-Labs/anything-llm
File: collector/utils/extensions/PaperlessNgx/PaperlessNgxLoader/index.js (JavaScript)

Code before:
          );
        }

        const data = await response.json();
        documents.push(...(data.results || []));
        nextUrl = data.next;
        page++;
      }

      console.log(
        `Fetched ${documents.length} documents from Paperless-ngx (Pages: ${
          page - 1
        })`
      );

      const documentsWithContent = await Promise.all(
        documents.map(async (doc) => {
          const content = await this.fetchDocumentContent(doc.id);
          return { ...doc, content };
        })
      );

      return documentsWithContent.filter((doc) => !!doc.content);
    } catch (error) {
      throw new Error(
        `Failed to fetch documents from Paperless-ngx: ${error.message}`
      );
    }
  }

  /**
   * Fetches the content of a document from Paperless-ngx
   * @param {string} documentId - The ID of the document to fetch
   * @returns {Promise<string>} The content of the document
   */
  async fetchDocumentContent(documentId) {
    try {
      const response = await fetch(
        `${this.baseUrl}/api/documents/${documentId}/download/`,
        {
          headers: this.baseHeaders,
        }
      );

      if (!response.ok)
        throw new Error(`Failed to fetch document content: ${response.status}`);

      const contentType = response.headers.get("content-type");
      switch (contentType) {
        case "text/plain":
          return await response.text();

Review comment (bug):
This needs try/catch handler for bad requests like 429s or 404s or anything else that could go wrong so we can still break out the while loop

Code after:
              );
            return res.json();
          });

          const validResults = data.results.filter((doc) => doc?.id);
          if (!validResults.length) break;

          documents.push(...validResults);

          if (data.next === nextUrl) break;
          nextUrl = data.next || null;
          page++;
        } catch (error) {
          console.error(
            `Error fetching page ${page} from Paperless-ngx:`,
            error
          );
          break;
        }
      }

      console.log(
        `Fetched ${documents.length} documents from Paperless-ngx (Pages: ${
          page - 1
        })`
      );

      const documentsWithContent = await Promise.all(
        documents.map(async (doc) => {
          const content = await this.fetchDocumentContent(doc.id);
          return { ...doc, content };
        })
      );

      return documentsWithContent.filter((doc) => !!doc.content);
    } catch (error) {
      throw new Error(
        `Failed to fetch documents from Paperless-ngx: ${error.message}`
      );
    }
  }

  /**
   * Fetches the content of a document from Paperless-ngx
   * @param {string} documentId - The ID of the document to fetch
   * @returns {Promise<string>} The content of the document
   */
  async fetchDocumentContent(documentId) {
    try {
      const response = await fetch(
        `${this.baseUrl}/api/documents/${documentId}/download/`,
Repository: Mintplex-Labs/anything-llm
File: frontend/src/pages/OnboardingFlow/Steps/Survey/index.jsx (JavaScript)

Code before:
      !!formData.get("comment")
    ) {
      formRef.current.requestSubmit();
      return;
    }

    skipSurvey();
  }

  function skipSurvey() {
    navigate(paths.home());
  }

  function handleBack() {
    navigate(paths.onboarding.dataHandling());
  }

  useEffect(() => {
    setHeader({ title: TITLE, description: DESCRIPTION });
    setForwardBtn({ showing: true, disabled: false, onClick: handleForward });
    setBackBtn({ showing: true, disabled: false, onClick: handleBack });

    async function createDefaultWorkspace() {
      const workspaces = await Workspace.all();
      if (workspaces.length === 0) {
        await Workspace.new({ name: "My Workspace", onboardingComplete: true });
      }
    }
    createDefaultWorkspace();
  }, []);

  const handleSubmit = async (e) => {
    e.preventDefault();
    const form = e.target;
    const formData = new FormData(form);

    await sendQuestionnaire({
      email: formData.get("email"),
      useCase: formData.get("use_case") || "other",
      comment: formData.get("comment") || null,
    });

    navigate(paths.home());
  };

  if (!!window?.localStorage?.getItem(COMPLETE_QUESTIONNAIRE)) {
    return (
      <div className="w-full flex justify-center items-center py-40">
        <div className="w-full flex items-center justify-center px-1 md:px-8 py-4">
          <div className="w-auto flex flex-col gap-y-1 items-center">
            <CheckCircle size={60} className="text-green-500" />

Review comment (suggestion):
Use `t("new-workspace.placeholder")` as it will already be translated and be in the user native language.

Code after:
      !!formData.get("comment")
    ) {
      formRef.current.requestSubmit();
      return;
    }

    skipSurvey();
  }

  function skipSurvey() {
    navigate(paths.home());
  }

  function handleBack() {
    navigate(paths.onboarding.dataHandling());
  }

  useEffect(() => {
    setHeader({ title: TITLE, description: DESCRIPTION });
    setForwardBtn({ showing: true, disabled: false, onClick: handleForward });
    setBackBtn({ showing: true, disabled: false, onClick: handleBack });
  }, []);

  useEffect(() => {
    async function createDefaultWorkspace() {
      const workspaces = await Workspace.all();
      if (workspaces.length === 0) {
        await Workspace.new({
          name: t("new-workspace.placeholder"),
          onboardingComplete: true,
        });
      }
    }
    createDefaultWorkspace();
  }, []);

  const handleSubmit = async (e) => {
    e.preventDefault();
    const form = e.target;
    const formData = new FormData(form);

    await sendQuestionnaire({
      email: formData.get("email"),
      useCase: formData.get("use_case") || "other",
      comment: formData.get("comment") || null,
    });

    navigate(paths.home());
  };

  if (!!window?.localStorage?.getItem(COMPLETE_QUESTIONNAIRE)) {
Repository: Mintplex-Labs/anything-llm
File: frontend/src/pages/OnboardingFlow/Steps/Survey/index.jsx (JavaScript)

Code before:
      !!formData.get("use_case") ||
      !!formData.get("comment")
    ) {
      formRef.current.requestSubmit();
      return;
    }

    skipSurvey();
  }

  function skipSurvey() {
    navigate(paths.home());
  }

  function handleBack() {
    navigate(paths.onboarding.dataHandling());
  }

  useEffect(() => {
    setHeader({ title: TITLE, description: DESCRIPTION });
    setForwardBtn({ showing: true, disabled: false, onClick: handleForward });
    setBackBtn({ showing: true, disabled: false, onClick: handleBack });

    async function createDefaultWorkspace() {
      const workspaces = await Workspace.all();
      if (workspaces.length === 0) {
        await Workspace.new({ name: "My Workspace", onboardingComplete: true });
      }
    }
    createDefaultWorkspace();
  }, []);

  const handleSubmit = async (e) => {
    e.preventDefault();
    const form = e.target;
    const formData = new FormData(form);

    await sendQuestionnaire({
      email: formData.get("email"),
      useCase: formData.get("use_case") || "other",
      comment: formData.get("comment") || null,
    });

    navigate(paths.home());
  };

  if (!!window?.localStorage?.getItem(COMPLETE_QUESTIONNAIRE)) {
    return (
      <div className="w-full flex justify-center items-center py-40">
        <div className="w-full flex items-center justify-center px-1 md:px-8 py-4">
          <div className="w-auto flex flex-col gap-y-1 items-center">

Review comment (suggestion):
The creation of a workspace should be its own `useEffect` just for a more clear seperation of concerns (UI v business logic)

Code after:
      !!formData.get("use_case") ||
      !!formData.get("comment")
    ) {
      formRef.current.requestSubmit();
      return;
    }

    skipSurvey();
  }

  function skipSurvey() {
    navigate(paths.home());
  }

  function handleBack() {
    navigate(paths.onboarding.dataHandling());
  }

  useEffect(() => {
    setHeader({ title: TITLE, description: DESCRIPTION });
    setForwardBtn({ showing: true, disabled: false, onClick: handleForward });
    setBackBtn({ showing: true, disabled: false, onClick: handleBack });
  }, []);

  useEffect(() => {
    async function createDefaultWorkspace() {
      const workspaces = await Workspace.all();
      if (workspaces.length === 0) {
        await Workspace.new({
          name: t("new-workspace.placeholder"),
          onboardingComplete: true,
        });
      }
    }
    createDefaultWorkspace();
  }, []);

  const handleSubmit = async (e) => {
    e.preventDefault();
    const form = e.target;
    const formData = new FormData(form);

    await sendQuestionnaire({
      email: formData.get("email"),
      useCase: formData.get("use_case") || "other",
      comment: formData.get("comment") || null,
    });

    navigate(paths.home());
  };
Repository: Mintplex-Labs/anything-llm
File: frontend/src/utils/username.js (JavaScript)

Code before:
/**
 * Unix-style username validation utilities
 *
 * Requirements:
 * - 2-32 characters long
 * - Must start with a lowercase letter
 * - Can contain lowercase letters, digits, underscores, hyphens, and periods
 */

export const USERNAME_REGEX = /^[a-z][a-z0-9._-]*$/;
export const USERNAME_MIN_LENGTH = 2;
export const USERNAME_MAX_LENGTH = 32;

/**
 * HTML5 pattern attribute for username inputs (without ^ and $)
 */
export const USERNAME_PATTERN = "[a-z][a-z0-9._-]*";

/**
 * Validates a username string
 * @param {string} username - The username to validate
 * @returns {{ valid: boolean, error: string | null }}
 */
export function validateUsername(username) {
  if (!username || typeof username !== "string") {
    return { valid: false, error: "Username is required" };

Review comment (refactor):
I can see the usefulness of this utility. It does feel like we should be only doing validations on the backend, though. This keeps the actual logic in a single place and the frontend is simply displaying the input with its known field restrictions.

This would be additional work though since we dont have an endpoint to do this and I wonder if that is even worth doing.

Code after:
/**
 * Unix-style username validation utilities
 *
 * Requirements:
 * - 2-32 characters long
 * - Must start with a lowercase letter
 * - Can contain lowercase letters, digits, underscores, hyphens, @ signs, and periods
 */

export const USERNAME_REGEX = /^[a-z][a-z0-9._@-]*$/;
export const USERNAME_MIN_LENGTH = 2;
export const USERNAME_MAX_LENGTH = 32;

/**
 * HTML5 pattern attribute for username inputs (without ^ and $)
 */
export const USERNAME_PATTERN = "[a-z][a-z0-9._@-]*";