Upload 11 files

CONTRIBUTING.md

@@ -0,0 +1,40 @@
+# How to Contribute to Perplexica
+
+Hey there, thanks for deciding to contribute to Perplexica. Anything you help with will support the development of Perplexica and will make it better. Let's walk you through the key aspects to ensure your contributions are effective and in harmony with the project's setup.
+
+## Project Structure
+
+Perplexica's design consists of two main domains:
+
+- **Frontend (`ui` directory)**: This is a Next.js application holding all user interface components. It's a self-contained environment that manages everything the user interacts with.
+- **Backend (root and `src` directory)**: The backend logic is situated in the `src` folder, but the root directory holds the main `package.json` for backend dependency management.
+
+## Setting Up Your Environment
+
+Before diving into coding, setting up your local environment is key. Here's what you need to do:
+
+### Backend
+
+1. In the root directory, locate the `sample.config.toml` file.
+2. Rename it to `config.toml` and fill in the necessary configuration fields specific to the backend.
+3. Run `npm install` to install dependencies.
+4. Run `npm run db:push` to set up the local sqlite.
+5. Use `npm run dev` to start the backend in development mode.
+
+### Frontend
+
+1. Navigate to the `ui` folder and repeat the process of renaming `.env.example` to `.env`, making sure to provide the frontend-specific variables.
+2. Execute `npm install` within the `ui` directory to get the frontend dependencies ready.
+3. Launch the frontend development server with `npm run dev`.
+
+**Please note**: Docker configurations are present for setting up production environments, whereas `npm run dev` is used for development purposes.
+
+## Coding and Contribution Practices
+
+Before committing changes:
+
+1. Ensure that your code functions correctly by thorough testing.
+2. Always run `npm run format:write` to format your code according to the project's coding standards. This helps maintain consistency and code quality.
+3. We currently do not have a code of conduct, but it is in the works. In the meantime, please be mindful of how you engage with the project and its community.
+
+Following these steps will help maintain the integrity of Perplexica's codebase and facilitate a smoother integration of your valuable contributions. Thank you for your support and commitment to improving Perplexica.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 ItzCrazyKns
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,11 +1,175 @@
----
-title: PerplexicaDocker
-emoji: 📈
-colorFrom: red
-colorTo: red
-sdk: docker
-pinned: false
-license: apache-2.0
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🚀 Perplexica - An AI-powered search engine 🔎 <!-- omit in toc -->
+
+![preview](.assets/perplexica-screenshot.png?)
+
+## Table of Contents <!-- omit in toc -->
+
+- [Overview](#overview)
+- [Preview](#preview)
+- [Features](#features)
+- [Installation](#installation)
+  - [Getting Started with Docker (Recommended)](#getting-started-with-docker-recommended)
+  - [Non-Docker Installation](#non-docker-installation)
+  - [Ollama Connection Errors](#ollama-connection-errors)
+- [Using as a Search Engine](#using-as-a-search-engine)
+- [Using Perplexica's API](#using-perplexicas-api)
+- [Expose Perplexica to a network](#expose-perplexica-to-network)
+- [One-Click Deployment](#one-click-deployment)
+- [Upcoming Features](#upcoming-features)
+- [Support Us](#support-us)
+  - [Donations](#donations)
+- [Contribution](#contribution)
+- [Help and Support](#help-and-support)
+
+## Overview
+
+Perplexica is an open-source AI-powered searching tool or an AI-powered search engine that goes deep into the internet to find answers. Inspired by Perplexity AI, it's an open-source option that not just searches the web but understands your questions. It uses advanced machine learning algorithms like similarity searching and embeddings to refine results and provides clear answers with sources cited.
+
+Using SearxNG to stay current and fully open source, Perplexica ensures you always get the most up-to-date information without compromising your privacy.
+
+Want to know more about its architecture and how it works? You can read it [here](https://github.com/ItzCrazyKns/Perplexica/tree/master/docs/architecture/README.md).
+
+## Preview
+
+![video-preview](.assets/perplexica-preview.gif)
+
+## Features
+
+- **Local LLMs**: You can make use local LLMs such as Llama3 and Mixtral using Ollama.
+- **Two Main Modes:**
+  - **Copilot Mode:** (In development) Boosts search by generating different queries to find more relevant internet sources. Like normal search instead of just using the context by SearxNG, it visits the top matches and tries to find relevant sources to the user's query directly from the page.
+  - **Normal Mode:** Processes your query and performs a web search.
+- **Focus Modes:** Special modes to better answer specific types of questions. Perplexica currently has 6 focus modes:
+  - **All Mode:** Searches the entire web to find the best results.
+  - **Writing Assistant Mode:** Helpful for writing tasks that does not require searching the web.
+  - **Academic Search Mode:** Finds articles and papers, ideal for academic research.
+  - **YouTube Search Mode:** Finds YouTube videos based on the search query.
+  - **Wolfram Alpha Search Mode:** Answers queries that need calculations or data analysis using Wolfram Alpha.
+  - **Reddit Search Mode:** Searches Reddit for discussions and opinions related to the query.
+- **Current Information:** Some search tools might give you outdated info because they use data from crawling bots and convert them into embeddings and store them in a index. Unlike them, Perplexica uses SearxNG, a metasearch engine to get the results and rerank and get the most relevant source out of it, ensuring you always get the latest information without the overhead of daily data updates.
+- **API**: Integrate Perplexica into your existing applications and make use of its capibilities.
+
+It has many more features like image and video search. Some of the planned features are mentioned in [upcoming features](#upcoming-features).
+
+## Installation
+
+There are mainly 2 ways of installing Perplexica - With Docker, Without Docker. Using Docker is highly recommended.
+
+### Getting Started with Docker (Recommended)
+
+1. Ensure Docker is installed and running on your system.
+2. Clone the Perplexica repository:
+
+   ```bash
+   git clone https://github.com/ItzCrazyKns/Perplexica.git
+   ```
+
+3. After cloning, navigate to the directory containing the project files.
+
+4. Rename the `sample.config.toml` file to `config.toml`. For Docker setups, you need only fill in the following fields:
+
+   - `OPENAI`: Your OpenAI API key. **You only need to fill this if you wish to use OpenAI's models**.
+   - `OLLAMA`: Your Ollama API URL. You should enter it as `http://host.docker.internal:PORT_NUMBER`. If you installed Ollama on port 11434, use `http://host.docker.internal:11434`. For other ports, adjust accordingly. **You need to fill this if you wish to use Ollama's models instead of OpenAI's**.
+   - `GROQ`: Your Groq API key. **You only need to fill this if you wish to use Groq's hosted models**.
+   - `ANTHROPIC`: Your Anthropic API key. **You only need to fill this if you wish to use Anthropic models**.
+
+     **Note**: You can change these after starting Perplexica from the settings dialog.
+
+   - `SIMILARITY_MEASURE`: The similarity measure to use (This is filled by default; you can leave it as is if you are unsure about it.)
+
+5. Ensure you are in the directory containing the `docker-compose.yaml` file and execute:
+
+   ```bash
+   docker compose up -d
+   ```
+
+6. Wait a few minutes for the setup to complete. You can access Perplexica at http://localhost:3000 in your web browser.
+
+**Note**: After the containers are built, you can start Perplexica directly from Docker without having to open a terminal.
+
+### Non-Docker Installation
+
+1. Install SearXNG and allow `JSON` format in the SearXNG settings.
+2. Clone the repository and rename the `sample.config.toml` file to `config.toml` in the root directory. Ensure you complete all required fields in this file.
+3. Rename the `.env.example` file to `.env` in the `ui` folder and fill in all necessary fields.
+4. After populating the configuration and environment files, run `npm i` in both the `ui` folder and the root directory.
+5. Install the dependencies and then execute `npm run build` in both the `ui` folder and the root directory.
+6. Finally, start both the frontend and the backend by running `npm run start` in both the `ui` folder and the root directory.
+
+**Note**: Using Docker is recommended as it simplifies the setup process, especially for managing environment variables and dependencies.
+
+See the [installation documentation](https://github.com/ItzCrazyKns/Perplexica/tree/master/docs/installation) for more information like exposing it your network, etc.
+
+### Ollama Connection Errors
+
+If you're encountering an Ollama connection error, it is likely due to the backend being unable to connect to Ollama's API. To fix this issue you can:
+
+1. **Check your Ollama API URL:** Ensure that the API URL is correctly set in the settings menu.
+2. **Update API URL Based on OS:**
+
+   - **Windows:** Use `http://host.docker.internal:11434`
+   - **Mac:** Use `http://host.docker.internal:11434`
+   - **Linux:** Use `http://<private_ip_of_host>:11434`
+
+   Adjust the port number if you're using a different one.
+
+3. **Linux Users - Expose Ollama to Network:**
+
+   - Inside `/etc/systemd/system/ollama.service`, you need to add `Environment="OLLAMA_HOST=0.0.0.0"`. Then restart Ollama by `systemctl restart ollama`. For more information see [Ollama docs](https://github.com/ollama/ollama/blob/main/docs/faq.md#setting-environment-variables-on-linux)
+
+   - Ensure that the port (default is 11434) is not blocked by your firewall.
+
+## Using as a Search Engine
+
+If you wish to use Perplexica as an alternative to traditional search engines like Google or Bing, or if you want to add a shortcut for quick access from your browser's search bar, follow these steps:
+
+1. Open your browser's settings.
+2. Navigate to the 'Search Engines' section.
+3. Add a new site search with the following URL: `http://localhost:3000/?q=%s`. Replace `localhost` with your IP address or domain name, and `3000` with the port number if Perplexica is not hosted locally.
+4. Click the add button. Now, you can use Perplexica directly from your browser's search bar.
+
+## Using Perplexica's API
+
+Perplexica also provides an API for developers looking to integrate its powerful search engine into their own applications. You can run searches, use multiple models and get answers to your queries.
+
+For more details, check out the full documentation [here](https://github.com/ItzCrazyKns/Perplexica/tree/master/docs/API/SEARCH.md).
+
+## Expose Perplexica to network
+
+You can access Perplexica over your home network by following our networking guide [here](https://github.com/ItzCrazyKns/Perplexica/blob/master/docs/installation/NETWORKING.md).
+
+## One-Click Deployment
+
+[![Deploy to RepoCloud](https://d16t0pc4846x52.cloudfront.net/deploylobe.svg)](https://repocloud.io/details/?app_id=267)
+
+## Upcoming Features
+
+- [x] Add settings page
+- [x] Adding support for local LLMs
+- [x] History Saving features
+- [x] Introducing various Focus Modes
+- [x] Adding API support
+- [x] Adding Discover
+- [ ] Finalizing Copilot Mode
+
+## Support Us
+
+If you find Perplexica useful, consider giving us a star on GitHub. This helps more people discover Perplexica and supports the development of new features. Your support is greatly appreciated.
+
+### Donations
+
+We also accept donations to help sustain our project. If you would like to contribute, you can use the following options to donate. Thank you for your support!
+
+| Ethereum                                              |
+| ----------------------------------------------------- |
+| Address: `0xB025a84b2F269570Eb8D4b05DEdaA41D8525B6DD` |
+
+## Contribution
+
+Perplexica is built on the idea that AI and large language models should be easy for everyone to use. If you find bugs or have ideas, please share them in via GitHub Issues. For more information on contributing to Perplexica you can read the [CONTRIBUTING.md](CONTRIBUTING.md) file to learn more about Perplexica and how you can contribute to it.
+
+## Help and Support
+
+If you have any questions or feedback, please feel free to reach out to us. You can create an issue on GitHub or join our Discord server. There, you can connect with other users, share your experiences and reviews, and receive more personalized help. [Click here](https://discord.gg/EFwsmQDgAu) to join the Discord server. To discuss matters outside of regular support, feel free to contact me on Discord at `itzcrazykns`.
+
+Thank you for exploring Perplexica, the AI-powered search engine designed to enhance your search experience. We are constantly working to improve Perplexica and expand its capabilities. We value your feedback and contributions which help us make Perplexica even better. Don't forget to check back for updates and new features!

app.dockerfile

@@ -0,0 +1,15 @@
+FROM node:20.18.0-alpine
+
+ARG NEXT_PUBLIC_WS_URL=ws://127.0.0.1:3001
+ARG NEXT_PUBLIC_API_URL=http://127.0.0.1:3001/api
+ENV NEXT_PUBLIC_WS_URL=${NEXT_PUBLIC_WS_URL}
+ENV NEXT_PUBLIC_API_URL=${NEXT_PUBLIC_API_URL}
+
+WORKDIR /home/perplexica
+
+COPY ui /home/perplexica/
+
+RUN yarn install --frozen-lockfile
+RUN yarn build
+
+CMD ["yarn", "start"]

backend.dockerfile

@@ -0,0 +1,17 @@
+FROM node:18-slim
+
+WORKDIR /home/perplexica
+
+COPY src /home/perplexica/src
+COPY tsconfig.json /home/perplexica/
+COPY drizzle.config.ts /home/perplexica/
+COPY package.json /home/perplexica/
+COPY yarn.lock /home/perplexica/
+
+RUN mkdir /home/perplexica/data
+RUN mkdir /home/perplexica/uploads
+
+RUN yarn install --frozen-lockfile --network-timeout 600000
+RUN yarn build
+
+CMD ["yarn", "start"]

config.toml

@@ -0,0 +1,14 @@
+[API_KEYS]
+OPENAI = ""
+GROQ = "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"
+ANTHROPIC = ""
+GEMINI = ""
+
+[API_ENDPOINTS]
+OLLAMA = ""
+SEARXNG = "http://localhost:32768"
+
+[GENERAL]
+PORT = 3_001
+SIMILARITY_MEASURE = "cosine"
+KEEP_ALIVE = "5m"

docker-compose.yaml

@@ -0,0 +1,54 @@
+services:
+  searxng:
+    image: docker.io/searxng/searxng:latest
+    volumes:
+      - ./searxng:/etc/searxng:rw
+    ports:
+      - 4000:8080
+    networks:
+      - perplexica-network
+    restart: unless-stopped
+
+  perplexica-backend:
+    build:
+      context: .
+      dockerfile: backend.dockerfile
+    image: itzcrazykns1337/perplexica-backend:main
+    environment:
+      - SEARXNG_API_URL=http://searxng:8080
+    depends_on:
+      - searxng
+    ports:
+      - 3001:3001
+    volumes:
+      - backend-dbstore:/home/perplexica/data
+      - uploads:/home/perplexica/uploads
+      - ./config.toml:/home/perplexica/config.toml
+    extra_hosts:
+      - 'host.docker.internal:host-gateway'
+    networks:
+      - perplexica-network
+    restart: unless-stopped
+
+  perplexica-frontend:
+    build:
+      context: .
+      dockerfile: app.dockerfile
+      args:
+        - NEXT_PUBLIC_API_URL=http://127.0.0.1:3001/api
+        - NEXT_PUBLIC_WS_URL=ws://127.0.0.1:3001
+    image: itzcrazykns1337/perplexica-frontend:main
+    depends_on:
+      - perplexica-backend
+    ports:
+      - 3000:3000
+    networks:
+      - perplexica-network
+    restart: unless-stopped
+
+networks:
+  perplexica-network:
+
+volumes:
+  backend-dbstore:
+  uploads:

drizzle.config.ts

@@ -0,0 +1,10 @@
+import { defineConfig } from 'drizzle-kit';
+
+export default defineConfig({
+  dialect: 'sqlite',
+  schema: './src/db/schema.ts',
+  out: './drizzle',
+  dbCredentials: {
+    url: './data/db.sqlite',
+  },
+});

package.json

@@ -0,0 +1,53 @@
+{
+  "name": "perplexica-backend",
+  "version": "1.10.0-rc2",
+  "license": "MIT",
+  "author": "ItzCrazyKns",
+  "scripts": {
+    "start": "npm run db:push && node dist/app.js",
+    "build": "tsc",
+    "dev": "nodemon --ignore uploads/ src/app.ts ",
+    "db:push": "drizzle-kit push sqlite",
+    "format": "prettier . --check",
+    "format:write": "prettier . --write"
+  },
+  "devDependencies": {
+    "@types/better-sqlite3": "^7.6.10",
+    "@types/cors": "^2.8.17",
+    "@types/express": "^4.17.21",
+    "@types/html-to-text": "^9.0.4",
+    "@types/multer": "^1.4.12",
+    "@types/pdf-parse": "^1.1.4",
+    "@types/readable-stream": "^4.0.11",
+    "@types/ws": "^8.5.12",
+    "drizzle-kit": "^0.22.7",
+    "nodemon": "^3.1.0",
+    "prettier": "^3.2.5",
+    "ts-node": "^10.9.2",
+    "typescript": "^5.4.3"
+  },
+  "dependencies": {
+    "@iarna/toml": "^2.2.5",
+    "@langchain/anthropic": "^0.2.3",
+    "@langchain/community": "^0.2.16",
+    "@langchain/openai": "^0.0.25",
+    "@langchain/google-genai": "^0.0.23",
+    "@xenova/transformers": "^2.17.1",
+    "axios": "^1.6.8",
+    "better-sqlite3": "^11.0.0",
+    "compute-cosine-similarity": "^1.1.0",
+    "compute-dot": "^1.1.0",
+    "cors": "^2.8.5",
+    "dotenv": "^16.4.5",
+    "drizzle-orm": "^0.31.2",
+    "express": "^4.19.2",
+    "html-to-text": "^9.0.5",
+    "langchain": "^0.1.30",
+    "mammoth": "^1.8.0",
+    "multer": "^1.4.5-lts.1",
+    "pdf-parse": "^1.1.1",
+    "winston": "^3.13.0",
+    "ws": "^8.17.1",
+    "zod": "^3.22.4"
+  }
+}

tsconfig.json

@@ -0,0 +1,18 @@
+{
+  "compilerOptions": {
+    "lib": ["ESNext"],
+    "module": "Node16",
+    "moduleResolution": "Node16",
+    "target": "ESNext",
+    "outDir": "dist",
+    "sourceMap": false,
+    "esModuleInterop": true,
+    "experimentalDecorators": true,
+    "emitDecoratorMetadata": true,
+    "allowSyntheticDefaultImports": true,
+    "skipLibCheck": true,
+    "skipDefaultLibCheck": true
+  },
+  "include": ["src"],
+  "exclude": ["node_modules", "**/*.spec.ts"]
+}