Meet Casper, the Astral Librarian ✨, A Discord Bot for Searching the Web
Tags: linodehackathon, python, kubernetes, discord
Published: 2023-02-21 00:26:34

## What I built
Casper, the Astral Librarian, is a [Discord bot](https://discord.com/developers/docs/intro) for searching the internet. Unlike other web searching discord bots, Casper does not restrict search results to a single search engine, instead returning results from a whole host of search engines.

Out of the box, Casper supports a number of features provided by the underlying search engine, [SearXNG](https://github.com/searxng/searxng), including searching specific engines by formatting queries like this:
```
!<ENGINE_NAME> my search
```
For example,
```
!reddit nvidia 3000 series review
```
These features are described in greater detail in the [SearXNG documentation
](https://docs.searxng.org/user/index.html)

Here is a list of some of the search engines Casper currently supports:
* google
* bing
* duckduckgo
* qwant
* reddit
* stackoverflow
* github
* bitbucket
* gitlab
* docker hub
* deviant art
* imdb
* pypi
* npm
* packagist
* rubygems
* yahoo
* vimeo
* youtube
* twitter
* wolfram alpha

In addition to these features, I plan to add some exciting features like allowing users to add their own search engines to Casper as well as user and channel based customization options.

If things go really well, I may also port Casper to other messaging platforms too!

### Category Submission:
* Wacky Wildcards
* Integration Innovators

### App Link
Discord invite link:
[Invite Casper to your Server](https://discord.com/api/oauth2/authorize?client_id=1056051428575170752&permissions=2147485696&scope=bot)

Source code:
https://github.com/speratus/indexor-discord-bot

### Screenshots
Here is a series of screenshots demonstrating Casper in Use:

![Slash command suggestion](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f1ft9ze74hukhvq6yp8r.png)

![Typing a search query](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z2b2gz8wycf1fj6stmph.png)

![Results from the search](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gm2pwwix6xq0p13a4sef.png)

#### Screenshots of Kubernetes config
List of nodes in Kubernetes cluster according to Kubernetes control panel (identifying information has been removed):

![Nodes in Kubernetes resource pool](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nj6gavbzvbseew8mutxy.png)

Same list, but showing the nodes running as instances running in Linode's control panel (identifying information has been removed):

![Linodes incorporated into Kubernetes resource pool](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/samn6ber6m103138tjvh.png)

### Description
Casper is a Discord bot built with [Python](https://python.org), [discord.py](https://discordpy.readthedocs.io/en/stable/), [Docker](https://www.docker.com/), [Kubernetes](https://kubernetes.io/), and [SearXNG.](https://github.com/searxng/searxng/tree/master). The main goal of Casper is to allow users to search for web results directly from within Discord. Secondary goals include a high degree of flexibility, enabling users to search using their preferred search engine, and, to that end, supporting the widest array of search engines possible.

Finally, Casper allows users to share search results with each other, enabling them to establish trust and common sources of information as well as a discussion of the quality of sources.

There are several major components of Casper:
* [The bot itself](https://github.com/speratus/indexor-discord-bot)
* The [search library containing the main search features](https://github.com/speratus/indexor-core). This library is what will enable Casper to be ported to other messaging platforms
* The SearXNG Instance, powering web-wide searches

### Link to Source Code
Main Casper source code:
{% github https://github.com/speratus/indexor-discord-bot %}

Search functionality core library:
{% github https://github.com/speratus/indexor-core %}
The Indexor core library contains the primary search feature and will contain other planned features such as results caching and user based searching preferences.

### Permissive License
[MIT License](https://mit-license.org/)

## Background
I came up with the idea for Casper on a whim, thinking how interesting it would be to get search results directly in a Discord server.

I also saw this as an opportunity learn a couple of technologies I had been interested in, but had never created any projects with: [Docker](https://www.docker.com/) and [Kubernetes](https://kubernetes.io/).


### How I built it
I deployed Casper on top of Linode's Kubernetes engine. I chose Linode Kubernetes engine for this project so that it would be easy to scale, update and rollback as necessary. This also lets me exercise my interest in learning Kubernetes and deploying an application using it.

To deploy Casper on Linode's Kubernetes engine, I had to overcome several obstacles:
* Packaging the source code as a Docker image to which I could transfer necessary project information while keeping it out of the source code for the sake of privacy
```bash
docker build -t casper:1.0.2
```
* Creating a deployment for the SearXNG instances that had the configuration required to be compatible with the Casper source code
```yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: searxng-deploy
spec:
  replicas: ...
  selector:
    matchLabels:
      app: searxng
  template:
    metadata:
      labels:
        app: searxng
    spec:
      containers:
        - name: searxng
          image: searxng/searxng
          env:
            - name: INSTANCE_NAME
              value: 'casper-engine'
            - name: BASE_URL
              value: 'http://n.n.n.n'
          volumeMounts:
            - name: settings
              mountPath: /etc/searxng/settings.yml
              subPath: settings.yml
      volumes:
        - name: settings
          configMap:
            name: searxng-settings
```
* Creating a Kubernetes service for the SearXNG instances so that they could be load balanced as well as simultaneously enabling the Casper source to communicate with the instances and preventing the SearXNG instances from being accessible outside of the Kubernetes Cluster
```yml
apiVersion: v1
kind: Service
metadata:
  name: lb-searxng
spec:
  selector:
    app: searxng
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 8080
```
* Creating a `configmap` for the SearXNG instance that could be read as the `settings.yml` file and which also includes the required values to enable Casper to function.

```bash
kubectl create configmap searxng-settings --from-file=PATH/TO/FILE/searxng/settings.yml
```
* Creating a Kubernetes secret to store the Discord bot's token without exposing it to outside access.

```bash
kubectl create secret generic ...
```
* Creating a deployment for the bot itself which drew from the necessary secrets and had the proper environment variables to communicate with the SearXNG instances.

```yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: casper-deploy
spec:
  replicas: ...
  selector:
    matchLabels:
      app: casper
  template:
    metadata:
      labels:
        app: casper
    spec:
      containers:
        - name: casper-bot
          image: speratus/casper:1.0.2
          env:
            - name: DISCORD_TOKEN
              valueFrom:
                secretKeyRef:
                  name: ...
                  key: ...
            - name: ENGINE_URL
              value: 'http://n.n.n.n'
```

Once I had tested out each of these steps in detail and identified working settings for each component, deployment was as easy as running a few commands. Once that was done, Casper was live with no headaches.

**NOTE:** In these samples, I replaced IP address with `n` to indicate that anybody wishing to follow along should use their own information instead. I also used `...` to indicate regular values that need to be replaced with the information specific to your context as well.


### Additional Resources/Info
SearXNG's documentation for creating and running an instance:
https://docs.searxng.org/admin/index.html

SearXNG Source:
{% github https://github.com/searxng/searxng %}

#### Planned Features
* Multiple pages of results
* Results caching for faster response times
* Improved support for getting results from specific search engines
* User-added search engines (Any sites supporting the [OpenSearch](https://developer.mozilla.org/en-US/docs/Web/OpenSearch) standard)
* Server and User based settings
* Direct messaging Casper to get results in addition to using the slash command
* Support for other messaging platforms such as Slack, Telegram, Matrix.org, and others.

##### Unsupported features
Currently, Casper does not support SearXNG's special queries features.