Repository: truera/trulens
Issue: [FEAT] Change database configuration
Root cause: Architectural Constraint

**Feature Description**
We are trying to implement Trulens-eval in an existing project that already uses a Postgresql database and Alembic to manage the migrations.
We have changed the database used by Trulens but, when we start the application, it runs [some checks](https://github.com/truera/trulens/blob/8826a03cb7881f690cd54c438ff844a6d229a89c/trulens_eval/trulens_eval/database/sqlalchemy_db.py#L54) to detect the status of the migrations applied to the database and, as we are already using Alembic with the default configuration, it tries to read the revision from the `alembic_version` table and compare it to the last revision in Trulens migrations, which make it fail.
We tried to change the Alembic configuration used by Trulens so we could change the revision table name but we don't see any option to do it.

**Reason**
 It would be great if Trulens allowed us to change these database/migrations configurations so we could use it in the same database that the application uses right now.

As a quick fix, we try to apply some monkey patching to some classes, for example, [adding the `version_table` param to the migrations file](https://github.com/truera/trulens/blob/8826a03cb7881f690cd54c438ff844a6d229a89c/trulens_eval/trulens_eval/database/migrations/env.py#L41). It almost works but the dashboard doesn't run (as it runs in a subprocess, I think that it is not able to read the patch applied and uses the default implementation).

**Importance of Feature**
We could have the Trulens feedback data in the same database where we have the application data, which allows us to create some reports about the performance of the application.
Also, we don't need to create a new database to store this information (with the maintenance associated).


<!-- Answer -->
Based on your description, it seems you're trying to configure Trulens to use a different database and migrations settings to avoid conflicts with your existing application that uses a PostgreSQL database and Alembic for migrations. Here are some steps you can follow:

1. **Specify a Different Database URL**: You can change the `sqlalchemy.url` in the `alembic.ini` file to point to a different database or schema. This will isolate Trulens' migrations from your main application. For PostgreSQL, the URL format is `postgresql://user:password@localhost/dbname`. You might need to create a separate schema or database for Trulens and adjust the URL accordingly.

    ```ini
    sqlalchemy.url = postgresql://user:password@localhost/trulens_db
    ```

2. **Customize Migration Script Location**: To keep Trulens' migration scripts separate from your main application's migration scripts, ensure the `script_location` under the `[alembic]` section points to a unique directory within the Trulens project structure.

    ```ini
    script_location = trulens_eval/trulens_eval/database/migrations
    ```

3. **Use a Different `alembic_version` Table**: To avoid conflicts with the `alembic_version` table, you can configure Alembic to use a different table name for Trulens. This requires modifying the `env.py` script within the migrations directory to specify a different version table name when configuring the `context.configure()` call. Unfortunately, the provided `alembic.ini` file does not directly support this change, as it's a change that needs to be made in the `env.py` script, not in the `alembic.ini`.

    ```python
    context.configure(
        ...
        version_table='trulens_alembic_version',
        ...
    )
    ```

4. **Adjust the `version_locations`**: If you're using multiple version directories (for example, if Trulens and your main application need to maintain separate migration histories), you can specify these directories in the `version_locations` setting. This allows you to isolate Trulens' migrations from your main application's migrations. Make sure to use the correct path separator as per your operating system or the `version_path_separator` setting.

    ```ini
    version_locations = %(here)s/trulens_eval/trulens_eval/database/migrations/versions
    ```

5. **Review and Adjust Post Write Hooks**: If you're using post-write hooks (e.g., for code formatting with Black), ensure that these hooks are correctly configured for your environment. If Trulens' migrations are in a separate directory, you might need to adjust the paths or settings to ensure that the hooks are applied correctly to the new migration files.

By following these steps and adjusting the `alembic.ini` file accordingly, you can configure Trulens to use a separate database and manage its migrations independently, avoiding conflicts with your existing application's `alembic_version` table and ensuring smooth operation of the dashboard subprocess.

Regarding the database connection in the Trulens codebase, it is established through the `LocalSQLite` class in the `trulens_eval/trulens_eval/db.py` file. This class is responsible for handling SQLite database operations, including connecting to the database, creating tables, inserting records, and querying data. The connection to the SQLite database is established using the `_connect` method of the `LocalSQLite` class. This method creates a connection to the SQLite database file specified by the `filename` attribute of the class instance. The method returns a tuple containing the connection object and a cursor object, which can be used to execute SQL commands.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

## Sources
- [trulens_eval/trulens_eval/database/migrations/alembic.ini](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/database/migrations/alembic.ini)
- [trulens_eval/trulens_eval/db.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/db.py)

<!-- Dosu Comment Footer -->
<details open>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>

Hi; I'm prototyping various ways in which we can let this be configurable. Let us know if you have any suggestions.

<!-- Answer -->
Hi, @adrycerdeiro

I'm helping the trulens team manage their backlog and am marking this issue as stale. The issue originally requested the ability to change database and migration configurations in Trulens to integrate with an existing project using a Postgresql database and Alembic for migrations. I provided a detailed response with steps to configure Trulens to use a separate database and manage its migrations independently. Additionally, piotrm0 mentioned that they are prototyping various ways to make this configurable and invited suggestions from the community.

Could you please confirm if this issue is still relevant to the latest version of the trulens repository? If it is, please let the trulens team know by commenting on the issue. Otherwise, feel free to close the issue yourself or the issue will be automatically closed in 7 days.

Thank you!




This issue is still relevant to the latest version

Hi @adrycerdeiro , can you check out this PR: https://github.com/truera/trulens/pull/971 . It adds a table prefix option and changes the default to "trulens_" which should make it not interfere with existing alembic-managed tables.

Hi @piotrm0, I'm not a Python expert (I just started to use it about 5 months ago) but the PR changes look good to me! Thanks for your work 🙏🏻