Repository: promptfoo/promptfoo
Issue: Deploying Promptfoo to Railway
Root cause: Configuration Error

I've just deployed promptfoo on Railway. I created a volume and my evals are saved and persistant between deploys. But when I access to prompts and datasets there is nothing there.
I've also configured the Share variables, but got an error from Railway when I click share button on UI.
Is there any super savvy guru here who can help me?

Hi @tonimatas1, what's the error when you click the share button in the UI?  Are you setting the `PROMPTFOO_REMOTE_API_BASE_URL` and `PROMPTFOO_REMOTE_APP_BASE_URL` environment variables locally before running `promptfoo view` or `promptfoo share`?

<img width="473" alt="Screenshot 2024-03-28 at 14 49 25" src="https://github.com/promptfoo/promptfoo/assets/127955819/5fb6ca8c-d775-4834-9220-1afe29b57bae">
Completely opaque, as you can see.


I think I have issues understanding the path and the folders in Railway deployment and, as the system does everything by itself and I have no access to folders info, I feel completely lost.
The evaluations run fine, and the evals are persistent, but prompts and datasets does not appear in their sections and share yields the error I pasted before.

Anything in the browser console/network tab?

Are you running evals through the web UI, or are you sending them to the server with `promptfoo share`?

I'm running the evals through the web UI.
I'm not sure what I've changed but now there is not the same error when sharing but the 404 Eval not found.
Again everything is running fine but prompts and datasets are not showing up and evals are persisting when I do a new deployment in Railway so the files should be there.

This is the mount path where I mounted the volume:

Mount Path
Directory that this volume is mounted to in promptfoo
/root/.promptfoo

And these are the environtment variables except OPENAI:
NEXT_PUBLIC_PROMPTFOO_REMOTE_API_BASE_URL=https://promptfoo.up.railway.app
PROMPTFOO_REMOTE_API_BASE_URL=${{RAILWAY_PUBLIC_DOMAIN}}
PROMPTFOO_REMOTE_APP_BASE_URL=${{RAILWAY_PUBLIC_DOMAIN}}
PROMPTFOO_SHARE_STORE_PATH=/root/.promptfoo/share-store
PROMPTFOO_SHARE_STORE_TYPE=filesystem

${{RAILWAY_PUBLIC_DOMAIN}} is promptfoo.up.railway.app






Yes, tons of errors in the browser console:

setup/:1 The resource https://promptfoo.up.railway.app/_next/static/css/87fb00eed311e9cd.css was preloaded using link preload but not used within a few seconds from the window's load event. Please make sure it has an appropriate `as` value and it is preloaded intentionally.
api/prompts/:1 
        
       Failed to load resource: the server responded with a status of 404 ()
prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/datasets/:1 

       Failed to load resource: the server responded with a status of 404 ()
datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/prompts/:1 

       Failed to load resource: the server responded with a status of 404 ()
prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/datasets/:1 

       Failed to load resource: the server responded with a status of 404 ()
datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/prompts/:1 

       Failed to load resource: the server responded with a status of 404 ()
prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/prompts/:1 

       Failed to load resource: the server responded with a status of 404 ()
prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON
api/datasets/:1 
       
        
       Failed to load resource: the server responded with a status of 404 ()
datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

<img width="493" alt="Screenshot 2024-03-28 at 15 25 33" src="https://github.com/promptfoo/promptfoo/assets/127955819/988b37b9-f872-4c05-b971-bae66472da10">


Are you setting the NEXT_PUBLIC_PROMPTFOO_REMOTE_API_BASE_URL docker build
arg? https://promptfoo.dev/docs/usage/self-hosting#setup.  I don't use
Railway so I'm not sure how that's done/whether build args are supported.

On Thu, Mar 28, 2024 at 7:25 AM Toni Matas ***@***.***> wrote:

> Yes, tons of errors in the browser console:
>
> setup/:1 The resource
> https://promptfoo.up.railway.app/_next/static/css/87fb00eed311e9cd.css
> was preloaded using link preload but not used within a few seconds from the
> window's load event. Please make sure it has an appropriate as value and
> it is preloaded intentionally.
> api/prompts/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/datasets/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/prompts/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/datasets/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/prompts/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/prompts/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> prompts/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
> api/datasets/:1
>
>    Failed to load resource: the server responded with a status of 404 ()
>
> datasets/:1 Uncaught (in promise) SyntaxError: Unexpected token '<',
> "<!DOCTYPE "... is not valid JSON
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/595#issuecomment-2025326833>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJRRS2S4JNKFYYEVB5TY2QK3TAVCNFSM6AAAAABFMV6HG6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRVGMZDMOBTGM>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


As an evirontment variable, you mean?
Yes, it's:
NEXT_PUBLIC_PROMPTFOO_REMOTE_API_BASE_URL=https://promptfoo.up.railway.app

Ok, sorry about that, I understand the issues you're reporting now:
- The share modal always displays the `app.promptfoo.dev` prefix, even though it's shared to the local instance (`promptfoo.up.railway.app`)
- The self-hosted instance currently doesn't support the prompts and datasets API endpoint

Both are fairly easy fixes that can be prioritized.  I'll keep this issue open and let you know once the fixes are in!

That sounds great!
So I understand there is no wrong configuration on my side, so I can go on working with this version till these issues will be fixed. I can work with it as is.
If it's ok for you, I'm going to change the deployment URL to avoid leaving it public here.

Of course.  Sounds good 👍 

These fixes are released in [0.50.1](https://github.com/promptfoo/promptfoo/releases/tag/0.50.1)

Note that the build arg variable has changed to `NEXT_PUBLIC_PROMPTFOO_BASE_URL`, because it applies to both the app and api :)