Made updates to support automatic reload of the TestGroups after a test run

Updated from using random.choices to random.sample throughout where I need a random distinct set as choices does replacement so you can get the same item twice. Discovered in pricing testing.

Marked the logger as a daemon thread so it doesn't prevent the exit of the python interpreter

Fixed bug where the Logger was logging its own name and not that of the architecture.

Implemented single threaded worker on writing the logs to the json file for controlled access to the resource on the file system now we are multi-threading the tests.

Modified test runner to dispatch requests in parallel to make use of the fact that there is a lot of wait time for the LLM. Defaulting to 16 threads.

Added runner for pricing fact checks to assess the level of fact embedding in the latest model

Saved file records to DB. Fixed a print to show the correct test-group name.

Updated the testing page to show the request/response pairs

Updated the offline save to save the actual request and response text

Fixed display of architecture name

Logged start of architecture invocation so if one stalls you can see what it is in the logs

Update to the new URL for model v5

Trying fine-tuning yet another way - all run, now testing v3 of the model

Added raw prompt format to just do passthrough so I can test a number of different examples just typed in

Added another test prompt format

Tweaked the test generator and updated the tests

Added the test question generator and increased the size of the question bank to 500

Added a missing comment

Fixed a bug where if the architecture had entirely failed and not generated a response the whole load of TestGroups would crash. Need to fix the root cause of the failure to generate a response, but also should be caught gracefully here in any event.

Refactored loading the TestGroups to make the structure of the json load and the DB load the same and clearer

Added comments throughout

Switched endpoint control to use the writeable token as it was inconsistent with the normal token.

Fixed bug with default prompt style not being valid

Configured more architectures to try and debug the fine-tuning issue each with different prompt styles

Trying a different prompting style

Tweaked the training data format to try and fix the issue of the model repeating the question over and over

Added loading of test groups from both the DB and the local file and merging these two

Adding the sqlite db where I will archive the test results and added the archiving code

Added option to directly pass the HF hub token when wiping the trace file, so I can use it locally outside of streamlit. Defaulted to None avoid changing existing behaviour.

Fixed bugs from the refactor of repo access. Now it should save trace again.

Added the test reporting structure

Refactored to bring common variables together. Also added a utility to get all the trace records as a list of records

Fixed the time.time bug here. Also a call to reset the Chroma DB

Tweaked the way the prompt is formatted going into the LLM query, to avoid the fine-tuned model giving nonsense answers

Flipped the default dataset to be the baseline not the "All products"

Added ability to wipe the logs from the system status page

Added utility to serve up the test questions

Added the comment to the actual save

Added ability to include comments on the saved trace

Fixed typo in train script generation

Added adapter merging to the fine-tuning script generation

Added awareness of the 'failed' status for an endpoint

Removed decimals from group tag and moved where it is called in the side by side flow so they actually share the same ID

Fixed bug

Changed function name as getting a conflict on the server which I am not getting locally

Added saving of the trace data

Added ability to request the hf token be writable, in combination with new environment secret

Fixed less than/greater than bug where I was dropping the wrong reviews to achieve a target average review. Update the sql data set too.

Added the script to shape the data for testing and the associated sqlite containg the test data