Specification for TAPAS LLM Application 1. Overview: The application will be a web-based application hosted on HuggingFace "Space" using either streamlit or gradio UI. It will allow users to upload CSV files and ask questions about the data in the files. The application will use Google's TAPAS LLM model to answer these questions. 2. Core Classes and Functions: - `CSVLoader`: This class will handle the uploading and loading of CSV files. It will convert the CSV files into pandas dataframes with the required structure for the SentenceTransformer retriever. - `EmbeddingGenerator`: This class will generate embeddings for the data using the SentenceTransformer model "deepset/all-mpnet-base-v2-table". - `ChromaDBIndexer`: This class will create a chroma db index and upload the generated embeddings to the index. - `QuestionAnswerer`: This class will handle the processing of user's queries, querying the chroma db for the relevant table, and running the table and query through the TAPAS pipeline for answering. - `AnswerPresenter`: This class will handle the presentation of the answers returned from TAPAS in the UI. It will also handle the execution of any suggested aggregator functions. 3. Non-Standard Dependencies: - pandas: For handling CSV files and data manipulation. - HuggingFace Transformers: For the TAPAS model, tokenizer, pipeline, and SentenceTransformer model. - streamlit or gradio: For the UI of the application. - chroma db: For creating the index and querying for relevant tables. 4. Detailed Functionality: - The application will start with a UI where users can upload one or more CSV files. The `CSVLoader` will handle the uploading and loading of these files, converting them into pandas dataframes with the required structure. - The `EmbeddingGenerator` will then generate embeddings for the data using the SentenceTransformer model "deepset/all-mpnet-base-v2-table". - The `ChromaDBIndexer` will create a chroma db index with a metric type of "cosine" and a dimension of "768", and upload the generated embeddings to the index. - The user will be able to input a query in natural language. The `QuestionAnswerer` will take this query, generate embeddings for it, and query the chroma db for the relevant table that should contain the answer. - The `QuestionAnswerer` will then run the relevant table and the query through the TAPAS pipeline for answering. The TAPAS model, tokenizer, and pipeline will be loaded from HuggingFace. - The `AnswerPresenter` will take the answer returned from TAPAS and present it neatly in the UI. If the TAPAS answer includes an aggregator, the `AnswerPresenter` will first execute the aggregator function on the cell values before presenting the answer. 5. Additional Considerations: - Error handling should be implemented to handle cases where the CSV file cannot be loaded, the embeddings cannot be generated, the chroma db index cannot be created, or the TAPAS pipeline cannot answer the question. - The UI should be user-friendly and intuitive, with clear instructions on how to upload CSV files and input queries. - The application should be designed with scalability in mind, as the number of CSV files and queries could potentially be large.