Update README.md

README.md

@@ -1,8 +1,42 @@
 ---
 license: mit
 ---
-This model was created for Cosmocook 2, a project at the 2024 Berkeley AI Hackathon.
+## Development Process
+This model was created with the intention of supplmenting our mobile application as an ingredient counter. Such that a user could take an image and have AI determine whether they have the necessary ingredients to create a recipe. As a result of the straightforward usecase, we were motivated to use a lightweight model for this classification task. We were recommended Moondream by a mentor from Intel, which we fine-tuned in this project.
 
-Cosmo-demo is a fine-tuned version of the Moondream vision model. This model was trained on a set of images of avocadoes, bananas, doritos, and onions. With special attention to counting the number of each food item in an image.
+## Model Details
 
-This model was trained in Intel's Jupyter notebooks using IPEX.
+- **Model Name**: Moondream Fine-tuned Variant
+- **Task**: Ingredient Classification and Counting
+- **Repository Name**: nyap/cosmo-demo
+- **Original Model**: vikhyatk/moondream2
+
+## Intended Use
+
+- **Primary Use Case**: This model counts and classifies ingredients (exclusively avocados, bananas, doritos, and onions during CalHacks) in images.
+- **Associated Application**: This model is applied through a cooking assistant mobile application built in React-native.
+
+## Data
+
+- **Dataset**: Used a very small dataset of a little less than 100 images collected during the Hackathon. Including images of avocados, bananas, doritos, and onions from various angles and in different combinations.
+- **Data Collection**: Images were taken of various tables with the ingredients arranged, before being manually labelled with the count/type of ingredients in each image.
+
+## Evaluation Metrics
+
+- **Performance Metrics**: Performance metrics did not provide any valuable feedback due to the size of the dataset (overfit)
+
+## Training
+
+- **Training Procedure**: Created a basic dataloader class to read image data and expected output. PIL images were then passed through the VIT provided with Moondream. Moondream was then fine-tuned using the default Adam optimizer with a batch size of 8 for 30 epochs.
+- **Hardware**: The model was trained with XPU (Intel(R) Data Center GPU Max 1100) provided by Intel mentors at the Berkeley AI Hackathon.
+
+## Limitations
+
+- **Current Limitations**: The model may struggle with images containing overlapping ingredients or items not seen during training. It also only has a limited dataset, so different backgrounds or types of ingredients would not likely not be counted.
+- **Potential Improvements**: Incorporate a full-scale dataset with labelled ingredient counts to make the model more knowledgable and applicable within our application. Tune hyperparameters beyond what was done during the Hackathon to optimize performance.
+
+## Additional Information
+
+- **License**: Model released under MIT License.
+- **Contact Information**: For inquiries, contact [nyap@umich.edu](mailto:nyap@umich.edu).
+- **Acknowledgments**: Acknowledge Intel team for starter code and assistance in fine-tuning.