{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inference and getting ready for deployment\n",
"\n",
"Let’s check if our models work in inferences.\n",
"\n",
"We only test one image and do a visual inspection of the results.\n",
"As already mentioned before, I did not provide a test set. \n",
"\n",
"This is the biggest open TODO.\n",
"\n",
"Another important aspect would be how certain the prediction is. How high is the probability for the second candidate?\n",
"Many improvements are possible in problem definition and post-processing. \n",
"\n",
"### Comparison of three models\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from fastcore.all import *\n",
"from fastai.vision.all import *"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from fastai.learner import load_learner\n",
"\n",
"# Load the FastAI Learner\n",
"learn_inf_tiny = load_learner(\"models/tiny.pkl\")\n",
"learn_inf_base= load_learner(\"models/base.pkl\")\n",
"learn_inf_resnet = load_learner(\"models/resnet.pkl\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"('Cantal',\n",
" tensor(4),\n",
" tensor([9.7780e-05, 9.0306e-06, 2.1395e-05, 1.0606e-05, 9.9840e-01, 1.2682e-07,\n",
" 4.7644e-04, 1.4753e-06, 9.9773e-06, 4.0509e-06, 2.3105e-05, 8.3267e-05,\n",
" 8.9159e-05, 2.2647e-06, 3.8224e-06, 4.5492e-07, 2.8718e-04, 2.2553e-06,\n",
" 7.6010e-07, 3.5029e-04, 2.3085e-07, 1.2567e-04]))"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn_inf_tiny.predict(\"working/which_cheese_cleaned/which_cheese_first/which_cheese/Cantal/0c81aeec-c0a6-421e-844f-3e6e240885a8.jpg\")"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"('Cantal',\n",
" tensor(4),\n",
" tensor([1.5877e-06, 5.6175e-05, 1.3185e-06, 4.0135e-06, 9.9739e-01, 1.9972e-06,\n",
" 1.6469e-03, 1.1616e-05, 1.5650e-04, 2.0251e-05, 1.6810e-05, 3.3364e-04,\n",
" 1.2042e-05, 1.8571e-06, 7.5011e-06, 5.5109e-07, 1.8472e-04, 2.0955e-06,\n",
" 1.5077e-05, 8.5131e-05, 1.5477e-07, 4.9878e-05]))"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn_inf_base.predict(\"working/which_cheese_cleaned/which_cheese_first/which_cheese/Cantal/0c81aeec-c0a6-421e-844f-3e6e240885a8.jpg\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"('Cantal',\n",
" tensor(4),\n",
" tensor([1.2273e-06, 1.0518e-04, 2.1162e-05, 9.5564e-07, 9.9687e-01, 5.3281e-06,\n",
" 2.2306e-03, 5.5073e-08, 9.4349e-05, 1.3493e-06, 2.2718e-04, 2.3397e-04,\n",
" 8.0347e-06, 5.4313e-06, 4.4896e-06, 8.3956e-07, 3.2568e-05, 1.5521e-05,\n",
" 2.5339e-06, 1.2194e-04, 1.5376e-05, 4.0610e-07]))"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"learn_inf_resnet.predict(\"working/which_cheese_cleaned/which_cheese_first/which_cheese/Cantal/0c81aeec-c0a6-421e-844f-3e6e240885a8.jpg\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Comparison of ONNX and Pytorch\n",
"\n",
"We’ll require an `onnx` model at a later time. Let’s evaluate the `resnet` model’s prediction accuracy."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install onnx"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"import torch"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"model = learn_inf_resnet.model\n",
"dummy_input = torch.randn(1, 3, 256, 256) # Use batch size 1 for export\n",
"torch.onnx.export(\n",
" model, \n",
" dummy_input, \n",
" \"model.onnx\", \n",
" export_params=True, \n",
" opset_version=11, \n",
" do_constant_folding=True, \n",
" input_names=[\"input\"], \n",
" output_names=[\"output\"], \n",
" dynamic_axes={\"input\": {0: \"batch_size\"}, \"output\": {0: \"batch_size\"}} # Allow variable batch size\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install onnxruntime numpy pillow torchvision"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['Banon', 'Bleu d’Auvergne', 'Brie de Meaux', 'Camembert', 'Cantal', 'Chabichou du Poitou', 'Comté', 'Fourme d’Ambert', 'Gruyere', 'Livarot', 'Manchego', 'Mimolette', 'Munster', 'Neufchâtel', 'Pont-l’Évêque', 'Pélardon', 'Reblochon', 'Roquefort', 'Selles-sur-Cher', 'Tomme de Savoie', 'Valençay', 'Époisses de Bourgogne']\n"
]
}
],
"source": [
"class_names = learn_inf_resnet.dls.vocab\n",
"print(class_names) # List of class names"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Predicted Class: Cantal (Confidence: 0.971699)\n"
]
}
],
"source": [
"import onnxruntime as ort\n",
"import numpy as np\n",
"from PIL import Image\n",
"import torchvision.transforms as transforms\n",
"# Load ONNX model\n",
"session = ort.InferenceSession(\"model.onnx\", providers=[\"CPUExecutionProvider\"])\n",
"\n",
"# Preprocessing function\n",
"def preprocess_image(image_path):\n",
" transform = transforms.Compose([\n",
" transforms.Resize((256, 256)),\n",
" transforms.ToTensor(),\n",
" transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
" ])\n",
" image = Image.open(image_path).convert(\"RGB\")\n",
" image = transform(image).unsqueeze(0).numpy().astype(np.float32) # Add batch dim and convert to NumPy\n",
" return image\n",
"\n",
"# Load and preprocess image\n",
"image_path = \"working/which_cheese_cleaned/which_cheese_first/which_cheese/Cantal/0c81aeec-c0a6-421e-844f-3e6e240885a8.jpg\" # Replace with your image path\n",
"input_tensor = preprocess_image(image_path)\n",
"\n",
"# Run inference\n",
"outputs = session.run(None, {\"input\": input_tensor})\n",
"\n",
"\n",
"import torch\n",
"outputs = session.run(None, {\"input\": input_tensor})[0] # Raw logits\n",
"probabilities = torch.nn.functional.softmax(torch.tensor(outputs), dim=1) # Convert to probabilities\n",
"predicted_class = torch.argmax(probabilities, dim=1).item()\n",
"# Get predicted class index and label\n",
"predicted_idx = np.argmax(probabilities)\n",
"predicted_label = class_names[predicted_idx]\n",
"\n",
"print(f\"Predicted Class: {predicted_label} (Confidence: {probabilities[0][predicted_idx]:.6f})\")\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The prediction is correct, but the confidence is slightly different. We will try it anyway in deployment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Deployment: Delivering an experience\n",
"\n",
"Of course we want to share our model and not only by posting the source code on GitHub or hugging face. What we want is a live version of the model. Something users can experience.\n",
"\n",
"You can deploy via cloud computing or on-device/edge computing.\n",
"The used technologies are different.\n",
"\n",
"### Cloud based deployment\n",
"\n",
"You cannot evaluate PyTorch ML models using simple JavaScript. A server runs a python backend, which provides an endpoint that is only doing the code of the previous section.\n",
"\n",
"[Here](https://www.tanishq.ai/blog/posts/2021-11-16-gradio-huggingface.html) is a good tutorial how to get a simple setup running on hugging face with gradio.\n",
"\n",
"I developed a webcam based app; the code is in the repo. [And the app is live](https://huggingface.co/spaces/dolind/whichcheese).\n",
"\n",
"In the app you can select the convnext-base, convnext-tiny and the resnet model. All models are trained with 256px images. Just point the camera towards a cheese. \n",
"\n",
"I used the `Gradio` framework, popular in ML and featured on Hugging Face. The processing takes several milliseconds, despite being server based. The Gradio app offers no frame dropping. I try to include dynamic throttling to avoid frame congestion.\n",
"\n",
"One thing I observed from this app is that the Convnext models have high numbers for the second and third best candidate. The app tries to predict a cheese when there is no cheese present. Two points worth to examine.\n",
"\n",
"### Edge based deployment\n",
"\n",
"The issue with edge-based deployment is python. Python is by default not available on mobile. And because of secure concerns, it is becoming more and more complex to run a full blown linux with a python installation.\n",
"\n",
"The other two ways are mobile apps and browser-based inference. We limit ourselves to browser based, because this is accessible via desktop and mobile.\n",
"\n",
"ONNX format is necessary for web browser model deployment. \n",
"At the end of this, we will evaluate if inferring `onnx` gives us the same probability. \n",
"Due to differences in preprocessing between fast.ai and manual methods, some variations may occur.\n",
"\n",
"Using my knowledge of web development, I constructed a basic [app](https://www.storymelange.com/cheese_classifier/) that does inferring the resnet model.\n",
"\n",
"My impressions were that the results were less good. But, the startup and inference time were acceptable and comparable to the python app.\n",
"This is a point worth to examine once I have defined a proper test set.\n",
"\n",
"## The End\n",
"\n",
"This was my first basic study in low level ML. I dabbled in pose recognition before and did work manage AI projects. I’m very impressed with the progress these tools have made.\n",
"\n",
"If you have enjoyed the read, come back for the next project. We will revisit a recipe classification app, which I programmed in 2021 and which we will improve with AI.\n",
"\n",
"### Links\n",
"\n",
"[Github Repo](https://github.com/dolind/cheese_classifier)\n",
"\n",
"[Javascript App](https://www.storymelange.com/cheese_classifier/) \n",
"\n",
"[Gradio App](https://huggingface.co/spaces/dolind/whichcheese)"
]
}
],
"metadata": {
"kaggle": {
"accelerator": "none",
"dataSources": [
{
"datasetId": 6811989,
"sourceId": 10951089,
"sourceType": "datasetVersion"
}
],
"dockerImageVersionId": 30919,
"isGpuEnabled": false,
"isInternetEnabled": true,
"language": "python",
"sourceType": "notebook"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}