Zooify - Gamifying a zoo visit
Tags: octograd2020
Published: 2020-05-20 19:20:53

[Comment]: # (All of this is placeholder text. Use this format or any other format of your choosing to best describe your project and experience.)

[Note]: # (If you used the GitHub Student Developer Pack on your project, add the "#githubsdp" tag above. We’d also love to know which services you used and how you used it!)

## My Final Project

The final project that I had worked on during my undergrad was to create a Pokedex like experience that would gamify zoo visits and create a more interactive experience for youth at the zoo. It uses image recognition to identify the animals found within the photo taken and then provides the information offered by the Zoo based on that animal in a mobile friendly environment. It also allows for the tracking of which animals you have encountered and what animals you have yet to encounter in order to make you interested in finding them all and learning along the way. 

## Demo Link
 
https://www.youtube.com/watch?v=1qmYpaYMwXc&feature=youtu.be

## Link to Code

Front-end: https://github.com/tchappus/BlobfishZooify.git
Back-end: https://github.com/jdomaga/Zooify.git
Model Creation: https://github.com/mprice33/blobfishAI.git

[Note]: # (Our markdown editor supports pretty embeds. If you're sharing a GitHub repo, try this syntax: `{% github link_to_your_repo %}`)

## How I built it (what's the stack? did I run into issues or discover something new along the way?)

The stack used for the application was React Native for the front-end to allow deployment of the application to either android or IOS, .Net Core for the back-end hosted on AWS and a model trained using Keras converted into ONNX format for ML.Net to use. My main contribution had to do with the ML aspect of the application in which I obtained the photos on the animals through scraping, cleaned the data/assured the photos were appropriate, built/trained the model based on the data I obtained and then converted the application to the appropriate format. The main issues I encountered were in 3 areas; the first being obtaining the data. Initially, we were supposed to be receiving animal photos/a labelled dataset from the Toronto Zoo but we encountered issues of unresponsiveness on their side so I decided to attempt to get as much data as I could myself. I created a few scripts that would essentially scrape all the photos off Google images related to a few keywords and then save them locally. The scraping lead to the second issue, Google changing its tags to try to dissuade scraping off Google images. Towards the midway point of obtaining data/attempting to improve the amount of data I had, google had changed their structure and made it so my scripts no longer worked. At this time, I had looked for alternatives but many developers/programs that scraped off google were encountering similar issues making it so I had to change the focus of my scripts to get information off Bing and Flickr. Finally, I had to deal with issues of converting my Keras model to a ONNX model; initially this was not difficult at all but as I introduced transfer learning, the regular conversion method no longer worked as my model now had lambda/custom layers. The new process I had to take was to roll back my Keras to an older version of Tensorflow to then save the model as a TF Frozen Graph and then create a conversion from TF Frozen Graph to the ONNX model. Many of these changes/steps were not documented well and it took a lot of attempts/changes it finally make it work. 

During this project, I had discovered a lot of different things as I was inexperienced with both scraping and image recognition at this time. Since this was my first time doing image recognition, I tried learning from the ground up and progressed my model gradually. My model went through many different iterations and even more tweaks, such as adjusting dropout, adding image augmentation/generation, leveraging a bunch of different models for feature extraction through transfer learning, etc. The final product was one I could be proud of since my model which initially got around 45% validation accuracy with a binary classifier ended up as a model with a 98% validation accuracy with multiclass classifier with all the animals we had photos for.  

## Additional Thoughts / Feelings / Stories

Overall, this was a super exciting experience and it allowed me to learn a lot more. Both my team members and I had grown so much and learned to roll with the punches during this experience. This project left us leaving undergrad with our heads held high as we all worked with technologies that were unknowns to us and created a product we were happy with. 

[Final Note]: # (CONGRATULATIONS!!! You are amazing!)