Introducing AI Vision Assistant, an open source project to help people with visual impairments
Tags: code, computervision, microsoftcognitive, opensource
Published: 2018-02-10 22:32:05

---
title: "Introducing AI Vision Assistant, an open source project to help people with visual impairments"
published: true
tags: code,Computer Vision,Microsoft Cognitive ,open source
canonical_url: https://devread.net/2016/11/12/introducing-ai-vision/
---

Three weeks ago, I started to work on a side project to test and experience a new service from Azure called [**Azure Functions**](https://azure.microsoft.com/en-us/services/functions/). The idea of the project was to create a Twitter account (bot) that will randomly get an image from [500px.com](http://500px.com) and describe it \*caption it \*using AI (**[Microsoft Cognitive Services](http://microsoft.com/cognitive)**) and tweet the image with the caption.

Simple idea! It worked as expected, you can check the account here: **[@aipics](http://twitter.com/aipics)**.

## Inspiration…

A friend of mine saw the idea and told me that he knows a person with visual impairments in Twitter, and he’s&nbsp;asking people to describe tweets that contains images, he can _hear_ the tweets (using the iPhone accessibility tools)&nbsp;but he can’t see the image content, his name is Mohammed. Then I thought to make a twitter bot for that purpose, **we have the technology!**

I started to implement that in a new Twitter account, I tested it and I asked Mohammed to test and provide ideas that could help more, And it was so awesome to work and talk to such a person, really!

## The result!

I called it **â€˜AI Vision Assistant’** , you can check it here: **[@aivisionasst](http://twitter.com/aivisionasst)**.

You can tweet the hashtag ( **#aivision** for English _Or_ **#ÙˆØµÙ** for Arabic) with an image _OR_ reply to a tweet that contains image (Including Instagram links) and within 2 minutes, the account will reply with the caption or the description of the image, it supports 2 languages mainly, English and Arabic.

But unfortunately, the Arabic version is not 100% accurate as the Arabic language is not yet supported in Microsoft Cognitive Services APIs, but I used Bing translator to translate the English caption to Arabic one and then build the tweet text.

My favorite part, The account served more than 150 tweets (_tweet = caption request_), in fact, there’s some users&nbsp;that relay on this account and they’re using it almost in daily basis!

Maybe the numbers are low, but for me, seeing one person taking advantage of this account and get help as he needed, it’s like changing the whole world!

## Technical details

There are two components mainly, described below:

[![archt-d-aivision](https://i1.wp.com/res.cloudinary.com/mmgrt/image/upload/v1501203434/archt-d-aivision_j76tjw.png?resize=840%2C473)](https://i1.wp.com/res.cloudinary.com/mmgrt/image/upload/v1501203434/archt-d-aivision_j76tjw.png)

_The architecture diagram of the complete idea..&nbsp;_

### Mentions Monitor:

A timer Function that will run every 2 minutes, to perform a search in Twitter for tweets that contains (#aivision OR #ÙˆØµÙ) with these conditions: Tweeted within the&nbsp;last 3 minutes & Not yet handled.

If found any, the tweet id will be saved in a local log file (_mark it as handled_) and then it’ll call the second Function, the **AI Vision Core**.

### AI Vision Core:

An Http-trigger Function, requires one parameter: tweet id.

Whenever this function receives a request, it’ll call Twitter API to get the image from the tweet object OR if it’s a reply, it’ll get the image from InReplyToTweet object OR it’ll search for an Instagram link if any.

Then it’ll prepare the image, call the Vision API to get the caption, translate it if the hashtag used was&nbsp;Arabic, build the tweet text and finally reply with the image description with any mentions!

## Open source…

This project is an **open source project** , you can browse it&nbsp;on Github: **[/AIVision.](https://github.com/mmgrt/AIVision)**

Please feel free to pivot it, provide it in your language/ culture or contribute in making it better.. or at least,&nbsp;I hope It’ll inspire you!

## We have the technology
> According to WHO, there’s more than 39 Million _blind_ people in the world… this project might help few individuals in Twitter, but still, there are millions out there.&nbsp; **Imagine** what help or support that could be provided by us, programmers!

Thank you

Muhamed –&nbsp;**[@mmg\_rt](http://twitter.com/mmg_rt)**.



_This post was originally published on [devread.net](https://devread.net/2016/11/12/introducing-ai-vision/)_