Learn to automatically analyze your images using AI services to find keywords, title and other useful metadata
Tags: serverless, tutorial, java, webdev
Published: 2019-12-16 15:57:04

---
title: Learn to automatically analyze your images using AI services to find keywords, title and other useful metadata
published: true
cover_image: https://thepracticaldev.s3.amazonaws.com/i/6z5dct67wfpfbxnmivht.jpg
description: In todays challenge we analyse an image and give a description for it using Cognitive Services and the Computer Vision API.
tags: serverless, tutorial, java, webdev
---

This article is part of [#25DaysOfServerless](https://25daysofserverless.com). New challenges will be published every day from Microsoft Cloud Advocates throughout the month of December. Find out more about how Microsoft Azure enables your [Serverless functions](https://docs.microsoft.com/azure/azure-functions/?WT.mc_id=25days_devto-blog-cxa).

Have an idea or a solution? <a class="twitter-share-button" href="https://twitter.com/intent/tweet?text=I'm joining the @azureadvocates %2325DaysOfServerless challenge!! Learn more at https://aka.ms/25daysofserverless or see solutions at https://dev.to/search?q=25DaysOfServerless! Join me!" data-url="https://aka.ms/25daysofserverless' " data-hashtags="25DaysOfServerless"> Share your thoughts on Twitter! 
</a><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

<hr/>

Do you want to allow your app to see, hear, listen, speak and even to begin to reason? No worries! You don't need to have a degree in Machine Learning. Nowadays, you can choose betweeen different AI services. One of them are [Azure Cognitive Services](https://docs.microsoft.com/azure/cognitive-services?WT.mc_id=25days_devto-blog-sakriema), available as APIs, SDKs and services. They help developers build intelligent applications. Cognitive Services are grouped. Each Group support different, generalized prediction capabilities. Computer Vision is the service to choose to process and analyse images.

The usage of the [Computer Vision API](https://westcentralus.dev.cognitive.microsoft.com/docs/services/5adf991815e1060e6355ad44/operations/56f91f2e778daf14a499e1fa?WT.mc_id=25days_devto-blog-sakriema) allows to analyse and describe images, to detect objects, recognize texts and more. You can either upload your image to the service or pass the image url.

In this blog post I want to demonstrate the usage of the Cognitive Services API by solving one of the [#25DaysOfServerless](https://25daysofserverless.com) challenges.

>Here in Munich, Germany, Felix is excited to go to a traditional Weihnachtsmarkt, a holiday market! He keeps texting photos to his friend Anna about all the fun things he's doing: drinking hot mulled Glühwein, going ice skating, shopping for presents. But Anna can't find her glasses, and can't clearly see what's in the pictures!

>For today's challenge, Anna needs a service that, given an image, describe the image and gives some keywords about what it contains.


To solve this [15th challenge] (https://25daysofserverless.com/calendar/15) we need to:

1. Find an image randomly or by keywords using an image API.
2. Process and Analyse the given image to give a matching description for the given picture.
3. (Optional) Display the image with the caption and keywords.

## Author's Solution 🔥

This challenge can be seen as a continuation of challenge 7. Feel free to extend your own solution or you can use the [author's solution](https://github.com/SandraAhlgrimm/serverlesschallenge) which is written in Java.

We will use Cognitive Services and integrate Computer Vision to calculate the description and tags for the images. Therefore, enter the [portal](https://portal.azure.com)
and search for _Computer Vision_.

![Screenshot of the Azure Portal, search for Computer Vision and choose it from Azure Marketplace](https://thepracticaldev.s3.amazonaws.com/i/x38im9ky6900cvptky1h.JPG 'Screenshot Computer Vision Search')


The free Tier allows you to execute 20 Calls per minute or 5000 per month. That is way more than we need for the challenge.

![Price Tier Screenshot](https://thepracticaldev.s3.amazonaws.com/i/jqie9jszkvt1elg8yuep.JPG 'Price Tier Screenshot')

All you need is the Key1 and your API Endpoint that you can find in the portal at `Quick start`.

![Computer Vision Screenshot](https://thepracticaldev.s3.amazonaws.com/i/k9s8x16d90g46gggu3bh.JPG 'Computer Vision Screenshot')

In the source code, all we need to do is to authenticate with the generated keys and then let the API do it's work 🐱‍🏍. Therefore, think which `VisualFeatureTypes` are interesting for the specific usecase as we want to keep it as _kiss_💋 as possible. For this challenge, we only need the description and tags, explaining what you can see on the image passed. To receive the image analysis call the Computer Vision service.

```java

    // Get environment variables and authenticate with computer vision
    String CV_KEY = System.getenv("COMPUTER_VISION_SUBSCRIPTION_KEY");
    String CV_API = System.getenv("COMPUTER_VISION_ENDPOINT");

    ComputerVisionClient compVisClient = ComputerVisionManager.authenticate(CV_KEY)
            .withEndpoint(CV_API);

    // This list defines the features to be extracted from the image.
    List<VisualFeatureTypes> extractDescriptionAndTags = new ArrayList<>();
    extractDescriptionAndTags.add(VisualFeatureTypes.DESCRIPTION);
    extractDescriptionAndTags.add(VisualFeatureTypes.TAGS);

    // Call the Computer Vision service and tell it to analyze the loaded image.
    ImageAnalysis analysis = compVisClient.computerVision().analyzeImage().withUrl(imageUrl)
            .withVisualFeatures(extractDescriptionAndTags).execute();

```

**Good to know:** What we will get in return are not only the captions and descriptions but also the matching confidence values. That can be very useful for scenarios where we do need to know with specific certainty whether the random image displays, for instance, a Christmas tree or it does not.
For this challenge, we will just print our findings with the confidence values in the console to satisfy our developers curiosity.


``` java
    // Display image captions and confidence values.
    System.out.println("\nCaptions: ");
    String captionString = "";
    for (ImageCaption caption : analysis.description().captions()) {
        captionString = caption.text();
        System.out.printf("\'%s\' with confidence %f\n", caption.text(), caption.confidence());
    }
    // Display image tags and confidence values.
    System.out.println("\nTags: ");
    StringBuilder keywords = new StringBuilder();
    for (ImageTag tag : analysis.tags()) {
        keywords.append(tag.name());
        keywords.append(" ");
        System.out.printf("\'%s\' with confidence %f\n", tag.name(), tag.confidence());
    }
```

Finally, we return the image with the matching caption and keywords to complete the challenge. ✨

```java
return request.createResponseBuilder(HttpStatus.OK).body(String.format("Search for image with keywords:%s. You can see on this picture: %s. With the keywords %s. Got url: %s", resultText, captionString, keywords, imageUrl)).build();
```

### In Conclusion
We learned what Cognitive Services are and looked in particular into the artificial comprehension of images.
Therefore, we have seen an example how to use the [Computer Vision Service](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts/java-analyze?WT.mc_id=25days_devto-blog-sakriema).
1. Register your service to receive your API keys.
2. Create a list of visual feature types you want to extract.
3. Analyze the image. 🐱‍👤

<hr/>
Want to submit your solution to this challenge? Build a solution locally and then [submit an issue](https://github.com/microsoft/25-days-of-serverless/issues/new?assignees=&labels=challenge-submission&template=challenge-solution-submission.md&title=%5BCHALLENGE+SUBMISSION%5D+). If your solution doesn't involve code you can record a short video and submit it as a link in the issue desccription. Make sure to tell us which challenge the solution is for. We're excited to see what you build! Do you have comments or questions? Add them to the comments area below.

<hr/>
Watch for surprises all during December as we celebrate 25 Days of Serverless. Stay tuned here on dev.to as we feature challenges and solutions! Sign up for a [free account on Azure](https://azure.microsoft.com/free/?WT.mc_id=25days_devto-blog-cxa) to get ready for the challenges!