GPT-4 Vision API Is a Game Changer
Published: 2023-11-08 20:48:59

<div class="entry-content">
			<div class="row">
				<div class="col-lg-8 col-xl-8 blog-content-col">
					
<p><strong>gpt-4-vision-preview</strong> is the latest and (arguably) the most powerful <a href="https://blog.nextideatech.com/openai-gpt-3-5-turbo-instruct/" class="ek-link">model </a>released on November 7 2023 during <a href="https://devday.openai.com/" class="ek-link">OpenAI’s DevDay</a> presentation and it has been the talk of social media merely hours after it became available.</p><section class="cta-card">
 



<p>Developers have already created apps that actively recognize what’s happening during a web live stream in real-time.</p>



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<div class="twitter-tweet twitter-tweet-rendered" style="display: flex; max-width: 550px; width: 100%; margin-top: 10px; margin-bottom: 10px;"><iframe id="twitter-widget-0" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" class="" style="position: static; visibility: visible; width: 550px; height: 688px; display: block; flex-grow: 1;" title="Twitter Tweet" src="https://platform.twitter.com/embed/Tweet.html?creatorScreenName=NextIdeaTweets&amp;dnt=true&amp;embedId=twitter-widget-0&amp;features=eyJ0ZndfdGltZWxpbmVfbGlzdCI6eyJidWNrZXQiOltdLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X2ZvbGxvd2VyX2NvdW50X3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9iYWNrZW5kIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19yZWZzcmNfc2Vzc2lvbiI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfZm9zbnJfc29mdF9pbnRlcnZlbnRpb25zX2VuYWJsZWQiOnsiYnVja2V0Ijoib24iLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X21peGVkX21lZGlhXzE1ODk3Ijp7ImJ1Y2tldCI6InRyZWF0bWVudCIsInZlcnNpb24iOm51bGx9LCJ0ZndfZXhwZXJpbWVudHNfY29va2llX2V4cGlyYXRpb24iOnsiYnVja2V0IjoxMjA5NjAwLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X3Nob3dfYmlyZHdhdGNoX3Bpdm90c19lbmFibGVkIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19kdXBsaWNhdGVfc2NyaWJlc190b19zZXR0aW5ncyI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdXNlX3Byb2ZpbGVfaW1hZ2Vfc2hhcGVfZW5hYmxlZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdmlkZW9faGxzX2R5bmFtaWNfbWFuaWZlc3RzXzE1MDgyIjp7ImJ1Y2tldCI6InRydWVfYml0cmF0ZSIsInZlcnNpb24iOm51bGx9LCJ0ZndfbGVnYWN5X3RpbWVsaW5lX3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9mcm9udGVuZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9fQ%3D%3D&amp;frame=false&amp;hideCard=false&amp;hideThread=false&amp;id=1721716511247921277&amp;lang=en&amp;origin=https%3A%2F%2Fblog.nextideatech.com%2Fgpt-4-vision-api-is-a-game-changer%2F&amp;sessionId=3c3a58866f23675c3e6111e58e2f900aaf93153d&amp;siteScreenName=NextIdeaTweets&amp;theme=light&amp;widgetsVersion=01917f4d1d4cb%3A1696883169554&amp;width=550px" data-tweet-id="1721716511247921277"></iframe></div><script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



<p>Or this person that incorporated just about every OpenAI API to analyze Messi highlights video using <strong>gpt-4-vision-preview</strong> model, create voiceover script based on the video frames then generate audio using OpenAI <a href="https://platform.openai.com/docs/guides/text-to-speech" class="ek-link">Text-to-Speech</a> </p>



<figure class="wp-block-embed is-type-rich is-provider-twitter wp-block-embed-twitter"><div class="wp-block-embed__wrapper">
<div class="twitter-tweet twitter-tweet-rendered" style="display: flex; max-width: 550px; width: 100%; margin-top: 10px; margin-bottom: 10px;"><iframe id="twitter-widget-1" scrolling="no" frameborder="0" allowtransparency="true" allowfullscreen="true" class="" style="position: static; visibility: visible; width: 550px; height: 697px; display: block; flex-grow: 1;" title="Twitter Tweet" src="https://platform.twitter.com/embed/Tweet.html?creatorScreenName=NextIdeaTweets&amp;dnt=true&amp;embedId=twitter-widget-1&amp;features=eyJ0ZndfdGltZWxpbmVfbGlzdCI6eyJidWNrZXQiOltdLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X2ZvbGxvd2VyX2NvdW50X3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9iYWNrZW5kIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19yZWZzcmNfc2Vzc2lvbiI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfZm9zbnJfc29mdF9pbnRlcnZlbnRpb25zX2VuYWJsZWQiOnsiYnVja2V0Ijoib24iLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X21peGVkX21lZGlhXzE1ODk3Ijp7ImJ1Y2tldCI6InRyZWF0bWVudCIsInZlcnNpb24iOm51bGx9LCJ0ZndfZXhwZXJpbWVudHNfY29va2llX2V4cGlyYXRpb24iOnsiYnVja2V0IjoxMjA5NjAwLCJ2ZXJzaW9uIjpudWxsfSwidGZ3X3Nob3dfYmlyZHdhdGNoX3Bpdm90c19lbmFibGVkIjp7ImJ1Y2tldCI6Im9uIiwidmVyc2lvbiI6bnVsbH0sInRmd19kdXBsaWNhdGVfc2NyaWJlc190b19zZXR0aW5ncyI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdXNlX3Byb2ZpbGVfaW1hZ2Vfc2hhcGVfZW5hYmxlZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9LCJ0ZndfdmlkZW9faGxzX2R5bmFtaWNfbWFuaWZlc3RzXzE1MDgyIjp7ImJ1Y2tldCI6InRydWVfYml0cmF0ZSIsInZlcnNpb24iOm51bGx9LCJ0ZndfbGVnYWN5X3RpbWVsaW5lX3N1bnNldCI6eyJidWNrZXQiOnRydWUsInZlcnNpb24iOm51bGx9LCJ0ZndfdHdlZXRfZWRpdF9mcm9udGVuZCI6eyJidWNrZXQiOiJvbiIsInZlcnNpb24iOm51bGx9fQ%3D%3D&amp;frame=false&amp;hideCard=false&amp;hideThread=false&amp;id=1721705524176257296&amp;lang=en&amp;origin=https%3A%2F%2Fblog.nextideatech.com%2Fgpt-4-vision-api-is-a-game-changer%2F&amp;sessionId=3c3a58866f23675c3e6111e58e2f900aaf93153d&amp;siteScreenName=NextIdeaTweets&amp;theme=light&amp;widgetsVersion=01917f4d1d4cb%3A1696883169554&amp;width=550px" data-tweet-id="1721705524176257296"></iframe></div><script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div></figure>



<h3>Using gpt-4-vision-preview Model With Python</h3>



<p>First install the <em>openai</em> pip module</p>



<pre class="wp-block-code language-bash" tabindex="0"><code lang="bash" class="language-bash">pip <span class="token function">install</span> --upgrade openai <span class="token comment">#to ensure we are using the latest version</span></code></pre>



<p>Next, we initialize OpenAI object with our API key and create an instance.</p>



<p>How to obtain your OpenAI API key?</p>



<ul><li>Go to <a href="https://platform.openai.com/api-keys" target="_blank" aria-label=" (opens in a new tab)" rel="noreferrer noopener" class="ek-link">API keys – OpenAI API</a></li><li>Click “Create new secret key”</li><li>Provide a name and be sure to copy your new key.</li></ul>



<pre class="wp-block-code language-python" tabindex="0"><code lang="python" class="language-python">client <span class="token operator">=</span> OpenAI<span class="token punctuation">(</span>api_key<span class="token operator">=</span><span class="token string">"sk_YOUR_OPENAI_KEY"</span><span class="token punctuation">)</span>
</code></pre>



<h3>Making your first call with gpt-4-vision-preview Model</h3>



<pre class="wp-block-code language-python" tabindex="0"><code lang="python" class="language-python"><span class="token keyword">import</span> os
<span class="token keyword">from</span> openai <span class="token keyword">import</span> OpenAI
<span class="token keyword">from</span> dotenv <span class="token keyword">import</span> load_dotenv
<span class="token keyword">import</span> base64
<span class="token keyword">import</span> mimetypes
load_dotenv<span class="token punctuation">(</span><span class="token punctuation">)</span>

client <span class="token operator">=</span> OpenAI<span class="token punctuation">(</span>api_key<span class="token operator">=</span>os<span class="token punctuation">.</span>getenv<span class="token punctuation">(</span><span class="token string">"OPENAI_API_KEY"</span><span class="token punctuation">)</span><span class="token punctuation">)</span>

<span class="token keyword">def</span> <span class="token function">image_to_base64</span><span class="token punctuation">(</span>image_path<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token comment"># Guess the MIME type of the image</span>
    mime_type<span class="token punctuation">,</span> _ <span class="token operator">=</span> mimetypes<span class="token punctuation">.</span>guess_type<span class="token punctuation">(</span>image_path<span class="token punctuation">)</span>
    
    <span class="token keyword">if</span> <span class="token keyword">not</span> mime_type <span class="token keyword">or</span> <span class="token keyword">not</span> mime_type<span class="token punctuation">.</span>startswith<span class="token punctuation">(</span><span class="token string">'image'</span><span class="token punctuation">)</span><span class="token punctuation">:</span>
        <span class="token keyword">raise</span> ValueError<span class="token punctuation">(</span><span class="token string">"The file type is not recognized as an image"</span><span class="token punctuation">)</span>
    
    <span class="token comment"># Read the image binary data</span>
    <span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span>image_path<span class="token punctuation">,</span> <span class="token string">'rb'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> image_file<span class="token punctuation">:</span>
        encoded_string <span class="token operator">=</span> base64<span class="token punctuation">.</span>b64encode<span class="token punctuation">(</span>image_file<span class="token punctuation">.</span>read<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">.</span>decode<span class="token punctuation">(</span><span class="token string">'utf-8'</span><span class="token punctuation">)</span>
    
    <span class="token comment"># Format the result with the appropriate prefix</span>
    image_base64 <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f"data:</span><span class="token interpolation"><span class="token punctuation">{</span>mime_type<span class="token punctuation">}</span></span><span class="token string">;base64,</span><span class="token interpolation"><span class="token punctuation">{</span>encoded_string<span class="token punctuation">}</span></span><span class="token string">"</span></span>
    
    <span class="token keyword">return</span> image_base64

base64_string <span class="token operator">=</span> image_to_base64<span class="token punctuation">(</span><span class="token string">"image1.jpg"</span><span class="token punctuation">)</span>


response <span class="token operator">=</span> client<span class="token punctuation">.</span>chat<span class="token punctuation">.</span>completions<span class="token punctuation">.</span>create<span class="token punctuation">(</span>
    model<span class="token operator">=</span><span class="token string">"gpt-4-vision-preview"</span><span class="token punctuation">,</span>
    messages<span class="token operator">=</span><span class="token punctuation">[</span>
        <span class="token punctuation">{</span>
            <span class="token string">"role"</span><span class="token punctuation">:</span> <span class="token string">"user"</span><span class="token punctuation">,</span>
            <span class="token string">"content"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>
                <span class="token punctuation">{</span><span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"text"</span><span class="token punctuation">,</span> <span class="token string">"text"</span><span class="token punctuation">:</span> <span class="token string">"Describe the attached image"</span><span class="token punctuation">}</span><span class="token punctuation">,</span>
                <span class="token punctuation">{</span>
                    <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"image_url"</span><span class="token punctuation">,</span>
                    <span class="token string">"image_url"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span>
                        <span class="token string">"url"</span><span class="token punctuation">:</span> base64_string<span class="token punctuation">,</span>
                        <span class="token string">"detail"</span><span class="token punctuation">:</span> <span class="token string">"low"</span>
                    <span class="token punctuation">}</span>
                <span class="token punctuation">}</span><span class="token punctuation">,</span>
            <span class="token punctuation">]</span><span class="token punctuation">,</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">]</span><span class="token punctuation">,</span>
    max_tokens<span class="token operator">=</span><span class="token number">300</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span>

<span class="token keyword">print</span><span class="token punctuation">(</span>response<span class="token punctuation">.</span>choices<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>message<span class="token punctuation">.</span>content<span class="token punctuation">)</span></code></pre>



<p><strong>Stepping Through the Code</strong></p>



<p>The <code>image_to_base64</code> function is defined to convert an image file to a base64-encoded string. It first checks if the file is an image and then reads and encodes the binary data of the image. Use this option to submit local images. </p>



<pre class="wp-block-code"><code class="">def image_to_base64(image_path):
    ...
</code></pre>



<p>You may also submit image URLs using the below example:</p>



<pre class="wp-block-code language-python" tabindex="0"><code lang="python" class="language-python">response <span class="token operator">=</span> client<span class="token punctuation">.</span>chat<span class="token punctuation">.</span>completions<span class="token punctuation">.</span>create<span class="token punctuation">(</span>
    model<span class="token operator">=</span><span class="token string">"gpt-4-vision-preview"</span><span class="token punctuation">,</span>
    messages<span class="token operator">=</span><span class="token punctuation">[</span>
        <span class="token punctuation">{</span>
            <span class="token string">"role"</span><span class="token punctuation">:</span> <span class="token string">"user"</span><span class="token punctuation">,</span>
            <span class="token string">"content"</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>    
                <span class="token punctuation">{</span>
                    <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"text"</span><span class="token punctuation">,</span>
                    <span class="token string">"text"</span><span class="token punctuation">:</span> <span class="token string">"What’s in these images? Is there any difference between them?"</span><span class="token punctuation">,</span>
                <span class="token punctuation">}</span><span class="token punctuation">,</span>
                <span class="token punctuation">{</span>
                    <span class="token string">"type"</span><span class="token punctuation">:</span> <span class="token string">"image_url"</span><span class="token punctuation">,</span>
                    <span class="token string">"image_url"</span><span class="token punctuation">:</span> <span class="token string">"https://www.goodfreephotos.com/albums/other-landscapes/rover-and-landscape-scenery.jpg"</span><span class="token punctuation">,</span>
                <span class="token punctuation">}</span>
            <span class="token punctuation">]</span><span class="token punctuation">,</span>
        <span class="token punctuation">}</span>
    <span class="token punctuation">]</span><span class="token punctuation">,</span>
    max_tokens<span class="token operator">=</span><span class="token number">300</span><span class="token punctuation">,</span>
<span class="token punctuation">)</span></code></pre>



<p><strong>The Completion parameters</strong></p>



<p>You may be familiar with the <em>message</em> object If you have used the <a href="https://blog.nextideatech.com/chat-with-documents-using-langchain-gpt-4-python/" class="ek-link">OpenAI API</a> in the past. With the release of <em>gpt-4-vision-preview</em> model, OpenAI has introduced a new message type; “image_url”. </p>



<p>The <em>url </em>parameter accepts base 64 encoded images or image URLs.</p>



<p>Adjusting the ‘<em>detail</em>‘ parameter in the GPT-4 Vision API affects the resolution and token budget used to interpret images. Setting ‘detail’ to ‘low’ restricts the model to a 512 x 512 pixel, low-resolution version of the image, using 65 tokens. This mode is quicker and more token-efficient, suitable for scenarios where detailed analysis isn’t critical. Opting for ‘high’ engages the high-resolution mode, which initially presents the model with the low-resolution image, followed by high-resolution 512px segments of the image, with each segment allocated a 130-token budget for a thorough examination.</p>



<pre class="wp-block-code"><code class="">messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the attached image"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": base64_string,
                        "detail": "low"
                    }
                },
            ],
        }
    ]</code></pre>



<h3>GPT-4 Vision API Pricing</h3>



<p>As shown below, OpenAI prices the vision capability based on image resolution. For example a 512×512 image will cost $0.00255 in “high” detail more, or 0.00085 for “low” model. </p>



<figure class="wp-block-image size-large"><img loading="lazy" width="1024" height="365" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-1024x365.png" alt="GPT-4 Turbo pricing" class="wp-image-31322" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-980x349.png 980w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-480x171.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw"><figcaption>GPT-4 Vision API Pricing</figcaption></figure>



<p>This may not seem like a lot but cost does add up when processing hundreds or thousands of images. To control your input image sizes, use the below code to resize them:</p>



<p>Ensure you have the Pillow library installed</p>



<pre class="wp-block-code language-bash" tabindex="0"><code lang="bash" class="language-bash">pip <span class="token function">install</span> Pillow </code></pre>



<pre class="wp-block-code language-python" tabindex="0"><code lang="python" class="language-python"><span class="token keyword">from</span> PIL <span class="token keyword">import</span> Image
<span class="token keyword">import</span> os

<span class="token keyword">def</span> <span class="token function">resize_image</span><span class="token punctuation">(</span>image_path<span class="token punctuation">,</span> new_width<span class="token punctuation">,</span> new_height<span class="token punctuation">)</span><span class="token punctuation">:</span>
    <span class="token keyword">with</span> Image<span class="token punctuation">.</span><span class="token builtin">open</span><span class="token punctuation">(</span>image_path<span class="token punctuation">)</span> <span class="token keyword">as</span> img<span class="token punctuation">:</span>
        <span class="token comment"># Resize the image</span>
        img <span class="token operator">=</span> img<span class="token punctuation">.</span>resize<span class="token punctuation">(</span><span class="token punctuation">(</span>new_width<span class="token punctuation">,</span> new_height<span class="token punctuation">)</span><span class="token punctuation">,</span> Image<span class="token punctuation">.</span>ANTIALIAS<span class="token punctuation">)</span>

        <span class="token comment"># Save the resized image</span>
        base<span class="token punctuation">,</span> ext <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>splitext<span class="token punctuation">(</span>image_path<span class="token punctuation">)</span>
        new_image_path <span class="token operator">=</span> <span class="token string-interpolation"><span class="token string">f"</span><span class="token interpolation"><span class="token punctuation">{</span>base<span class="token punctuation">}</span></span><span class="token string">_resized</span><span class="token interpolation"><span class="token punctuation">{</span>ext<span class="token punctuation">}</span></span><span class="token string">"</span></span>
        img<span class="token punctuation">.</span>save<span class="token punctuation">(</span>new_image_path<span class="token punctuation">)</span>

        <span class="token keyword">print</span><span class="token punctuation">(</span><span class="token string-interpolation"><span class="token string">f"Image saved as </span><span class="token interpolation"><span class="token punctuation">{</span>new_image_path<span class="token punctuation">}</span></span><span class="token string">"</span></span><span class="token punctuation">)</span>

</code></pre>



<h3>Real World Use of GPT-4 Vision API: Enhancing Web Experience with a Chrome Extension</h3>



<p>Ok so GPT-4 Vision API is cool and all – people have used it to seamlessly create soccer highlight commentary and interact with Webcams but let’s put the <em>gpt-4-vision-preview</em> to the test and see how it fairs with real world problems.</p>



<h3>Browser Extension – GPT Vision Assistant</h3>



<figure class="wp-block-image size-large"><img loading="lazy" width="1024" height="450" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-2-1024x450.png" alt="" class="wp-image-31331" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-2-980x431.png 980w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-2-480x211.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw"></figure>



<p>The Chrome extension is designed to harness the GPT-4 Vision API works in a streamlined three-step process: </p>



<p><strong>First</strong>, it captures a screenshot of the current tab. </p>



<figure class="wp-block-image size-large"><img loading="lazy" width="1024" height="528" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-3-1024x528.png" alt="" class="wp-image-31332" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-3-980x505.png 980w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-3-480x248.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw"></figure>



<p><strong>Second</strong>, the user is prompted to input a specific question or instruction, articulating what they seek to learn or accomplish with regard to the captured image. </p>



<figure class="wp-block-image size-full"><img loading="lazy" width="806" height="610" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-4.png" alt="" class="wp-image-31333" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-4.png 806w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-4-480x363.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 806px, 100vw"></figure>



<p><strong>Finally</strong>, this input, along with the screenshot, is sent as a request to the OpenAI API. The <em>gpt-4-vision-preview</em> models returns its findings or outputs directly within the extension interface, displaying the results for the user to review. This seamless integration offers a powerful tool for real-time image analysis and interaction, all without leaving the browser tab.</p>



<figure class="wp-block-image size-full"><img loading="lazy" width="805" height="608" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-5.png" alt="" class="wp-image-31334" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-5.png 805w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-5-480x363.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 805px, 100vw"></figure>



<h2>Other Use Cases With GPT-4 Vision API</h2>



<p>The GPT-4 Vision API’s capabilities extend beyond simple image recognition and analysis; they open up a world of possibilities for enhancing and streamlining our interactions with digital content. For this purpose, we created a Chrome Extension that leverages this advanced API stands as a testament to its practical utility in everyday web usage. Here are other use cases:</p>



<ol><li><strong>Quality Assurance for Websites</strong>: Website developers and QA testers can use the extension to take screenshots of web pages and submit them directly to the GPT-4 Vision API with prompts like “Identify any visual inconsistencies across these screenshots” or “Does this layout comply with web accessibility standards?” The API can analyze the images for color contrast, element alignment, responsive design issues, and more, providing quick feedback that would traditionally require meticulous manual review.</li><li><strong>UX/UI Design Feedback</strong>: Designers can capture snapshots of their work and ask the model questions such as “What improvements can be made to this user interface to enhance user experience?” or “What are the best practices missing from this design?” This not only speeds up the iterative process of design but also injects an objective, data-driven perspective into creative workflows.</li><li><strong>Content Management and Moderation</strong>: For content managers and online moderators, the extension could be used to screen website content. By taking screenshots of various posts, images, or comments and querying the GPT-4 Vision API, the system could assist in identifying inappropriate content or copyright issues, streamlining the moderation process.</li><li><strong>Educational Content Interaction</strong>: Students and educators could use the extension to capture diagrams, equations, or charts from educational websites and ask the model to explain or solve them. This interactive approach could enhance online learning by providing instant assistance and explanations for complex visual information.</li></ol>



<figure class="wp-block-image size-large"><img loading="lazy" width="1024" height="626" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-7-1024x626.png" alt="" class="wp-image-31337" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/11/image-7-980x600.png 980w, https://blog.nextideatech.com/wp-content/uploads/2023/11/image-7-480x294.png 480w" sizes="(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1024px, 100vw"><figcaption>GPT-4 Vision API answering a math question</figcaption></figure>



<ol><li><strong>Competitor Analysis</strong>: Marketing professionals might employ the extension to capture the layout of competitors’ websites, asking the model to analyze and compare branding consistency, messaging clarity, and call-to-action placements. This competitive intelligence can be invaluable for strategic planning.</li><li><strong>Accessibility Auditing</strong>: The extension could also be used to assist with ensuring web accessibility. Users could take snapshots of websites and ask the model to check if the images contain proper alt text or if the color schemes used are suitable for color-blind individuals.</li><li><strong>Automated Documentation</strong>: IT professionals and developers could use the extension to document the behavior of web applications. By taking screenshots and prompting the API to describe the process flow or detect anomalies, they could generate documentation or troubleshooting guides more efficiently.</li><li><strong>Code Generation from Designs</strong>: A revolutionary use case would be for front-end developers to send UX designs to the OpenAI API and ask it to generate the necessary HTML/CSS/JavaScript code. This could potentially reduce development time by providing a starting point for building out web interfaces.</li></ol>



<p>If interested in the extension or other custom solutions using OpenAI, please email info@nextideated.com</p>
					<!-- <section class="cta-card">
					  <div class="cta-card__container">
					    <div class="cta-card__header">
					      <p class="cta-card__title">Tell us the skills you need and we'll find the best developer for you in days, not weeks.</p>
					      <a href="https://nextideatech.com/contact/">Hire Developers</a>
					    </div>
					  </div>
					</section> -->

					<!-- subscribe form -->
					<div id="mc_embed_signup" class="subscribe-form">
						<form action="https://nextideatech.us7.list-manage.com/subscribe/post?u=ae45185df605e4c3b1371d09f&amp;id=22c96b1894&amp;f_id=0004c5e4f0" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="novalidate">
							<div id="mc_embed_signup_scroll">
								<h4>Subscribe to our newsletter and get<br> exclusive content</h4>
								<div class="mc-field-group">
									<input type="email" value="" name="EMAIL" class="required email" placeholder="Email Address" id="mce-EMAIL" aria-required="true">
									<span id="mce-EMAIL-HELPERTEXT" class="helper_text"></span>
									<input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button">
								</div>
								<div id="mce-responses" class="clear foot">
									<div class="response" id="mce-error-response" style="display:none"></div>
									<div class="response" id="mce-success-response" style="display:none"></div>
								</div>
						    <div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_ae45185df605e4c3b1371d09f_22c96b1894" tabindex="-1" value=""></div>
						   </div>
						</form>
					</div>
					<script type="text/javascript" src="//s3.amazonaws.com/downloads.mailchimp.com/js/mc-validate.js"></script>
					<script type="text/javascript">(function($) {window.fnames = new Array(); window.ftypes = new Array();fnames[0]='EMAIL';ftypes[0]='email';fnames[1]='FNAME';ftypes[1]='text';fnames[2]='LNAME';ftypes[2]='text';fnames[3]='ADDRESS';ftypes[3]='address';fnames[4]='PHONE';ftypes[4]='phone';}(jQuery));var $mcj = jQuery.noConflict(true);</script>

				</div>

				<div class="col-lg-4 col-xl-4 blog-sidebar-col">
						<div id="sidebar">
		<div id="block-17" class="et_pb_widget widget_block">
<h5>Related Posts</h5>
</div><div id="block-18" class="et_pb_widget widget_block widget_recent_entries"><ul class="wp-block-latest-posts__list has-dates wp-block-latest-posts"><li><div class="wp-block-latest-posts__featured-image alignleft"><a href="https://blog.nextideatech.com/gpt-4-vision-api-is-a-game-changer/" aria-label="GPT-4 Vision API Is a Game Changer"><img width="300" height="171" src="https://blog.nextideatech.com/wp-content/uploads/2023/11/b2ac07c3-e18e-4228-999c-4a94f500a06b-300x171.webp" class="attachment-medium size-medium wp-post-image" alt="gpt-4 vision api" loading="lazy" style=""></a></div><a class="wp-block-latest-posts__post-title" href="https://blog.nextideatech.com/gpt-4-vision-api-is-a-game-changer/">GPT-4 Vision API Is a Game Changer</a><time datetime="2023-11-08T20:47:09+00:00" class="wp-block-latest-posts__post-date">November 8, 2023</time></li>
<li><div class="wp-block-latest-posts__featured-image alignleft"><a href="https://blog.nextideatech.com/hire-nearshore-ai-developers/" aria-label="Hire Nearshore AI Developers"><img width="300" height="300" src="https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-300x300.png" class="attachment-medium size-medium wp-post-image" alt="hire nearshore ai developers" loading="lazy" style="" srcset="https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-300x300.png 300w, https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte.png 1024w, https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-150x150.png 150w, https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-768x768.png 768w, https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-980x980.png 980w, https://blog.nextideatech.com/wp-content/uploads/2023/10/DALL·E-2023-10-16-15.25.11-Professional-illustration-of-Latin-America-with-countries-clearly-marked-in-vibrant-colors.-Floating-above-the-continent-is-a-polished-artificial-inte-480x480.png 480w" sizes="(max-width: 300px) 100vw, 300px"></a></div><a class="wp-block-latest-posts__post-title" href="https://blog.nextideatech.com/hire-nearshore-ai-developers/">Hire Nearshore AI Developers</a><time datetime="2023-10-18T07:00:00+00:00" class="wp-block-latest-posts__post-date">October 18, 2023</time></li>
<li><div class="wp-block-latest-posts__featured-image alignleft"><a href="https://blog.nextideatech.com/gpt-3-5-turbo-instruct-with-node-js-python-and-mern-stack-for-advanced-web-applications/" aria-label="GPT-3.5 Turbo Instruct with Node.js, Python, and MERN Stack for Advanced Web Applications"><img width="300" height="158" src="https://blog.nextideatech.com/wp-content/uploads/2023/10/node-Js-300x158.png" class="attachment-medium size-medium wp-post-image" alt="gpt-3.5 turbo instruct nodejs python mern" loading="lazy" style=""></a></div><a class="wp-block-latest-posts__post-title" href="https://blog.nextideatech.com/gpt-3-5-turbo-instruct-with-node-js-python-and-mern-stack-for-advanced-web-applications/">GPT-3.5 Turbo Instruct with Node.js, Python, and MERN Stack for Advanced Web Applications</a><time datetime="2023-10-17T05:47:00+00:00" class="wp-block-latest-posts__post-date">October 17, 2023</time></li>
</ul></div><div id="block-19" class="et_pb_widget widget_block">
<div class="sticky-block-wrapper sticky-block-wrapper-0"><div class="sticky-block-placeholder sticky-block-placeholder-0" style="width:0; height:0; margin:0; padding:0; visibility:hidden;"></div><div class="wp-block-senff-sticky-block senff-sticky-block-0 sticky-block-original-0 block-is-not-sticky sticky-block-active-0" data-topspace="100" data-cfa="true" data-minwidth="980" data-maxwidth="99999" data-pushup="#main-footer" data-zindex="1" style="">
<h5>Share</h5>


<p></p><div class="addtoany_shortcode"><div class="a2a_kit a2a_kit_size_32 addtoany_list" data-a2a-url="https://blog.nextideatech.com/gpt-4-vision-api-is-a-game-changer/" data-a2a-title="GPT-4 Vision API Is a Game Changer" style="line-height: 32px;"><a class="a2a_button_facebook" href="/#facebook" title="Facebook" rel="nofollow noopener" target="_blank"><span class="a2a_svg a2a_s__default a2a_s_facebook" style="background-color: rgb(10, 107, 216);"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path fill="#FFF" d="M17.78 27.5V17.008h3.522l.527-4.09h-4.05v-2.61c0-1.182.33-1.99 2.023-1.99h2.166V4.66c-.375-.05-1.66-.16-3.155-.16-3.123 0-5.26 1.905-5.26 5.405v3.016h-3.53v4.09h3.53V27.5h4.223z"></path></svg></span><span class="a2a_label">Facebook</span></a><a class="a2a_button_twitter" href="/#twitter" title="Twitter" rel="nofollow noopener" target="_blank"><span class="a2a_svg a2a_s__default a2a_s_twitter" style="background-color: rgb(10, 107, 216);"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path fill="#FFF" d="M28 8.557a9.913 9.913 0 0 1-2.828.775 4.93 4.93 0 0 0 2.166-2.725 9.738 9.738 0 0 1-3.13 1.194 4.92 4.92 0 0 0-3.593-1.55 4.924 4.924 0 0 0-4.794 6.049c-4.09-.21-7.72-2.17-10.15-5.15a4.942 4.942 0 0 0-.665 2.477c0 1.71.87 3.214 2.19 4.1a4.968 4.968 0 0 1-2.23-.616v.06c0 2.39 1.7 4.38 3.952 4.83-.414.115-.85.174-1.297.174-.318 0-.626-.03-.928-.086a4.935 4.935 0 0 0 4.6 3.42 9.893 9.893 0 0 1-6.114 2.107c-.398 0-.79-.023-1.175-.068a13.953 13.953 0 0 0 7.55 2.213c9.056 0 14.01-7.507 14.01-14.013 0-.213-.005-.426-.015-.637.96-.695 1.795-1.56 2.455-2.55z"></path></svg></span><span class="a2a_label">Twitter</span></a><a class="a2a_button_pinterest" href="/#pinterest" title="Pinterest" rel="nofollow noopener" target="_blank"><span class="a2a_svg a2a_s__default a2a_s_pinterest" style="background-color: rgb(10, 107, 216);"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path fill="#FFF" d="M16.539 4.5c-6.277 0-9.442 4.5-9.442 8.253 0 2.272.86 4.293 2.705 5.046.303.125.574.005.662-.33.061-.231.205-.816.27-1.06.088-.331.053-.447-.191-.736-.532-.627-.873-1.439-.873-2.591 0-3.338 2.498-6.327 6.505-6.327 3.548 0 5.497 2.168 5.497 5.062 0 3.81-1.686 7.025-4.188 7.025-1.382 0-2.416-1.142-2.085-2.545.397-1.674 1.166-3.48 1.166-4.689 0-1.081-.581-1.983-1.782-1.983-1.413 0-2.548 1.462-2.548 3.419 0 1.247.421 2.091.421 2.091l-1.699 7.199c-.505 2.137-.076 4.755-.039 5.019.021.158.223.196.314.077.13-.17 1.813-2.247 2.384-4.324.162-.587.929-3.631.929-3.631.46.876 1.801 1.646 3.227 1.646 4.247 0 7.128-3.871 7.128-9.053.003-3.918-3.317-7.568-8.361-7.568z"></path></svg></span><span class="a2a_label">Pinterest</span></a><a class="a2a_button_linkedin" href="/#linkedin" title="LinkedIn" rel="nofollow noopener" target="_blank"><span class="a2a_svg a2a_s__default a2a_s_linkedin" style="background-color: rgb(10, 107, 216);"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path d="M6.227 12.61h4.19v13.48h-4.19V12.61zm2.095-6.7a2.43 2.43 0 0 1 0 4.86c-1.344 0-2.428-1.09-2.428-2.43s1.084-2.43 2.428-2.43m4.72 6.7h4.02v1.84h.058c.56-1.058 1.927-2.176 3.965-2.176 4.238 0 5.02 2.792 5.02 6.42v7.395h-4.183v-6.56c0-1.564-.03-3.574-2.178-3.574-2.18 0-2.514 1.7-2.514 3.46v6.668h-4.187V12.61z" fill="#FFF"></path></svg></span><span class="a2a_label">LinkedIn</span></a><a class="a2a_button_email" href="/#email" title="Email" rel="nofollow noopener" target="_blank"><span class="a2a_svg a2a_s__default a2a_s_email" style="background-color: rgb(10, 107, 216);"><svg focusable="false" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32"><path fill="#FFF" d="M26 21.25v-9s-9.1 6.35-9.984 6.68C15.144 18.616 6 12.25 6 12.25v9c0 1.25.266 1.5 1.5 1.5h17c1.266 0 1.5-.22 1.5-1.5zm-.015-10.765c0-.91-.265-1.235-1.485-1.235h-17c-1.255 0-1.5.39-1.5 1.3l.015.14s9.035 6.22 10 6.56c1.02-.395 9.985-6.7 9.985-6.7l-.015-.065z"></path></svg></span><span class="a2a_label">Email</span></a></div></div><p></p>
</div></div>
</div>	</div>
				</div>

			</div>
		</div>