Cookbook for preparing your own assistant for meetings
Tags: ai, python, embeddedpython, programming
Published: 2024-03-23 16:40:44

  <p>Hi folks</p>
<p>I want to tell you how you can make your own assistant based on IRIS and OpenAI (perhaps you can then move to your own AI models)</p>
<p><a href="https://openexchange.intersystems.com/package/iris-recorder-helper">iris-recorder-helper</a></p>
<p><img src="https://github.com/alexnxela/iris-recorder-helper/blob/master/demo/main.png?raw=true" width="500"></p>
<p>This is the first time I have fully tried developing an application for IRIS and I want to point out steps that may also be useful to you</p>
<ol>
<li>Deploy to google cloud via github workflows (One container with an IRIS database and a web server in Python (flask)) and placing secrets inside the application not through an environment variable</li>
<li>Record audio from the browser and send to the server</li>
<li>Launching cron jobs via the ZPM module</li>
<li>Integration with OpenAi</li>
</ol>
<h2>First, about the application itself</h2>
<p>You can try it using the link and see how it works - <a href="https://iris-recorder-helper.demo.community.intersystems.com/" target="_blank">https://iris-recorder-helper.demo.community.intersystems.com/</a></p>
<p><strong>Here are the main scenarios to use</strong></p>
<ol>
<li>You can record offline meetings by opening the page on your phone</li>
<li>You can record an online meeting by also opening the site on your phone and turning on the sound</li>
</ol>
<p>After you make an entry, you can process it with your prompts</p>
<ul>
<li>Make a summary</li>
<li>Highlight agreements on who should do what</li>
<li>Find dates and promises</li>
</ul>
<h2>Now about the problems that I got</h2>
<h3>1) Deployment&nbsp;</h3>
<p>The <a href="https://github.com/intersystems-community/demo-deployment/blob/master/.github/workflows/deployment.yml" target="_blank">basic deployment</a>, which is here, is great for a simple application, but I needed to put my secrets in the application itself (not through the environment)</p>
<p>For this I made a modification to the <a href="https://github.com/alexnxela/iris-recorder-helper/blob/master/.github/workflows/build-default.yaml">github workflow</a></p>
<pre class="codeblock-container" idlang="4" lang="YAML" tabsize="4"><code class="language-yaml hljs"><span class="hljs-string">...</span>
<span class="hljs-attr">- name:</span> <span class="hljs-string">Build</span> <span class="hljs-string">and</span> <span class="hljs-string">Push</span> <span class="hljs-string">image</span>
<span class="hljs-attr">        run:</span> <span class="hljs-string">|
          echo "$<span class="hljs-template-variable">{ secrets.CONFIG_BASE64 }</span>" &gt; ./rh/flask/app/config.b64
          docker buildx build -t ${GCR_LOCATION}/${PROJECT_ID}/${IMAGE_NAME}:${GITHUB_SHA} --push .
</span><span class="hljs-attr">- name:</span> <span class="hljs-string">Deploy</span> <span class="hljs-string">to</span> <span class="hljs-string">Cloud</span> <span class="hljs-string">Run</span>
<span class="hljs-string">...</span></code></pre>
<p>In my repository you can find an example in Python that converts the config from the config.json file to base64<br><u>Please note that base64 is used to bypass problems with "line breaks" and other special characters</u></p>
<p>You can also use secrets&nbsp;<a href="https://cloud.google.com/run/docs/configuring/services/secrets" target="_blank">https://cloud.google.com/run/docs/configuring/services/secrets</a> but the “deployment demo key” does not have permission to create secrets :(</p>
<h3>Now about the problem with audio recording</h3>
<p>If you want to send an audio stream to the server in chunks, then the first chunk will arrive perfectly and can be sent for decoding, but subsequent chunks will arrive “broken”, since there are headers at the beginning of the audio file. Therefore, I made my own timer that takes the first part of the audio file and adds it to all subsequent ones (<a href="https://github.com/alexnxela/iris-recorder-helper/blob/2fc3f5f41eeda60716f0d9fd84b8339808efa205/rh/flask/app/static/record .js#L124" target="_blank">you can see it here</a>)</p>
<pre class="codeblock-container" idlang="5" lang="JavaScript" tabsize="4"><code class="language-javascript hljs">mediaRecorder.ondataavailable = <span class="hljs-function"><span class="hljs-keyword">function</span> (<span class="hljs-params">e</span>) </span>{
    <span class="hljs-keyword">if</span> (e.data.size &gt; <span class="hljs-number">0</span>) {
        <span class="hljs-keyword">if</span> (mediaChunkFirst === <span class="hljs-literal">null</span>) {
            mediaChunkFirst = e.data
            <span class="hljs-keyword">return</span>;
        }

        mediaChunks.push(e.data);
        mediaChunkNum++;
        <span class="hljs-built_in">console</span>.log(<span class="hljs-string">'iteration: '</span>, mediaChunkNum);
        <span class="hljs-keyword">if</span> (mediaChunkNum &gt; <span class="hljs-number">10</span>) {
            saveAudio();
        }
    }
};

<span class="hljs-comment">//...</span>
<span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">saveAudio</span>() </span>{
    <span class="hljs-keyword">let</span> data = [mediaChunkFirst];
    data = data.concat(mediaChunks);
    mediaChunks = [];
    mediaChunkNum = <span class="hljs-number">0</span>;

    <span class="hljs-keyword">const</span> audioBlob = <span class="hljs-keyword">new</span> Blob(data, {<span class="hljs-attr">type</span>: <span class="hljs-string">'audio/wav'</span>});
    <span class="hljs-keyword">const</span> formData = <span class="hljs-keyword">new</span> FormData();
    formData.append(<span class="hljs-string">'content'</span>, audioBlob);
    formData.append(<span class="hljs-string">'date_created'</span>, <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>().toISOString());
    formData.append(<span class="hljs-string">'key'</span>, <span class="hljs-built_in">window</span>.session_key);

    $.ajax({
        <span class="hljs-attr">type</span>: <span class="hljs-string">'POST'</span>,

<span class="hljs-comment">// ...</span></code></pre>
<h4>Cron</h4>
<p>I decided to do data cleaning on the demo stand to minimize data leaks<br>To do this, I used the module <a href="https://community.intersystems.com/post/schedule-tasks-one-line-using-iris-cron-task" target="_blank">iris-cron-task</a> and when calling the /autoclean method in IRIS, a task is added to clear all records older than two days<br>The code can be <a href="https://github.com/alexnxela/iris-recorder-helper/blob/2fc3f5f41eeda60716f0d9fd84b8339808efa205/rh/flask/app/__init__.py#L139" target="_blank">found here</a></p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-meta">@app.route('/autoclean')</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">autoclean</span><span class="hljs-params">()</span>:</span>
        <span class="hljs-keyword">from</span> iris_cron <span class="hljs-keyword">import</span> Task
        <span class="hljs-keyword">import</span> iris

        task_name = <span class="hljs-string">'iris-recorder-helper-cleaner'</span>
        task_found = <span class="hljs-keyword">False</span>
        tasks = Task.get_tasks()
        <span class="hljs-keyword">for</span> item <span class="hljs-keyword">in</span> tasks:
            <span class="hljs-keyword">if</span> item[<span class="hljs-string">'name'</span>] == task_name:
                task_found = <span class="hljs-keyword">True</span>
        task_id = <span class="hljs-number">0</span>
        <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> task_found:
            task = <span class="hljs-string">"""import iris 
            iris.sql.exec("DELETE FROM Record WHERE date_created &lt;= DATEADD(DAY, -2, CURRENT_TIMESTAMP)")
            """</span>
            tid = iris.ref(<span class="hljs-number">0</span>)
            command = <span class="hljs-string">f'w ##class(%SYS.Python).Run("<span class="hljs-subst">{task}</span>")'</span>
            <span class="hljs-comment"># 0 0 6 * * * - at 6 am every day</span>
            iris.cls(<span class="hljs-string">'dc.cron.task'</span>).Start(task_name, <span class="hljs-string">'0 0 6 * * *'</span>, command, <span class="hljs-keyword">True</span>, tid)
            task_id = tid.value

        <span class="hljs-keyword">return</span> <span class="hljs-string">f"Task id: <span class="hljs-subst">{task_id}</span>"</span>, <span class="hljs-number">200</span></code></pre>
<p>&nbsp;</p>
<h3>Integration with openai</h3>
<p>in fact, everything is very simple here, I used a ready-made library (pip install openai) and the whole process consists of a <a href="https://github.com/alexnxela/iris-recorder-helper/blob/2fc3f5f41eeda60716f0d9fd84b8339808efa205/rh/flask/app/__init__.py#L81" target="_blank">few lines</a></p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-meta">    @app.route('/save', methods=['POST'])</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">save</span><span class="hljs-params">()</span>:</span>
        <span class="hljs-keyword">try</span>:
            file_content = request.files[<span class="hljs-string">'content'</span>]

            date_created = datetime.strptime(request.form[<span class="hljs-string">'date_created'</span>], <span class="hljs-string">'%Y-%m-%dT%H:%M:%S.%fZ'</span>)
            rec = Record(key=request.form[<span class="hljs-string">'key'</span>],
                         content=file_content,
                         text=<span class="hljs-keyword">None</span>,
                         filepath=<span class="hljs-keyword">None</span>,
                         date_created=date_created)

            db.session.add(rec)
            db.session.commit()

            tmp_file = NamedTemporaryFile(suffix=<span class="hljs-string">".wav"</span>)

            file_content.save(tmp_file)

            <span class="hljs-keyword">with</span> open(tmp_file.name, <span class="hljs-string">"rb"</span>) <span class="hljs-keyword">as</span> tmp_file_audio:
                transcript = openai.Audio.transcribe(<span class="hljs-string">"whisper-1"</span>, tmp_file_audio)

            <span class="hljs-keyword">if</span> transcript.text:
                rec.text = transcript.text

            db.session.commit()

            result = [{<span class="hljs-string">'id'</span>: rec.id, <span class="hljs-string">'key'</span>: rec.key,
                       <span class="hljs-string">'date_created'</span>: rec.date_created,
                       <span class="hljs-string">'text'</span>: rec.text}]
            <span class="hljs-keyword">return</span> jsonify(result)
        <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:
            <span class="hljs-keyword">return</span> <span class="hljs-string">f'Error: <span class="hljs-subst">{str(e)}</span>'</span>, <span class="hljs-number">500</span></code></pre>
<p><br>and also subsequent processing of all <a href="https://github.com/alexnxela/iris-recorder-helper/blob/2fc3f5f41eeda60716f0d9fd84b8339808efa205/rh/flask/app/__init__.py#L33" target="_blank">messages via prompt</a></p>
<pre class="codeblock-container" idlang="2" lang="Python" tabsize="4"><code class="language-python hljs"><span class="hljs-meta">    @app.route('/summary', methods=['POST'])</span>
    <span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">summary</span><span class="hljs-params">()</span>:</span>
        <span class="hljs-keyword">try</span>:
            key = request.form[<span class="hljs-string">'key'</span>]
            prompt = request.form[<span class="hljs-string">'prompt'</span>]

            <span class="hljs-keyword">if</span> key <span class="hljs-keyword">is</span> <span class="hljs-keyword">None</span>:
                <span class="hljs-keyword">return</span> jsonify({<span class="hljs-string">'message'</span>: <span class="hljs-string">'key is empty'</span>}), <span class="hljs-number">400</span>
            <span class="hljs-keyword">if</span> prompt <span class="hljs-keyword">is</span> <span class="hljs-keyword">None</span>:
                <span class="hljs-keyword">return</span> jsonify({<span class="hljs-string">'message'</span>: <span class="hljs-string">'prompt is empty'</span>}), <span class="hljs-number">400</span>
            records = Record.query.filter_by(key=key).order_by(Record.date_created).all()
            msgs = <span class="hljs-string">''</span>
            <span class="hljs-keyword">if</span> records:
                <span class="hljs-keyword">for</span> record <span class="hljs-keyword">in</span> records:
                    <span class="hljs-keyword">if</span> record.text:
                        msgs += <span class="hljs-string">"\n"</span> + record.text
            <span class="hljs-keyword">else</span>:
                <span class="hljs-keyword">raise</span> <span class="hljs-string">"Msgs is empty"</span>
            prompt = prompt.replace(<span class="hljs-string">"${MSGS}"</span>, msgs)

            <span class="hljs-string">"""
            print('PROMPT:', prompt)

            response = openai.Completion.create(
                model="text-davinci-003",
                prompt=prompt,
                temperature=JSON_CONFIG['OPENAI_MAX_TEMP']
            )
            print(response)
            """</span>
            response = openai.ChatCompletion.create(
                model=<span class="hljs-string">"gpt-3.5-turbo"</span>,
                messages=[
                    {<span class="hljs-string">"role"</span>: <span class="hljs-string">"user"</span>, <span class="hljs-string">"content"</span>: prompt},
                ],
            )

            print(response)

            msg = response[<span class="hljs-string">"choices"</span>][<span class="hljs-number">0</span>][<span class="hljs-string">"message"</span>][<span class="hljs-string">"content"</span>].strip()

            <span class="hljs-keyword">return</span> jsonify({<span class="hljs-string">"msg"</span>: msg})
        <span class="hljs-keyword">except</span> Exception <span class="hljs-keyword">as</span> e:
            <span class="hljs-keyword">return</span> <span class="hljs-string">f'Error: <span class="hljs-subst">{str(e)}</span>'</span>, <span class="hljs-number">500</span></code></pre>
<p>&nbsp;</p>
<p>Thank you for reading to the end, I will be glad to receive your feedback.</p>