Doctor-Patient Conversations: AI-Powered Transcription and Summarization
Tags: ai, docker, javascript, chatgpt
Published: 2023-08-16 16:06:08


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sl1f9v9gqiz1lpc5lo27.png)

 

**Previous post** - [Using AI to Simplify Clinical Documents Storage, Retrieval and Search](https://community.intersystems.com/post/using-ai-simplify-clinical-documents-storage-retrieval-and-search)

This post explores the potential of OpenAI's advanced language models to revolutionize healthcare through AI-powered transcription and summarization. We will delve into the process of leveraging OpenAI's cutting-edge APIs to convert audio recordings into written transcripts and employ natural language processing algorithms to extract crucial insights for generating concise summaries.

While existing solutions like Amazon Medical Transcribe and MedVoice offer similar capabilities, the focus of this post will be on implementing these powerful features using OpenAI technology within Intersystems FHIR.

## Recording audio on Vue.js

The voice recorder on the Vue.js app if fully native and written using the [Mediarecorder](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder) interface in javascript. This was done to keep the application light-weight and also get full control over the recording options. Below are snippets to start and stop recording audio input.

<pre class="codeblock-container" idlang="5" lang="JavaScript" tabsize="4"><code class="language-javascript hljs"><span class="hljs-comment">// Start recording method that stores audio stream as chunks</span>
<span class="hljs-keyword">async</span> startRecording() {
      <span class="hljs-keyword">try</span> {
        <span class="hljs-keyword">const</span> stream = <span class="hljs-keyword">await</span> navigator.mediaDevices.getUserMedia({
          <span class="hljs-attr">audio</span>: <span class="hljs-literal">true</span>,
        });
        <span class="hljs-keyword">this</span>.mediaRecorder = <span class="hljs-keyword">new</span> MediaRecorder(stream);
        <span class="hljs-keyword">this</span>.mediaRecorder.start();

        <span class="hljs-keyword">this</span>.mediaRecorder.ondataavailable = <span class="hljs-function">(<span class="hljs-params">event</span>) =></span> {
          <span class="hljs-keyword">this</span>.chunks.push(event.data);
        };
        <span class="hljs-keyword">this</span>.isRecording = <span class="hljs-literal">true</span>;
      } <span class="hljs-keyword">catch</span> (error) {
        <span class="hljs-built_in">console</span>.error(<span class="hljs-string">"Error starting recording:"</span>, error);
      }
}

<span class="hljs-comment">// Stop recording method that creates a blob after stop (and calls the transcription method)</span>
stopRecording() {
      <span class="hljs-keyword">if</span> (<span class="hljs-keyword">this</span>.mediaRecorder) {
        <span class="hljs-keyword">this</span>.isLoading = <span class="hljs-literal">true</span>;
        <span class="hljs-keyword">this</span>.mediaRecorder.stop();
        <span class="hljs-keyword">this</span>.mediaRecorder.onstop = <span class="hljs-keyword">async</span> () => {
          <span class="hljs-keyword">const</span> blob = <span class="hljs-keyword">new</span> Blob(<span class="hljs-keyword">this</span>.chunks, {
            <span class="hljs-attr">type</span>: <span class="hljs-string">"audio/webm;codecs=opus"</span>,
          });
          <span class="hljs-keyword">await</span> <span class="hljs-keyword">this</span>.sendAudioToWhisper(
            <span class="hljs-keyword">new</span> File([blob], <span class="hljs-string">`file<span class="hljs-subst">${<span class="hljs-built_in">Date</span>.now()}</span>.m4a`</span>)
          );

          <span class="hljs-keyword">this</span>.getSummary(<span class="hljs-keyword">this</span>.transcription);
        };
      }
}</code></pre>

## Transcription Component

Transcription of audio data using OpenAI's Whisper model involves several essential components. The following code snippet demonstrates the steps involved in the transcription process.

<pre class="codeblock-container" idlang="5" lang="JavaScript" tabsize="4"><code class="language-javascript hljs"><span class="hljs-keyword">const</span> apiKey = process.env.OPENAI_API_KEY;

<span class="hljs-keyword">const</span> formData = <span class="hljs-keyword">new</span> FormData();
formData.append(<span class="hljs-string">"file"</span>, blob);
formData.append(<span class="hljs-string">"model"</span>, <span class="hljs-string">"whisper-1"</span>);
formData.append(<span class="hljs-string">"response_format"</span>, <span class="hljs-string">"json"</span>);
formData.append(<span class="hljs-string">"temperature"</span>, <span class="hljs-string">"0"</span>);
formData.append(<span class="hljs-string">"language"</span>, <span class="hljs-string">"en"</span>);

<span class="hljs-keyword">try</span> {
  <span class="hljs-keyword">const</span> response = <span class="hljs-keyword">await</span> fetch(
    <span class="hljs-string">"https://api.openai.com/v1/audio/transcriptions"</span>,
    {
      <span class="hljs-attr">method</span>: <span class="hljs-string">"POST"</span>,
      <span class="hljs-attr">headers</span>: {
        <span class="hljs-attr">Accept</span>: <span class="hljs-string">"application/json"</span>,
        <span class="hljs-attr">Authorization</span>: <span class="hljs-string">`Bearer <span class="hljs-subst">${apiKey}</span>`</span>,
      },
      <span class="hljs-attr">body</span>: formData,
      <span class="hljs-attr">redirect</span>: <span class="hljs-string">"follow"</span>,
    }
  );

  <span class="hljs-keyword">const</span> data = <span class="hljs-keyword">await</span> response.json();
  <span class="hljs-keyword">if</span> (data.text) {
    <span class="hljs-keyword">this</span>.transcription = data.text;
  }
} <span class="hljs-keyword">catch</span> (error) {
  <span class="hljs-built_in">console</span>.error(<span class="hljs-string">"Error sending audio to Whisper API:"</span>, error);
}

<span class="hljs-keyword">return</span> <span class="hljs-keyword">this</span>.transcription;</code></pre>

  1. **API Key** - The `OPENAI_API_KEY` is a required authentication token for accessing OpenAI APIs.
  2. **Form Data** - The audio file to be transcribed is appended to a `FormData` object. Additional parameters like the chosen model `whisper-1`, the response format (`json`), temperature (``), and language (`en`) are also included.
  3. **API Request** - The `fetch` method is used to send a POST request to the OpenAI API endpoint `https://api.openai.com/v1/audio/transcriptions` with the specified headers and body containing the form data.
  4. **Response Handling** - The response from the API is captured, and the transcribed text is extracted from the `data` object. The transcription can be assigned to a variable `this.transcription` for further processing or usage.

## Summarization Component

The following code snippet demonstrates the core components involved in the text summarization process using OpenAI's `text-davinci-003` model.

<pre class="codeblock-container" idlang="5" lang="JavaScript" tabsize="4"><code class="language-javascript hljs">response = openai.Completion.create(
    model=<span class="hljs-string">"text-davinci-003"</span>,
    prompt=<span class="hljs-string">"Summarize the following text and give title and summary in json format. \
            Sample output - {\"title\": \"some-title\", \"summary\": \"some-summary\"}.\
            Input - "</span>
    + text,
    temperature=<span class="hljs-number">1</span>,
    max_tokens=<span class="hljs-number">300</span>,
    top_p=<span class="hljs-number">1</span>,
    frequency_penalty=<span class="hljs-number">0</span>,
    presence_penalty=<span class="hljs-number">1</span>
)

<span class="hljs-keyword">return</span> response[<span class="hljs-string">"choices"</span>][<span class="hljs-number">0</span>][<span class="hljs-string">"text"</span>].replace(<span class="hljs-string">'\n'</span>, <span class="hljs-string">''</span>)</code></pre>

  1. **Model Selection** - The `model` parameter is set to `text-davinci-003`, indicating the use of OpenAI's text completion model for summarization.
  2. **Prompt** - The prompt provided to the model specifies the desired outcome, which is to summarize the input text and return the title and summary in JSON format. The input text is concatenated to the prompt for processing. It’s interesting how we can handle response transformation through OpenAI. Only validation at the receiving end is enough and we might almost never need converters in the future.
  3. **Generation Parameters** - Parameters such as `temperature`, `max_tokens`, `top_p`, `frequency_penalty`, and `presence_penalty` are set to control the behaviour and quality of the generated summary.
  4. **API Request and Response Handling** - The `openai.Completion.create()` method is called to make the API request. The response is captured, and the generated summary text is extracted from the response object. Any newline characters (`\n`) in the summary text are removed before returning the final result.

## Document Reference in FHIR

In the context of implementing doctor-patient conversation transcription and summarization using OpenAI technologies, an important consideration is the storage of clinical notes within the FHIR standard. FHIR provides a structured and standardized approach for exchanging healthcare information, including clinical notes, among different healthcare systems and applications. The [Document Reference](https://build.fhir.org/documentreference.html) resource in FHIR serves as a dedicated section for storing clinical notes and related documents.

When integrating transcription and summarization capabilities into the doctor-patient conversation workflow, the resulting transcriptions and summaries can be stored as clinical notes within the FHIR Documents resource. This allows for easy access, retrieval, and sharing of the generated insights among healthcare providers and other authorized entities.

<pre class="codeblock-container" idlang="3" lang="JSON" tabsize="4"><code class="language-json hljs">{
    <span class="hljs-attr">"resourceType"</span>: <span class="hljs-string">"Bundle"</span>,
    <span class="hljs-attr">"id"</span>: <span class="hljs-string">"1a3a6eac-182e-11ee-9901-0242ac170002"</span>,
    <span class="hljs-attr">"type"</span>: <span class="hljs-string">"searchset"</span>,
    <span class="hljs-attr">"timestamp"</span>: <span class="hljs-string">"2023-07-01T16:34:36Z"</span>,
    <span class="hljs-attr">"total"</span>: <span class="hljs-number">1</span>,
    <span class="hljs-attr">"link"</span>: [
        {
            <span class="hljs-attr">"relation"</span>: <span class="hljs-string">"self"&</span>,
            <span class="hljs-attr">"url"</span>: <span class="hljs-string">"http://localhost:52773/fhir/r4/Patient/1/DocumentReference"</span>
        }
    ],
    <span class="hljs-attr">"entry"</span>: [
        {
            <span class="hljs-attr">"fullUrl"</span>: <span class="hljs-string">"http://localhost:52773/fhir/r4/DocumentReference/1921"</span>,
            <span class="hljs-attr">"resource"</span>: {
                <span class="hljs-attr">"resourceType"</span>: <span class="hljs-string">"DocumentReference"</span>,
                <span class="hljs-attr">"author"</span>: [
                    {
                        <span class="hljs-attr">"reference"</span>: <span class="hljs-string">"Practitioner/3"</span>
                    }
                ],
                <span class="hljs-attr">"subject"</span>: {
                    <span class="hljs-attr">"reference"</span>: <span class="hljs-string">"Patient/1"</span>
                },
                <span class="hljs-attr">"status"</span>: <span class="hljs-string">"current"</span>,
                <span class="hljs-attr">"content"</span>: [
                    {
                        <span class="hljs-attr">"attachment"</span>: {
                            <span class="hljs-attr">"contentType"</span>: <span class="hljs-string">"application/json"</span>,
                            <span class="hljs-attr">"data"</span>: <span class="hljs-string">"<base64-payload>"</span>
                        }
                    }
                ],
                <span class="hljs-attr">"id"</span>: <span class="hljs-string">"1921"</span>,
                <span class="hljs-attr">"meta"</span>: {
                    <span class="hljs-attr">"lastUpdated"</span>: <span class="hljs-string">"2023-07-01T16:34:33Z"</span>,
                    <span class="hljs-attr">"versionId"</span>: <span class="hljs-string">"1"</span>
                }
            },
            <span class="hljs-attr">"search"</span>: {
                <span class="hljs-attr">"mode"</span>: <span class="hljs-string">"match"</span>
            }
        }
    ]
}</code></pre>

## Try it out

  1. **Clone the Project** - Clone the project repository from the following GitHub link: <https://github.com/ikram-shah/iris-fhir-transcribe-summarize-export>.
  2. **Setup Locally** - Follow the provided instructions to set up the project locally on your machine. Let us know if you face any issues during set up.
  3. **Select a Patient** - Choose a patient from the provided sample list within the project. This patient will be associated with the doctor-patient conversation for transcription and summarization.
  4. **Interactions Page** -Once the patient is selected, navigate to the interactions page within the project. On this page, locate and click on the "Take Notes" option to initiate the transcription process for the doctor-patient conversation.
  5. **View and Edit Transcription** - After the transcription process is completed, an option will be presented to view the generated transcription. You can also edit the title and summary associated with the transcription for better organization and clarity.
  6. **Saving to FHIR Document Reference** - Upon finalizing the title and summary, saving the changes will automatically store them within the FHIR document reference. This ensures that the relevant clinical notes are captured and associated with the respective patient's record. Currently, the project does not save the entire transcription text. However, it can be modified to include the storage of the complete transcription as well.

## Demo

Watch a demo of the overall application here - <https://youtu.be/3Th6bR4rw0w>

## Future Directions

Expanding the application of AI-powered transcription and summarization to telehealth communication holds immense potential. Integrating these capabilities with popular meeting platforms like Zoom, Teams, and Google Meet can streamline remote doctor-patient interactions. Automatic transcription and summarization of telehealth sessions offer benefits such as accurate documentation and enhanced analysis. However, data privacy remains a crucial challenge. To address this, implementing measures to filter or anonymize personally identifiable information (PII) before transmitting data to external servers is essential.

Future directions include exploring on-device AI models for local processing, improved support for multilingual communication, and advancements in privacy-preserving techniques.

**If you find this implementation useful, consider [voting for this app](https://openexchange.intersystems.com/package/IRIS-FHIR-Transcribe-Summarize-Export) at Grand Prix 2023.**