PrivateGPT exploring the Documentation
Tags: machinelearning, applemacos, documentation, programming
Published: 2024-03-23 16:57:44

<p>Considering new business interest in applying Generative-AI to local commercially sensitive private data and information, without exposure to public clouds. Like a match needs the energy of striking to ignite, the Tech lead new "activation energy" challenge is to reveal how investing in GPU hardware could support novel competitive capabilities. The capability can reveal the use-cases that provide new value and savings.</p>
<p>Sharpening this axe begins with a functional protocol for running LLMs on a local laptop.</p>
<p>My local Mac has an M1 processor. In early experiments had found from exploring Falcon models the flow of toolkit was primarily towards using the Cuda graphics card. There are "compressed" versions of models ( Quantized ). However from my current exploration path these could only be loaded into a Cuda GPU which I didn't have. No Bit-and-bytes support for M1 / M2 processors and AutoGPTQ quantization didn't support MPS processor either. Running the unquantized models in CPU was prohibitively slow.</p>
<p>I had spotted PrivateGPT project and the following steps got things running.</p>
<pre># install developer tools
xcode-select --install 

# create python sandbox
mkdir PrivateGTP
cd privateGTP/
python3 -m venv .
# actiavte local context
source bin/activate

# privateGTP uses poetry for python module management
privateGTP&gt; pip install poetry

# sync privateGTP project
privateGTP&gt; git clone https://github.com/imartinez/privateGPT

# enable MPS for model loading and processing
privateGTP&gt; CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python

privateGTP&gt; cd privateGPT

# Import configure python dependencies
privateGTP&gt; poetry run python3 scripts/setup

# launch web interface to confirm operational on default model
privateGTP&gt; python3 -m private_gpt
# navigate safari browser to http://localhost:8001/

# To bulk import documentation needed to stop the web interface as vector database not in multi-user mode
privateGTP&gt; [control] + "C"

# import some PDFs
privateGTP&gt; curl "https://docs.intersystems.com/irislatest/csp/docbook/pdfs.zip" -o /tmp/pdfs.zip
privateGTP&gt; unzip /tmp/pdfs.zip -d /tmp
# took a few hours to process
privateGTP&gt; make ingest /tmp/pdfs/pdfs/

# launch web interface again for query documentation
privateGTP&gt; python3 -m private_gpt</pre>
<h2>Experiments with the default model mistral-7B-Instruct</h2>
<p>Some things that worked reasonably were looking for more textual content.</p>
<h3>1. What is a lock Table?</h3>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3zvjuo1cnbmhvcn16w78.jpeg)
<h3>2. Write Object Script "Hello World"</h3>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i1ol362hyzsvulzbjslr.jpeg)
<h3>3. INSERT in SQL Statement</h3>
<p>The trailing semicolon is a trained habit from training on different implementations of SQL.</p>
<p>Current deployed versions of IRIS would have an issue with the trailing semicolon but this is addressed as discarded rather that being an error in newer version specifically to address this common generative nuance.</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4hxw6zypv54lklcuxihp.jpeg)
<h3>4. ObjectScript FOR loop</h3>
<p>The challenge here is treating the increment as the max counter. So it creates an endless loop incrementing by "10" instead of a loop that increments by "1" ten times. As a minor point the trailing quit is not necessary to the loop, but is tidy in terms of a well contained line label.</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ivptxmo6yl1nrhu508q4.jpeg)
<h3>5 . ObjectScript continue</h3>
<p>Here the use of documentation is clearly confused the ObjectScript with IRIS BASIC language examples ( THEN keyword ). A query docs approach, possibly needs to use "ObjectScript" as a metadata filter or have upstream generated sets of help PDFs that are limited to a particular language implementation. Anticipating Python and Java examples could impose a similar effect. The use of "i%2" is python syntax for modular operator where as would be expecting "i#2" for object script.</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0wjqz2o9sdryfl6ko5xb.jpeg)
<p>&nbsp;</p>
<h2>Swapping out models</h2>
<p>New models can be added by downloading GGUF format models to the models sub-directory from https://huggingface.co/</p>
<p>Here the naming convention contained "Q"+level to indicate quantization loss versus size. Where lower "Q" is effectively a smaller download model with more quality loss.</p>
<h3>settings.yaml</h3>
<pre>local:
&nbsp; llm_hf_repo_id: TheBloke/Mistral-7B-Instruct-v0.1-GGUF
&nbsp; llm_hf_model_file: mistral-7b-instruct-v0.1.Q4_K_M.gguf
&nbsp; #llm_hf_repo_id: TheBloke/Orca-2-13B-GGUF
&nbsp; #llm_hf_model_file: orca-2-13b.Q6_K.gguf
&nbsp; #llm_hf_repo_id: TheBloke/XwinCoder-34B-GGUF
&nbsp; #llm_hf_model_file: xwincoder-34b.Q6_K.gguf
&nbsp; embedding_hf_model_name: BAAI/bge-small-en-v1.5ssss</pre>
<p>Using "LLM Chat" mode ( No query documents) with "xwincoder-34b" model, suggests much "code" recommendations can come from the existing trained model.</p>
<p>It demonstrated an interesting learned confusion between Globals and Routines which both are referenced by the carrot ( "^" ) .</p>
<p>The enthusiasm for "%" prefix for method name invocation may be learned invocation patterns from system class documentation instead of learning the feature of method name relates to invoking said method.</p>
<p>There are configuration references with system management that undertake actions in the %SYS namespace and this is generally quite separate from activities of third-party code.</p>
<p>It was interesting to see the invocation as well as implementation example to uncover such disparity ( Invoked as a Routine instead of class method ).</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9nfmk58v17twmw55x32u.png)
<h2>General Generative capabilities ( Model Orca 13b Q6 )</h2>
<h3>1. Generating questions and answers from supplied text</h3>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/doakq6a0k3s2gb5c2mf2.jpeg)
<h3>2. Explaining differences of concepts from supplied text</h3>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/verxhnqsba3czpvbkytb.jpeg)
<h3>3. Summarizing information into a shorter form</h3>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/abcpug9baykspt522215.jpeg)
<h2>Integrate with IRIS</h2>
<p>PrivateGPT can be put to work from IRIS by exercising available JSON end point with surprisingly little code.</p>
<p>The following code snippets demonstrate direct usage from ObjectScript.</p>
<p>This can easily be wrapped by IRIS Integration Operations and Messages to provide a reusable facility with configuration, message trace, etc.,.</p>
<h3>IRIS Conversation example</h3>
<div style="color: #ffffff;background-color: #000000;font-family: Consolas, 'Courier New', monospace;font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;">
<div><span style="color: #7ca668;">// Construct a request message via IRIS building Dynamic Object in ObjectScript</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The property "use_context" = true, this ensures that preloaded documents are searched</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// for similarity, autoloaded into context to support response generation </span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The property "prompt", This is the full prompt provided to the LLM</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">request</span><span style="color: #ffffff;">={</span><span style="color: #ce9178;">"use_context"</span><span style="color: #ffffff;">:</span><span style="color: #569cd6;">true</span><span style="color: #ffffff;">,</span><span style="color: #ce9178;">"prompt"</span><span style="color: #ffffff;">:</span><span style="color: #ce9178;">"What is an IRIS lock table"</span><span style="color: #ffffff;">}</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Using %Net.HttpRequest for direct access</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">=</span><span style="color: #569cd6;">##class</span><span style="color: #ffffff;">(</span><span style="color: #4ec9b0;">%</span><span style="color: #4ec9b0;">Net</span><span style="color: #4ec9b0;">.</span><span style="color: #4ec9b0;">HttpRequest</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">%New</span><span style="color: #ffffff;">()</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Server</span><span style="color: #ffffff;">=</span><span style="color: #ce9178;">"127.0.0.1"</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Port</span><span style="color: #ffffff;">=</span><span style="color: #b5cea8;">8001</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">ContentType</span><span style="color: #ffffff;">=</span><span style="color: #ce9178;">"application/json"</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">SetHeader</span><span style="color: #ffffff;">(</span><span style="color: #ce9178;">"Accept"</span><span style="color: #ffffff;">,</span><span style="color: #ce9178;">"application/json"</span><span style="color: #ffffff;">)</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Embed the request </span><span style="color: #7ca668;">as</span><span style="color: #7ca668;"> </span><span style="color: #7ca668;">JSON</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">EntityBody</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Write</span><span style="color: #ffffff;">(</span><span style="color: #9cdcfe;">request</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%</span><span style="color: #dcdcaa;">ToJSON</span><span style="color: #ffffff;">())</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Make request</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The optional "2" argument causes the response to be output to the default device ( </span><span style="color: #7ca668;">terminal</span><span style="color: #7ca668;"> )</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Useful for analysing returned output</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Post</span><span style="color: #ffffff;">(</span><span style="color: #ce9178;">"/v1/completions"</span><span style="color: #ffffff;">,</span><span style="color: #b5cea8;">2</span><span style="color: #ffffff;">) </span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The stream should already be rewound, but stated here </span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// should stream need to be reused in testing</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">HttpResponse</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Data</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Rewind</span><span style="color: #ffffff;">()</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Turn the response back into IRIS %Dynamic Object</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">response</span><span style="color: #ffffff;">={}.</span><span style="color: #dcdcaa;">%FromJSON</span><span style="color: #ffffff;">(</span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">HttpResponse</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Data</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Read</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">32000</span><span style="color: #ffffff;">))</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Grab the generated response</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">outputMessage</span><span style="color: #ffffff;">=</span><span style="color: #9cdcfe;">response</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">choices</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%Get</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">0</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">message</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Grab the metadata for references used by generation</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">outputSources</span><span style="color: #ffffff;">=</span><span style="color: #9cdcfe;">response</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">choices</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%Get</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">0</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">sources</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%Get</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">0</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">document</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Write</span><span style="color: #ffffff;"> !,</span><span style="color: #ce9178;">"Output:"</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">outputMessage</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">content</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Write</span><span style="color: #ffffff;"> !,</span><span style="color: #ce9178;">"File reference:"</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">outputSources</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">"doc_metadata"</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">"page_label"</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Write</span><span style="color: #ffffff;"> !,</span><span style="color: #ce9178;">"Filename:"</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">outputSources</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">"doc_metadata"</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">"file_name"</span></div>
</div>
<p>Example output:</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cp1khwfyzei0gfstd7rj.png)
<p>&nbsp;</p>
<h3>IRIS Retrieve Embedding Example</h3>
<div style="color: #ffffff;background-color: #000000;font-family: Consolas, 'Courier New', monospace;font-weight: normal;font-size: 14px;line-height: 19px;white-space: pre;">
<div><span style="color: #7ca668;">// Construct a request message via IRIS building Dynamic Object Construct</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">textForEmbedding</span><span style="color: #ffffff;">={</span><span style="color: #ce9178;">"input"</span><span style="color: #ffffff;">:</span><span style="color: #ce9178;">"This is some test text from IRIS database"</span><span style="color: #ffffff;">}</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Using %Net.HttpRequest for direct access</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">=</span><span style="color: #569cd6;">##class</span><span style="color: #ffffff;">(</span><span style="color: #4ec9b0;">%Net</span><span style="color: #4ec9b0;">.</span><span style="color: #4ec9b0;">HttpRequest</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">%New</span><span style="color: #ffffff;">()</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Server</span><span style="color: #ffffff;">=</span><span style="color: #ce9178;">"127.0.0.1"</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Port</span><span style="color: #ffffff;">=</span><span style="color: #b5cea8;">8001</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">ContentType</span><span style="color: #ffffff;">=</span><span style="color: #ce9178;">"application/json"</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">SetHeader</span><span style="color: #ffffff;">(</span><span style="color: #ce9178;">"Accept"</span><span style="color: #ffffff;">,</span><span style="color: #ce9178;">"application/json"</span><span style="color: #ffffff;">)</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Embedd the request as JSON</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">EntityBody</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Write</span><span style="color: #ffffff;">(</span><span style="color: #9cdcfe;">textForEmbedding</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%ToJSON</span><span style="color: #ffffff;">())</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Make request</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The optional "2" argument causes the response to be output to the default device ( terminal )</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Useful for analysing returned output</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Post</span><span style="color: #ffffff;">(</span><span style="color: #ce9178;">"/v1/embeddings"</span><span style="color: #ffffff;">,</span><span style="color: #b5cea8;">2</span><span style="color: #ffffff;">)</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// The stream should already be rewound, but stated here </span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// should stream need to be reused in testing</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Do</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">HttpResponse</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Data</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Rewind</span><span style="color: #ffffff;">()</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Turn the response back into IRIS %Dynamic Object</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">response</span><span style="color: #ffffff;">={}.</span><span style="color: #dcdcaa;">%FromJSON</span><span style="color: #ffffff;">(</span><span style="color: #9cdcfe;">hr</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">HttpResponse</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Data</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">Read</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">32000</span><span style="color: #ffffff;">))</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Example of iterator to loop over embeddings array</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">Set</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">iter</span><span style="color: #ffffff;">=</span><span style="color: #9cdcfe;">response</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">data</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%Get</span><span style="color: #ffffff;">(</span><span style="color: #b5cea8;">0</span><span style="color: #ffffff;">).</span><span style="color: #dcdcaa;">embedding</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%GetIterator</span><span style="color: #ffffff;">()</span></div>
<div><span style="color: #7ca668;">&nbsp; &nbsp; </span><span style="color: #7ca668;">// Output of floating point numbers of the returned embedding</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; </span><span style="color: #c586c0;">While</span><span style="color: #ffffff;"> </span><span style="color: #9cdcfe;">iter</span><span style="color: #ffffff;">.</span><span style="color: #dcdcaa;">%GetNext</span><span style="color: #ffffff;">(.</span><span style="color: #9cdcfe;">key</span><span style="color: #ffffff;">, .</span><span style="color: #9cdcfe;">value</span><span style="color: #ffffff;">, .</span><span style="color: #9cdcfe;">type</span><span style="color: #ffffff;">) {</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; &nbsp; &nbsp; </span><span style="color: #c586c0;">Write</span><span style="color: #ffffff;"> !,</span><span style="color: #ce9178;">"key="</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">key</span><span style="color: #ffffff;">,</span><span style="color: #ce9178;">", value="</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">value</span><span style="color: #ffffff;">,</span><span style="color: #ce9178;">", type="</span><span style="color: #ffffff;">,</span><span style="color: #9cdcfe;">type</span></div>
<div><span style="color: #ffffff;">&nbsp; &nbsp; }</span></div>
</div>
<p>Example output:</p>

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a946yn7xr4o2nh168ozr.png)woriginal="570"></p>
<h3>Further IRIS integration</h3>
<p>The web API also supports:</p>
<ul>
<li>dynamically loading new source documents</li>
<li>listing existing source document</li>
<li>deleting existing source documents</li>
<li>a health API to indicate availability</li>
</ul>
<p>Further details available at: https://docs.privategpt.dev/api-reference/api-reference/ingestion</p>
<h2>Summary thoughts and ideas</h2>
<p>Local inference ( running quantized LLMs on laptop for productivity ) can be a useful way to initially scale the application of existing and internally retrained models. It allows flexibility for business users to privately explore and share new usercases and prompt engineering recipes.</p>
<p>To use PrivateGPT better for documentation, would need to delve deeper to reconfigure generative temperature lower, to reduce the creativity and improve accuracy of answers.</p>
<p>Technical Documentation and user manuals are no longer intended simply for human readers.</p>
<p>Can existing documentation pipelines be easily repurposed with metadata to shape documentation output that is better consumed and repurposed for generative AI?</p>
<p>A single documentation resource for multiple code languages makes it difficult to generalize usefully without conflating code examples. Hence hypothesis is that both documentation "code language" and "source retraining" of models would be better suited in short-term to be mono-code-language resources and assistants.</p>
<p>How well can retraining an existing model "unlearn" existing code conflations to be replaced with the useful and expected code syntax?</p>
<p>Hope this inspires new explorations.</p>
<h2>References</h2>
<p>Martínez Toro, I., Gallego Vico, D., &amp; Orgaz, P. (2023). PrivateGPT [Computer software]. https://github.com/imartinez/privateGPT</p>
<h2>Resources</h2>
<p>https://github.com/imartinez/privateGPT</p>
<p>https://docs.privategpt.dev/</p>
<p>https://huggingface.co/models</p>
<p>https://github.com/ggerganov/llama.cpp</p>