Languages Interoperability
Published: 2021-03-22 08:28:38

# Table Of Contents
        * [Introduction to Visual Microservices](#introduction-to-visual-microservices)
        * [Environment Setup](#environment-setup)
        * [Executing Algonia project](#executing-algonia-project)
        * [Project description](#project-description)
        * [Conclusion](#conclusion)

#Introduction to Visual Microservices <a name="introduction-to-visual-microservices"></a>

##Modern world challenges

The requirements and expectations of the modern world drive us to the point where specialists from different fields must collaborate and work in tandem.

Software engineers are focused on developing backends and UI engineers on designing applications. Data scientists and analysts are gathering and analyzing data. System integrators are implementing solutions in end customer environments.

The specialists mostly use different and incompatible technologies as there is no one that rules them all.

Python is the most sought-after among data professionals because it provides an environment for easily setting up foundations for Machine and Deep learning. Software engineers may prefer C# which has great support for developing multithreaded applications. On the other hand, C/C++  and Rust are very good at solving performance bottlenecks. 

##Algonia vision
Algonia's vision is to overcome those technology barriers and create tool for smooth collaboration between different technical profiles.

The first step onward to accomplish this goal is ensuring interoperability between programming languages. 
 
Data scientists must be able to seamlessly share their algorithms and models with Software Engineers. On the other hand Software Engineers must provide data sources to data scientists. 
UI engineers must be able to flawlessly integrate developed designs with backends, without much of concern about technology powering the backend.

Another problem is that complex systems are hard to maintain. Software is a living creature, constantly developed and driven by new requirements that can easily break existing codebase. Sometimes new features just cannot be added due to current codebase foundations.

Lets see how Algonia deals with those problems.

###Language interoperability
Everyone who has ever worked on a system that involves a mix of different languages and technologies can witness that project can easily turn into a nightmare. Calling functions from one language to another can easily lead to improper memory releasing causing hard-to-find bugs, memory leaks and crashes. 
Algonia’s main advantage is a solution where engineers can easily pack their code into drag'n droppable visual boxes. Once packed, they become a part of the existing ecosystem. They can be connected to other algorithms in any order, regardless of programming language.
Many programming languages are supported and new ones are adding constantly. 

###Easy maintenance
Visual programming languages offer easy maintenance and there is no exception here. Visual boxes can be re-connected, deleted from and added to visual workflows.

###Performance
If you used some of the visual programming tools already, then you know that performance was sacrificed for that benefit. 
Algonia is lightning fast and low memory usage on-premise environment that executes visual workflows. It is by far the most powerful visual programming language ever created, with more raw processing horsepower than anything available at any price.

###For non developers
System integrators, end users and other non-developers can create new functionality by purchasing visual boxes from the marketplace and connecting them into existing workflows. 

#Setting up Algonia Environment

An introductory example will be an application for displaying charts of the historical stock prices.

Architecture, demonstrated on practical Stock Prices project could serve for other projects, even larger ones. Of course you can adopt it also to your needs.

C# is going to be used for fetching data from api, Python for creating charts and JS/HTML for designing user interface. 
##Prerequisites
Make sure you have met following requirements:
- Windows or MacOS
- [Git](https://git-scm.com/downloads)
- [VS Code](https://code.visualstudio.com/)
- [Python 36, 37 or 38](https://www.python.org/downloads/)
- [NetCore 3.1](https://dotnet.microsoft.com/download/dotnet/3.1)
- [Alpha Vantage api key](https://www.alphavantage.co/support/#api-key). It's for free. We’ll use this API for getting Stock prices.

#Environment setup <a name="environment-setup"></a>
###Clone StockPrices project

Open command prompt and type command 
*git clone https://github.com/tvinko/algonia.git*
Go to the repository folder after cloning.

###Install Python Packages

We’ll use **virtualenv** to install Python packages. There are many benefits of using virtualenv for package management. It really shines in an environment with many projects that don’t share the same Python installation. This way we solve conflicting requirements.

- Run *virtualenv --version* command to check if you have installed virtualenv.
If you don’t have it then install it with command *pip install virtualenv*

- Next step is to create a virtual environment with the name **venv_38**. Run the *python --version* command to check your python default version.

Then run the command according to your default Python version 
*virtualenv venv_38 --python=python3.8*
*virtualenv venv_37 --python=python3.7*
*virtualenv venv_38 --python=python3.6*

This command will create a new folder in your project root:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8ycxfp4h8kofxelo171u.png)

- Activate virtual environment with *.\venv_38\Scripts\activate* 
  (Windows)  or 
*source venv_38/bin/activate* (macos) command

- Install packages from requirements: *pip install -r requirements.txt*

Now the packages are installed in your virtual, isolated python environment.

###Download Algonia Bundle
Download **Algonia Bundle** from  [github repo](https://github.com/tvinko/algonia/releases)


**Windows**
Use the latest release and extract **algonia-bundle-win64.zip** to your repository folder. 
Your folder structure should now look like this:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3guo1a0flwlycj6ikvjp.png)

**MacOS**
Download **algonia-bundle-macos.zip**, extract it and drag **algonia-designer** to your **Applications** folder.

##Project structure description
First and foremost, the weapon of choice here is VSCode because of its versatility. It is a source code editor that can be used with a variety of programming languages and supports multiple platforms such as Linux, Windows, and macOS.

###algonia-bundle-win64 / algonia-bundle-macos
Bundle contains a **development tool** (IDE) for gluing your algorithms (Algonia Visual Designer) and an **environment** for executing visual scripts (Algonia Engine).
Algonia Visual Designer contains (among other things) bundled .NET Core SDK, so it doesn't have to be installed on production environment. However, it must be installed on development machine.

When you ship your project to customers, you’ll create a package. This is a lightweight executing environment containing only Algonia Engine, without Algonia Designer. 

Using your own .NET Core version and creating Algonia Engine packages will be explained in the following tutorials.

###algos
Your team’s projects. Stock Prices contains three projects. They are located in *algos* directory of repository:
- Python project for graphs imaging *algos/py/graphs/plot_graph.py*
- NetCore project for fetching data from api *algos/netcore/StockPrices/StockPrices.csproj*
- HTML/Js for user interface *algos/html/assets*

Those projects will be glued together in **Algonia Visual Designer** and executed by **Algonia Engine**.

###templates
The **Algonia Visual Designer** projects are called **visual templates** - drawing canvases for connecting your projects. Templates folder contains those Algonia Visual Designer projects.

###venv_36/venv_37/venv_38
Python packages, required for the Stock Prices project.

###requirements.txt
Python requirements file

###sync.bat / sync.sh
Scripts for synchronizing your teams and 3rd party projects with **Algonia Designer**. Use **sync.bat** on Windows and **sync.sh** on MacOS systems.
Algonia Designer "glues" your project results (*.dll, *.so., *.py, *.js,*.html and similar files).
When you compile code in your preferred IDE (VisualStudio Code, Visual Studio...) the library is normally created in the bin folder of your project which is out of Algonia Designer reach. 
Those scripts synchronize your libraries with Algonia Designer.
The rule of thumb is: run sync script when you make changes outside of Algonia Designer (in VS Code or your preferred IDE)

#Executing Algonia project<a name="executing-algonia-project"></a>
###Compile StockPrices project (C#)
Open algonia.code-workspace in VSCode.
Click on **CTRL+Shift+B** (Win) or **Shift+Command+B** (MacOS) to get a list of build tasks and select **StockPricesTest** project. 

###Sync libraries and Algonia Designer
Synchronize .Net Core library with Algonia Designer by executing sync.bat/sync.sh script. Script will also synchronize our Python and HTML files.

###Set python virtual environment packages
You need to set the location to virtualenv packages so that Algonia Engine will add it to it’s packages path.

Open algonia-designer.
On Windows navigate to the algonia-bundle-win64 folder and click on algonia-designer executable to launch Algonia Designer.

On MacOS go to Applications and open algonia-designer. On the first run you’ll get following warning:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/grqwr8ngaa78bxvzwfdy.png)

Click on *Cancel*, navigate to *Security & Privacy -> General* and click *Open Anyway* 

Click on the *Project* menu tab and select *Open*. Navigate to the *templates/StockPrices* folder  of your cloned repository and select *StockPrices.algo* to open the Algonia project.

Packages are located in the *site-packages* directory. 
On Mac this is normally in your cloned repository under  *StockPrices/venv_38/lib/python3.8/site-packages* and on Windows *venv_38\Lib\site-packages*

Click on the *Settings* tab, select *Open* and enter **full** path to the packages:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pt05u46mp7z1bqkf0fvx.png)

###Set Alpha Vantage api key
*Alpha Vantage API* is used to fetch Stock prices.

Select *PRICES_OBJECT* visual box:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sa9dnn8yc9ef8e500pju.png)

Click on the *Add args* in the properties window:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bvzuvunh75x4y6ji7vrj.png)

Enter your API key in the value field of *API_KEY* property
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f2fvwucl1832cgbvcte7.png)

Click on the *Save* button
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hr4zk4dqo8ff6dmqd9i9.png)


Click *Project* and *Save* to save the project. You can also use shortcut *CTRL+S*.

###Set Python version if needed
Project is currently set for the Python 3.8 environment. If you’re using Python 3.6 or 3.7 then the version must be set on Python visual boxes. 
Select *GUI* visual box and change Python version:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/unwhjedbq31rnfnpivgu.png)

Select *SAVE_GRAPH* visual box and repeat procedure. 
Click on the *Project* menu item and click on *Save* to save the project.
 
###Run project
Press F5 to run the application and open main application window
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nnsq5tjy7crn06yzj266.png)
Click on the *Draw Graph* button

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ztrnlonbdqxtcoppbcdj.png)

You can change the value from MSFT to let's say ETH and see the result.

#Project description <a name="project-description"></a>
In the main project window you’ll see green boxes, some of them connected and some of them just floating around:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjcsn0ask6fi68s0wwi7.png)
Those boxes are wrappers around our C# and Python projects.

For example, *SET_SYMBOL* visual box calls C# function *ChangeSymbol* from *StockPrices* project (**algos\netcore\StockPrices**):
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3wzvq2cbiz7f8u0hdupw.png)

And the *GET_PRICES* visual box calls function *GetPrices* from the same solution.

*SAVE_GRAPH* visual box for instance calls *plotGraph* method from Python project (**algos\py\graphs\plot_graph.py**):
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3oa148mkm4br9kcjzs37.png)

Confused? How can “visual box” just call the function from our, or someone else’s project? Create an object instance? 
Don’t worry, it’s going to be explained in the next sections.

But I hope that you briefly got some clue at this point: projects created with different technologies and programming languages can be glued together inside **Algonia Designer**. You can connect them in any meaningful order you want. 

Now, let’s rewind to the beginning and take a look at the first row of the boxes:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o7jvdufk6n6vn4uvjqke.png)

##Start box
*Start* box is the entry point to our application. Applications can have multiple entry points running in parallel, but there is no need for this in our Stock Prices project.

##PRICES_OBJECT
Next, *PRICES_OBJECT* box, creates an object from the *StockPrices* class defined in **algos/netcore/StockPrices/StockPrices.csproj** project.

Select the box and click on the *Add code* button in it’s properties window:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9i68worcevme0l3fld37.png)

This is ordinal C# code, with some extensions:
- In **header** section we reference C# dlls and write using directives 
- Code is written in the **code** section. Tags containing attribute **run** with value **true** are entry points to visual boxes

Rest of the code should  look familiar to you. Except for the two functions, **get_arg** and **set_result**.
Those two functions are part of the **Algonia SDK** and their purpose is to control visual boxes and workflows. There are few more functions for that purpose and will be explained later.

**get_arg**
this function receives the argument key and returns it’s value. We can set those arguments programmatically or through Algonia Designer user interface. In fact you already set argument on Algonia Designer UI when you set your API key:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/515osla37qs5lpjl57ec.png)


**set_result** 
this function takes an object and saves it into the current *Visual Box*. Each *Visual Box* can store some result that is accessible to other *Visual Boxes*.

##GUI Box
This is a Python visual box containing a user interface. It uses the Python Eel package for making Electron-like offline HTML/JS GUI apps. I’m not going into Eel details as this would be out of scope for our tutorial, but I strongly recommend that you check it out [here](https://martin-thoma.com/eel/)

The wrapper code inside Visual Box is:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a1x84rvihu7ufogjx6z6.png)

Let’s just go through a few interesting points.
**def registerGui(self)**
Entry point to Visual Box is *registerGui* function, set in it’s properties window:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ka0hx4e69zmsqz1rw4f.png)

This function starts gui with *main.html*. This is Eel’s way of starting UI: 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cbo4by1un8kn9l7i1ro5.png)

**def getPricesRequest(symbol)**
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/crm2zcyec3s7uf0dcmey.png)

Algonia SDK functions here are:
**algonia_helpers.set_element_arg**
sets the value of the visual box’s argument. First parameter is the name of the visual box that argument is set and the second parameter is it’s value (besides storing results, visual boxes can also store arguments).

This function is exposed to Javascript with *eel.expose* attribute.
It receives a Stock symbol parameter from html input and it’s called by jQuery (we’ll get to HTML and jQuery part later).
The call to **set_element_arg** stores this symbol as an argument.

**algonia_helpers.exec**  executes Visual Box and all chained boxes connected into the same workflow. The parameter of this function is the visual box name.

Lets just recap what we just did:
- We set the *SET_SYMBOL* argument to the current Stock symbol that we got from the user interface and then executed it. 
- After *SET_SYMBOL* finishes, *GET_PRICES* is automatically executed (because those two boxes are visually connected).


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zdksztc6ul4hdzgjvxrd.png)

**def registerExecNodes(self)**
There is another Algonia system function, or more precise system callback:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ye1t8au0z59c68u1c5k.png)

All Visual Boxes names, that are executed dynamically with the **algonia_helpers.exec** method, must be registered and returned as string.

**def saveGraphReuest()**
This function returns graph image, encoded as base64 string and will be explained in following steps




Lets go to the next row of Visual Boxes 
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j9rekdlkk3npro4oujkz.png)

##SET_SYMBOL box
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oli91t3gemvgic2w42yn.png)

There is a new algonia SDK function here that gets the result of Visual Box - **get_result_raw**.
It takes the name of the box as a parameter.

Remember, Visual boxes can see each other’s results. *StockPrices* object is stored in a **PRICES_OBJECT** box.
Now we are going to use this object and call the *ChangeSymbol* function.

**get_element_arg**  gets the box’s current argument. We set it in previously described eel exposed Python function:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qq52ba9gvw9g8jktu2w3.png)

##GET_PRICES box

This box calls the *GetPrices* method. There is nothing new here, as we already described all Algonia SDK functions contained in this box:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7yfpelgabqi6h9dxcwdh.png)


Lets recap what we did:
- *getPricesRequest* Python function is called from Javascript
- This function set the symbol name of the *SET_SYMBOL* box argument and executed it
- *SET_SYMBOL* called our C# StockPrices *ChangeSymbol* method.
- After the *SET_SYMBOL*, *GET_PRICES* is executed, because those two boxes are directly connected with virtual wire. *GET_PRICES* call our *GetPrices* method defined in *StockPrices* object

###SAVE_GRAPH box
Let's move to the last Visual Box. If you open the code, you can see following Python script:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y1zwefruwhf90zq97hxw.png)

Visual Box reads result from *GET_PRICES* (prices in CSV format):
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jhnlqpy7aaptt1kwilw0.png)

Interesting part is call to function *plotGraph*, which is defined in our *plot_graph* Python project and returns an image in base64 string format:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9u550gtiek7witebrupv.png)


The image is then shown on UI.

##User Interface
All six Visual Boxes from Algonia Designer canvas were described, but there is one topic left - User Interface.

It is located in **libs\algonia\html\assets\main.html** file.
The Javascript part that communicates with Visual Boxes is:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mq9kbf4ymz04hnjdx0ur.png)

###STEP 1
*Draw Graph* button on the user interface executes *eel.getPricesRequest* function and passes currently selected symbol from html input (ETH, MSFT...)
This call is handled by Eel in Python *GUI box* that executes *SET_SYMBOL* visual box:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6axym9u9eufnjil4wk2t.png)


###STEP 2
Python code executes callback to javascript *onGetPricesCompleted* function:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mccpqghlj99a09ieqvna.png)

This function then calls *saveGraphRequest* in *GUI box*:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qswr6fhr653ftbxhg66c.png)

Here we execute the *SAVE_GRAPH box* to get a base64 encoded image string and call the javascript function *onSaveGraphCompleted*.

###STEP 3
*onSaveGraphCompleted* function shows a graph image in *divGraph* div
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v4ejl8v2butnik1vtidt.png)

#Conclusion<a name="conclusion"></a>
We just went through all steps of our Stock Prices example. In real life we should handle exceptions and errors more conveniently. We could also simplify workflow by reconnecting Visual Boxes differently. But the purpose of the introduction tutorial was to show different ways of Visual Boxes execution.

Main Algonia purpose is technology interoperability, collaboration foundations between different specialists. In our example we used just C# and Python, but of course there are many more choices.

Algonia also forces loosely coupled and modular architecture from the start. You can easily rearrange visual boxes, adding new and deleting existing ones.
Full blown code including all object oriented goodies can be written right inside Visual Boxes. But the biggest strength of Visual Boxes is that they act like wrappers to your and 3rd party libraries. Libraries are developed independently but their functions are wrapped inside boxes.
Visual Boxes hold their object state - there are not just fire’n’forget function executions.

And last but not least, system integrators and non-developers can maintain systems on their own by adding or removing Visual Boxes purchased from Algonia Marketplace.

We are just entering the next era of programming - Visual Microservices.