Twelve months of UI hacks
Tags: java, vaadin, javascript, design
Published: 2020-07-22 15:33:00

Being a part of the Tools team at Vaadin I often get interesting ideas that I want to try out outside the usual stream of product development. Here are some that I've done between July 2019 and July 2020.


Sketch mode integration into Vaadin Designer
--------------------------------------------
Sketch Mode is a tool for free-form drawing of UIs. In this hack I integrated it into Vaadin Designer. The video shows how a layout can be sketched rather quickly.

{% twitter 1167054753160978432 %}

Designer Components
-------------------
Designer Components is an idea based on Sketch Mode where there are UI components and also "action" components that produce code. Each component has a custom editor for it, which allows great flexibility beyond the usual tree-and-attributes approach. The example shows how a trivial REST application can be created using Designer Components.

{% twitter 1167372745421860864 %}

Unide in Electron
-----------------
Sketch Mode was eventually integrated into Unide, a side project GUI builder application I've made. I also made it run using Electron, as is demonstrated in the following video.

{% twitter 1173830517491425281 %}

Flex box with Brute force
-------------------------
While considering the various configurations of flex box layouts, it dawned on me that one could draw some boxes in some configuration and then use plain old brute force to see what flex box settings would produce the closest result. The algorithm is trivial and slow, but fast enough and the results are reasonable.

{% twitter 1184324434406072320 %}


Brute and Sketch Mode
---------------------
By using Brute to perform the layout duties in Sketch Mode it became possible to produce decent flex box layouts just by drawing them. 

{% twitter 1192800063355006976 %}


Exporting Vaadin projects from Figma designs
--------------------------------------------
The free form drawing capabilities of Sketch Mode can be used to implement exporters for other free form things. With the
addition of rectangles for layout and defining tags for each component, one could get working Vaadin applications out of Figma designs.

{% twitter 1197801041036746752 %}

Drawing UIs on napkins
----------------------
Another application of Sketch Mode is synthesis of layouts from hand drawn pictures of UIs. Together with rectangle detection and some additional heuristics, it becomes possible to take a picture of the drawing using a phone and create a preview of the UI on that same phone.

{% twitter 1207770269806465031 %}

Unide in VSCode
----------------
After making Unide run in Electron, it was not too complicated to do the same with VSCode.

{% twitter 1214215719832948737 %}


Standalone Vaadin Designer 
--------------------------
With the data model from Unide it became possible to make a version of Vaadin Designer that runs on the web independently. This resulted in an excellent vehicle for further experimentation.

{% twitter 1219572979882954753 %}


Simple element size editing
---------------------------
A short hack where the image used to show the size, border, padding and marging of an element could be used to edit those values by dragging with the mouse.

{% twitter 1227649535268225025 %}


Graphics programming and visual editing
---------------------------------------
A bit more esoteric stuff: a Lisp-like language that is compiled into JavaScript. Using some ideas from Sketch Mode it is possible to write Lisp using B-expressions (B for Box) in a more visual manner. Also, just like the Designer Components above, each B-expression can contain more complicated editors such as the height map painter shown in the video below.

Lastly, there's a generalization of visual design by connecting pixels in the generated image to code via a trace buffer that records the position of the component instance that produced each pixel. Const expressions can be used to define editable points in the code which is a generalization of the concept of element properties.

{% twitter 1242202561777668099 %}


Exporting Java from Vaadin Designer
-----------------------------------
One of the first things to come out of the previous standalone Vaadin Designer was the ability to generate Java from Designs using code from Unide.

{% twitter 1250873440720113664 %}

And back
--------
An advanced version of the above allows modifying the Java code and importing it back into the design in real time.

{% twitter 1252087639316008962 %}


In place editing of a live Vaadin application
---------------------------------------------
The coolest hack to date with the standalone Vaadin Designer: a version of Vaadin Designer running in Electron that can be used to modify a live Vaadin application. Any component in view can be switched into edit mode
and modified. Switching back into normal mode shows the modified design in the application as usual. This could eventually allow creating a navigable prototype of an application by running it, starting the designer application
and then adding views to it as the application is navigated.

{% twitter 1257934382695940100 %}

Full support for visual design of LitElement components?
--------------------------------------------------------
While I previously made some statements about the difficulty of writing a visual design application for LitElement vs P3 and other template based systems, a method came to mind that makes writing such editors only slightly more complex.

{% twitter 1263767394801143810 %}


A GUI builder in dev tools
--------------------------
One potential future place where a GUI builder could be found is the Chrome dev tools. I wrote an initial prototype of the concept, and it seems promising.

{% twitter 1266222879378677760 %}

Upcoming things
----------------
In the future I will be working more on Unide. Future features will include CRUD generation from Java beans.

{% twitter 1280819645419790336 %}

And Sketch Mode improvements, like a better preview while sketching

{% twitter 1285513939162411008 %}

And others
----------
These are only some of the things I've hacked together during the last year. Check my twitter at [@mjvesa](https://twitter.com/mjvesa) and [youtube](https://www.youtube.com/channel/UCODe-CBYuMh2Pb-QaJibl2w) for all the things not mentioned here with more to come!