POC Custom Street View for Vienna
Tags: webdev, javascript, opendata, node
Published: 2023-04-04 20:32:52

## TLDR
A POC of a Street View client using open data: [https://webkappa.karten.wien](https://webkappa.karten.wien)

## Intro
In 2020 my beautiful hometown - the City of Vienna, Austria ("Stadt Wien") completed mapping the streets with Lidar Scanners and Cameras mounted on cars.

The good thing: **The collected data is completely Open Data** and so you can create your own "Street View" based on it, completely independent from Google. Using the laser scanner data even more useful things can be created, the City of Vienna will use it e.g. for traffic planning in the future.

I tried to create a "Street View" client based on the provided images.

## Data
The images where taken by 6 cameras, all in all ~250 megapixels.

Data can be requested via an online tool - after some hours or days of internal processing, you get a link of a .tar file containing images and meta data:

![directory structure of example tar file](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4d86qij2wtngv0mgf7vy.png)

The file `image_meta.txt` is a TSV of all images taken for the requested set:

![image_meta.txt TSV](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hgd6kdlwkfum4u8mcm03.png)

It contains the following columns (unused columns are omitted):

- `trajectory_id` ID of trajectory (part of folder name)
- `sensor_id` ID of the camera (essential to get the direction of the camera/part in the cubemap - see later)
- `x_m/y_m` Latitude and longitude/position of the car (projection EPSG:31256)
- `rz_rad` Z-Rotation = facing of car in radiants

The photos are stored in way, that a "cubemap" consisting of 6 single photos can be created (negative X,Y,Z and positive X,Y,Z) - the "sensor" = camera used are mapped to the cubemap position as follows:

- sensor id ending in 0: positive Y
- ending in 1: positive Z
- ending in 2: positive X
- ending in 3: negative Z
- ending in 4: negative X
- ending in 5: negative Y

The unfolded cubemap would look like this:

![cubemap positions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/huoes9zkennbq50k1pd8.jpg)

Each photo has a resolution of 7130x7130 and has about 5MB, thus the complete cubemap has up to 60MB. So even about **100 meters** of street consists of about **2 GB of raw photo data** (one cubemap every ~ 3 meters). That's why for this POC the images are **scaled down** to 2048x2048 later.

## Compiling the data
Enough of theory, let's get physical

A node.js was created to do the following:

- Retrieve list of images from `image_meta.txt` TSV
- Scale images down to 2048x2048 using graphicsmagick and rename to a unique filename (`<hash of position>_{p,n}{x,y,z}.jpg`)
- Transform position of cubemap to EPSG:4326
- Create a JSON file for each cubemap to store meta data (e.g. facing)
- Save position, image filenames to postgres/postgis database

## The backend
Now that we have the data in place, there's a need of a small backend to provide the frontend with 2 sets of information

- Get all possible positions to mark them on a map
- Get the nearest position based on a given coordinate

As we have all the information for that in our database, a small express based backend was set up fast.

- One API route just queries `SELECT array[st_y(position), st_x(position)] as ll FROM cubemaps` and returns the positions.
- The second route basically does the some, but orders by distance to the given position using `st_distance`

## The frontend
A full screen map based on **leaflet** was set up. It initialy retrieves information about all possible positions and displays it as blue circles using `L.circle`.

![Base map with positions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0zzgwh6fraiqkbc11y12.png)

A click on the map gets the nearest position from the backend and opens a panorama viewer based on **three.js** and **panolens**.

![Panoroma based on Cubemap](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dokvynvhzxt2oppxdqqe.gif)

A small map in the bottom right corner shows the current facing of the panorama viewer and also all other possible positions within 500 meters.

![Mini map with positions](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oxkcl2gjzusrqywshh33.png)
![Mini map animated](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l80vxikych25czpkczlz.gif)

Navigation is done by clicking the positions there - you can also navigate by clicking inside the cubemap - but that's not done yet and kind of buggy right now.

## Where can I see it?
![Result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vqp1nv58eydw9q3j7elo.png) Here (It's german, sorry): [https://webkappa.karten.wien](https://webkappa.karten.wien)

## Still todo

- Cleanup, cleanup, refactor, refactor
- Think about mobile version
- Find memory leaks (switching between cubemaps is done pretty dumb)
- Import more Cubemaps (Unfortunately the download tool seems to be overloaded currently)
- Make navigation within the panorama possible (like Google does it...)

## Thanks!
- Creator and maintainers of [leaflet] (https://github.com/Leaflet/Leaflet)
- Creator and maintainers of [panolens.js]  (https://pchen66.github.io/Panolens/)
- Anita Graser for exploring the sample data of 2020 [blog article](https://anitagraser.com/2021/09/25/exploring-viennas-street-level-lidar-kappazunder-data-sample/)
- The City of Vienna for making this data open!