Data Streaming between Every Device over HTTP/HTTPS
Tags: showdev, typescript, linux, opensource
Published: 2019-02-09 10:41:17

---
title: Data Streaming between Every Device over HTTP/HTTPS
published: true
description: A server for data stream transfer between every device over HTTP/HTTPS
tags: showdev,typescript,linux,opensource
---

Hi all! I'm very new to the DEV community. Today, I'd like to introduce data stream transfer between almost every device over HTTP/HTTPS made for use with Unix Pipe and also for browser!


## What do I want to solve?

We sometimes want to transfer data between Mac, Windows, Linux, Unix, iPhone, Android... Although we have AirDrop, Google Drive, Dropbox, Slack, WhatsApp, Skype, `netcat`, `ssh` or etc, we need to

* Find a common service used by both sender and receiver
* Install additional GUI software
* Sign up some service
* Solve NAT traversal

In addition, almost existing data transfer services are not CUI-friendly or not GUI-friendly.

So, I made a solution to the problems above. The system allows you to

* Transfer data with almost every device
* Use it without additional installation
* Use it with Unix/Linux Pipe and it's engineer-friendly

## Actual Usages

I'd like to introduce how to use the system I made.

Here is a demo to transfer `seq 100000`.

![piping seq](https://thepracticaldev.s3.amazonaws.com/i/kwm8mdrlu8hr5l38oovl.gif)
Send: `seq 100000 | curl -T - https://ppng.io/myseq`
Get: `curl https://ppng.io/myseq`

---


Here is a demo to transfer a 100MB file.

![piping 100 MB file](https://thepracticaldev.s3.amazonaws.com/i/8qkvg7tkmep5ryeh2ofx.gif)
Send: `cat 100MB.dat | curl -T - https://ppng.io/mydata`
Get: `curl https://ppng.io/mydata > mydata`

---

You can transfer any data by using pipe. For example,

Compress & Send: `cat myfile | gzip | curl -T ...`
Get & Decompress: `curl ... | zcat > myfile`
and
Encrypt & Send: `cat myfile | openssl aes-256-cbc | curl -T ...`
Get & Decript: `curl ... | openssl aes-256-cbc -d > myfile`
and
Send directory(zip): `zip -q -r - ./mydir | curl -T - https://ppng.io/mydir.zip`
Send directory(tar.gz): `tar zfcp - ./mydir | curl -T - https://ppng.io/mydir.tar.gz`

---

Here is a demo to transfer data **to a browser**.

![piping from CUI to broswer](https://thepracticaldev.s3.amazonaws.com/i/8yiu3zxvmm2cxdowg0ig.gif)


---

Here is a demo to transfer data **from a browser**.

![piping from browser to CUI](https://thepracticaldev.s3.amazonaws.com/i/f9hzl0b812zvloos34g8.gif)

This means you can transfer a file from Android/iPhone/iPad... to another device!

---

### Transfer to Multiple Receivers

![piping multiple](https://thepracticaldev.s3.amazonaws.com/i/a337vdmul0ozqyk7vd5r.gif)


### Terminal Screen Sharing!

![piping terminal share](https://thepracticaldev.s3.amazonaws.com/i/9blobmr61goaesjcqcxg.gif)


Share: `script -f >(curl -T - https://ppng.io/myscript >&/dev/null)`
View: `curl https://ppng.io/myscript`
(Use `script -F` in Mac instead of `script -f`)


## GitHub Repository

The project is Piping Server found in the link below.
GitHub: <https://github.com/nwtgck/piping-server>

{% github nwtgck/piping-server %}

This project is written in TypeScript.

## Public Servers

Here are public servers. My recommendation is <https://ppng.io> because it has a short name.

* https://ppng.io/
* https://ppng.herokuapp.com
* https://piping-92sr2pvuwg14.runkit.sh/

In my policy, you can freely run the servers. I'd like to describe how to run later.


You can get engineer-friendly help from the following.

```bash
$ curl https://ppng.io/help
<Help will be displayed here>
```

## How it works

I think Piping Server is like a TURN server in Pear-to-Pear. Piping Server relays your data to another device to solve NAT traversal.

A body of POST/PUT HTTP request is passed to a body of GET HTTP response.


## How to Run Piping Server

We have several ways to run Piping Server. I'd like to introduce one by one.

### Heroku

Push [Deploy to Heroku] button in <https://github.com/nwtgck/piping-server>.

### Portable Binary Executable

The following commands make a Piping Server run on <http://localhost:8888>.

```bash
wget https://github.com/nwtgck/piping-server-pkg/releases/download/v0.8.7/piping-server-macos
chmod +x piping-server-macos
./piping-server-macos --http-port=8888
```

You can also find other platform releases in GitHub Releases in <https://github.com/nwtgck/piping-server-pkg>. The portable executables are produced by [zeit/pkg](https://github.com/zeit/pkg).


### Docker

A `docker run` make a Piping Server run on <http://localhost:8888>.

```bash
docker run -d --restart=always -p 8888:80 nwtgck/piping-server --http-port=80
```


### npm

The following commands make Piping Server run on http://localhost:8888.

```
# install
npm install -g piping-server
# run
piping-server --http-port=8888
```


## Possibility of Data Transfer over HTTP

I'm conducting an experiment to reveal how many data can be transferred over HTTP.
I started the experiment 55 days and 7 hours ago and over 1000 terabyte data were transferred.

The method of the experiment is as follows.
Infinitely send: `cat /dev/urandom | curl -T - localhost:8888/rand`
Infinitely discard: `curl localhost:8888/rand > /dev/null`


## Transfer Speed Compared with Go

Here is a simple comparison in transfer speed between TypeScript/Node.js and Go implementations. As a result of the videos, there seem to be almost no differences.

**TypeScript/Node.js**:
{% youtube ufijpqHlJww %}

**Go:**
{% youtube NuKvDwB9JSU %}

Repo Go ver.: https://github.com/nwtgck/go-piping-server


## Japanese Post

Here is my original Japanese post: <https://qiita.com/nwtgck/items/78309fc529da7776cba0>.
You can get more information there.


Thank you very much for reading. Have a wonderful day!