[Android Software Development] Koloses:
Introduction

Hello XDA. My name’s Chris and in this guide I’ll try to show you how to set up your own independent “Data Saver” proxy similar to the one we know from Google Chrome browser. I have to warn you that the steps needed require moderate knowledge and might not be easy to follow for the beginners. It also requires setting up some software on your server (Ubuntu 16.04 in my case) prior to following this guide. You can find tutorials on how to do so in Prerequisites section below.

Prerequisites

-	Server running Linux distro (Ubuntu 16.04 x64 in my case as I said above)

-	Wireguard or other VPN (OpenVPN) setup on the server that is useable on your device (I’ll go with wireguard, so here’s a great tutorial to set that up: https://www.digitalocean.com/commun...t-to-point-vpn-with-wireguard-on-ubuntu-16-04 . It's pretty straightforward.)

-	Machine to build (preferably gcloud or some other buildserver)

-	Time

Environment

We’ll start of with installing libraries & additional tools needed to build Android itself just to make sure everything’s ready to move on later:

sudo apt-get install git ccache automake lzop bison gperf build-essential zip curl zlib1g-dev zlib1g-dev:i386 g++-multilib python-networkx libxml2-utils bzip2 libbz2-dev libbz2-1.0 libghc-bzlib-dev squashfs-tools pngcrush schedtool dpkg-dev liblz4-tool make opting libjpeg8 libjpeg-dev openssl ssl-cert maven mercurial libnss3-tools

Okay, so that’s about it for automated install of packages. We’ll also need newest version of golang and Ubuntu repos do not provide that by default so we will install It and set work directory manually:

mkdir -p ~/golang
cd ~/golang
wget https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz
tar -xvf go1.10.1.linux-amd64.tar.gz
sudo mv go /usr/local
mkdir -p ~/go

Now that it is installed let’s add it to our PATH environment by editing .profile:

sudo nano ~/.profile

Add these lines at the end of the file:

export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go
export GOPATH=$HOME/go

Save the file and run

source ~/.profile

To refresh your env setup.

That’s it for now when it comes to your environment setup. We may be installing other packages needed in this guide later.

Installing the Proxy server

Let’s proceed to setting our proxy server that will do all the compression work we want it to. We’ll be using Compy proxy here since it supports ssl which is a must for almost all modern websites nowadays. Let’s download the source we need:

go get github.com/barnacs/compy

Now we are ready to compile our proxy server binary:

cd go/src/github.com/barnacs/compy/
go install

Our binary should be located in ~/go/bin so lets try to run it and see if it’s okay

cd ~/go/bin
./compy -help

This should print all of the arguments we can pass to our proxy. I prefer to have easy access to running it on boot without setting any paths so I’ll move my binary to /usr/bin of my server.

sudo mv compy /usr/bin
sudo chmod a+x /usr/bin/compy

Generating Self-signed Certificate & Running the Server

Since w plan on using our proxy with SSL traffic we have to provide our own SSL certificate. I just went ahead an generated self-signed to not bother myself with getting it the other way:

mkdir -p ~/keys
cd ~/keys
openssl req -x509 -newkey rsa:2048 -nodes -keyout cert.key -out cert.crt -days 36500

openssl is going to ask you a few details. Fill those in however you want as it doesn’t matter much since you will be the only one using that certificate anyway. After doing so you should have two files in your keys folder:

cert.key
cert.crt

Now you may test run compy with those keys to see if everything’s okay. I’ll run it on port 8888:

compy -host :8888 -ca ~/keys/cert.crt -cakey ~/keys/cert.key -gif false -brotli 8 -gzip 7 -jpeg 75 -png -minify

As you can see we have our keys specified twice. That’s due to workaround applied earlier, don’t worry about it. If everything’s okay you should now see something like this:

2018/04/05 09:46:49 compy listening on :8888

Since our proxy server seems to run correctly, we may also add some iptables to make it accessible only from our VPN peer address:

sudo apt-get install iptables-persistent
sudo iptables -A INPUT -p udp -s 10.0.0.0/24 --dport 8888 -j ACCEPT
sudo iptables -A INPUT -p tcp -s 10.0.0.0/24 --dport 8888 -j ACCEPT
sudo iptables -A INPUT -p udp -s 0.0.0.0/0 --dport 8888 -j DROP
sudo iptables -A INPUT -p tcp -s 0.0.0.0/0 --dport 8888 -j DROP
sudo dpkg-reconfigure iptables-persistent

From now on your proxy should be accessible only through your Wireguard VPN connection. At this point it’s safe to run our proxy in the background at system bootup. I do that via crontab:

sudo crontab -e

Add this line in the end of a file (change the settings if you want to):

@reboot compy -host :8888 -ca ~/keys/cert.crt -cakey ~/keys/cert.key -gif false -brotli 8 -gzip 7 -jpeg 75 -png -minify

Save the file and you're done with server config.

Server is ready. What now?

Now we need a way to make use of our server and that’s the moment where the fun begins. We could of course add our SSL certificate to Android system certs and try to use our proxy globally BUT that will result in most of apps refusing to connect to their services due to SSL Pinning as our MiTM proxy signs the traffic with our self-signed certificate so non-browser traffic would be essentially dead. Since I haven’t found reliable way to set proxy for individual apps nor disable SSL Pinning on Android 8.1 I had to try my luck finding browser that support proxy setup and Firefox does just that via about:config.

Here comes another problem. Firefox doesn’t use system certificate store so trying to setup our proxy there will result in “Unsafe connection” errors that won’t allow us to visit any website due to the fact the browser doesn’t trust our certificate. I also haven’t found any reliable way to add the cert into newest versions of the browser so the easiest way around that was compiling it from source with the certificate bundled in. Well, let’s go then.

Sync Fennec source

Development variant of Firefox is named Fennec. I recommend doing these steps on some build server as it will be faster than on regular machine. We will use Mercurial to download newest source. It got installed when we were setting up our Env so you can just proceed to clone the repo:

mkdir -p ~/fennec
cd ~/fennec
hg clone https://hg.mozilla.org/mozilla-central

Wait for the sync to complete. It can take a while so in the meantime we can go forward with other things.

Prepare certificate for Fennec

So we have our cert files we generated earlier used by the proxy. Now we need to convert our cert to format used by mozilla in their browser sources. First we need to convert our certificate into .der format

cd ~/keys
openssl x509 -in cert.crt -out cert.der -outform DER

Then we will convert it into certdata.txt format for us to use.

nss-addbuiltin -n ProxyCert -t "C,C,C" -i cert.der >> certdata.txt

We should get quite lengthy file certdata.txt that contains our certificate in the format needed by Fennec sources. We will add it there in a few moments.

Adding the certificate into browser source

Okay so the sync finished and our certdata is ready. Now we need to append it to file provided in Fennec sources:

cat ~/keys/certdata.txt >> ~/fennec/mozilla-central/security/nss/lib/ckfw/builtins/certdata.txt

Now our browser is ready to be built.

Building Fennec

Let’s move on to our final part: Building the browser. First let the building environment setup by itself.

cd ~/fennec/mozilla-cental
./mach bootstrap

You will be prompted with a question which version of Firefox you want to build. Naturally, I chose 4. Firefox for Android. Some additional package installs and dependencies downloads will occur at this point. This might take some time. You may need to agree to Android SDK licence and make some other chocies like Mercurial install. Choose Do not install Mercurial as we have it already. Then in another Mercurial dialogue choose No.

After the sync is complete we will get the suggested content of mozconfig file that is needed for the build:

# Build Firefox for Android:
ac_add_options --enable-application=mobile/android
ac_add_options --target=arm-linux-androideabi
# With the following Android SDK and NDK:
ac_add_options --with-android-sdk="/home/chris/.mozbuild/android-sdk-linux"
ac_add_options --with-android-ndk="/home/chris/.mozbuild/android-ndk-r15c"

Create the file with nano and paste your config there:

nano mozconfig

Now before we start the build we will need to update gcc toolchain since Ubuntu 16.04 comes with gcc 5.4 and Fennec needs at least 6.1 to build:

sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get update
sudo apt-get install gcc-6 g++-6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6

Verify our install with gcc -v

gcc version 6.3.0 20170519 (Ubuntu/Linaro 6.3.0-18ubuntu2~16.04)

We also need to download rust toolchain for arm target and then we may start the build:

source ~/.profile
rustup target add armv7-linux-oideabi
./mach build

This will take a lot of time. On Google Cloud build took me around 50 minutes so you may take your rest now and go out for a walk/grab some coffee etc. Anyway, after it’s done run:

./mach package

To generate your new browser fennec-61.0a1.en-US.android-arm.apk file. It should be somewhere in

~/fennec/mozilla-central/ objdir-droid/dist

Copy it over to your device and install. Time for some tests.

Setting up the browser & Testing

After installing your fresh Fennec build proceed to open about:config in the address bar and setup proxy as follows (replace ip and port with the ones you used):

network.proxy.http
10.0.0.1
network.proxy.http_port
8888
network.proxy.ssl
10.0.0.1
network.proxy.ssl_port
8888
network.proxy.type
1
network.proxy.no_proxies_on
10.0.0.1

Try to load any https website while running compy proxy on your server. It should work now:

Screenshot of browser and console output

Enjoy your self-made data compressing browser. Remember to support the developers of open source software used in this guide. Perhaps later I'll attach some more screenshot of the final product.

Links & References:

Compy proxy

Wireguard

Mozilla Cert Storage Info