[Galaxy S III Android Development] JustArchi:
ArchiDroid Features

Know your ArchiDroid, learn how to use it

Introduction / Basics

Welcome to ArchiDroid's world mortal. In this tutorial we will show you what ArchiDroid has "inside" and what it really offers. All of things included in this post are ArchiDroid-specific, which means that you won't find any ROM with the same features, as they're written from scratch.

Here you can find some definitions of the words used in sections below. You should know most of them, but in case somebody is lost here you can get back on track.

Terminal, Shell - Typical android shell, which may be obtained in three ways.

1. Through native Android Debug Bridge (ADB) with command "adb shell" from the PC or even "ADB through network" (if supported).

2. Through any Android terminal app, f.e. Android Terminal Emulator bundled with ArchiDroid.

3. Through secure shell daemon (sshd), which needs to be turned on firstly. This is extremely useful in terms of pocket debian, which will be described later.

You can use any of these methods to access android's terminal, however usually Android Terminal Emulator will be the easiest one, as it's android app bundled with ArchiDroid. WARNING! Most of the commands below WILL require root access. You can log in to super user shell by "su" command right after obtaining shell. If you're unsure if you're logged as root or not, "whoami" command should print actual user, "root" or "unknown uid 0" are OK, others are not.

ArchiDroid's Pocket Debian

From wikipedia:

From debian site:

How exactly this covers our beloved SGS3 (and countless number of other android arm-based phones)?

As you may (or even should!) know, Android operates on it's own Linux-based kernel. Android's kernel is literally a fork of Linux kernel, with a few special/unique functions which are required, mostly hardware-specific. Because of that kernel itself is VERY compatible with everything based on Linux.

However there have always existed one typical problem, lack of proper environment. We have a great kernel, great power, linux-based android environment, but this environment lacks of very common and required libraries/binaries. If you ever wondered what is or what does busybox, this is the answer. Busybox is just a small package which offers a few standalone GNU/Linux binaries, which are required to make certain things work. For example, swap priorities. Android knows what swap is, and nothing else. It doesn't know that swap could have a priority, so if you use android's swapon command on 4 devices, it will firstly fill first device, and then proceed to the next. That's why we need busybox in ALL custom kernels, because android environment isn't enough.

However busybox sometimes isn't enough. If we focus only on Android itself, it is. But if you for example want to run stricly linux-based service, I don't know, a web server for example... Is it possible to run a native linux web server on an android? No, it's not. You should firstly compile such service on arm architecture, including all dynamic and static libraries (wrrr :mad:) in it only to finally get mad because of missing libraries or other dependencies. Of course if you're patient you'd finally compile everything and set up, however how long would it take? A few days maybe? If you're skilled in what you're doing...

This is why I included built-in "pocket" debian in ArchiDroid. It's FULLY compatible with everything compiled/based on armhf/armel GNU/Linux architecture, for example Raspberry Pi. With two easy commands you can literally jump into debian environment and use every typical GNU/Linux utilities known from debian itself. Of course this means nothing for most of the users, because they'll never have any reason to use such debian environment but from the developer side, it's big step forward. The best example is with github. As you know ArchiDroid has it's own repo on github, from where you can download/manage stuff. There also exists git app for linux and windows. If you want to follow "expert" way of flashing experimental ArchiDroid version, such program is required. The scenario is the same as compiling web server for an android, it requires much more effort than it's worth. And even then you can end up with syncing external dependencies and searching for solutions for the problems you've never seen before... And with ArchiDroid's pocket debian? It's as simple as in any debian/ubuntu distro. "apt-get update && apt-get install git" and voila. Your git is installed and ready for work. Going further I've even included git in pocket debian itself. Okay, I have debian, I have git, and what next? With git utility I can for example provide you with delta updates for ArchiDroid! ArchiDroid can easily use pocket debian to set up and sync ArchiDroid's repository and then pack and flash latest version without even needing of a PC, using 7-zip or anything else. Another example? A web server. I know that it's very dev-specific but if you for any reason need a web server running, just for example to test simple website, you can have it with just one command. Going further, VNC? MySQL server? PHP? Python? Perl? Ruby? Maybe conditional tasks with cron? Persistent minimal IRC client? rtorrent with rutorrent GUI over WWW? The list goes on... Anything based on linux will work. You can even host a server for your favourite game, as long as it has armhf/armel binaries (unfortunately most of the games don't).

So that's it. In short, debian is an operating system built-in in ArchiDroid to provide you with (unfortunately missing) GNU/Linux environment, with full power, ready to handle anything you could request. I made my best to include fully working debian in ArchiDroid for a minimal cost. Whole OS is packed in one big tar file, compressed using highest bzip2. As for now pocket debian has ONLY 40 megabytes of size, maybe in future it will have up to 50 megabytes, but no more. It's a VERY small cost for having such great power, especially if you know how to use it.

This is a really cutting-edge feature, mostly because I have no limitiations what I can include in my ROM right now, and while other developers are dealing with OpenDelta updates and many Android-based problems, I'm just launching my pocket debian and manages linux stuff.

I'm SURE that most of the advanced ArchiDroid user will just LOVE this feature, as much as I love it. I'm looking forward to your responses how YOU use pocket debian with your ArchiDroid. It's also a great time to learn what does the debian offer and how you can simplify your common tasks with just one example debian utility ;).

Technical informations:

1. Pocket Debian does not cause any additional overhead. We don't need to use emulation, neither virtualization to boot our monster. I used chroot technology to "jump" into debian environment with already running kernel and Android. That means additional required CPU/RAM is based on what you run in pocket debian. Booting itself doesn't require anything, just about one megabyte of ram for /bin/bash shell :).

2. Android has some restrictions, mostly sockets. It doesn't allow to create inet sockets by default, even for root users. You will need to add your custom debian users to special group called "inet" (GID 3003) to allow creating of inet sockets, and you may also need to add a group to net_raw group (GID 3004) to allow creating of raw sockets. Please keep in mind that it's only required if you're running an app which required it's own socket, for example mysql server. So apt-get install mysql-server will fail right after booting, you will need to use "addgroup mysql inet" and then apt-get -f install to complete installation. Of course "mysql" is the new user under which mysql-server really operates. I've added root to both of these groups by default.

3. The only "real" restriction is the kernel. Our debian uses Android kernel and it's filesystem. It should work with most common tasks but in some cases our kernel may lack specific modules or built-in code, for example tun/tap required for OpenVPN. Still it's enough to run pretty much everything and if you get in touch with your favourite kernel developer you can also kindly ask for specific missing things.

4. Debian is built and included thanks to debootstrap utility, ArchiDroid command used for creating debian environment is debootstrap --verbose --arch armhf --include=git,ca-certificates,ssh,htop,tightvncserver,xterm,xfonts-base --exclude=manpages,man-db,rsyslog,vim-common,vim-tiny testing debian http://ftp.fr.debian.org/debian

HowTo:

Pocket Debian contains two main terminal commands, "adlinux" and "debian". Both of them are described below. By adlinux and debian you boot and jump into debian's chroot, which means you can use any debian-specific commands.

Examples:

passwd - changes password of actual user. This is needed to login as specific user, for example through ssh.

service ssh start - starts local SSH (secure shell) daemon on native port :22, to which you can easily access via any client supporting ssh, f.e. PuTTY. So basicly after you start shell you can literally connect to your local area network (LAN) IP on port 22 f.e. through PuTTY from your PC.

ifconfig - prints network-related informations about online interfaces, including your local IP, which may be useful for connecting to SSH.

htop - Enhanced top utility. Gives you very good terminal-based view on actual running processes, used ram, load, and more.

apt-get update - Syncs with debian's apt repository. This is mandatory to use many of apt commands because ArchiDroid's debian comes without local repo available, however fully configured to download and access it with just one command

apt-get install XXX - installs packet XXX from debian's repository.

apt-cache search XXX - searches for all packets including keyword "XXX". Ultra useful in terms of searching for specific packet.

Please note that pocket debian is VERY similar to normal native Debian/Ubuntu distribution, therefore above commands are not ArchiDroid's magic, they're very widely used in Debian/Ubuntu distros. If you want to learn more, most of the Debian/Ubuntu tutorials will be very helpful.

ArchiDroid's Pocket Debian Booter (adlinux)

You can call "adlinux" command from your favourite terminal.

adlinux is designed to boot and prepare ArchiDroid's Pocket Debian environment. It requires mode to be specified, and also respects any extra arguments passed.

If you call standalone "adlinux" command then it will print usage and then ask you what you want to do with giving proper informations about every choice. Additionally if you know what you want to do, you can also pass arguments directly to adlinux command, f.e. by executing "adlinux default", which will execute adlinux with "default" mode.

Available modes:

default - Will mount /data /system /storage/sdcard0 /storage/sdcard1 and core filesystems in chroot. Default suggested mode

safe	- Will mount only core filesystems in chroot. Useful if you don't want to share your storage in chroot

bare	- Won't mount even core filesystems such as /proc /dev or /sys. Requires "debian force" to enter chroot. This is the "real" safe mode. You won't be able to interact with an android in any way, while debian itself will work in very limited environment, making core functions unavailable. Suggested only for testing purposes

rebuild - Will automatically reboot your device and remove debian folder in the safe way. WILL CAUSE ALL DEBIAN DATA TO BE LOST!

unmount - Will automatically reboot your device to safely unmount debian environment

Extra options:

extsd - Use external sd card (/storage/sdcard1 /storage/extSdCard), if possible

intsd - Use internal sd card (/data/media/0)

Additional information about modes:

Debian shares core kernel filesystems in "safe" and "default" modes, while it also shares your internal and external sd card in "default" mode. This is nothing to be scared of, as you have full control of what you run in debian, however please note that you CAN'T do whatever you want. All mounted partitions in debian are "binded". "Bind" means that it's mirrored to the mount point and all changes on mounted partition WILL affect the mount point, which is logical. This is nothing to be scared of, as long as you know that debian only extends your environment, it does not fully works in it's own and you CAN cause serious problems from inside of chroot. The only really safe mode is "bare" mode, however in "bare" mode debian can't really do anything, as kernel filesystems are absolutely required for most of the functions. Okay so, you need to know one thing. If you have booted debian you SHOULD NOT touch debian's folder, which is ArchiDroid/debian (on your internal or external sd card, depends what you choosed).. As you know debian for example binds /data to it's folder /data, which is physically ArchiDroid/debian/data. If you for example delete ArchiDroid/debian through root explorer WITH mounted debian then it will ALSO delete debian/data folder, which is binded to /data, and therefore will delete your whole internal sd card, that's why it's extremely important to take care because booted debian becomes part of the android and deleting it can cause at least soft bricks, with a possibility of hard as well. If you want to delete debian folder PLEASE use "rebuild" mode, only through this way you're absolutely sure that nothing bad happens and you won't delete your whole system partition by accident.

Note about extsd option:

Debian requires symlink functionality, typically native windows filesystems DON'T support symlinks, therefore you need to have your external sd card formatted in one of the native linux filesystems, f.e. ext4. adlinux will automatically tell you if debian can be unpacked and used on your external sd card, however it won't be possible under most common filesystems, such as exFAT or FAT32.

Technical informations:

1. Pocket debian archive is located in ArchiDroid/System/debian.tar.gz file. This is "bare" system used for creating environment for the first time, you should not touch it.

2. adlinux detects if debian is already extracted when booting, if not, it's firstly extracted from the file described above.

3. After extracting (if required), core filesystems are mounted with "bind" option based on the mode you've selected in "mode" question above. Typically it mounts /data /system /storage/sdcard0 /storage/sdcard1 /storage/extSdCard /dev /proc /sys.

4. Unmounting is not fully supported right now (linux barrier), therefore both "unmount" and "rebuild" options require a restart to execute properly.

ArchiDroid's Pocket Debian Shell/Chroot (debian)

You can call "debian" command from your favourite terminal.

debian command is designed to allow you "jumping" into debian chroot created by adlinux. Please read how adlinux command works firstly if you haven't done that already. debian command checks if core filesystems are available (if debian is booted), and if they are then it firstly modifies required environment variables to make debian happy (such as TERM, HOME, PATH), then it changes root (chroots) into debian folder, therefore allowing you to execute everything from inside of chroot. It's very generic command, therefore standalone "debian" command won't give you a choice the way adlinux did.

Available options (parameters):

force - required for jumping into bare debian, created with "adlinux bare" command above. This skips debian checks for mounted core filesystems, normally you should avoid it at all cost, unless you know what you're doing. If core filesystems are missing then it's very likely that your debian will be disabled in more than 90%.

extsd - Use external sd card (/storage/sdcard1 /storage/extSdCard), if possible

intsd - Use internal sd card (/data/media/0)

cmd - Executes command in debian chroot

WARNING! cmd parameter will cause all further parameters to be threated as a command passed to debian, therefore you need to make sure that this is the last debian parameter which you want. For example "debian force cmd service ssh start" will skip filesystems checks and execute "service ssh start" in debian's chroot, however "debian cmd force service ssh start" will pass "force service ssh start" to debian, therefore respecting filesystems checks and passing invalid command.

This function is extremely useful for making init.d and other startup scripts. For example you can easily call "adlinux default" and then "debian cmd service ssh start" to call secure shell daemon on every boot with two easy steps.

Technical informations:

1. debian command uses chroot technology to change root of current shell to debian shell.

2. After chrooting to debian directory, /bin/bash shell is automatically called as default debian shell.

ArchiDroid's Flasher (adflash)

You can call "adflash" command from your favourite terminal.

adflash is a great small utility, which allows you to easily update your ArchiDroid to latest stable or experimental version with one easy command and delta upgrade. It utilizes ArchiDroid functions, therefore you must be running ArchiDroid to use it.

If you call standalone "adflash" command then it will print usage and then ask you what you want to do with giving proper informations about every choice. Additionally if you know what you want to do, you can also pass arguments directly to adflash command, f.e. by executing "adflash 2e git", which will execute adflash with 2.X-EXPERIMENTAL version using git mode.

Available versions:

2e - 2.X-EXPERIMENTAL

2s - 2.X-STABLE

1e - 1.X-EXPERIMENTAL

1s - 1.X-STABLE

Extra options:

git - Sets up local git repository, which gives you delta upgrades and bandwidth saving

direct - Downloads targeted branch as .zip file directly from github

clean	- Cleans everything up, including local repo and tmp folder from ArchiDroid directory specified below

extsd - Use external sd card (/storage/sdcard1 /storage/extSdCard)

intsd - Use internal sd card (/data/media/0)

nozip	- Shows changelog and changes only

Okay so, the most interesting option is the mode...

Direct mode is simple, fast and effective. It downloads target version (stable or experimental) from GitHub server, then it repacks downloaded zip file and makes it available for flash. You should use this mode for one-time downloads, such as once per stable version or two. The only advantage of this method is the ability to download from github (and with one command).

Git mode is complex. It uses ArchiDroid's Pocket Debian (read above) for cloning and updating local ArchiDroid repo. This gives several number of advantages, mostly for using experimental versions. Firstly, by having local ArchiDroid repo you have to download ONLY changes between your snapshot and server's snapshot, which means delta upgrades. Secondly, you have access to all commits from target branch, so you know exactly what has changed since your latest download. Again, this is extremely useful for experimental branch, as changelog may not be up-to-date. Keep in mind that git mode will require additional space on your device for keeping ArchiDroid repository, therefore you sacrifice some space for delta upgrades. This mode is extremely useful for flashing ArchiDroid often, for example daily experimental versions, because in fact you download only new commits instead of whole repo/archive.

ArchiDroid's RunOnce (Backend)

ArchiDroid's Init (Backend)

ArchiDroid's Backend Control

ArchiDroid Backend Control is a set of settings, which controls behaviour of ArchiDroid's Init. It's located in /system/archidroid/dev and contains a number of files, which are recognized by ArchiDroid's Init. You shouldn't directly touch /system/archidroid/dev, instead you can control behaviour of ArchiDroid's Backend through /system/archidroid/scripts. They can be easily executed through any script manager, f.e. Root Browser or Android Terminal Emulator. Some of the settings are also located in /system/archidroid/etc folder, mostly configurations for binaries utilized by ArchiDroid's Init.

ArchiDroid's HArdware Volatile Entropy Gathering and Expansion Daemon (Haveged)

The haveged project is an attempt to provide an easy-to-use, unpredictable random number generator based upon an adaptation of the HAVEGE algorithm. Haveged was created to remedy low-entropy conditions in the Linux random device that can occur under some workloads, especially on headless servers. Current development of haveged is directed towards improving overall reliablity and adaptability while minimizing the barriers to using haveged for other tasks.

The original HAVEGE research dates back to 2003 and much of the original haveged documentation is now quite dated. Recent work on haveged has included an effort to provide more recent information on the project and its applications.

The original research behind HAVEGE use was based upon studies of the behavior of processor caches from a hardware level. The 'Flutter' documents attempt to provide a modern view of HAVEGE at software level through the use of a diagnostic build of haveged that captures the non deterministic inputs to haveged for analysis by external tools.

ArchiDroid has built-in haveged entropy generator. It's controlable through ArchiDroid's Backend Control - ArchiDroid_Haveged_EnableDisable.sh. It's turned on in default configuration, through HAVEGED_ENABLED

ArchiDroid's Fast Random Number Generator (Frandom)

Frandom is a Linux kernel random number generator, which is 10-50 times faster than what you get from Linux' built-in /dev/urandom. And it uses very little (/dev/frandom) or none (/dev/erandom) of the kernel's entropy pool, so it is very useful for applications that require a handy source for lots of random data.

ArchiDroid has built-in frandom activator. It's controlable through ArchiDroid's Backend Control - ArchiDroid_Frandom_EnableDisable.sh. It's turned on in default configuration, through FRANDOM_ENABLED.

Notice: Kernel must support frandom module to actually make use of that. Init will try to search for frandom.ko module and load it, then use /dev/erandom for both /dev/random and /dev/urandom. If your kernel supports frandom, it will work. If it doesn't, obviously this will be skipped even if you have FRANDOM_ENABLED. Check ArchiDroid Init log located in /data/media/0/ArchiDroid/Init.log to check if frandom works properly for you.

ArchiDroid's Adblock (dnsmasq/dnrd, dnsproxy2, pixelserv)

dnsproxy2 is a replacement DNS proxy for Android 4.3+

This currently allows the user to manually override the DNS server IP,

and it sets the correct UID on outbound requests so they can be filtered

via iptables / AFWall+ / DroidWall / etc.

Dnsmasq is a lightweight server designed to provide DNS, DHCP and TFTP services to a small-scale network. It can serve the names of local machines which are not in the global DNS. The DHCP server integrates with the DNS server and allows machines with DHCP-allocated addresses to appear in the DNS with names configured either in each host or in a central configuration file. Dnsmasq supports static and dynamic DHCP leases and BOOTP for network booting of diskless machines.

Dnrd, Domain Name Relay Daemon is a caching, forwarding DNS proxy server. Most useful on vpn or dialup firewalls but it is also a nice DNS cache for minor networks and workstations.

Pixelserv is a super minimal webserver, it's one and only purpose is serving a 1x1 pixel transparent gif file. Using some creative firewalling (netfilter/iptables) rules you can redirect some webrequests (for adds for example) to pixelserv.

ArchiDroid has built-in Adblock. It's controlable through ArchiDroid's Backend Control:

ArchiDroid_Adblock_DnsmasqDnrdModeSwitch.sh

ArchiDroid_Adblock_EnableDisable.sh

ArchiDroid_Adblock_EnableDisableLocalDNSes.sh

ArchiDroid_Adblock_EnableDisableLocalDNSesDaemon.sh

ArchiDroid_Adblock_LockUnlockHosts.sh

ArchiDroid_Adblock_MoabAdawayHostsSwitch.sh

ArchiDroid_Adblock_Reload.sh

It's turned on in default configuration, through:

ADBLOCK_ENABLED

ADBLOCK_LOCAL_DNSES_DAEMON_ENABLED

ADBLOCK_LOCAL_DNSES_ENABLED

ADBLOCK_USE_ADAWAY_HOSTS

ADBLOCK_USE_DNSMASQ

In short. This is a very advanced and powerful solution for blocking ads through DNS queries. First of all we're forwarding all DNS traffic to localhost (127.0.0.1). Then we're handling them through local DNS server - dnsmasq (default), or dnrd (option). Our local DNS server reads blocked hostnames through special /system/archidroid/etc/hosts file, then if no record is found, it forwards DNS query to OpenDNS/Google DNS servers, or if it's found, returns 127.0.0.1 as the address. Lastly, pixelserv is providing a 1x1 NULLGIF response on local web server, so instead of big black/white screen instead of the AD, we get 1x1 transparent pixel, which usually perfectly hides ad from the app or the website.

Extra features:

1. You can specify if you want to use dnsmasq (default), or dnrd (option) as a local dns server. Dnsmasq is more flexible, modern, faster and has less memory footprint, however I also left dnrd as an option, because it's proven to work stable.

2. You can specify hosts file, which you want to use. In default configuration we use AdAway's hosts file, with more than 30 thousand of records, which results in extra ~2.5 MB memory usage. You have also an option to use MOAB (Mother Of Ad Blocking) hosts file, with more than 330 thousand of records, which will result in about ~30 MB memory usage. Eventually you can append your own rules or use non-standard hosts file, available in /system/archidroid/etc/hosts. Pro tip: You can point AdAway to use this hosts file (/system/archidroid/etc/hosts_adaway), which will result in automatic updates. /system/archidroid/etc/hosts is a symbolic link, either to hosts_away or hosts_moab, if you want to specify your own hosts, you can delete symbolic link and write your own rules.

3. Original /system/etc/hosts file has been locked from editing. This is to ensure that AdAway or other adblockers won't use obsolete and slow method of blocking ads through hosts. The whole point of implementing Adblock in ArchiDroid is to provide you with super-fast, flexible and effective way of blocking ads, also with getting rid of black/white ad screen. In 99% situations you don't want to touch ArchiDroid's default behaviour, as it blocks ads perfectly. Eventually, if you have a very good reason, you can unlock original hosts file through ArchiDroid's Backend Control and modify them, however keep in mind that every additional rule WILL slow down your network speed.

4. In default configuration local dns server uses two OpenDNS servers at port 5353, two Google DNS servers at port 53 and up to two local DNS servers provided by your Wi-Fi/3G connection, which overall gives a sum of 6 remote dns servers. In some rare scenarios (f.e. some wi-fi hotspots) you can notice that a moron, administrator of this wi-fi, blocked all dns queries and forces you to use his DNSes. This is BAD because of freedom and so on, but it's very common practice, that's why I turned on local DNSes as well. If you want to improve your privacy at least a bit, you can disable local DNS servers and then use only OpenDNS and Google DNS.

5. Above option initialy has been written to allow you one-time access to such non-trusty wi-fi's. But if you for any reason need automatic update of your local DNSes (3G and Wi-Fi's will use different local DNSes), you can also turn on Local DNSes Daemon, which will automatically query and update local DNSes if needed. This is also turned on in addition to local dnses above, of course in default preset.

ArchiDroid's Forced Update (RunOnce)

Forced update selected during mode selection in aroma tells RunOnce to work in "INSTALL" mode even on "UPDATE" mode, apart from that it works exactly the same as update mode, only RunOnce is affected.