[XDA-University] Red Devil:
Why this thread?

​

Hello xda-university !

As a newbie ( still remain a big newbie ) , I always had a tough time managing my ROMs and kernel projects.

I blindly followed what guides said .. and then realized how much time and energy I was wasting on downloading unnecessary $#!t from the internet as I have a very limited bandwidth and a particularly slow speed. ( around 80 kbps download speed :o )

While I am not a pro at this stuff, I would like to share a few tips and tricks about ROM building which will save some of your time , energy, space as well as a lot of the money while returning back to our great community out here.

After all efficiency is what matters in the end right ? ;)

Also , understanding how the process works is more important that the process itself .

For what its worth , even a build bot can compile a ROM.. thats no biggie.

Learning how it works matters more ! :)

Tired of waiting for that repo sync to end ?!!!

READ ON

! ​

Prerequisites / Requirements​

Some rough knowledge on compiling a ROM or experience of compiling a ROM.

Check out this thread ! : COMPILE From Source For Any Phone On Linux Distros [BUILD ENVIRONMENT TO COMPILE] by @mithun46​

​

That was just to scare you ;). Though this will be more useful if you do have that knowledge . :)

Ok now moving on to the guide :

Google's Repo Tool

​

I ll just start off with a basic introduction of

Repository :

Repository is basically like a store room for code. It can be either selfmade i.e. you have yourself contributed it or claim to have written that code .. ( circled in yellow )

Or it can also be forked from someone else ... that is .. it is a version of someone else's code that you have modified a bit and pushed it to the remote server/ website. ( circled in black )

Look at the icons on the left of the name of the repository to know .​

There are a variety of websites that offer such free hosting for code for eg : Github.com(which I will be using in this tutorial) and BitBucket.org among others..

Repo tool :

You maybe very familiar with the words repo init , repo sync , etc while downloading source.

Where did that repo command come from ?

Remember this ? :

curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo

Yep thats where it came from .

So in order to use the repo tool you need to run this command at least once.

What this does is it download the repo tool from Google's source and makes it executable for your own use.​

How does the repo tool work ?

The repo tool is used for downloading a bunch of remote repositories that are on the internet thru one single command :

repo sync -j4

Details , details and more details

To explain this , I will take an example of a ROM, say AOKP.

For building AOKP you need to download its source code.

AOKP ROM developers have maintained the ROM's source code in a large number of Repositories that are located at www.github.com/AOKP.

What they have done is they have organized their source code into a number of folders for and uploaded it to separate repositories online.

What this does is , it enables all the ROM developers to work on a specific part of the source on their own computers and then push the source online to remote repository online so that he can share the code with all the developers of the same ROM. ( just a basic idea. it involves a lot more including pull requests and code reviews but we will stick with this for now )

Social coding at its best. :highfive:

So during the command :

repo sync -j4

What is done is the repo tool fetches all the remote repositories that the AOKP ROM devs have deemed necessary for building the ROM for a device through 4 threads.

If you have a slow internet connection then it is advised to use -j1 or -j2.

If you have a fast internet connection then you can use a -j16 or a -j32 whatever suits you the best.

Where does the repo tool take all the reference when downloading ??

The repo tool takes all the reference for the code to be downloaded from a

Default.xml

or a

Manifest.xml

.

The manifest.xml is a file which is written in xml markup and directs the repo tool to the repositories and their respective branches that need to be downloaded .

This is basically a huge list of all the repositories that need to be downloaded by the repo tool.​

Where can this list of yours be found ?

​

Once you have initialized your repo sync which will be somewhat similar to this :

akshat@vaio:~$ mkdir -p kang
akshat@vaio:~$ cd kang
akshat@vaio:~/kang$ repo init -u https://github.com/AOKP/platform_manifest.git -b jb-mr1
Get https://gerrit.googlesource.com/git-repo
remote: Counting objects: 101, done
remote: Finding sources: 100% (101/101)
Receiving objects: 100% (2526/2526), 1.69 MiB | 67 KiB/s, done.
remote: Total 2526 (delta 1442), reused 2526 (delta 1442)
Resolving deltas: 100% (1442/1442), done.
From https://gerrit.googlesource.com/git-repo
* [new branch] maint -> origin/maint
* [new branch] master -> origin/master
* [new branch] stable -> origin/stable
* [new tag] v1.0 -> v1.0
* [new tag] v1.0.1 -> v1.0.1
* [new tag] v1.0.2 -> v1.0.2
* [new tag] v1.0.3 -> v1.0.3
* [new tag] v1.0.4 -> v1.0.4
* [new tag] v1.0.5 -> v1.0.5
* [new tag] v1.0.6 -> v1.0.6
* [new tag] v1.0.7 -> v1.0.7
* [new tag] v1.0.8 -> v1.0.8
* [new tag] v1.0.9 -> v1.0.9
* [new tag] v1.1 -> v1.1
* [new tag] v1.10.0 -> v1.10.0
* [new tag] v1.10.1 -> v1.10.1
* [new tag] v1.10.2 -> v1.10.2
* [new tag] v1.10.3 -> v1.10.3
* [new tag] v1.10.4 -> v1.10.4
* [new tag] v1.11.0 -> v1.11.0
* [new tag] v1.11.1 -> v1.11.1
* [new tag] v1.12.0 -> v1.12.0
* [new tag] v1.12.1 -> v1.12.1
* [new tag] v1.12.2 -> v1.12.2
* [new tag] v1.2 -> v1.2
* [new tag] v1.3 -> v1.3
* [new tag] v1.3.1 -> v1.3.1
* [new tag] v1.3.2 -> v1.3.2
* [new tag] v1.4 -> v1.4
* [new tag] v1.4.1 -> v1.4.1
* [new tag] v1.4.2 -> v1.4.2
* [new tag] v1.4.3 -> v1.4.3
* [new tag] v1.4.4 -> v1.4.4
* [new tag] v1.5 -> v1.5
* [new tag] v1.5.1 -> v1.5.1
* [new tag] v1.6 -> v1.6
* [new tag] v1.6.1 -> v1.6.1
* [new tag] v1.6.10 -> v1.6.10
* [new tag] v1.6.10.1 -> v1.6.10.1
* [new tag] v1.6.10.2 -> v1.6.10.2
* [new tag] v1.6.2 -> v1.6.2
* [new tag] v1.6.3 -> v1.6.3
* [new tag] v1.6.4 -> v1.6.4
* [new tag] v1.6.5 -> v1.6.5
* [new tag] v1.6.6 -> v1.6.6
* [new tag] v1.6.7 -> v1.6.7
* [new tag] v1.6.7.1 -> v1.6.7.1
* [new tag] v1.6.7.2 -> v1.6.7.2
* [new tag] v1.6.7.3 -> v1.6.7.3
* [new tag] v1.6.7.4 -> v1.6.7.4
* [new tag] v1.6.7.5 -> v1.6.7.5
* [new tag] v1.6.8 -> v1.6.8
* [new tag] v1.6.8.1 -> v1.6.8.1
* [new tag] v1.6.8.10 -> v1.6.8.10
* [new tag] v1.6.8.11 -> v1.6.8.11
* [new tag] v1.6.8.2 -> v1.6.8.2
* [new tag] v1.6.8.3 -> v1.6.8.3
* [new tag] v1.6.8.4 -> v1.6.8.4
* [new tag] v1.6.8.5 -> v1.6.8.5
* [new tag] v1.6.8.6 -> v1.6.8.6
* [new tag] v1.6.8.7 -> v1.6.8.7
* [new tag] v1.6.8.8 -> v1.6.8.8
* [new tag] v1.6.8.9 -> v1.6.8.9
* [new tag] v1.6.9 -> v1.6.9
* [new tag] v1.6.9.1 -> v1.6.9.1
* [new tag] v1.6.9.2 -> v1.6.9.2
* [new tag] v1.6.9.3 -> v1.6.9.3
* [new tag] v1.6.9.4 -> v1.6.9.4
* [new tag] v1.6.9.5 -> v1.6.9.5
* [new tag] v1.6.9.6 -> v1.6.9.6
* [new tag] v1.6.9.7 -> v1.6.9.7
* [new tag] v1.6.9.8 -> v1.6.9.8
* [new tag] v1.7 -> v1.7
* [new tag] v1.7.1 -> v1.7.1
* [new tag] v1.7.2 -> v1.7.2
* [new tag] v1.7.3 -> v1.7.3
* [new tag] v1.7.3.1 -> v1.7.3.1
* [new tag] v1.7.4 -> v1.7.4
* [new tag] v1.7.4.1 -> v1.7.4.1
* [new tag] v1.7.4.2 -> v1.7.4.2
* [new tag] v1.7.4.3 -> v1.7.4.3
* [new tag] v1.7.5 -> v1.7.5
* [new tag] v1.7.6 -> v1.7.6
* [new tag] v1.7.6.1 -> v1.7.6.1
* [new tag] v1.7.7 -> v1.7.7
* [new tag] v1.7.7.1 -> v1.7.7.1
* [new tag] v1.7.7.2 -> v1.7.7.2
* [new tag] v1.7.7.3 -> v1.7.7.3
* [new tag] v1.7.7.4 -> v1.7.7.4
* [new tag] v1.7.7.5 -> v1.7.7.5
* [new tag] v1.7.7.6 -> v1.7.7.6
* [new tag] v1.7.8 -> v1.7.8
* [new tag] v1.7.8.1 -> v1.7.8.1
* [new tag] v1.7.8.2 -> v1.7.8.2
* [new tag] v1.8.0 -> v1.8.0
* [new tag] v1.8.1 -> v1.8.1
* [new tag] v1.8.2 -> v1.8.2
* [new tag] v1.9.0 -> v1.9.0
* [new tag] v1.9.1 -> v1.9.1
* [new tag] v1.9.2 -> v1.9.2
* [new tag] v1.9.3 -> v1.9.3
* [new tag] v1.9.4 -> v1.9.4
* [new tag] v1.9.5 -> v1.9.5
* [new tag] v1.9.6 -> v1.9.6
Get https://github.com/AOKP/platform_manifest.git
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
curl: (22) The requested URL returned error: 404 Not Found
Server does not provide clone.bundle; ignoring.
remote: Counting objects: 1722, done.
remote: Compressing objects: 100% (1012/1012), done.
remote: Total 1722 (delta 839), reused 1552 (delta 709)
Receiving objects: 100% (1722/1722), 249.52 KiB | 50 KiB/s, done.
Resolving deltas: 100% (839/839), done.
From https://github.com/AOKP/platform_manifest
* [new branch] ics -> origin/ics
* [new branch] jb -> origin/jb
* [new branch] jb-mr1 -> origin/jb-mr1
* [new tag] Build-2 -> Build-2
* [new tag] Build-3 -> Build-3
* [new tag] Build-4 -> Build-4
* [new tag] Build-5 -> Build-5
* [new tag] Milestone-1 -> Milestone-1
* [new tag] jb-mr1_build-1 -> jb-mr1_build-1
* [new tag] jb-mr1_build-4 -> jb-mr1_build-4
* [new tag] jb-mr1_build-6 -> jb-mr1_build-6
* [new tag] jb-mr1_milestone-1 -> jb-mr1_milestone-1

Your identity is: akshatshenoy <akshat.shenoy@gmail.com>
If you want to change this, please re-run 'repo init' with --config-name

repo has been initialized in /home/akshat/kang

So I have to go to : the /kang folder in my home directory and open the .repo file. Which will look somewhat like the following images :

Important tip ! : While browsing through files in a directory , to view hidden files press control + H and to view the exact directory of the folder which you are browsing through press control + L ..

Opening this file with a text editor will tell you the list of the repos that need to be downloaded by the repo tool :

For AOKP JB-MR1 it will look somewhat like this

Now just a flashback to the original repo init command we had applied earlier

akshat@vaio:~/kang$ repo init -u https://github.com/AOKP/platform_manifest.git -b jb-mr1

When the repo init command is carried out , the repo tool grabs the default.xml / manifest.xml from AOKP's github which is mentioned in the init command :

So open up a web-browser and navigate to that page and you will realize that it is the same manifest.xml that is present in the .repo of the source directory where you have performed a repo init at. :)

Now when we repo sync ,

the repo tool downloads each repo as a specific project as shown in the picture : (highlighted)

So, basically, there are 415 projects to be synced in order to build AOKP that has been defined in the manifest.xml

Once those 415 projects have been downloaded , they will be stored in the .repo and a copy of it will be synced as a work-tree .

This work-tree comprises of the files and folders that appear automagically once the repo sync is done with all 415 projects synced !

It is that part of the source code which you actually work on and make your device specific changes to !​

Understanding the manifest :

​

​

1) The remote tag :

syntax :

<remote name=" any name here "
fetch="url of the remote website/hosting/server "
review="url of the code review of the repo :optional: " />

This tag just specifies a " remote " location for the repo tool to get your repositories from . This allows you to then associate a particular repository with that remote.

examples of remotes :

<remote name="github"
fetch="https://github.com/" />

<remote name="bitbucket" fetch="https://bitbucket.org"/>

<remote name="aokp"
fetch="https://github.com/"
review="gerrit.aokp.co" />

This allows you to use multiple code hosting sites , private or public and thereby enabling a trouble-free maintenance of your team's source code.

Now like the remote is specified , you can base you're manifest at a particular remote and define it as your default remote.

The advantages of this will be seen in (2) and (3) during the adding of the project tags to the manifest.xml

The syntax of the default remote is as follows :

<remote name=" any name here "
fetch="url of the remote website/hosting/server "
review="url of the code review of the repo :optional: " />

<default revision="any/path/here"
remote="remote hosting name defined earlier"
sync-j="x" />

where x is the default number of threads to be used when it is not defined in the repo sync command.

best example of a default revision that is widely used in almost every ROM default.xml is

<remote name="aosp"
fetch="https://android.googlesource.com/"
review="https://android-review.googlesource.com/" />

<default revision="refs/tags/android-4.2.2_r1.2"
remote="aosp"
sync-j="4" />

2) The base projects

As explained earlier the projects refer to the repositories that have to be fetched by the repo tool.

The 2) tag in the picture above is used to download the repository from your default remote's server.

The basic syntax of it is :

<project path=" path/in/work/tree" name="/repo/name/in/your/default" />

Examples :

<project path="abi/cpp" name="platform/abi/cpp" />
<project path="bootable/diskinstaller" name="platform/bootable/diskinstaller" />

This downloads those particular repos from the actual android source .

It is advisable to use this particular tag from the remote you will download the most number of repositories from as it cuts down on the effort of editing the manifest.xml

Many ROMs use the AOSP as the base as it is easier for them to keep their source code upto date without wasting their bandwidth on uploading the same repositories to their own gits .​

3) The other projects

This is for the non default remote projects

Their general syntax is :

<project path=" directory in work tree" name="name of the repository in remote" remote="name of remote defined earlier" revision="branch of the repository" />

NOTE : Default revision is master. So if you dont define the revision it is considered as the master branch!

Examples for this are :

<project path="bootable/recovery" name="CyanogenMod/android_bootable_recovery" remote="cm" revision="cm-10.1" />

For this of course the earlier remote "cm" should be mentioned.​

Quick tip !

​

Indirectly , the total url path to the repository should be = the fetched URL of the remote + the name of the repo in the project line.

In this case

https://github.com/ && CyanogenMod/android_bootable_recovery

in total :

https://github.com/CyanogenMod/android_bootable_recovery

For the sole purpose of making the life of a developer easier , CyanogenMod / AOKP and almost all other ROMs named their own repositories as

android_path_to_directory_in_work_tree || path_to_directory_in_work_tree

Such as :

If you notice the name of the repository : vendor_aokp corresponds to the aokp folder in the source folder in my source.

Thus the name of the repository gives you the path to the same files in your repository.

So much win ! :highfive:​

Maintaining your own projects using manifest.xml

​

Now that you have understood how the manifest.xml works .. you can even maintain your own projects with the help of a manifest.xml ! it is not necessary it has to be used only for ROMs !

Thats the best bit !

So the next time you have to pull a large number of repositories and maintain them you can create your own default manifest in your own github and add the repos as mentioned above.

Example I use sometimes :

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

<remote name="red-devil"
fetch="https://github.com/red-devil" />

<project path="Cherrypicking/frameworks/base" name="AeonROM/android_frameworks_base" remote="red-devil" revision="master" />
<project path="Cherrypicking/frameworks/native" name="AeonROM/android_frameworks_native" remote="red-devil" revision="master" />
<project path="Cherrypicking/frameworks/av" name="AeonROM/android_frameworks_av" remote="red-devil" revision="master" />

</manifest>

Understanding local manifests :

​

What is the local_manifest.xml ?

A local manifest is like a specific copy of the manifest of a particular ROM that is maintained by YOU the ROM compiler in your own computer.

This local manifest allows you to override some of the projects mentioned in the manifest and hence allows you to delete / add / modify the source code according to your preferences.

Best part about this is that every time you sync your source , even the onces mentioned in the local manifest get updated by themselves.

Hence, you dont need to keep track of all your device trees and kernel sources manually, let the repo tool do the work for you. :cool:

Let us see how this wonderful tool works :

Go to the folder where you have inited the repo and open the .repo ( press ctrl + H if it is NOT visible ).

Create a folder named local_manifests

Download this View attachment 2052061 and rename it to local_manifest.xml and copy it in the / .repo / local_manifests / directory

NOTE : In some older ROMs where an older version of the repo tool is used the local manifests has to be placed like

As opposed to a folder named local_manifests as illustrated in the above pictures :)​

Functions of the local manifest :

​

The local manifest looks somewhat similar to the main manifest.xml ... however in this you can even remove projects or add them

> To remove projects the basic syntax is :

<remove-project name="name of the project" />

What to remove :

Consider my case

I am building AOKP JB-MR1 for the HTC Explorer.

As a result of that I do not need the various device trees that are officially supported by AOKP.

Such all these :

OR

You will have to revert back to the original default.xml that you was present in your .repo and create a local_manifest.xml and remove projects / add them .

How do I do that ?

Basic syntax for removing them is :

<remove-project name="name_of_project" />

As shown in the image since I dont need those projects the syntax will be somewhat like :

<remove-project name="AOKP/device_samsung_d2att" />
<remove-project name="AOKP/device_samsung_d2tmo" />

For removing the 1st two projects as shown in the picture above :)

In a similar way , I went on removing projects that were not required... only device trees and saved the local_manifests.xml

Remember when I earlier synced there were 415 projects ?

Look at it now .. slimmed down around 50 projects ! :)

That means a much quicker repo sync for me ! :D​

> How to add projects to get it synced :

To add projects just use the code like I explained in the manifests.xml

Basic syntax is :

<remote name=" any name here "
fetch="url of the remote website/hosting/server "
review="url of the code review of the repo :optional: " />

<project path=" directory in work tree" name="name of the repository in remote" remote="name of remote defined earlier" revision="branch
of the repository" />

EG :

I wanted to add my own device tree of AOKP and the kernel source for my device in the local manifests cause I am too lazy to manually download them :silly:

<project path="device/htc/pico" name="android_device_htc_pico_aokp" remote="red-devil" revision="master"/>
<project path="kernel/htc/pico" name="android_kernel_htc_pico" remote="red-devil" revision="cm-10.1" />

So ultimately this is the sample local_manifest.xml I used :

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<remote name="red-devil"
fetch="https://github.com/red-devil/" />
<project path="device/htc/pico" name="android_device_htc_pico_aokp" remote="red-devil" revision="master"/>
<project path="kernel/htc/pico" name="android_kernel_htc_pico" remote="red-devil" revision="cm-10.1" />
<remove-project name="AOKP/device_acer_a510" />
<remove-project name="AOKP/device_asus_tilapia" />
<remove-project name="AOKP/device_htc_evita" />
<remove-project name="AOKP/device_htc_s4-common" />
<remove-project name="AOKP/device_lge_iprj-common" />
<remove-project name="AOKP/device_lge_mako" />
<remove-project name="AOKP/device_lge_p930" />
<remove-project name="AOKP/device_lge_su640" />
<remove-project name="AOKP/device_lge_vs920" />
<remove-project name="AOKP/device_motorola_maserati" />
<remove-project name="AOKP/device_motorola_omap4-common" />
<remove-project name="AOKP/device_motorola_solana" />
<remove-project name="AOKP/device_motorola_spyder" />
<remove-project name="AOKP/device_motorola_targa" />
<remove-project name="AOKP/device_motorola_umts_spyder" />
<remove-project name="AOKP/device_samsung_aries-common" />
<remove-project name="AOKP/device_samsung_d2-common" />
<remove-project name="AOKP/device_samsung_celox-common" />
<remove-project name="AOKP/device_samsung_d2att" />
<remove-project name="AOKP/device_samsung_d2tmo" />
<remove-project name="AOKP/device_samsung_d2usc" />
<remove-project name="AOKP/device_samsung_d2vzw" />
<remove-project name="AOKP/device_samsung_galaxys2-common" />
<remove-project name="AOKP/device_samsung_hercules" />
<remove-project name="AOKP/device_samsung_i605" />
<remove-project name="AOKP/device_samsung_i9100" />
<remove-project name="AOKP/device_samsung_i9100g" />
<remove-project name="AOKP/device_samsung_i9300" />
<remove-project name="AOKP/device_samsung_l900" />
<remove-project name="AOKP/device_samsung_maguro" />
<remove-project name="AOKP/device_samsung_manta" />
<remove-project name="AOKP/device_samsung_msm8660-common" />
<remove-project name="AOKP/device_samsung_msm8960-common" />
<remove-project name="CyanogenMod/android_device_samsung_omap4-common" />
<remove-project name="AOKP/device_samsung_smdk4412-common" />
<remove-project name="CyanogenMod/android_device_samsung_smdk4412-qcom-common" />
<remove-project name="AOKP/device_samsung_t0lte" />
<remove-project name="AOKP/device_samsung_t0lteatt" />
<remove-project name="AOKP/device_samsung_t0ltetmo" />
<remove-project name="AOKP/device_samsung_toro" />
<remove-project name="AOKP/device_samsung_toroplus" />
<remove-project name="AOKP/device_samsung_tuna" />
<remove-project name="AOKP/device_samsung_vibrantmtd" />
<remove-project name="AOKP/device_sony_hayabusa" />
<remove-project name="AOKP/device_sony_mint" />
<remove-project name="AOKP/device_sony_odin" />
<remove-project name="AOKP/device_sony_tsubasa" />
<remove-project name="AOKP/device_sony_yuga" />
<remove-project name="AOKP/device_sony_blue-common" />
<remove-project name="AOKP/device_sony_fusion3-common" />
<remove-project name="CyanogenMod/android_device_sony_qcom-common" />
<remove-project name="AOKP/device_sony_common" />
<remove-project name="AOKP/packages_apps_Torch" />
</manifest>

Result of this : the number of projects being downloaded reduced from 415 to 364 !

So , I hope you guys understood at least a bit about manifests and how the repo sync process actually works..​

Read on for some tips ! :)