[Android] Kishan14:
Repo and Manifests

​

Hello guys..:D:D

There are many upcoming developers and many dev on xda.Till now you all have learned how to setup build enviroment and packages required and build process. Now here I will be sharing few tips and tricks that will help some new developer for developing their ROMs and Kernels. Well, I am not pro in all these stuff even I am a newbie. I have came across many guides and my best friend Google.

These tips may save your Time, Internet Bandwidth, HDD space etc.. :p

Requirements:

Basic Knowledge of compiling ROM/kernel:)

Ok so here I Start::cowboy:

Repository (Repo)

​

What is Repo?:rolleyes:

Repo is a repository management tool that we built on top of Git. Repo unifies the many Git repositories when necessary, does the uploads to our revision control system, and automates parts of the Android development workflow. Repo is not meant to replace Git, only to make it easier to work with Git in the context of Android. The repo command is an executable Python script that you can put anywhere in your path. In working with the Android source files, you will use Repo for across-network operations. For example, with a single Repo command you can download files from multiple repositories into your local working directory

Was that a bit tough definition.. no worries ... here's Screenshots explaining roughly.

The image shown above is repositories (Platform_mainfest,C-Rom Device tree, android etc.)

In Simple words you can say 'repo is a folder that stores code'. In the above image, the two repo's are highlighted. the black highlighted repo is the one which one have written or contributed. The other (red) one shows that the repo is forked from someone else repo.

Repo Tool:

This tool plays a vey important role in downloading rom source.. So the word ‘Repo’ is common with you guys as u do ‘repo init’ ‘repo sync’ ..

Remember,From the above guide, while setting up you Linux you entered a command

mkdir ~/bin && curl [URL]http://commondatastorage.googleapis.com/git-repo-downloads/repo[/URL] > ~/bin/repo && chmod a+x ~/bin/repo

In simple language This Step was to download and install the Repo tool..

Why This Tool is used?:rolleyes:

The Repo Tool helps you to download the bunch of repositories by entering just a single command, which helps us to make our downloading part easier.:DTo download the source following command is used:

repo sync -j#

# - number of jobs you want to give for your downloading source.

Now A question may arise that What does Repo Tool actually do??:confused::D

Umm, Let me explain with an example, Let's take the above example. The repo tool takes all the reference for the code to be downloaded from a Manifest.xml. Suppose you wanna download cm source. So as mentioned above, you have to give 'repo init' command. after this command '.repo' folder is created. you ll find 'manifest.xml' file in '.repo' folder. Now when you open Manifest.xml file you'll find some path of files and folders. The command 'repo sync' downloads the folders and files that is mentioned in manifest.xml to the specific location mentioned.

Now, the '-j#' tag represents the number of jobs you allow your pc to download the source. For Normal internet speed it is recommended to use -j2 or -j4. For faster connection -j16 or -j32 is recommended.

The Manifest

​

The manifest.xml is a file which is written in xml markup and directs the repo tool to the repositories and their respective branches that need to be downloaded. This manifest is located under the head of the android source tree in the .repo/manifests/ directory. The manifest is formatted in XML markup and contains information regarding which git repositories to use, where they are located on the internet, where to put them in the source code directory, and what branches of the git repositories are used. Some of these repositories are pulled, unmodified from AOSP, but most come from CyanogenMod on GitHub. We can take repositories from different git servers, use different branches for each one of them, gather them into groups for easy management and more. Here’s an example for a manifest xml file:

<manifest>
<remote name="aosp"
fetch="https://android.googlesource.com/"/>
<remote name="aogp"
fetch="https://github.com/aogp/"/>

<default revision="master"
remote="aosp"
sync-j="4" />

<project path="art" name="platform/art" />
<project path="bionic" name="platform/bionic" groups="pdk" />
<project path="dalvik" name="platform_dalvik” remote=“aogp”/>
<project path="frameworks/base" name="platform_frameworks_base" remote=“ aogp" revision=“statusbar_fixes"/>
<project path="tools/adt/eclipse" name="platform/tools/adt/eclipse" groups="notdefault,tools" />
</manifest>

Let’s review its structure and see why it is used :laugh:

<remote>

The remote tag is describing the remote git servers we support pulling repositories from. In this case, we have a remote named aosp, linking to the Google’s AOSP git server, and another one named AOGP, linking to a Github account of the user “AOGP”.

<project>

Defines a single repository. These are the main attributes:

path – Where the repository will be checked out into, relative to the current working directory.

name – The name of the project on our git server

remote – The name of the remote server where the repository can be found.

revision – The branch/tag name we want to checkout.

group – The group name(s) for this project. We can declare a set of projects as members of a group, then sync only them. Omitting this attribute makes the project a member of the “default” group.

<default>

Now like the remote is specified , you can base you're manifest at a particular remote and define it as your default remote. it defines a default values for attributes we are using, when syncing the system. In this case, we set the default branch to checkout each project on its “master” branch and the default remote as “aosp”. If we omit those attributes when defining a project, the defaults defined here will be used. Other interesting attributes:

sync-j

The value for this will be the number of jobs to use when syncing the system. Parallelization helps get the job done quicker, but could also get the computer or the network stuck. Using 4 threads is what commonly used for syncing.

sync-c

syncing only the current branch/tag from git. This will checkout for each project only the current branch/tag that we specify for it, and not any other branches that exists on the repository. This will help save some space and bandwidth (and also time), but if you’ll need to switch between branches on a specific project later on – you’ll need to fetch it manually.

Local_Manifest :

​

If you wanted to change the manifest.xml list, you could edit default.xml directly. However, this would make your local copy of the manifest conflict with the official version on github. This could create issues when the official manifest is updated, as it would clash with your changes.

:eek:SO NEVER THINK OF EDITING DEFAULT.XML:silly:

Now The Question Arises how we can control the sync?:rolleyes:

This is where local manifests come into play.

Creating a local manifest allows you to customize the list of repositories used in your copy of the source code by overriding or supplementing the default manifest. In this way, you can add, remove, or replace source code in the official manifest with your own. By including repositories (which need not even reside on GitHub) in a local manifest, you can continue to synchronize with the repo sync command just as you would have previously. Only now, both the official repositories from the default manifest and the additional repositories you specify will be checked for updates.

Hence Local_manifest.xml is BOON to Developers:highfive::cool:

Uses of Local_manifest.xml:cowboy:

1) Adding and removing repositories

To add to the contents of the default manifest, create a file called local_manifests.xml under the .repo directory, this can also be created via terminal by using following steps:

A) open terminal (Ctrl+Alt+T) and type the following command and hit Enter.

cd path_of_ur_folder/.repo

B) Type following command

nano local_manifest.xml

Now type the content you wanna add to local_manifest.xml. After completing press "ctrl+O" and then hit enter to save it. to exit nano editor press "ctrl+X"

Simple Isn't it?:laugh:

Let's start with an Example which we can use to describe the scenario:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

<remote name="github" fetch="git://www.github.com/" />

<remove-project name="CyanogenMod/android_hardware_qcom_fm" />

<project path="hardware/qcom/fm" name="aogp/hardware/qcom/fm" remote="github" revision="cm-11.0"/>

</manifest>

Meaning of Above Terms

1) <?xml version="1.0" encoding="UTF-8"?>

It is a standard XML declaration, telling interpreters this is an Extensible Markup Language file. Once this is established, the <manifest> and </manifest> tags enclose some contents which the repo command will recognize.

2) <remote name="github" fetch="git://www.github.com/" />

First, a remote for git is declared and given the name "github". In git, a remote essentially refers to a place and method for accessing a git repository. In this case, www.github.com contains special up-to-date repositories for fm qcom hardware from "AOGP" platform. This is equivalent to the following git command:

git remote add github git://www.github.com/

3) <remove-project name="CyanogenMod/android_hardware_qcom_fm" />

This line removes a project (specifically, cyanogenmod/android_hardware_qcom_fm) declared in the default manifest. After running repo sync, it will no longer be available in the source tree.

4) <project path="hardware/qcom/fm" name="aogp/hardware/qcom/fm" remote="github" revision="cm-11.0"/>

This line defines a new project. In this case, it replaces the removed project android_hardware_qcom/fm with one from other, using the remote that was defined above.

When adding a new project that replaces an existing project, you should always remove that project before defining the replacement.

You can simply add a new project to the source code, such as when you want to add your own app to the build.

Note that when adding new projects, there are at least three parts defined:

• remote -- the name of the remote. this can be one that was defined in either the default manifest or local_manifest.xml.

• name -- the name of the git project-- for github it has the format account_name/project_name.

• path -- where the git repository should go in your local copy of the source code.

• revision -- (optional) which branch or tag to use in the repository. If this attribute is omitted, repo sync will use the revision specified by the <default ... /> tag in the default manifest.

After creating .repo/local_manifests.xml, you should be able to repo sync and the source code will be updated accordingly:D:D

This was all about Repo and Manifest:D

Now i'm gonna give you some tips that will help you a more better way in developing.!!!

TIPS AND TRICKS

Tip#1 Syncing a full new source tree in minutes!

​

Suppose you wanna compile multiple roms for device. For example, you may want to have an AOGP source tree as well as a CM tree.

Obvisouly, you could repeat the same steps, to repo init and then repo sync -j8 for both. But that would be twice as time consuming. isn't it?

No issues, there is a simple trick to download the source within few minutes(depending on your internet speed). Lets say that you have already initialized and synced a CM source tree, in ~/roms/cm11.

Now, we will prepare a new repo for AOGP tree, in ~/roms/aogp. Here comes the trick: We will use the --reference flag of repo init to indicate to repo that we already have a local repository that it can take information from, instead of downloading everything from Internet again:

So Lets say u wanna begin with AOGP, So here's the steps you got to follow :

mkdir roms/aogp &&cd roms/aogp

repo init --reference=~/roms/cm11 -u git://github.com/AOGP/android.git -b cm-11.0

You should notice a big time difference, from hours for a full repo sync, down to minutes using this trick!

Did u understood what it exactly did? NO!:p

here's simple words explaining about reference flags. In this tip (reference flag) the repo tool does it checks the locally downloaded source and skips downloading that it Symlinks the code from the existing source to the new source.

Tip#2 Using CCACHE a.k.a Compiler's Cache tool

​

CCACHE also known as a compiler cache tool, which caches the output of the compiler, to save time when compiling a file which has not been changed since the last compile. It is really a best companion of rom developers

To enable CCACHE,

use your favorite editor.. I ll be using nano editor

1) Open a terminal ( cntrl + alt + t ) and type

nano .bashrc

2) Add the following code :

export USE_CCACHE=1

After you log out and back in again, this change will take effect.

By default, the Android build process creates a subfolder named “out” within the root of your source tree. While for most users this is best, in some situations it can be advantageous to move the output directory to another file system. If, for example, you have a striped RAID array, it is beneficial to store the output directory on this array. All of the files within the out directory can be re-generated in the event of filesystem problems (if the sources were held on another filesystem).

To change the output directory to another filesystem, use the command

export OUT_DIR_COMMON_BASE= /<path_of_your_choice>/.ccache

prebuilts/misc/linux-x86/ccache/ccache -M 50G

The suggested cache size is 50-100G.

On OSX, you should replace linux-x86 with darwin-x86

The above defined output directory will created a new folder, named after your current source tree directory. For instance, if you have source trees as /source/tree1 and /source/tree2 and OUT_DIR_COMMON_BASE is set to /output, then output directories will be/output/tree1 and /output/tree2.

It is therefore important to ensure you do not have multiple sources stored in directories with the same name, as they would end up sharing an output directory, with unpredictable results.

Tip#3 Playing with local_manifest.xml

​

In many Manifest.xml you ll find many stuff not needed by our device. Also there are files needed and that is not present in the default.xml.

* Adding Projects to local_manifest.xml

So, to add projects to your local_manifest.xml for your device refer the following format that explains the same.

<?xml version="1.0" encoding="UTF-8"?>
<manifest>

<project path=" directory in work tree" name="name of the repository in remote" remote="name of remote defined earlier" revision="branch of the repository" />

</manifest>

Still have some doubts?? the following example will clear it.. ;)

Suppose i wanna add device tree, kernel and vendor for my device which is needed in Building ROM Guide Above i ll follow the above format and will make a new one for my device.

Device Example Used Here : Samsung Galaxy Grand Duos GT-I9082

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
<project path="device/samsung/i9082" name="pawitp/android_device_samsung_i9082" remote="github" revision="cm-11.0" />
<project path="vendor/samsung/i9082" name="k2wl/android_vendor_samsung_i9082" remote="github" revision="master" />
<project path="kernel/samsung/i9082" name="pawitp/android_kernel_samsung_i9082" remote="github" revision="cm-11.0" />
</manifest>

This will download your device Identity needed for Rom Building:cowboy::cowboy:

* Remove Unwanted Stuff From default.xml

As I mentioned above the command for removing project, you can use that command to remove the unwanted stuff from default.xml

As majority of the people here are using Linux. As dawin tool chains are used by MAC OS, So darwin tool chains and darwin packages are useless for linux users. You can remove the same by using remove project command.

For Example :

<project path="prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6" name="platform/prebuilts/gcc/darwin-x86/arm/arm-eabi-4.6" groups="pdk,darwin,arm" />
<project path="prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.6" name="platform/prebuilts/gcc/darwin-x86/arm/arm-linux-androideabi-4.6" groups="pdk,darwin,arm" />
<project path="prebuilts/gcc/darwin-x86/mips/mipsel-linux-android-4.6" name="platform/prebuilts/gcc/darwin-x86/mips/mipsel-linux-android-4.6" groups="pdk,darwin,mips" />
<project path="prebuilts/gcc/darwin-x86/x86/i686-linux-android-4.6" name="platform/prebuilts/gcc/darwin-x86/x86/i686-linux-android-4.6" groups="pdk,darwin,x86" />

Before Removing Darwin Lines see below

After Removing Darwin Lines

From above you can notice earlier there were 462 projects to be downloaded but after removing darwin lines its slimmed to 444 projects.

Less projects to download sync completes faster:D:D

*Remove all the official device not required by you for compiling.

There will be many devices that are officially supported by ROM developers and there Tree will be added to Default.xml which are useless for your device.

The device may be in the following format

android_device_companyname_codename

For Example :

android_device_samsung_codename

*Similarly Remove all the kernel source not required by you as it will take more time and your precious bandwidth.

* Remove all unnecessary vendor trees from the manifest.. Why do you need the blobs for a device you are not building for !!:cowboy::D

Hope these tips helps you a lot...!!! :D:D

Any doubts.. post on this thread.. !!:)

credits:; [MENTION=1534880]Jacle[/MENTION]agle for editing and screenshots.. :D