Git Explained: A Project Umbrella Structure Using Git Submodules
Tags: git, github, beginners, docker
Published: 2020-05-30 19:23:45

Oftentimes, a team of developers will be faced with the decision of implementing a Git structure to handle the different parts that a project entails. Some teams opt for a ***monorepo***, while others decide to break it down into ***sub-projects***, for instance, a repository for the UI and another one for the API. Both Git structures have advantages and drawbacks. 

***Monorepos*** offer benefits such as code sharing, less code duplication, a single build, and atomic commits. However, ***monorepos*** also deal with a large number of commits in unrelated parts of the tree, as well as potential performance issues when cloning, fetching, and pushing code. On the other hand, maintaining ***sub-projects*** also translates into multiple repositories to clone, multiple sets of instructions to follow in order to install and build, and multiple directories to keep up with. 

Recently, I’ve been introduced to the use of **Git Submodules** to create a central umbrella repository that points to a particular commit in both the UI and API external repositories. This umbrella structure allows you to keep a sub-project composition with some of the advantages of using a monorepo such as managing documentation and configurations in one central place, and the ability to pull the whole project, including sub-projects, in one step. This post is going to discuss the [benefits of using an umbrella repository structure using Git Submodules](#the-benefits), the [struggles that come with it](#the-struggles), and the [commands that you need to learn to start using this Git structure](#the-commands).

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/487gofjnuacom8xphemo.png)

## The benefits

**Sub-project composition.** Having multiple repositories for the different parts of your project allows you to give limited access on a “need to code” basis. For instance, you might be in a situation where a frontend developer is added to your team for a few weeks to speed up the UI implementation. This developer can easily choose to pull only your UI repository, without having to mess with the API.

**Continuous Development.** The proposed umbrella structure using Git Submodules keeps a sub-project composition that allows each repository to have it’s own deployment process.

**Documentation in one place.** The umbrella repository gives you space to store documentation that applies to the whole project in a single location. I personally like to keep a global markdown file with the instructions to build and run the whole project using [Docker](https://www.docker.com) in the umbrella repository and a markdown file in each repository with instructions on how to install and use each sub-project locally. 

**Configurations in one place.** Our team uses Docker containers to create, deploy and run applications. The umbrella repository is a fantastic spot to place a `docker-compose.yml` file that builds and runs a database image, your api, and ui environments with one command.

**Atomic code pulling.** By using the `--recurse-submodules` flag when cloning an umbrella repository, you get all the nested directories that contain submodules in one command. In other words, you get your UI and API repositories all at once.

---

## The struggles

**Need to specify the use of Git Submodules in instructions.** If your team is not used to this Git structure, they most likely won’t know how to clone or handle them correctly. Git does not download submodule contents by default, so make sure to include this information in your README.

**Manual retrieval of submodules updates.** Git won’t automatically register the updates to submodules. You need to run `git submodule update --remote` to see changes made to a submodule.

**Simple implementations only.** So far, I’ve only used this umbrella structure to link the UI and API repositories per project. I can see how keeping up with more than two submodules could get challenging and might require you to look into a different implementation.

**Mostly useful when using Docker.** If you are not using Docker, the benefits of this structure might not be as impactful.

---

## The commands

Let’s discuss the possibilities and commands you have in your toolbox to implement this umbrella structure:

- [Create a project with submodules](#create-a-project-with-submodules)
- [Rename a submodule](#rename-a-submodule)
- [Clone a project with submodules](#clone-a-project-with-submodules)
- [Work in a submodule](#work-in-a-submodule)
- [Check and pull submodule updates](#check-and-pull-submodule-updates)
- [Update umbrella project with latest submodule commits](#update-umbrella-project-with-latest-submodule-commits)

### Create a project with submodules
To explain this action clearly, I will create a repository called umbrella and two repositories called umbrella-ui and umbrella-api in GitHub. You can check them out [here](https://github.com/milufranz08/umbrella).

	cd[umbrella-project-name]
	git submodule add [git@github.com:your-account/umbrella-ui.git] [name]

> Note: the name parameter is optional. If you don’t pass a name, you will get the submodule repository name. In this case, I will give it the “ui” name parameter because I don’t want a redundant directory name like “umbrella-ui” inside my “umbrella” project.

Run the `ls -a` command to see a new directory called `ui` and a new file called `.gitmodules`. The `.gitmodules` file is a configuration that stores the mapping between your main project and its submodules. It will look similar to the following:

	[submodule “ui”]
	path = ui
	url = git@github.com:your-account/umbrella-ui.git

After committing and pushing this change, you should see a folder for your submodule that points to the latest commit in the submodule's master branch.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/gez5k4ly4zyz4shrfsf1.png)

### Rename a submodule
If you forget to pass a name parameter when adding your submodule, you can still change your submodule directory name using the following command:

	git mv [old-name] [new-name]

### Clone a project with submodules
Pass the following command to automatically clone your project and its submodules. If you don’t pass the `--recurse-submodules` flag, your directory will be empty.

	git clone --recurse-submodules [git-repo-url]
 
### Work in a submodule
This is the same process that you follow when dealing with a regular repository. You make changes, commit them, push them to remote.

### Check for submodule updates
When changes in submodules are pushed to their remote repository, you are going to need to check for these changes and update your umbrella repository because the commit pointers will be out of date.

To make sure you understand this concept, I will show you the `umbrella-api` repository, which has been updated and the last commit in master is `6302c8…`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/wky2dtbge6hups6nf3ek.png)

Now let’s look at our umbrella repository, ***the `api` submodule is not pointing to the latest commit in `umbrella-api`. Instead, it is pointing at an old commit `15a146…`***

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/82a0ky61l0t249m5mnx1.png)

You can pull the changes made in all your submodules directly from the umbrella directory with the following command:

    // fetch all new updates in submodules
    git submodule update --remote

    // see which submodule has been modified
    git status

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/9byhvzorelfbrkh4aw9z.png)

    // see all new commits that have been pulled down
    git diff

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/6a09mb93epebdzwn226b.png)

### Update umbrella project with latest submodule commits
Now that we pulled changes made in our submodules, we need to update our umbrella project pointers. This is VERY IMPORTANT because ***if someone clones our umbrella project in this state, they won’t get the newest code under our submodules.***

	// add all submodules changes
	git add .

	// commit new pointers
	git commit -m “Update submodule pointers”

	// publish the changes
	git push origin master


> ***Note:*** You can also use the following shortcut to pull the latest changes made to your submodules and get your umbrella project to point to the last commits.

	git submodule update --remote --merge

Now our umbrella project points to the latest commits in both submodules, umbrella-ui and umbrella api.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/i60y8pweft6sg0slrucp.png)

Have you used Git Submodules in a similar umbrella structure before? If not, I hope it sparks your interest and you give it a try.

Thank you for reading, liking, and sharing my Git Explained series posts, it truly means a lot to me. I enjoy putting them together and love your questions and suggestions.

Checkout my other Git Explained posts and keep an eye for the next one. I explore, define, and explain Git concepts every Saturday!