Initializing a `git` repository with `git-lfs` (large file storage)
Tags: git, programming, datascience, gitlfs
Published: 2023-01-18 14:53:25

As it happens, you would like to initialize a `git` repository with `git-lfs` (large file storage) functionality to store some really large files. For someone a file of a 10MB size may seem large, for somebody else it would have to be as 1GB in size, but regardless of that, this tutorial explains how to efficiently initialize a repository with `git-lfs`.

## Prerequisites

First, make sure that you have `git lfs` installed (how-to install for: [macOS](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage), [Linux](https://github.com/git-lfs/git-lfs/wiki/Installation#debian-and-ubuntu)). In short, for macOS: `brew install git-lfs`.

## Initializing `git-lfs`

Assuming that you have not initialized your repository with the "normal" `git` command yet, run the following commands:

```
git init
git lfs install
```

For the sake of the example (to show you some difficulties you may encounter), let's assume that the files that you wish to add to `git-lfs` are stored in two separate directories (e.g., `data` and `supplementary_materials`). The tree of your repository looks somehow like that:

```
.
├── ...
├── README.md
├── data
│   ├── data1.png
│   ├── data2.zip
│   └── label.txt
├── images
│   └── some_pict.png
└── supplementary_materials
    ├── first_subdir
    │   └── somefile.png 
    ├── second_subdir
    │   └── somefile.png 
    └── third_subdir
        └── somefile.zip 
```

First, some basic stuff about synchronizing with your remote repository:

```
git remote add origin https://github.com/mikbuch/your-repository.git
```

Then:

```
git branch -M main
git pull origin main
```

Now, there are basically two ways of adding files to `git-LFS`.


### The first (recommended) method

Create `.gitattributes` file:

```
vim .gitattributes
```

and add the following lines:

```
data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text
```

**Note!** In our example we have two separate directories that you wish to add to `git-lfs` in which you have some subdirectories. (1) The syntax has to be exactly like the one in the code above, or else you won't add all files in all subdirectories. (2) If you have several directories, from the point of view of your current working directory, you have to list them all, one by one.

### The second (not recommended) method

Use the following commands to add directory and all files in this directory and its subdirectories:

```
git lfs track data/**/*
git lfs track supplementary_materials/**/*
```

**Note!** The command `git lfs track supplementary_materials/**` will not be sufficient to add all files in all subdirectories.

Why this method is **not** recommended?: It will list/add **all** tracked files to your `.gitattributes` (one by one!), e.g.:

```
➜  git:(main) ✗ cat .gitattributes

data/data1.png filter=lfs diff=lfs merge=lfs -text
data/data2.zip filter=lfs diff=lfs merge=lfs -text
data/label.txt filter=lfs diff=lfs merge=lfs -text
...
supplementary_materials/first_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/second_subdir/somefile.png filter=lfs diff=lfs merge=lfs -text
supplementary_materials/third_subdir/somefile.zip filter=lfs diff=lfs merge=lfs -text
...
```

What you would like to have in your `.gitattributes` is just:

```
data/**/* filter=lfs diff=lfs merge=lfs -text
supplementary_materials/**/* filter=lfs diff=lfs merge=lfs -text
```

It is much more clearer approach. See the first (recommended) method described above.

### Preview the `.gitattributes` file

```
cat .gitattributes
```

### Add all files to your repository:
```
git add --all
```

### Show which files are tracked with (`git lfs ls-files`)

```
git lfs ls-files
```

**Note!** `git lfs ls-files` works only after you have added the files to the repository with `git add`.

Show which files were added to the repository (regardless of whether LFS or not):

```
git status
```

Make sure that the files are tracked with `git-LFS`:

```
du -sh .git
du -sh .git/lfs
```

### Note on `GitHub`

If you are using `GitHub`, please make sure that you are using the proper SSH file:

```
cat ~/.ssh/config
```

In `GitHub`, also make sure that your key is added to you profile (not just one repository!). See the output of a command:

```
ssh -T -ai ~/.ssh/my-key.pem git@github.com
```

Source: https://superuser.com/a/232406/950943

If you used this key for one repository only you will see somethink like this:

```
Hi mikbuch/your-repository! You've successfully authenticated, but GitHub does not provide shell access.
```

You should see output like this:

```
Hi mikbuch! You've successfully authenticated, but GitHub does not provide shell access.
```

### Committing and pushing to the server

Commit the files (both LFS and "normal"):

```
git commit -a -m "Initializing commit with git LFS"
```

Finally, you can push your reporitory to the remote server:

```
git push -u origin main
```

---

In order to reset (if needed/optional) during the process:

```
rm -rf .git
rm .gitattributes
```

---

And that's it, I hope you will find this tutorial useful, have a nice day!

---

## Additional resources

 * 045 Introduction to Git LFS (Large File Storage) by Dan Gitschooldude: https://youtu.be/xPFLAAhuGy0

 * Example use: https://github.com/git-lfs/git-lfs/wiki/Installation#debian-and-ubuntu

---

Source for the cover photo image: Photo by <a href="https://unsplash.com/it/@wesleyphotography?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Wesley Tingey</a> on <a href="https://unsplash.com/photos/snNHKZ-mGfE?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>