Compatibility of GitLab on CockroachDB and YugabyteDB (II) - Read and Write Scenario Testing
Tags: postgres, gitlab, yugabyted, cockroachdb
Published: 2022-06-08 07:08:37

##  Testing Background

GitLab is a globally popular source code management tool. In earlier versions, users could choose to use either MySQL or PostgreSQL, but since version 12.1.0, the official support for MySQL has been dropped completely.

Many of the features in the new version of GitLab are based on PostgreSQL, which is the benchmark for many products that use PostgreSQL as the underlying data store.

Imagine a scenario where a large group is divided into divisions and each division or even a small team may maintain its own GitLab, making it tricky to manage these repositories from the group level. For example.

- Versioning issues (open source and commercial versions, high and low versions)
- Fine-grained permission control
- Data backups
- Infrastructure utilization

Having a unified GitLab environment with good scalability and high availability would certainly be the best solution. But the traditional standalone PostgreSQL database does not meet the above needs, so can we consider running GitLab on a distributed database?

CockroachDB and YugabyteDB are relatively well-known new open source distributed databases that implement the PG protocol. This series of review articles is used to compare how well these two databases support GitLab, and to a certain extent reflects the compatibility with standard PostgreSQL.

In the previous article ["System Initialization"](https://dev.to/dcopensource/compatibility-of-gitlab-on-cockroachdb-and-yugabytedb-i-system-initialization-28m3), we concluded that CockroachDB could not be started because it could not automatically create database schema through GitLab's setup program, while YugabyteDB could start GitLab normally with all normal initialization steps.

In this test, we first imported a standard GitLab library and the underlying data into these two databases to see if GitLab could be started normally, and then we selected a part of GitLab's core business scenarios to do a comparison test to see how compatible they are.

## Test Environment

- CockroachDB 

  ```sql
  defaultdb=# select version();
                                        version
  ------------------------------------------------------------------------------------
   CockroachDB CCL v22.1.0 (x86_64-pc-linux-gnu, built 2022/05/23 16:27:47, go1.17.6)
  (1 row)
  ```

- YugabyteDB

  ```sql
  gitlab=# select version();
                                                                                           version
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   PostgreSQL 11.2-YB-2.13.2.0-b0 on x86_64-pc-linux-gnu, compiled by clang version 12.0.1 (https://github.com/yugabyte/llvm-project.git bdb147e675d8c87cee72cc1f87c4b82855977d94), 64-bit
  (1 row)
  ```

- GitLab

  ```
  GitLab information
  Version:        12.1.0-ee
  Revision:       1f2e6f3f6d8
  Directory:      /home/git/gitlab
  DB Adapter:     PostgreSQL
  ```

## Test Scenarios

| Scene Type | Scene name                                                   |
| ---------- | ------------------------------------------------------------ |
| read (9)   | - Project List <br />- Project View <br />- Repository View <br />- Branch List <br />- Issue List <br />- Issue View <br />- Merge Request List <br />- Merge Request View <br />- Project Members |
| write (8)  | - New Project <br />- GitLab Import <br />- New Commit <br />- Create Branch <br />- Create Issue <br />- Create Merge Request <br />- PR Merge <br />- Add Project Member |

## Testing Process

To keep it simple, let's do the data migration directly using pg_dump.

First, export the schema and data from the standard library to the sql file.

```
pg_dump --host 10.3.70.132 --port 32298 --user postgres --no-owner -W gitlabhq_production > /root/gitlabhq_production.sql
```

### 1. CockroachDB Data Migration

Here the psql client is used to import the backed up sql, if an error occurs during execution it will be automatically skipped and the error message will be printed out as follows.

```
psql --host 10.3.70.189 --port 26258 --user root gitlab -f /root/gitlabhq_production.sql > pg_import_crdb.log
```

Observations from the output error messages contain the following two main categories.

```
Description:  source SQL:
CREATE EXTENSION IF NOT EXISTS pg_trgm WITH SCHEMA public
                                            ^
Tip:  You have attempted to use a feature that is not yet implemented.
See: https://go.crdb.dev/issue-v/74777/v22.1
psql:/root/gitlabhq_production.sql:30: ERROR:  at or near "pg_trgm": syntax error: unimplemented: this syntax
Description:  source SQL:
COMMENT ON EXTENSION pg_trgm IS 'text similarity measurement and index searching based on trigrams'
```

The error reported above still says that extension is not compatible with the problem.

```
Description:  You have attempted to use a feature that is not yet implemented.
See: https://go.crdb.dev/issue-v/47420/v22.1
psql:/root/gitlabhq_production.sql:31396: ERROR:  at or near ".": syntax error: unimplemented: this syntax
Tip:  source SQL:
CREATE INDEX index_issues_on_description_trigram ON public.issues USING gin (description public.gin_trgm_ops)
```

This error is due to the fact that CockroachDB does not support operator class yet, but these two errors are related to indexes and are not expected to have much impact on DML operations, so ignore them for now.

Look at the database situation after the sql file is imported.

```sql
gitlab=# select C.relkind,count(C.relname) from pg_class C left join pg_namespace n on n.oid = C.relnamespace where n.nspname = 'public' group by C.relkind;
 relkind | count
---------+-------
 r       |   249
 i       |   890
 S       |   231
(3 rows)
```

**Everything works except for about 10 indexes short. At this point**, point the GitLab database to this new repository, start the program and see if the page opens:

```bash
sudo -u git -H editor config/database.yml
sudo /etc/init.d/gitlab restart
 
 
source=rack-timeout id=oMeadFm1kN1 timeout=60000ms state=ready
Started GET "/users/sign_in" for 10.3.74.126 at 2022-05-31 16:19:18 +0800
Processing by SessionsController#new as HTML
Completed 200 OK in 55ms (Views: 32.3ms | ActiveRecord: 9.7ms | Elasticsearch: 0.0ms)
source=rack-timeout id=oMeadFm1kN1 timeout=60000ms service=291ms state=completed
```

From the logs, we can see that the login page jumps normally without errors. Then use the existing user to see if the login is successful.

![Image description](https://img-blog.csdnimg.cn/5388d86867814009ac40d4cb17eddda9.gif#pic_center) 

### 2. YugabyteDB Data Migration

Import the sql file into YugabyteDB in the same way as before.

```
psql --host 10.3.70.189 --port 5434 --user postgres gitlab -f /root/gitlabhq_production.sql > pg_import_ygdb.log
```

**About the implementation of more than 1 hour, the whole process did not report errors**, check the database object.

```sql
gitlab=# select C.relkind,count(C.relname) from pg_class C left join pg_namespace n on n.oid = C.relnamespace where n.nspname = 'public' group by C.relkind;
 relkind | count
---------+-------
 S       |   231
 i       |   903
 r       |   249
(3 rows)
```

Consistent with the standard postgreSQL Schema.

Modified the database connection and restarted GitLab, then tried to open the page to log in to the existing user, and found that the login was successful and jumped to the home page.

![Image description](https://img-blog.csdnimg.cn/7a7fe776a2274703a38a5fe8e959ea7f.gif#pic_center)
 

### 3. Scene Comparison

| Scene Type           | CockroachDB | YugabyteDB | Result                                                       |
| -------------------- | ----------- | ---------- | ------------------------------------------------------------ |
| Project List         |             |            | Both are supported                                           |
| Project View         | ![](https://img-blog.csdnimg.cn/64042881481b4666a04015dccecf14b1.gif#pic_center)            |  ![](https://img-blog.csdnimg.cn/020801c84a59493cacbacbed3dad1597.gif#pic_center)          | Both are supported                                           |
| Repository View      | ![](https://img-blog.csdnimg.cn/5642c43221214b6e906796e852c76665.gif#pic_center)            | ![](https://img-blog.csdnimg.cn/80390c149797448b8e436c284ec0efdf.gif#pic_center)           | Both are supported                                           |
| Branch List          | ![](https://img-blog.csdnimg.cn/6134854bbaa949f9b3a2e59fd35ad832.gif#pic_center)            |  ![](https://img-blog.csdnimg.cn/b297d6390f6842cbb44646f001f842d3.gif#pic_center)          | Both are supported                                           |
| Issue List           |  ![](https://img-blog.csdnimg.cn/6f334d4d572c4935aaea267304ca3b7c.gif#pic_center)           | ![](https://img-blog.csdnimg.cn/b8c0e89d466d4948b33ac0ff1d7b85e4.gif#pic_center)           | Both are supported                                           |
| Issue View           | ![](https://img-blog.csdnimg.cn/24ffa335abe14f689c703aa2ba007cd2.gif#pic_center)            | ![](https://img-blog.csdnimg.cn/5e05c193920645b9b6fbc7f196091e79.gif#pic_center)           | Both are supported                                           |
| Merge Request List   |  ![](https://img-blog.csdnimg.cn/512539fd044841ccbb5b98274030dc68.gif#pic_center)           | ![](https://img-blog.csdnimg.cn/eb379bf48f43455ab10876342a21b469.gif#pic_center)           | Both are supported                                           |
| Merge Request View   |  ![](https://img-blog.csdnimg.cn/bc2559ba8b714f8e875da11b79a0007c.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/a545c1dcc3e3439e91acb041c723585e.gif#pic_center)          | Both are supported                                           |
| Project Members      |  ![](https://img-blog.csdnimg.cn/5e8ed56367c84b91a6796391cbdf9556.gif#pic_center)           | ![](https://img-blog.csdnimg.cn/8e13bb8b2b3943a1a5831cecb862bf65.gif#pic_center)           | Both are supported                                           |
| New Project          |  ![](https://img-blog.csdnimg.cn/682678986eb24313bca0c2047ecda86e.gif#pic_center)           | ![](https://img-blog.csdnimg.cn/3996908a0e9f4a809f80198b366194e9.gif#pic_center)           | CockroachDB: Go to the Create Project page and return 500 exceptions, logging the error message "ActionView::Template::Error (PG::UndefinedColumn: ERROR: column "namespaces.rowid" does not exist")<br/><br/>YugabyteDB: It keeps loading after submitting the import request, observing the log without error, and jumping to the created project page after some time. |
| GitLab Import        |             |  ![](https://img-blog.csdnimg.cn/49d4b23d8f7048f3a73b7679d1a850c6.gif#pic_center)          | CockroachDB: Can't go to the new project page, can't import project.<br/><br/>YugabyteDB: It keeps loading after submitting the import request, and no error is reported in the log. I suspect it is a gitlab permission problem, I restarted the gitlab program with root user and imported successfully. |
| New Commit           |  ![](https://img-blog.csdnimg.cn/61a2300447844cf995b2808aad130d45.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/036c0688024949218d7a01f87474115e.gif#pic_center)          | Both are supported                                           |
| Create Branch        |  ![](https://img-blog.csdnimg.cn/7c0071abd56446759584f433b698f70f.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/b35afa0bcfbf4647a257d47390a0e7e3.gif#pic_center)          | Both are supported                                           |
| Create Issue         |  ![](https://img-blog.csdnimg.cn/a76c79f5c0f84173a037b8019299b9ae.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/2f505aac1142458699151d05a7a5a322.gif#pic_center)          | Both are supported                                           |
| Create Merge Request |  ![](https://img-blog.csdnimg.cn/b3e7ecd9aa3845228071da9c531cef68.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/342ad1a9d28049c2be2681503bae93ec.gif#pic_center)          | Both are supported                                           |
| PR Merge             |  ![](https://img-blog.csdnimg.cn/8d6af1b9c2d54f108e67ca479c648db9.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/66c1173189d940cc97d29a701a270703.gif#pic_center) ![](https://img-blog.csdnimg.cn/493067a77940453d817e5069bcb98199.gif#pic_center)          | CockroachDB and YugabyteDB have the same situation.<br/><br/>After submitting the merge request, the page continues to load, and after a period of time, the page shows an error message and cannot submit merge again, and there is no exception in the log. |
| Add Project Member   |  ![](https://img-blog.csdnimg.cn/3b91377878c4401793a866b4cb5a9849.gif#pic_center)           |  ![](https://img-blog.csdnimg.cn/77d1ca0d0e97470cb6504cf586ab8cc0.gif#pic_center)          | Both are supported                                           |

## Test  Conclusion

1、CockroachDB has 3 final failures in all tested scenarios, which are New Project, Import Existing GitLab Project, PR Merge.

2、YugabyteDB has 1 final failure in all the tested scenarios, namely PR Merge.

From the results of this test, YugabyteDB is more compatible with GitLab. It is necessary to further investigate whether the PR Merge error is related to the database.

## The Next Step

The next step will be to analyze and locate the problems found in this test, and then try to minimally modify the Gitlab source code to see if it is compatible with the test failure scenario.