The YugabyteDB configuration parameters (GFLAGs and GUCs)
Tags: yugabytedb, postgres, distributed, database
Published: 2022-07-05 10:53:52


A database is a complex software with many layers and, when distributed, multiple instances. As with any software, there's a lot of configuration knobs, at many levels. People used to them, like database engineers, DBA, SRE may just mention their name thinking everybody knows what they are talking about. But for a developer, who doesn't spend her day tuning the knobs, it must be clear on how and where to do it.

In short, I'll describe the settings that concern:
- the whole **YugabyteDB universe** (a cluster of shared-nothing nodes connected through a public or private network). They are defined when starting the YugabyteDB processes (`yb-master` and `yb-tserver`) with some possibilities to update them later. Geeks reference them as 'G-Flags'.
- the **PostgreSQL backend** (which provides the YSQL API). They are defined, like in Postgres, with the `SET` command in your session, with many ways to define the defaults at higher level. Geeks regerence them as 'GUCs'.

Here, I'll reference them as _YugabyteDB setting (or GFLAG)_ and _PostgreSQL setting (or GUC)_ and I may update this post with more questions coming.

## PostgreSQL setting (or GUC)

In the PostgreSQL community, the PostgreSQL setting parameters are often called GUC, the name of the 'Grand Unified Configuration scheme' project that introduced them. They can be displayed with [SHOW](https://docs.yugabyte.com/preview/api/ysql/the-sql-language/statements/cmd_show/) or by calling `current_setting()`, are visible from `pg_settings`. They can be set in a session with [SET](https://docs.yugabyte.com/preview/api/ysql/the-sql-language/statements/cmd_set/). The default value can also be set for a user with `ALTER USER ... SET`or for a database with `ALTER DATABASE ... SET`. The default values for a universe can also be set globally, as a comma separated list with the `ysql_pg_conf_csv` YugabyteDB setting (or GFLAG) but we will see that after explaining those.

As YugabyteDB reuses PostgreSQL you will find all PostgreSQL settings (or GUC) in YSQL but they are not all relevant to set. For example you will not set `shared_buffers` or WAL settings because YugabyteDB uses a distributed storage which has its own caching and logging.

There are also some additional settings specific to YugabyteDB. They often start with `yb_` but, as both YugabyteDB and PostgreSQL are open source, you can also compare the `guc.c` definition. This is a quick awk script I use to compare PostgreSQL 11.2 with the master branch of YugabyteDB:

```awk
awk -F '"' '
/^static struct config_bool/ {
 struct=$0 
}
/^};/ {
 struct = "" ; guc="" 
}
/{"/ && struct=="static struct config_bool ConfigureNamesBool[] =" {
 guc=$2
}
/},/{
 guc=""
}
/gettext_noop/ && guc!="" {
 # record when first encountered
 if ( gucs[guc]=="" ) gucs[guc]=FILENAME
 desc[guc]=$2
}
END{
 for (i in gucs) if (length(i)>maxlen) maxlen=length(i)
 # list only first encountered in the last file
 for (i in gucs) if ( gucs[i]==FILENAME) printf "%-"maxlen"s  %-s\n", i ,desc[i]
}
' <(
# PostgreSQL
curl -s https://raw.githubusercontent.com/postgres/postgres/REL_11_STABLE/src/backend/utils/misc/guc.c
) <(
# YugabyteDB
curl -s https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/src/postgres/src/backend/utils/misc/guc.c
) | sort

```
This shows the settings that are specific to YugabyteDB:
```
force_global_transaction                      Forces use of global transaction table.
suppress_nonpg_logs                           Suppresses non-Postgres logs from appearing in the Postgres log file.
yb_binary_restore                             Enter a special mode designed specifically for YSQL binary restore.
yb_debug_log_catcache_events                  Log details for every catalog cache event such as a cache miss or cache invalidation/refresh.
yb_debug_log_docdb_requests                   Log the contents of all internal (protobuf) requests to DocDB.
yb_debug_log_internal_restarts                Log details for internal restarts such as read-restarts, cache-invalidation restarts, or txn restarts.
yb_debug_report_error_stacktrace              Append stacktrace information for error messages.
yb_disable_transactional_writes               Sets the boolean flag to disable transaction writes.
yb_enable_create_with_table_oid               Enables the ability to set table oids when creating tables or indexes.
yb_enable_expression_pushdown                 Push supported expressions down to DocDB for evaluation.
yb_enable_geolocation_costing                 Allow the optimizer to cost and choose between duplicate indexes based on locality
yb_enable_optimizer_statistics                Enables use postgres selectivity model.
yb_enable_upsert_mode                         Sets the boolean flag to enable or disable upsert mode for writes.
yb_format_funcs_include_yb_metadata           Include DocDB metadata (such as tablet splits) in formatting functions exporting system catalog information.
yb_non_ddl_txn_for_sys_tables_allowed         Enables the use of regular transactions for operating on system catalog tables in case a DDL transaction has not been started.
yb_planner_custom_plan_for_partition_pruning  If enabled, choose custom plan over generic plan
yb_read_from_followers                        Allow any statement that generates a read request to go to any node.
yb_test_fail_next_ddl                         When set, the next DDL (only CREATE TABLE for now)
yb_test_system_catalogs_creation              Relaxes some internal sanity checks for system catalogs to
ysql_upgrade_mode                             Enter a special mode designed specifically for YSQL cluster upgrades.
```

Always check the documentation or Yugabyte Community channels (https://www.yugabyte.com/community) as some settings are for specific situations only.

## YugabyteDB setting (or GFLAG)

The above was limited to the YSQL layer. The settings that are global to all Yugabyte layers are set differently. You will find them referenced as "flags" like the [yugabyted](https://docs.yugabyte.com/preview/reference/configuration/yugabyted/#start) `--master_flags master_flags` and `--master_flags tserver_flags` and sometimes G-Flags:

![YugabyteDB Anywhere](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2b0ixhpevkugbv3m8n20.png)
 
You also find them in the `/varz` endpoint on the master and tserver. 

The YugabyteDB settings (or GFLAGs) are set when starting the yb-master and yb-tserver. Some parameters are common to the master and tserver. Some make sense only for one of them. The [reference documentation](https://docs.yugabyte.com/preview/reference/configuration/) is the place to read about them. Some parameters can be changed dynamically with [yb-ts-cli ](https://docs.yugabyte.com/preview/admin/yb-ts-cli/#set-flag) but the right way for persistent change is to set them at startup. And, this can be done while the application is running thanks to rolling restarts. You probably want to set the same value on all nodes, except the placement info ones, and the IP addresses.

## ysql_pg_conf_csv

This is a YugabyteDB settings (or GFLAGs) that can set the default PostgreSQL setting (or GUC) by adding parameters to the `ysql_pg.conf`, the YugabyteDB equivalent of `postgresql.conf`. The value in `ysql_pg_conf_csv` declares them as Comma-Separated values. Then each `name=value` can be enclosed in double-quotes if, for example, there's a comma in the value. If they contain a double-quote, it must be doubled. And, as you put that in a bash command line, you may enclose all that with single quotes. Then, if there are single quotes in the value, I close the outer bash single quotes, and open double quotes to put the single quote not to be interpreted by bash...

Ok, here is an example that will add `search_path = '"$user", public'` in `/yb/pg_data/ysql_pg.conf`:
```sh
  yb-tserver --fs_data_dirs /yb --ysql_pg_conf_csv \
  '"search_path='"'"'""$user"",public'"'"'",yb_enable_expression_pushdown=on' 
# '             '" "'                '" "'   quotes for bash                '
#                 '                    '     single quotes (protected by double quotes)
#  "                                      "  doule quote protection for CSV value that contains commas
#                    ""     ""               doubled double-quotes (doubled as their are inside the CSV ones)
```
I have put, in comments, some explanations. But if you are not as fan as I am with escaping quotes in the command line, you can also put flags in file referenced by `--flagfile`. For example:
```sh

cat > flagfile.txt <<'CAT'
--ysql_pg_conf_csv="search_path='""$user"",public'",yb_enable_expression_pushdown=on
CAT

yb-tserver --fs_data_dirs /yb --flagfile flagfile.txt

```

From `yugabyted`, this might require additional escapes (the CSV list being enclosed between `{` and `}`), either in `--tserver_flags` or in the JSON configuration file. Be careful that `{` and `}` are interpreted by bash to expand a comma separated list and may have to be escaped.

To make it simpler, I use `yugabyted` to quick-start my labs. When I need more control on flags, I use a single `--tserver_flags=flagfile=./tserver.flags` set with flags in this file:
```sh
cat > ./tserver.flags <<'GFLAGS'
--ysql_enable_packed_row=true
--ysql_beta_features=true
--yb_enable_read_committed_isolation=true
--enable_deadlock_detection=true
--enable_wait_queues=true
--ysql_beta_feature_tablespace_alteration=true
--ysql_pg_conf_csv=log_statement='all',shared_preload_libraries="'auto_explain','pg_stat_monitor'"
GFLAGS
``` 

But if you want to put all in a single line, this should also work: `--tserver_flags="ysql_enable_packed_row=true,ysql_beta_features=true,yb_enable_read_committed_isolation=true,enable_deadlock_detection=true,enable_wait_queues=true,ysql_beta_feature_tablespace_alteration=true,ysql_pg_conf_csv=\{log_statement='all',shared_preload_libraries="'auto_explain','pg_stat_monitor'"}\"`


If you find this too complex, yes, it is. Escaping special characters, including those used to escape, has always been a mess—except with PostgreSQL strings. I would like to see any software use a PostgreSQL dollar-quoted string.