Summary of Chapter# 8 : "Buffer Manager" from the book "The Internals of PostgreSQL" Part-2
Tags: postgres, apacheage, database, agenssql
Published: 2023-08-01 08:14:23

This blog aims to assist you in understanding the medial concepts of Chapter:8 [Buffer Manager] from the book **_[The Internals of PostgreSQL](https://www.interdb.jp/pg/)_**.

> _**Note: Ensure that you have a thorough understanding of 
 [Chapter 8 Part-1](https://dev.to/vkt1271/summary-of-chapter-8-buffer-manager-from-the-book-the-internals-of-postgresql-4i2d) and basics of PostreSQL before we proceed to Chapter 8 Part-2, as it forms the foundation for our exploration.**_

So, Let's Continue:

**Buffer Descriptors Layer**

- A collection of buffer descriptors forms an **array**. In this document, the array is referred to as the **buffer descriptors layer**.

- When the PostgreSQL server starts, the state of all buffer descriptors is **empty**. In PostgreSQL, those descriptors comprise a linked list called **freelist**.

> **Buffer manager initial state** in PostgreSQL is depicted in figure below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l9kzfucwbtpo6f9ro86s.png)

> **Loading the first page** in PostgreSQL is depicted in figure below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8535t7pc1qfoczzmshcg.png)

**Buffer Pool**

- The buffer pool is a simple **array** that stores data file pages, such as _tables and indexes_. Indices of the buffer pool array are referred to as **buffer_ids**.

- The buffer pool slot size is **8 KB**, which is equal to the size of a page. Thus, each slot can store an entire page.

---------------------------------------------------------------

## Buffer manager locks

**Buffer Table Locks**

- **BufMappingLock** protects the data integrity of the entire buffer table.

- It is a **light-weight** lock that can be used in both shared and exclusive modes. When searching an entry in the buffer table, a backend process holds a shared **BufMappingLock**.

- The BufMappingLock is split into **partitions** to reduce the contention in the buffer table (the default is **128 partitions**). Each BufMappingLock partition guards the portion of the corresponding hash bucket slots.

> **Two processes simultaneously acquire the respective partitions of BufMappingLock in exclusive mode to insert new data entries** in PostgreSQL is depicted in figure below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fyy4habxdgwgh3ru9bsc.png)

**Locks for Each Buffer Descriptor**

- Each buffer descriptor uses **two** light-weight locks, **content_lock** and **io_in_progress_lock**, to control access to the stored page in the corresponding buffer pool slot. When the values of own fields are checked or changed, a **spinlock** is used.

**Content_lock**

- The content_lock is a typical lock that enforces access limits. It can be used in **shared** and **exclusive** modes.

- When reading a page, a backend process acquires a shared **content_lock** of the buffer descriptor that stores the page.

- **Exclusive content_lock** is acquired in the following scenarios:

1. Inserting rows (tuples) into the stored page.
2. Changing the t_xmin/t_xmax fields of tuples within the stored page.
3. Removing tuples physically or compacting free space on the stored page.
4. Freezing tuples within the stored page.

**io_in_progress_lock**

- The io_in_progress lock is used to wait for **I/O** on a buffer to complete. 

- When a PostgreSQL process loads/writes page data from/to storage, the process holds an exclusive **io_in_progress lock** of the corresponding descriptor while accessing the storage.

**Spinlock**

- When the flags or other fields (e.g. refcount and usage_count) are checked or changed, a **spinlock** is used. 

- **Two** specific examples of spinlock usage are given below:

`1. The following shows how to pin the buffer descriptor:`

- Acquire a spinlock of the buffer descriptor.
- Increase the values of its refcount and usage_count by 1.
- Release the spinlock.

`2. The following shows how to set the dirty bit to '1':`

- Acquire a spinlock of the buffer descriptor.
- Set the dirty bit to '1' using a bitwise operation.
- Release the spinlock.

------------------------------------------------------------------

## How the buffer manager works

- When a backend process wants to access a desired page, it calls the **ReadBufferExtended** function.

- The behavior of the ReadBufferExtended function depends on **three** logical cases.

**1. Accessing a Page Stored in the Buffer Pool**

- First, the simplest case is described, i.e. the desired page is _already_ stored in the **buffer pool**.

-  In this case, the buffer manager performs the following steps:


- (1) Create the buffer_tag of the desired page (in this example, the buffer_tag is 'Tag_C') and compute the hash bucket slot, which contains the associated entry of the created buffer_tag, using the hash function.


- (2) Acquire the BufMappingLock partition that covers the obtained hash bucket slot in shared mode (this lock will be released in step (5)).


- (3) Look up the entry whose tag is 'Tag_C' and obtain the buffer_id from the entry. In this example, the buffer_id is 2.


- (4) Pin the buffer descriptor for buffer_id 2, i.e. the refcount and usage_count of the descriptor are increased by 1.


- (5) Release the BufMappingLock.


- (6) Access the buffer pool slot with buffer_id 2.

> **Accessing a page stored in the buffer pool** in PostgreSQL is depicted in figure below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/no9l8qnn8aojb0sfp57c.png)

- When reading rows from the page in the buffer pool slot, the PostgreSQL process acquires the shared **content_lock** of the corresponding buffer descriptor. Thus, **buffer pool** slots can be read by multiple processes simultaneously.

- When inserting (and updating or deleting) rows to the page, a Postgres process acquires the **exclusive content_lock** of the corresponding buffer descriptor (**note** that the dirty bit of the page must be set to '1').

- After accessing the pages, the refcount values of the corresponding buffer descriptors are **decreased by 1**.

**2. Loading a Page from Storage to Empty Slot**

- In this second case, assume that the desired page is **not** in the buffer pool and the freelist has free elements (empty descriptors). 

- In this case, the buffer manager performs the following steps:

- (1) Look up the buffer table (we assume it is not found).

1. Create the buffer_tag of the desired page (in this example, the buffer_tag is 'Tag_E') and compute the hash bucket slot.
2. Acquire the BufMappingLock partition in shared mode.
3. Look up the buffer table (not found according to the assumption).
4. Release the BufMappingLock.

- (2) Obtain the empty buffer descriptor from the freelist, and pin it. In this example, the buffer_id of the obtained descriptor is 4.

- (3) Acquire the BufMappingLock partition in exclusive mode (this lock will be released in step (6)).

- (4) Create a new data entry that comprises the buffer_tag 'Tag_E' and buffer_id 4; insert the created entry to the buffer table.

- (5) Load the desired page data from storage to the buffer pool slot with buffer_id 4 as follows:

1. Acquire the exclusive io_in_progress_lock of the corresponding descriptor.

2. Set the io_in_progress bit of the corresponding descriptor to '1 to prevent access by other processes.

3. Load the desired page data from storage to the buffer pool slot.

4. Change the states of the corresponding descriptor; the io_in_progress bit is set to '0', and the valid bit is set to '1'.

5. Release the io_in_progress_lock.

- (6) Release the BufMappingLock.
- (7) Access the buffer pool slot with buffer_id 4.

> **Loading a page from storage to an empty slot** in PostgreSQL is depicted in figure below:

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/70fe2mljm7lxk018z4bw.png)

-----------------------------------------------------------------

I hope, this blog has helped you in understanding the medial concepts of Buffer Manager in PostreSQL.

Check out summary of [Chapter : 8 Part-3](https://dev.to/vkt1271/summary-of-chapter-8-buffer-manager-from-the-book-the-internals-of-postgresql-part-3-3fhj)

If you want to understand [PostgreSQL In-Depth](https://www.youtube.com/playlist?list=PLhr1KZpdzukeAZuX0pNZ1lLDkaZWBOhQT).