Growing the PHP Core—One Test at a Time
Tags: php, testing, contributorswanted
Published: 2020-09-22 08:58:27

In September 2000, I started my vocational training at an internet agency with two teams: one doing JavaServer Pages, and one was doing PHP. I was assigned to the PHP team, and when presented with the language, I immediately knew that no one will ever use this. I was wrong. Today, my entire career is built on PHP. It’s time to give back to the community by writing tests for the PHP core itself!

## Prepare Your machine!

Before you start writing tests for PHP, let’s start with running the tests that already exist. Fetch the PHP source from GitHub and compile it to do so.

``` bash
$ git clone git@github.com:php/php-src.git
$ cd php-src
$ ./buildconf
$ ./configure --with-zlib
$ make -j `nproc`
```

I recommend creating a fork upfront because it makes creating a pull request with your test easier, later on.

If you do not have a compiler and build tools already installed on your Linux computer, you should install the `development-tools` group on Fedora or the `build-essential` on Debian Linux. The `./configure` command may exit with an error condition; this usually occurs when build requirements are not met, so install whatever `./configure` is missing and re-run that step. Keep in mind that you need to install the development packages — in my case, the `configure` script stated it was missing `libxml`, which was, in fact, installed. What it really missed was the header files, which are in the development package (usually named with a dev or devel suffix). Note that the `--with-zlib` is mandatory in this case, as you need the zlib extension which is not built by default.

After your build is complete, you can find the PHP binary in `./sapi/cli/php`. Go ahead and check what you just created:

``` bash
$ ./sapi/cli/php –v
PHP 8.4.0-dev (cli) (built: Feb 21 2024 16:29:11) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.4.0-dev, Copyright (c) Zend Technologies
```

Now that you have a freshly built and running PHP binary, you can finally run the tests included within the GitHub repository. Since PHP 7.4 tests may run in parallel, you can give the number of parallel jobs with the `-j` argument.

``` bash
$ make TEST_PHP_ARGS=-j`nproc` test
...
=============================================================
TEST RESULT SUMMARY
-------------------------------------------------------------
Exts skipped    :   46
Exts tested     :   26
-------------------------------------------------------------
Number of tests : 15670             10668
Tests skipped   : 5002 ( 31.9%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :   32 (  0.2%) (  0.3%)
Tests passed    : 10636 ( 67.9%) ( 99.7%)
-------------------------------------------------------------
Time taken      :   76 seconds
=============================================================
```

This looks good and it only took 76 seconds to run 10636 tests — quite fast. Zero warnings and zero failures, hooray! You can see that 5002 tests have been skipped and 32 have been expected to fail. You will learn about why this is in the next part.

## What Do Tests Look Like?

Let’s take a look at what such a test case may look like. PHP test files have the file ending `.phpt` and consist of several sections, three of which are mandatory: the `TEST`, the `FILE` and the `EXPECT` or `EXPECTF` section.

``` php
--TEST--
strlen() function
--FILE--
<?php
var_dump(strlen('Hello World!'));
?>
--EXPECT--
int(12)
```

#### The TEST Section

This is a short description of what you are testing. It should be as short as possible, as this is what is printed when running the tests. You can put additional details in the `DESCRIPTION` section.

#### The SKIPIF Section

This section is optional and is executed by the PHP binary under test. If the resulting output starts with the word `skip`, the test case is skipped. If it starts with `xfail` the test case is run, but it is expected to fail.

#### The FILE Section

This is the actual test case, PHP code enclosed by PHP tags. This is where you do whatever you want to test. As you have to create output that is then matched against your expectation, you usually find `var_dump` all over the place.

#### The EXPECT Section

This section must exactly match the output from the executed code in the `FILE` section to pass.

#### The EXPECTF Section

`EXPECTF` can be used as an alternative to the `EXPECT` section and allows the usage of substitution tags you may know from the `printf` functions family:

``` php
--EXPECTF--
int(%d)
```

#### The XFAIL Section 

`XFAIL` is another optional section. If present, the test case is expected to fail; you don’t need to echo out `xfail` in the `SKIPIF` section at all if you have this. It contains a description of why this test case is expected to fail. This feature is mainly used if the test is already finished, but the implementation isn’t yet, or for upstream bugs. It usually contains a link to where a discussion about this topic can be found.

#### The CLEAN Section

This exists so you can clean up after yourself. An example might be temporary files you created during the test. Keep in mind, this section's code is executed independently from the `FILES` section, so you have no access to variables declared over there. Also this section is run regardless of the outcome of the test.

#### What else?

You may find more in depth details on the file format at the [PHP Quality Assurance Team Web Page](http://qa.php.net).

## What Can You Test?

Now that you know what a PHP test looks like it is time to find something to test. Head over to [codecov.io](https://app.codecov.io/github/php/php-src).

When I started looking for something to test, I found that the `zlib_get_coding_type()` function in `ext/zlib/zlib.c` was not covered at all (this was back in the PHP 7.1 branch and on the now retired [gcov.php.net](http://gcov.php.net/PHP_7_1/lcov_html/ext/zlib/zlib.c.gcov.php#543)).

![Code coverage report shows the `zlib_get_coding_type()` function as completely uncovered.](https://dev-to-uploads.s3.amazonaws.com/i/bf9ha1cqkxapkkonjoq7.png)

The next step was to check out what this function was supposed to do in the [PHP Documentation](https://www.php.net/manual/en/function.zlib-get-coding-type.php). What I saw in the documentation, but also in the code itself, was that this function returns the string `gzip` or `deflate` or the Boolean value `false`. The linked [zlib.output_compression directive documentation](https://www.php.net/manual/en/zlib.configuration.php#ini.zlib.output-compression) gave one additional bit of information: the zlib output compression feature reacts on the HTTP `Accept-Encoding` header sent with the client HTTP request.

For the test, this means there are four possible cases to check for:

- the absence of the Accept-Encoding header
- the Accept-Encoding being `gzip`
- the Accept-Encoding being `deflate`
- the Accept-Encoding being anything else

The last case is treated the same as the first case: The function is expected to return the Boolean value `false`.

## Time To Write That Test!

Let's start with the first test in the file `test/zlib_get_coding_type.phpt` for the case that there is a `Accept-Encoding` header set to `gzip`. One question that might arise now is: How do I add an HTTP-Request in the tests? The answer is: you don't, but we know that HTTP-Request headers will be exposed via the `$_SERVER` super global, so we can inject the fake HTTP header there.

``` php
--TEST--
zlib_get_coding_type()
--EXTENSIONS--
zlib
--FILE--
<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
var_dump(zlib_get_coding_type());
ini_set('zlib.output_compression', 'On');
var_dump(zlib_get_coding_type());
?>
--EXPECT--
bool(false)
string(4) "gzip"
```

You can run this single test via:

``` bash
$ make test TESTS=test/zlib_get_coding_type.phpt
```

Sadly, this gives you a failed test. You can easily see why this is the case, by checking the `test` directory contents where you find your `.phpt` test file and some other files with various file endings. One that is particularly interesting, in this case, is the `zlib_get_coding_type.log` file:

```
---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
Warning: ini_set(): Cannot change zlib.output_compression - headers already sent in test/zlib_get_coding_type.php on line 4
bool(false)
---- FAILED
```

And this is your first time learning about PHP internals. You cannot change the output compression setting after your script created any form of output. For this to work, there seems to be a handler that is called when we change the `zlib.output_compression` directive. Try searching for "zlib.output_compression" in the `ext/zlib/zlib.c` source code file. You find it in the call to the `STD_PHP_INI_BOOLEAN` macro along with the pointer to the function `OnUpdate_zlib_output_compression` in which you can spot the warning you received. To work around this warning, you can change the `FILE` section:

``` php
<?php
$_SERVER["HTTP_ACCEPT_ENCODING"] = "gzip";
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$on = zlib_get_coding_type();
var_dump($off, $on);
?>
```

Rerunning the test case results in yet another failed test. But this time you know where to look and you will find this:

```
---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
bool(false)
bool(false)
---- FAILED
```

The warning is gone, but test failed again. That's progress!

Looking into the `zlib_get_coding_type.log` file, you notice that the second call to the `zlib_get_coding_type()` function returned `false` and not the expected string `gzip`. It seems like PHP is not reacting to the HTTP header that you clearly set just few lines before. Did we spot a bug in PHP?

The source code tells you: Open the `ext/zlib/zlib.c` source code file and search for the variable `compression_coding` as this is the variable that is evaluated in the function you are testing. You should find some matches, but there is one at the beginning of the file in the C function `php_zlib_output_encoding` which looks like the only place where something is assigned to the variable in question.

Analysing the source code, not understanding exactly what is going on, and finally asking for help on the [PHP Community Chat](https://phpc.chat) reveals the following: All of your userspace variables and PHP user-accessible autoglobals (`$_GET`, `$_SERVER`, ...) are copy-on-write. Altering those from your script creates a copy for you to work on in userspace, effectively **making the HTTP request headers immutable to the userspace**. The PHP core continues to use the original, unaltered version.

Now that you know you can not alter or set the HTTP header from within your script to influence PHP’s behavior, this might look like a sad end for your tests code coverage.

Wait, there is more! I did not tell you about another section in the PHPT file format that might come in handy now: The `ENV` section. This section is used to give environment variables to your script and are passed to the PHP process that runs your test code. This means you can go on and that you have to create a test file per test case. Let’s create a new test file for the gzip case and name it `zlib_get_coding_type_gzip.phpt`.

``` php
--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"
```

Run the test and see it fails once more. You might have expected this by now. 😝

Let’s see what happened by looking into the `zlib_get_coding_type_gzip.log` file.

```
---- EXPECTED OUTPUT
bool(false)
string(4) "gzip"
---- ACTUAL OUTPUT
�1�0
    ���B�P�Txꆘ8`5ؑ������s�7a�ze��B+���ϑ�,����^���~�^�J1����ʶ`�\0�v@cm��}�ap�X}��'4�ͩqG�^w
---- FAILED
```

Wow, this looks like some binary garbage, and yeah, this does not match the expected output.

Look at your test case. You told PHP that you accept gzip encoding and you turned on output compression. PHP is basically just doing what we told it to do. The binary garbage you see here is gzip encoded data. This means you succeeded in activating gzip output compression, but forgot to turn it off before dumping out the two variables. Now, all that’s left is to add another call to deactivate output compression again so the final test looks similar to this:

``` php
--TEST--
zlib_get_coding_type() is gzip
--EXTENSIONS--
zlib
--ENV--
HTTP_ACCEPT_ENCODING=gzip
--FILE--
<?php
ini_set('zlib.output_compression', 'Off');
$off = zlib_get_coding_type();
ini_set('zlib.output_compression', 'On');
$gzip = zlib_get_coding_type();
ini_set('zlib.output_compression', 'Off');
var_dump($off);
var_dump($gzip);
?>
--EXPECT--
bool(false)
string(4) "gzip"
```

Run this test again, and it finally passes! 🎊🥳🎉

This leaves you with writing one test case for the `Accept-Encoding` header being set to `deflate` and another test case for the `Accept-Encoding` header being set to an invalid string (basically something that is not `gzip` or `deflate`). But this is just diligence from this point on and if you like to cheat you can find the tests in the `ext/zlib/tests/` directory named `zlib_get_coding_type_basic.phpt`, `zlib_get_coding_type_br.phpt`, `zlib_get_coding_type_deflate.phpt` and `zlib_get_coding_type_gzip.phpt`.

## Collect Some Evidence!

Now that you have tests for all four cases you could hope you covered that function with tests or (a way better option if you ask me), you could continue and create a code coverage report! For this to work you need to install the `lcov` tool via your Linux package manager (`sudo dnf install lcov` for Fedora or `sudo apt-get install lcov` for Debian). As PHP was built with `gcov` disabled, we need to run configure again, with enabled `gcov` and recompile.

``` bash
$ make clean
$ CCACHE_DISABLE=1 ./configure --with-zlib --enable-gcov
$ make -j `nproc`
```

This compiles the PHP binary with enabled code coverage generation. Lcov will be used to create an HTML code coverage report afterwards. To create your code coverage report, you need to rerun the tests.

``` bash
$ make test TESTS=test/zlib_get_coding_type.phpt
```

While running the tests, `gcov` generates coverage files with a `gcda` file ending for every source code file. To generate the HTML code coverage report, run the following:

``` bash
$ make lcov
```

You may find the resulting HTML code coverage report in `lcov_html/index.html`.

![Code coverage report now shows 9 of 10 lines covered from the `zlib_get_coding_type()` function](https://dev-to-uploads.s3.amazonaws.com/i/d0w4ro1jiikjms3u5k01.png)

It looks like we covered all four cases.

## What’s in It for You?

With every test you write PHP becomes more stable and reliable, as functions may not change behavior suddenly between releases.

Writing tests for PHP gives you a deeper understanding of how your favorite language works internally. In fact, by reading up to this point you learned that HTTP request headers are immutable to userspace, that there is such a thing as userspace, and that there are handler functions called and check what you are doing when you want to change ini settings at runtime.

Also knowing the PHPT file format gives you other benefits as well: [PHPUnit](https://phpunit.de) not only supports the PHPT file format and can execute those tests, part of PHPUnit is tested with PHPT test cases. This led me to become a contributor to PHPUnit: Writig tests for PHPUnit in the PHPT file format.

## Shout Out!

I would like to thank everyone involved in the [PHP TestFest](https://phptestfest.org/start/#whos-behind-this), but especially [Ben Ramsey](https://twitter.com/ramsey), [Sammy Kaye Powers](https://www.sammyk.me/) and everyone else involved in the organization of the 2017 edition. This was what made me write tests for PHP and made me not only a PHP but also a PHPUnit contributor. In the long run, this brought me my first ElePHPant 🐘!

#### Closing notes

I would like to thank my proof readers [Kara Ferguson](http://www.karaferguson.net/) and [Pim Elsfhof](https://www.pelshoff.com/). It was a pleasure working with you!

"Wooble" is the property of [Sammy Kaye Powers](https://www.sammyk.me/) and is used with permission.

The German version is available through the [PHP Magazin](https://kiosk.entwickler.de/php-magazin/php-magazin-6-2020/php-core-erweitern-test-fuer-test/)

Interested in seeing the [talk to this blog post](https://nomadphp.com/video/305/Growing-the-PHP-Core--one-Test-at-a-Time)? Nomad PHP has you covered.