Repository: citusdata/citus

Issue #890: Infrastructure to automate release testing
Created: 2016-10-19T16:46:08Z
Building an infrastructure to automate our release testing is a comprehensive topic. I'm creating this as a parent issue to track all our release testing components (in addition to regression tests). We should then incrementally build an infrastructure to automate as much of the testing as possible.
We currently have a regression testing framework that we run after each build. Before each release, we also perform the following tests.
Citus 5.2 Release Test Plan

Test setup
Test instructions on documentation
Basic feature testing
Basic Integration testing
Specific integration testing
Feature Testing

New Features
Regressions


Failure testing
Concurrency testing

DML and Select Testing
DDL Testing


Performance testing

Select Performance
DML Performance


Memory testing (Valgrind)
Upgrade testing
Code coverage testing

Test setup

Multi node: (3+1) with different OSes on each (ubuntu, redhat), aws m3.large

ubuntu
redhat


Enable logging on both master and workers

Test instructions on documentation

Test with ASSERT_ENABLED

Tutorials (skipping not release related - binaries already in the downloaded folder)
Installation - Manual installation from source
Distributed Tables
Performance  - 5.2 release related parts
Administration (there is already a separate upgrade test task)



Basic feature testing

Verify that there are no warnings while compiling
Run all regression tests and verify that they pass. This test should be executed with and without asserts are enabled

make installcheck (make sure postgresql regression tests are running OK)
make check-full
make check-multi-hll (this requires downloading hll)


Create TPC-H tables and COPY TPC-H data - Verify metadata and worker node tables.  scale factor = 10, replication factor 2

Hash partitioned table - partition on an id column
Append partitioned table - partition on id column


Verify results of all supported TPC-H queries against regular Postgresql tables. (With some rounding errors)
JDBC: run select / insert / update / delete statements through jdbc connection

Use 3 different executors (router, real time, task tracker) for SELECTs



Basic Integration testing
Test Citus integration with the extensions

shard rebalancer -  (Regression and Basic Behaviour Tests)

Regression tests are failing. https://github.com/citusdata/shard_rebalancer/pull/35.
Basic functionality is okay for the branch “add_schema_support”


session analytics + hstore (Regression and  Basic Behaviour Tests)
cstore_fdw (Regression and Basic Behaviour Tests)

Block_filtering fails on Ubuntu (Known issue)


HLL (Regression Basic Behaviour Tests)

HLL Regressions are also failing on PostgreSQL 9.5.3 without Citus.



For all extensions, verify the following:

Extension regression tests pass
Basic behaviour test

Specific integration testing
Shard rebalancer - test on add_schema_support branch

Install shard rebalancer on the master node
Rebalance_table_shards
Replicate_table_shards
Concurrency. Verify that the following concurrent operations produce a consistent result:

rebalance_table_shards + replicate_table_shards (on the same table)
master_append_table_to_shard + rebalance_table_shards (on the same table. make sure the shard being added to will be moved
master_append_table_to_shard + replicate_table_shards
master_apply_delete_command + [rebalance_table_shards or replicate_table_shards]
Master_modify_multiple_shards +  [rebalance_table_shards or replicate_table_shards]


Test the behavior during concurrent insert operation
Verify that the canceling leaves the data in a consistent state

Test both replicate_table_shards & rebalance_table_shards



Feature Testing
New Features

master_modify_multiple_shards() feature. PR #486
With failure scenarios on 2PC
Shard placement policy aware master_create_empty_shard() PR #489
Complex count distinct support PR #516
Allow users to run unsupported DDL statements manually PR #548
Support tables with special characters PR #588
Comprehensive schema support #122 (Test every issue reported here)
Expand router executable support to recognize all "single machine" queries Issue #501, #448, PRs #647, #651, #610
Support RETURNING PR #578
Support STABLE functions in INSERT statements PR #488
LIMIT/OFFSET Support PR #596
Router executor prepare support for non-partition columns PR #649
Support single Insert, Update, Delete statements wrapped within transaction blocks Issue #505
Support Sequences PR #673

Regressions

Check that all citus functions are created in pg_catalog both for fresh install and upgrade. Fix for master_update_shard_statistics() is in PR #570
Check that we don’t send ONLY keyword to worker nodes PR #544
Update only shard length on statistics update for hash-partitioned PR #576
Improve error handling during failed modification PR #552
Check that we create correct storage type for foreign tables PR #539
Fix crash in count distinct with filters in repartition subqueries #579
Fix wrongly distributed LEFT JOIN quals PR #603
Fix join pruning performance #608 (Observe the difference between 5.1 and 5.2 for high number of shards)
Fix for `flex' is missing on your system. PR #574
Fix for PQresultErrorField() returning NULL causes crashes in router exec PR #636
Fix for hash-distributed anti-joins return wrong results Issue #628
Fix for COPY produces error when using hstore[] #463
Use 2PC to propagate DDL commands PR #618
Fixes for outer join crash when subquery is flatten PR #660
Fix for real-time executor blocks if all workers are down Issue #580
Test performance benefits of using remote_task_check_interval less. PR #391. In the PR, there is already some comments on how to measure the performance changes.

Create and empty distributed table with different number of shards
Run SELECT queries with real-time executor with the same remote_task_interval_check and see the performance change
Load some data and run the test again



Failure testing

Fail some nodes while running a query, but all shards have at least one active placements

Do this with real time executor: aggregates, simple joins, table fetching queries, functions
Do this with task tracker executor queries - Look at the task tracker design document for identifying/testing various failure scenarios: aggregates, simple joins, re-partition joins. Tested with all regression tests (a) terminate some workers randomly and start again (b) terminate workers randomly and not start again until query finished
Do this with router executor queries: Aggregates, window functions, simple-key value lookups


Fail some nodes while running a query, one shard doesn’t have any active placements
COPY on hash partitioned tables with default parameters (1PC)

If worker any worker is not active before COPY starts, mark placements on that workers as INACTIVE
If one worker fails during COPY, COPY errors out: Hash, Range, Append
If one worker fails during COMMITs, there is no recovery mechanism. Some of the data is already committed
If data is corrupted (i.e., 1 row is erroneous), COPY errors-out and no data is ingested to the workers


COPY on hash partitioned tables with (2PC)

If worker any worker is not active before COPY starts, mark placements on that workers as INACTIVE
If one worker fails during COPY, COPY errors out
If one worker fails during COMMITs, try to commit the remaining worker COPYies offline
If data is corrupted (i.e., 1 row is erroneous), COPY errors-out and no data is ingested to the workers


COPY on append partitioned failures

Shard creation fails: Create table with composite type and don’t create composite type on one node. Verify COPY handles failure as expected
Min/Max fetching fails: In the above don’t create min/max function for composite type on workers. Verify COPY handles failure as expected


Fail the source node when caching a table shard

Failed one of the source nodes and noted that the shard is fetched from the other replica
Failed the only source node and noted that query failed


Fail nodes during master_create_empty_shard

Fail one node and verify that it is able to create a shard. Verify metadata
Fail several nodes and verify that no metadata is inserted if it is not able to find shard_replication_factor nodes


Fail nodes during master_append_table_to_shard()

Fail all nodes having the shard and verify that we error out and no metadata is added
Fail some nodes having the shard and verify it is marked as invalid and metadata is updated accordingly
Ctrl-C during master_append_table_to_shard
If statistics fetching fails, verify that the metadata changes are as expected i.e. nodes to which append failed are marked as invalid: (a) Delete max/min functions from the workers, (b) shard is created, data is copied, pg_dist_shard contains null min/max


Fail nodes during master_apply_delete_command()

Verify that shards on failed nodes are not deleted and are marked as ‘to-delete’
Verify that pg_dist_shard_placement row is not deleted
Verify that shards on nodes which are up are deleted and their metadata is removed


Fail some nodes while running a subselect

Behavior should be same as executor behaviour
Run a single table subquery, single node failure is tolerated, 2 node failure caused query to fail: (a) Task-tracker and (b) Real-time


Fail some nodes while inserting/upserting/updating/deleting rows

test this by failing active, and inactive nodes


Fail on master_create_worker_shards

Set replication factor to 1 and stop one of the workers before the test starts. Ensure that all shard placements are created on the running worker
Set replication factor to 1 and stop one of the workers during the tests. In the end, ensure that all the shards are created on the two workers


Fail nodes during DDL command propagation

Verify that if node is failed before executing DDL command, then command can’t be executed
Verify that if node fails during execution or if you get a postgres error on a particular shard and not the others, the failed shards are marked as invalid



Concurrency testing
DML and Select Testing
Think of and try several concurrency scenarios like copy with queries,copy with table appends, appends with deletes, copy with deletes, master_modify_multiple_shards(), and so forth.

Copy, copy

On hash table
On append table


Copy, queries

Copy on hash first, then query
Copy on append first, then query
query first, then copy on hash
query first, then copy on append


Copy, master_append_table_to_shard

Copy on append first
master_append_table_to_shard first


copy, master_apply_delete_command

copy on hash first
Copy on append first
delete first, then copy on hash table (issue #288)
Delete first, then copy on append table


Copy, master_modify_multiple_shards()

Start copy, then master_modify_multiple_shards() before copy finishes
Start master_modify_multiple_shards(), then Copy before master_modify_multiple_shards() finishes


insert, master_append_table_to_shard
insert, delete
insert, query
insert, master_modify_multiple_shards()
copy, upsert
upsert, master_append_table_to_shard()
upsert, delete
upsert, query
upsert, insert
update and master_modify_multiple_shards() on the same row
delete and master_modify_multiple_shards() on the same row
two master_modify_multiple_shards() effecting same rows
two master_modify_multiple_shards() effecting different rows
query, query
query, master_append_table_to_shard

query first
append first


query, delete

query first: real-time executor, task-tracker executor, router executor
delete first: real-time executor, task-tracker executor, router executor


master_append_table_to_shard, master_append_table_to_shard
master_append_table_to_shard, master_apply_delete_command

append first
delete first


master_apply_delete_command, master_apply_delete_command
master_create_worker_shards, master_create_worker_shards
DML commands, master_copy_shard_placement UDF

Test DDL operations

Select, drop column
Copy on hash, add column
Copy on append, add column
Copy, drop column
repartition query, add column
repartition query, drop column
Copy on append, add index: Index creation is blocked until COPY finishes
Copy, drop index: Dropping index is blocked until COPY finishes

Performance testing
SELECT performance
Compare query times for each TPC-H query in 5.1 vs 5.2 (on postgres 9.5.3) using r3.2xlarge (1 + 3 nodes)

wget http://citus-benchmarks.s3.amazonaws.com/config/scripts/tpch/generate2.sh
SCALE_FACTOR=10 CHUNKS="o 24 c 4 P 1 S 4 s 1" sh generate2.sh
wget http://citus-benchmarks.s3.amazonaws.com/config/scripts/tpch/stage2.sh
emacs -nw stage2.sh (Edit psql path and use csql for \stage)
sh stage2.sh

RT: Benchmarked with real-time executor
TT: Benchmarked with task-tracker executor.
Note: One thought while going through it was our SELECT performance is tied to TPC-H right now. We aren't focusing on that as a workload, so it might be useful to rethink that for maybe router/real-time query performance.
DML Performance
For all DML tests:

c4.4xlarge (1 + 2)
pgbench -c 256 -j 32 -f insert.sql -n -t 5000 -P 5 postgres (on the master only, load ~1250000ls rows)
./pgbench -c 256 -j 32 -f update.sql -n -t 500 -P 5 postgres
All tables consists of only integer columns (partition col. is bigint only)
Sync Commit is off - kept this on
(for pg 5, max_wal_size 10GB), checkpoint timeout 1h
UPDATE/DELETE tests are executed such that they hit some or many rows.

Memory testing (PG 9.5.3)
Compile with --enable-debug (both postgres and citus) Note: citus gets configuration parameters from postgresql.
See this for enabling valgrind for postgres. You would need to install and configure citusdb onto postgresql instance.
Run all nodes using valgrind: /usr/bin/valgrind /usr/local/bin/postgres -D /data

Note: I was not able to run the test MacOs, instead, run the test on ubuntu.
Memory testing is done under check-multi regression test suite.
valgrind --read-var-info=no  --leak-check=full --show-leak-kinds=definite --trace-children=yes --show-reachable=no --time-stamp=yes --log-file=$HOME/pg-valgrind/%p.log --fullpath-after=/ --gen-suppressions=no --suppressions=/home/ubuntu/postgresql-9.5.3/src/tools/valgrind.supp make check-multi

Use postgres’ suppression file
Do not use -v option


valgrind --read-var-info=no  --leak-check=no --show-leak-kinds=definite --trace-children=yes --show-reachable=no --time-stamp=yes --log-file=$HOME/pg-valgrind/%p.log --fullpath-after=/ --gen-suppressions=no --suppressions=/home/ubuntu/postgresql-9.5.3/src/tools/valgrind.supp   make check-multi

We observed the following issues when we ran Valgrind with --leak-check=full. I shortened the following error messages from the original test document for brevity. For full traces, please refer to the document.

Jump on uninitialised value > 100. This is probably a bogus warning

Conditional jump or move depends on uninitialised value(s) at 0x5598427: __printf_fp


Libpq issue (1000+). Malloc in PQmakeEmptyPGresult. Similar call stack including the same malloc  was also found in almost all utility command handling inside “ExecuteDistributedDDLCommand()”,  “SendQueryToShards()”, “ConsumeQueryResult()”, “RemoteFinalizedShardPlacementList”  functions

at 0x4C2AB80: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) by 0x61A8ED3: PQmakeEmptyPGresult (home/ubuntu/postgresql-9.5.3/src/interfaces/libpq/fe-exec.c:144)


Malloc inside make_absolute_path. This was the most common call stack reported. It was also reported within “ProcessCopyStmt()”. This call stack is probably bogus since since “ParseWorkerNodeFile” explicitly free’s the memory. However ProcessCopyStmt assigns possibly palloc’ed struct to point to this memory location and returns.
There were few more reported inside csql

Upgrade testing

Test from 5.0 to 5.2 and from 5.1 to 5.2

ALTER EXTENSION citus UPDATE;
Verify extension version is changed \dx
Check new features


Test with cstore_fdw installed (5.1 -> 5.2 )
Test shard_rebalancer installed

Create table with replication factor 1
ALTER EXTENSION citus UPDATE;


Test running in mixed mode: (5.1 workers and 5.2 master)
Test router executor SELECT

Test real-time executor SELECT
Test task-tracker executor repartition SELECT
Test INSERTs
Test COPY on hash and append from master
Test Explain
Test Drop Table



Code coverage testing (PG 9.5.3 on Mac)
As of #994 Code Coverage testing is now done automatically
Steps to run code coverage tool:

./configure  --enable-coverage (do this for postgresql, since citus extension gets configure parameters from Postgres)
make clean
Add coverage flags to src/Backend/distributed/Makefile

PG_CPPFLAGS += -ftest-coverage -fprofile-arcs
SHLIB_LINK += -ftest-coverage -fprofile-arcs


make
make install
make -C src/test/regress/ check-full
lcov -d . -c -o lcov.info
genhtml --show-details --legend --output-directory=coverage --title=PostgreSQL --num-spaces=4 --prefix=./src/ find . -name lcov.info -print

Since check-multi fails due to timing issues, I was not able to run all tests.  Then I run all targets included in check-full seperately. Coverage document could be viewed here.
I compared the coverage results against the one we prepared for 5.1 testing. Some findings:

multi_copy.c coverage has been decreased. There were some error cases that could be tested but was not.
Transmit.c coverage has dropped to 44% from 91%. A newly added function  “RedirectCopyDataToRegularFile()” was not hit at all.
Coverage on multi_task_tracker_executor.c is now below 70% due to task failure handling codes being not hit.
multi_utility.c: ReplicateGrantStmt() was not tested at all. There is no entry point for this function in the codebase
master_metadata_utility.c: ShardIntervalCount() is unused. Coverage could be increased with more regression tests for error conditions.
Master_node_protocol.c: Coverage is good, but still can be increased with more regression tests.
Worker_node_manager.c: unused functions ResponsiveWorkerNodeList and WorkerNodeResponsive
Relay_event_utility.c: coverage is 50%. Could be improved.
Files under src/backend/distributed/utils have poor coverage due them being copied from postgresql code with very little modifications. Should not be a problem AFAIC.

Duplicate issue #990: Tests: Automate Integration Testing
Created: 2016-11-21T06:09:25Z
We would like to automate all integration tests;
Citus Enterprise
CloudFormation
Shard Rebalancer
cstore_fdw
postgresql_hll (already have regression tests)
session analytics + hstore

Closing comment (2016-11-21T06:09:37Z): Related to #890