Repository: apache/accumulo

Issue #1647: Compaction tests failing in TableOperationsIT
Created: 2020-07-01T17:47:07Z
TableOperationsIT is now failling with the merge of #1605.  I am seeing 3 tests failing:
testCompactEmptyTableWithGeneratorIterator
testCompactEmptyTableWithGeneratorIterator_Splits_Partial
testCompactEmptyTableWithGeneratorIterator_Splits
They have similar assertion errors:
java.lang.AssertionError: expected:<{a1 colF3:colQ3 [] 1593624908123 false=1, c1 colF3:colQ3 [] 1593624908123 false=1, m1 colF3:colQ3 [] 1593624908123 false=1}> but was:<{}>
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.apache.accumulo.test.TableOperationsIT.testCompactEmptyTableWithGeneratorIterator(TableOperationsIT.java:288)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:834)

Duplicate issue #1648: Generator compactions are failing
Created: 2020-07-01T21:16:52Z
After the changes in #1605 TableOperationsIT.testCompactEmptyTableWithGeneratorIterator started to fail.  This is because this test initiates a compaction with user iterators on an empty table.  Before the changes in #1605 the compaction would run even though there was no data and the iterators generated data.  After the changes in #1605 a tablet with no data will not run a compaction.
Should this behavior be implemented, it would require multiple special cases. While starting to look into implementing empty compactions, I found the following cases. There may be more.

Only allow a single concurrent empty compaction to run. The changes in #1605 enable multiple concurrent compactions to run against a single tablet as long they are compacting a disjoint set of files.  Two empty compactions have a disjoint set of files, so the current per tablet tracking of compacting files would not be sufficient to prevent concurrent empty compactions.  Would need special handling just for the empty set.
Only commit empty compaction when there are no files in tablet. Its possible that while an empty compaction is running, that a new files arrives for the tablet.  Given that empty compactions are special cases, should this compaction be allowed to complete in this case?  The only reason the special empty compaction was started was because there were no files, so it seems it should only complete successfully if there are no files when the compaction finishes.  I don't think the old code handled this case in anyway, but it could have happened in the old code.  The old code would have committed an empty compaction where a file arrived after it started.
User compactions can select a subset of files to compact.  When a tablet has files, but the user selects none for compaction should an empty compaction run? I don't think so.

These special cases will require specialized code, increasing the complexity of the compaction code.  I am not sure the complexity is worth it.  I was unaware of this capability until I ran into the test.  I have not yet found any user facing documentation that advertises this feature.

Closing comment (2020-07-01T21:52:34Z): Duplicate of #1647.