Like for almost all 0.94 and 0.96 HBase releases, I spent some time the last few days to test HBase 0.94.17RC2 release. Since I have not provided performances numbers for a while, I have decided to share here some information about what I test, and some of the result numbers. HBase release canditates are actively tested by many HBase contributors, which ensure their stability.
To test a release, I proceed with 2 main tracks.. First track is stability, to ensure release run correctly, no errors in the logs, documentation is correct, features are as expected, etc. Second track is performances. I ensure the release provides performances similar to the previous release from the same branch. After each release tested I try to find something which I can add to my testing plan. When a release is sunk because someone else find something wrong, I make sure I add a test in my plan to make sure this is covered into the next release.
So here is the result of the 0.94.17RC2 testing. For each step I give an overall description with the result, then more details on what is done.
- Jars and signatures checked => Passed
Checked the secured and not secured jars by calcultating the MD5 signatured, compared with the provided result. Tried to un-compressed the 2 jars to make sure they un-compress correctly.
- Content => Passed
Checked the CHANGES.TXT file to make sure release fixed are correctly listed. Randomly checked some documentation pages.
- Run default test suite => Passed
Ran the default HBase test suite using maven command and verified that all tests ran correctly.
- Start the release and check => Passed
Start the release in a 4 node cluster (1 master, 3 region servers). Checked the logs, checked the web interface. Check consistency using HBCK.
- Ran default/performances tests and compared => Passed
HBase come with multiple tools to test performances, like PerformanceEvaluation, LoadTestTool, IntegrationTestLoadAndVerify, HLogPerformanceEvaluation and IntegrationTestBigLinkedList (at least). I use all those tools to compare the stability and the performances between the test release and the previous stable one from the same branch. For the purpose of this release candidate, I ran all those tests with HBase 0.94.16 + Hadoop 1.2.1 and HBase 0.94.17RC2 + Hadoop 1.2.1 and below are the results of those tests for those 2 releases. Numbers below are operations per seconds for PerformanceEvaluation. Like, for RandomScanWithRange100Test, numbers mean we can do 147 scan of 100 rows per seconds.
IntegrationTestLoadAndVerify still give me an incorrect result. This test is supposed to give REFERENCES_CHECKED=10000000 however I still get results like REFERENCES_CHECKED=9855584 what ever version I try with (HBase 0.94.16, HBase 0.94.17, HBase 0.96.1).
LoadTestTool results are below. They are very close between the 2 version.
HLogPerformanceEvaluation results are also very similar between the 2 releases. Values are ops/s. I tried to compare with HBase 0.96.1 + HBase 2.2.0 but the 2 re not returning the same metrics so I have not been able to compare them correctly.
Last automated test is IntegrationTestBigLinkedList where I got the right result for the 2 release. (REFERENCED=6000000)
- Run manual tests => Passed
Using the shell, create a table, put some values, scan, flush, compact, validate VERSIONS behaviour, drop table, all passed succesfuly.
- Test splits, transition and balancing => Passed.
On a 8 node cluster, offline merge a 17 region table into a single region. Run HBCK to validate everything is fine. Then run man split and compaction to make sure table is automatically split by HBase back into multiple regions. Validate regions transition and balancing, and run HBCK again. Everything working fine.
- Run some MapReduce jobs => Passed
Ran the simple RowCounter map reduce job first because this one doesn't write anything into the cluster so is safer. Since it passed, run another MapReduce job doing reads, writes, lookups and deletes. All passed succesfuly.
All over the process, HBCK and logs are checked multiple times to make sure no exception is found and no unconsistencies are produced.
As a result, all tests with 0.94.17 are giving similar results as 0.94.16. Performances are equivalents, all features tests passed and are stable. I have provided a +1 in the HBase mailing list based on my results.