2020-08-19 10:11:15 +00:00
|
|
|
|
<!DOCTYPE html>
|
|
|
|
|
|
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
|
|
|
|
|
|
<head>
|
|
|
|
|
|
<meta charset="utf-8" />
|
|
|
|
|
|
<meta name="generator" content="pandoc" />
|
|
|
|
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
|
|
|
|
|
|
<title>Native/Unit Test Development Guidelines</title>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<style>
|
|
|
|
|
|
code{white-space: pre-wrap;}
|
|
|
|
|
|
span.smallcaps{font-variant: small-caps;}
|
|
|
|
|
|
div.columns{display: flex; gap: min(4vw, 1.5em);}
|
|
|
|
|
|
div.column{flex: auto; overflow-x: auto;}
|
|
|
|
|
|
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
|
|
|
|
|
|
ul.task-list{list-style: none;}
|
|
|
|
|
|
ul.task-list li input[type="checkbox"] {
|
|
|
|
|
|
width: 0.8em;
|
|
|
|
|
|
margin: 0 0.8em 0.2em -1.6em;
|
|
|
|
|
|
vertical-align: middle;
|
|
|
|
|
|
}
|
|
|
|
|
|
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</style>
|
|
|
|
|
|
<link rel="stylesheet" href="../make/data/docs-resources/resources/jdk-default.css" />
|
|
|
|
|
|
<!--[if lt IE 9]>
|
|
|
|
|
|
<script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
|
|
|
|
|
|
<![endif]-->
|
|
|
|
|
|
</head>
|
|
|
|
|
|
<body>
|
|
|
|
|
|
<header id="title-block-header">
|
|
|
|
|
|
<h1 class="title">Native/Unit Test Development Guidelines</h1>
|
|
|
|
|
|
</header>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<nav id="TOC" role="doc-toc">
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><a href="#good-test-properties" id="toc-good-test-properties">Good
|
|
|
|
|
|
test properties</a>
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li><a href="#lightness" id="toc-lightness">Lightness</a></li>
|
|
|
|
|
|
<li><a href="#isolation" id="toc-isolation">Isolation</a></li>
|
|
|
|
|
|
<li><a href="#atomicity-and-self-containment"
|
|
|
|
|
|
id="toc-atomicity-and-self-containment">Atomicity and
|
|
|
|
|
|
self-containment</a></li>
|
|
|
|
|
|
<li><a href="#repeatability"
|
|
|
|
|
|
id="toc-repeatability">Repeatability</a></li>
|
|
|
|
|
|
<li><a href="#informativeness"
|
|
|
|
|
|
id="toc-informativeness">Informativeness</a></li>
|
|
|
|
|
|
<li><a href="#testing-instead-of-visiting"
|
|
|
|
|
|
id="toc-testing-instead-of-visiting">Testing instead of
|
|
|
|
|
|
visiting</a></li>
|
|
|
|
|
|
<li><a href="#nearness" id="toc-nearness">Nearness</a></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><a href="#asserts" id="toc-asserts">Asserts</a>
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li><a href="#several-checks" id="toc-several-checks">Several
|
|
|
|
|
|
checks</a></li>
|
|
|
|
|
|
<li><a href="#first-parameter-is-expected-value"
|
|
|
|
|
|
id="toc-first-parameter-is-expected-value">First parameter is expected
|
|
|
|
|
|
value</a></li>
|
|
|
|
|
|
<li><a href="#floating-point-comparison"
|
|
|
|
|
|
id="toc-floating-point-comparison">Floating-point comparison</a></li>
|
|
|
|
|
|
<li><a href="#c-string-comparison" id="toc-c-string-comparison">C string
|
|
|
|
|
|
comparison</a></li>
|
|
|
|
|
|
<li><a href="#error-messages" id="toc-error-messages">Error
|
|
|
|
|
|
messages</a></li>
|
|
|
|
|
|
<li><a href="#uncluttered-output"
|
|
|
|
|
|
id="toc-uncluttered-output">Uncluttered output</a></li>
|
|
|
|
|
|
<li><a href="#failures-propagation"
|
|
|
|
|
|
id="toc-failures-propagation">Failures propagation</a></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><a href="#naming-and-grouping" id="toc-naming-and-grouping">Naming
|
|
|
|
|
|
and Grouping</a>
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li><a href="#test-group-names" id="toc-test-group-names">Test group
|
|
|
|
|
|
names</a></li>
|
|
|
|
|
|
<li><a href="#filename" id="toc-filename">Filename</a></li>
|
|
|
|
|
|
<li><a href="#file-location" id="toc-file-location">File
|
|
|
|
|
|
location</a></li>
|
|
|
|
|
|
<li><a href="#test-names" id="toc-test-names">Test names</a></li>
|
|
|
|
|
|
<li><a href="#fixture-classes" id="toc-fixture-classes">Fixture
|
|
|
|
|
|
classes</a></li>
|
|
|
|
|
|
<li><a href="#friend-classes" id="toc-friend-classes">Friend
|
|
|
|
|
|
classes</a></li>
|
|
|
|
|
|
<li><a href="#oscpu-specific-tests" id="toc-oscpu-specific-tests">OS/CPU
|
|
|
|
|
|
specific tests</a></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><a href="#miscellaneous" id="toc-miscellaneous">Miscellaneous</a>
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li><a href="#hotspot-style" id="toc-hotspot-style">Hotspot
|
|
|
|
|
|
style</a></li>
|
|
|
|
|
|
<li><a href="#codetest-metrics" id="toc-codetest-metrics">Code/test
|
|
|
|
|
|
metrics</a></li>
|
|
|
|
|
|
<li><a href="#access-to-non-public-members"
|
|
|
|
|
|
id="toc-access-to-non-public-members">Access to non-public
|
|
|
|
|
|
members</a></li>
|
|
|
|
|
|
<li><a href="#death-tests" id="toc-death-tests">Death tests</a></li>
|
|
|
|
|
|
<li><a href="#external-flags" id="toc-external-flags">External
|
|
|
|
|
|
flags</a></li>
|
|
|
|
|
|
<li><a href="#test-specific-flags"
|
|
|
|
|
|
id="toc-test-specific-flags">Test-specific flags</a></li>
|
|
|
|
|
|
<li><a href="#flag-restoring" id="toc-flag-restoring">Flag
|
|
|
|
|
|
restoring</a></li>
|
|
|
|
|
|
<li><a href="#googletest-documentation"
|
|
|
|
|
|
id="toc-googletest-documentation">GoogleTest documentation</a></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><a href="#todo" id="toc-todo">TODO</a></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul>
|
|
|
|
|
|
</nav>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>The purpose of these guidelines is to establish a shared vision on
|
|
|
|
|
|
what kind of native tests and how we want to develop them for Hotspot
|
|
|
|
|
|
using GoogleTest. Hence these guidelines include style items as well as
|
|
|
|
|
|
test approach items.</p>
|
|
|
|
|
|
<p>First section of this document describes properties of good tests
|
|
|
|
|
|
which are common for almost all types of test regardless of language,
|
|
|
|
|
|
framework, etc. Further sections provide recommendations to achieve
|
|
|
|
|
|
those properties and other HotSpot and/or GoogleTest specific
|
|
|
|
|
|
guidelines.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h2 id="good-test-properties">Good test properties</h2>
|
|
|
|
|
|
<h3 id="lightness">Lightness</h3>
|
|
|
|
|
|
<p>Use the most lightweight type of tests.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>In Hotspot, there are 3 different types of tests regarding their
|
|
|
|
|
|
dependency on a JVM, each next level is slower than previous</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
|
|
|
|
|
<li><p><code>TEST</code> : a test does not depend on a JVM</p></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><p><code>TEST_VM</code> : a test does depend on an initialized JVM,
|
|
|
|
|
|
but are supposed not to break a JVM, i.e. leave it in a workable
|
|
|
|
|
|
state.</p></li>
|
|
|
|
|
|
<li><p><code>TEST_OTHER_VM</code> : a test depends on a JVM and requires
|
|
|
|
|
|
a freshly initialized JVM or leaves a JVM in non-workable state</p></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul>
|
|
|
|
|
|
<h3 id="isolation">Isolation</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Tests have to be isolated: not to have visible side-effects,
|
|
|
|
|
|
influences on other tests results.</p>
|
|
|
|
|
|
<p>Results of one test should not depend on test execution order, other
|
|
|
|
|
|
tests, otherwise it is becoming almost impossible to find out why a test
|
|
|
|
|
|
failed. Due to hotspot-specific, it is not so easy to get a full
|
|
|
|
|
|
isolation, e.g. we share an initialized JVM between all
|
|
|
|
|
|
<code>TEST_VM</code> tests, so if your test changes JVM's state too
|
|
|
|
|
|
drastically and does not change it back, you had better consider
|
|
|
|
|
|
<code>TEST_OTHER_VM</code>.</p>
|
|
|
|
|
|
<h3 id="atomicity-and-self-containment">Atomicity and
|
|
|
|
|
|
self-containment</h3>
|
|
|
|
|
|
<p>Tests should be <em>atomic</em> and <em>self-contained</em> at the
|
|
|
|
|
|
same time.</p>
|
|
|
|
|
|
<p>One test should check a particular part of a class, subsystem,
|
|
|
|
|
|
functionality, etc. Then it is quite easy to determine what parts of a
|
|
|
|
|
|
product are broken basing on test failures. On the other hand, a test
|
|
|
|
|
|
should test that part more-or-less entirely, because when one sees a
|
|
|
|
|
|
test <code>FooTest::bar</code>, they assume all aspects of bar from
|
|
|
|
|
|
<code>Foo</code> are tested.</p>
|
|
|
|
|
|
<p>However, it is impossible to cover all aspects even of a method, not
|
|
|
|
|
|
to mention a subsystem. In such cases, it is recommended to have several
|
|
|
|
|
|
tests, one for each aspect of a thing under test. For example one test
|
|
|
|
|
|
to tests how <code>Foo::bar</code> works if an argument is
|
|
|
|
|
|
<code>null</code>, another test to test how it works if an argument is
|
|
|
|
|
|
acceptable but <code>Foo</code> is not in the right state to accept it
|
|
|
|
|
|
and so on. This helps not only to make tests atomic, self-contained but
|
|
|
|
|
|
also makes test name self-descriptive (discussed in more details in <a
|
|
|
|
|
|
href="#test-names">Test names</a>).</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="repeatability">Repeatability</h3>
|
|
|
|
|
|
<p>Tests have to be repeatable.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Reproducibility is very crucial for a test. No one likes sporadic
|
|
|
|
|
|
test failures, they are hard to investigate, fix and verify a fix.</p>
|
|
|
|
|
|
<p>In some cases, it is quite hard to write a 100% repeatable test,
|
|
|
|
|
|
since besides a test there can be other moving parts, e.g. in case of
|
|
|
|
|
|
<code>TEST_VM</code> there are several concurrently running threads.
|
|
|
|
|
|
Despite this, we should try to make a test as reproducible as
|
|
|
|
|
|
possible.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="informativeness">Informativeness</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>In case of a failure, a test should be as <em>informative</em> as
|
|
|
|
|
|
possible.</p>
|
|
|
|
|
|
<p>Having more information about a test failure than just compared
|
|
|
|
|
|
values can be very useful for failure troubleshooting, it can reduce or
|
|
|
|
|
|
even completely eliminate debugging hours. This is even more important
|
|
|
|
|
|
in case of not 100% reproducible failures.</p>
|
|
|
|
|
|
<p>Achieving this property, one can easily make a test too verbose, so
|
|
|
|
|
|
it will be really hard to find useful information in the ocean of
|
|
|
|
|
|
useless information. Hence they should not only think about how to
|
|
|
|
|
|
provide <a href="#error-messages">good information</a>, but also <a
|
|
|
|
|
|
href="#uncluttered-output">when to do it</a>.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="testing-instead-of-visiting">Testing instead of visiting</h3>
|
|
|
|
|
|
<p>Tests should <em>test</em>.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>It is not enough just to "visit" some code, a test should check that
|
|
|
|
|
|
code does that it has to do, compare return values with expected values,
|
|
|
|
|
|
check that desired side effects are done, and undesired are not, and so
|
|
|
|
|
|
on. In other words, a test should contain at least one GoogleTest
|
|
|
|
|
|
assertion and do not rely on JVM asserts.</p>
|
|
|
|
|
|
<p>Generally speaking to write a good test, one should create a model of
|
|
|
|
|
|
the system under tests, a model of possible bugs (or bugs which one
|
|
|
|
|
|
wants to find) and design tests using those models.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="nearness">Nearness</h3>
|
|
|
|
|
|
<p>Prefer having checks inside test code.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Not only does having test logic outside, e.g. verification method,
|
|
|
|
|
|
depending on asserts in product code contradict with several items above
|
|
|
|
|
|
but also decreases test’s readability and stability. It is much easier
|
|
|
|
|
|
to understand that a test is testing when all testing logic is located
|
|
|
|
|
|
inside a test or nearby in shared test libraries. As a rule of thumb,
|
|
|
|
|
|
the closer a check to a test, the better.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h2 id="asserts">Asserts</h2>
|
|
|
|
|
|
<h3 id="several-checks">Several checks</h3>
|
|
|
|
|
|
<p>Prefer <code>EXPECT</code> over <code>ASSERT</code> if possible.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>This is related to the <a href="#informativeness">informativeness</a>
|
|
|
|
|
|
property of tests, information for other checks can help to better
|
|
|
|
|
|
localize a defect’s root-cause. One should use <code>ASSERT</code> if it
|
|
|
|
|
|
is impossible to continue test execution or if it does not make much
|
|
|
|
|
|
sense. Later in the text, <code>EXPECT</code> forms will be used to
|
|
|
|
|
|
refer to both <code>ASSERT/EXPECT</code>.</p>
|
|
|
|
|
|
<p>When it is possible to make several different checks, but impossible
|
|
|
|
|
|
to continue test execution if at least one check fails, you can use
|
|
|
|
|
|
<code>::testing::Test::HasNonfatalFailure()</code> function. The
|
|
|
|
|
|
recommended way to express that is
|
|
|
|
|
|
<code>ASSERT_FALSE(::testing::Test::HasNonfatalFailure())</code>.
|
|
|
|
|
|
Besides making it clear why a test is aborted, it also allows you to
|
|
|
|
|
|
provide more information about a failure.</p>
|
|
|
|
|
|
<h3 id="first-parameter-is-expected-value">First parameter is expected
|
|
|
|
|
|
value</h3>
|
|
|
|
|
|
<p>In all equality assertions, expected values should be passed as the
|
|
|
|
|
|
first parameter.</p>
|
|
|
|
|
|
<p>This convention is adopted by GoogleTest, and there is a slight
|
|
|
|
|
|
difference in how GoogleTest treats parameters, the most important one
|
|
|
|
|
|
is <code>null</code> detection. Due to different reasons,
|
|
|
|
|
|
<code>null</code> detection is enabled only for the first parameter,
|
|
|
|
|
|
that is to said <code>EXPECT_EQ(NULL, object)</code> checks that object
|
|
|
|
|
|
is <code>null</code>, while <code>EXPECT_EQ(object, NULL)</code> checks
|
|
|
|
|
|
that object equals to <code>NULL</code>, GoogleTest is very strict
|
|
|
|
|
|
regarding types of compared values so the latter will generates a
|
|
|
|
|
|
compile-time error.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="floating-point-comparison">Floating-point comparison</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Use floating-point special macros to compare
|
|
|
|
|
|
<code>float/double</code> values.</p>
|
|
|
|
|
|
<p>Because of floating-point number representations and round-off
|
|
|
|
|
|
errors, regular equality comparison will not return true in most cases.
|
|
|
|
|
|
There are special <code>EXPECT_FLOAT_EQ/EXPECT_DOUBLE_EQ</code>
|
|
|
|
|
|
assertions which check that the distance between compared values is not
|
|
|
|
|
|
more than 4 ULPs, there is also <code>EXPECT_NEAR(v1, v2, eps)</code>
|
|
|
|
|
|
which checks that the absolute value of the difference between
|
|
|
|
|
|
<code>v1</code> and <code>v2</code> is not greater than
|
|
|
|
|
|
<code>eps</code>.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="c-string-comparison">C string comparison</h3>
|
|
|
|
|
|
<p>Use string special macros for C strings comparisons.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p><code>EXPECT_EQ</code> just compares pointers’ values, which is
|
|
|
|
|
|
hardly what one wants comparing C strings. GoogleTest provides
|
|
|
|
|
|
<code>EXPECT_STREQ</code> and <code>EXPECT_STRNE</code> macros to
|
|
|
|
|
|
compare C string contents. There are also case-insensitive versions
|
|
|
|
|
|
<code>EXPECT_STRCASEEQ</code>, <code>EXPECT_STRCASENE</code>.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="error-messages">Error messages</h3>
|
|
|
|
|
|
<p>Provide informative, but not too verbose error messages.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>All GoogleTest asserts print compared expressions and their values,
|
|
|
|
|
|
so there is no need to have them in error messages. Asserts print only
|
|
|
|
|
|
compared values, they do not print any of interim variables, e.g.
|
|
|
|
|
|
<code>ASSERT_TRUE((val1 == val2 && isFail(foo(8)) || i == 18)</code>
|
|
|
|
|
|
prints only one value. If you use some complex predicates, please
|
|
|
|
|
|
consider <code>EXPECT_PRED*</code> or <code>EXPECT_FORMAT_PRED</code>
|
|
|
|
|
|
assertions family, they check that a predicate returns true/success and
|
|
|
|
|
|
print out all parameters values.</p>
|
|
|
|
|
|
<p>However in some cases, default information is not enough, a commonly
|
|
|
|
|
|
used example is an assert inside a loop, GoogleTest will not print
|
|
|
|
|
|
iteration values (unless it is an assert's parameter). Other
|
|
|
|
|
|
demonstrative examples are printing error code and a corresponding error
|
|
|
|
|
|
message; printing internal states which might have an impact on results.
|
|
|
|
|
|
One should add this information to assert message using
|
|
|
|
|
|
<code><<</code> operator.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="uncluttered-output">Uncluttered output</h3>
|
|
|
|
|
|
<p>Print information only if it is needed.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Too verbose tests which print all information even if they pass are
|
|
|
|
|
|
very bad practice. They just pollute output, so it becomes harder to
|
|
|
|
|
|
find useful information. In order not print information till it is
|
|
|
|
|
|
really needed, one should consider saving it to a temporary buffer and
|
|
|
|
|
|
pass to an assert. <a
|
|
|
|
|
|
href="https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/shared/test_memset_with_concurrent_readers.cpp"
|
|
|
|
|
|
class="uri">https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/shared/test_memset_with_concurrent_readers.cpp</a>
|
|
|
|
|
|
has a good example how to do that.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="failures-propagation">Failures propagation</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Wrap a subroutine call into <code>EXPECT_NO_FATAL_FAILURE</code>
|
|
|
|
|
|
macro to propagate failures.</p>
|
|
|
|
|
|
<p><code>ASSERT</code> and <code>FAIL</code> abort only the current
|
|
|
|
|
|
function, so if you have them in a subroutine, a test will not be
|
|
|
|
|
|
aborted after the subroutine even if <code>ASSERT</code> or
|
|
|
|
|
|
<code>FAIL</code> fails. You should call such subroutines in
|
|
|
|
|
|
<code>ASSERT_NO_FATAL_FAILURE</code> macro to propagate fatal failures
|
|
|
|
|
|
and abort a test. <code>(EXPECT|ASSERT)_NO_FATAL_FAILURE</code> can also
|
|
|
|
|
|
be used to provide more information.</p>
|
|
|
|
|
|
<p>Due to obvious reasons, there are no
|
|
|
|
|
|
<code>(EXPECT|ASSERT)_NO_NONFATAL_FAILURE</code> macros. However, if you
|
|
|
|
|
|
need to check if a subroutine generated a nonfatal failure (failed an
|
|
|
|
|
|
<code>EXPECT</code>), you can use
|
|
|
|
|
|
<code>::testing::Test::HasNonfatalFailure</code> function, or
|
|
|
|
|
|
<code>::testing::Test::HasFailure</code> function to check if a
|
|
|
|
|
|
subroutine generated any failures, see <a href="#several-checks">Several
|
|
|
|
|
|
checks</a>.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h2 id="naming-and-grouping">Naming and Grouping</h2>
|
|
|
|
|
|
<h3 id="test-group-names">Test group names</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Test group names should be in CamelCase, start and end with a letter.
|
|
|
|
|
|
A test group should be named after tested class, functionality,
|
|
|
|
|
|
subsystem, etc.</p>
|
|
|
|
|
|
<p>This naming scheme helps to find tests, filter them and simplifies
|
|
|
|
|
|
test failure analysis. For example, class <code>Foo</code> - test group
|
|
|
|
|
|
<code>Foo</code>, compiler logging subsystem - test group
|
|
|
|
|
|
<code>CompilerLogging</code>, G1 GC — test group <code>G1GC</code>, and
|
|
|
|
|
|
so forth.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="filename">Filename</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>A test file must have <code>test_</code> prefix and <code>.cpp</code>
|
|
|
|
|
|
suffix.</p>
|
|
|
|
|
|
<p>Both are actually requirements from the current build system to
|
|
|
|
|
|
recognize your tests.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="file-location">File location</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Test file location should reflect a location of the tested part of
|
|
|
|
|
|
the product.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><p>All unit tests for a class from <code>foo/bar/baz.cpp</code>
|
|
|
|
|
|
should be placed <code>foo/bar/test_baz.cpp</code> in
|
|
|
|
|
|
<code>hotspot/test/native/</code> directory. Having all tests for a
|
|
|
|
|
|
class in one file is a common practice for unit tests, it helps to see
|
|
|
|
|
|
all existing tests at once, share functions and/or resources without
|
|
|
|
|
|
losing encapsulation.</p></li>
|
|
|
|
|
|
<li><p>For tests which test more than one class, directory hierarchy
|
|
|
|
|
|
should be the same as product hierarchy, and file name should reflect
|
|
|
|
|
|
the name of the tested subsystem/functionality. For example, if a
|
|
|
|
|
|
sub-system under tests belongs to <code>gc/g1</code>, tests should be
|
|
|
|
|
|
placed in <code>gc/g1</code> directory.</p></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Please note that framework prepends directory name to a test group
|
|
|
|
|
|
name. For example, if <code>TEST(foo, check_this)</code> and
|
|
|
|
|
|
<code>TEST(bar, check_that)</code> are defined in
|
|
|
|
|
|
<code>hotspot/test/native/gc/shared/test_foo.cpp</code> file, they will
|
|
|
|
|
|
be reported as <code>gc/shared/foo::check_this</code> and
|
|
|
|
|
|
<code>gc/shared/bar::check_that</code>.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="test-names">Test names</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Test names should be in small_snake_case, start and end with a
|
|
|
|
|
|
letter. A test name should reflect that a test checks.</p>
|
|
|
|
|
|
<p>Such naming makes tests self-descriptive and helps a lot during the
|
|
|
|
|
|
whole test life cycle. It is easy to do test planning, test inventory,
|
|
|
|
|
|
to see what things are not tested, to review tests, to analyze test
|
|
|
|
|
|
failures, to evolve a test, etc. For example
|
|
|
|
|
|
<code>foo_return_0_if_name_is_null</code> is better than
|
|
|
|
|
|
<code>foo_sanity</code> or <code>foo_basic</code> or just
|
|
|
|
|
|
<code>foo</code>,
|
|
|
|
|
|
<code>humongous_objects_can_not_be_moved_by_young_gc</code> is better
|
|
|
|
|
|
than <code>ho_young_gc</code>.</p>
|
|
|
|
|
|
<p>Actually using underscore is against GoogleTest project convention,
|
|
|
|
|
|
because it can lead to illegal identifiers, however, this is too strict.
|
|
|
|
|
|
Restricting usage of underscore for test names only and prohibiting test
|
|
|
|
|
|
name starts or ends with an underscore are enough to be safe.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="fixture-classes">Fixture classes</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Fixture classes should be named after tested classes, subsystems, etc
|
|
|
|
|
|
(follow <a href="#test-group-names">Test group names rule</a>) and have
|
|
|
|
|
|
<code>Test</code> suffix to prevent class name conflicts.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="friend-classes">Friend classes</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>All test purpose friends should have either <code>Test</code> or
|
|
|
|
|
|
<code>Testable</code> suffix.</p>
|
|
|
|
|
|
<p>It greatly simplifies understanding of friendship’s purpose and
|
|
|
|
|
|
allows statically check that private members are not exposed
|
|
|
|
|
|
unexpectedly. Having <code>FooTest</code> as a friend of
|
|
|
|
|
|
<code>Foo</code> without any comments will be understood as a necessary
|
|
|
|
|
|
evil to get testability.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="oscpu-specific-tests">OS/CPU specific tests</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Guard OS/CPU specific tests by <code>#ifdef</code> and have OS/CPU
|
|
|
|
|
|
name in filename.</p>
|
|
|
|
|
|
<p>For the time being, we do not support separate directories for OS,
|
|
|
|
|
|
CPU, OS-CPU specific tests, in case we will have lots of such tests, we
|
|
|
|
|
|
will change directory layout and build system to support that in the
|
|
|
|
|
|
same way it is done in hotspot.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h2 id="miscellaneous">Miscellaneous</h2>
|
|
|
|
|
|
<h3 id="hotspot-style">Hotspot style</h3>
|
|
|
|
|
|
<p>Abide the norms and rules accepted in Hotspot style guide.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Tests are a part of Hotspot, so everything (if applicable) we use for
|
|
|
|
|
|
Hotspot, should be used for tests as well. Those guidelines cover
|
|
|
|
|
|
test-specific things.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="codetest-metrics">Code/test metrics</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Coverage information and other code/test metrics are quite useful to
|
|
|
|
|
|
decide what tests should be written, what tests should be improved and
|
|
|
|
|
|
what can be removed.</p>
|
|
|
|
|
|
<p>For unit tests, widely used and well-known coverage metric is branch
|
|
|
|
|
|
coverage, which provides good quality of tests with relatively easy test
|
|
|
|
|
|
development process. For other levels of testing, branch coverage is not
|
|
|
|
|
|
as good, and one should consider others metrics, e.g. transaction flow
|
|
|
|
|
|
coverage, data flow coverage.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="access-to-non-public-members">Access to non-public members</h3>
|
|
|
|
|
|
<p>Use explicit friend class to get access to non-public members.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>We do not use GoogleTest macro to declare friendship relation,
|
|
|
|
|
|
because, from our point of view, it is less clear than an explicit
|
|
|
|
|
|
declaration.</p>
|
|
|
|
|
|
<p>Declaring a test fixture class as a friend class of a tested test is
|
|
|
|
|
|
the easiest and the clearest way to get access. However, it has some
|
|
|
|
|
|
disadvantages, here is some of them:</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
|
|
|
|
|
<li>Each test has to be declared as a friend</li>
|
|
|
|
|
|
<li>Subclasses do not inheritance friendship relation</li>
|
|
|
|
|
|
</ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>In other words, it is harder to share code between tests. Hence if
|
|
|
|
|
|
you want to share code or expect it to be useful in other tests, you
|
|
|
|
|
|
should consider making members in a tested class protected and introduce
|
|
|
|
|
|
a shared test-only class which expose those members via public
|
|
|
|
|
|
functions, or even making members publicly accessible right away in a
|
|
|
|
|
|
product class. If it is not an option to change members visibility, one
|
|
|
|
|
|
can create a friend class which exposes members.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="death-tests">Death tests</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>You can not use death tests inside <code>TEST_OTHER_VM</code> and
|
|
|
|
|
|
<code>TEST_VM_ASSERT*</code>.</p>
|
|
|
|
|
|
<p>We tried to make Hotspot-GoogleTest integration as transparent as
|
|
|
|
|
|
possible, however, due to the current implementation of
|
|
|
|
|
|
<code>TEST_OTHER_VM</code> and <code>TEST_VM_ASSERT*</code> tests, you
|
|
|
|
|
|
cannot use death test functionality in them. These tests are implemented
|
|
|
|
|
|
as GoogleTest death tests, and GoogleTest does not allow to have a death
|
|
|
|
|
|
test inside another death test.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="external-flags">External flags</h3>
|
|
|
|
|
|
<p>Passing external flags to a tested JVM is not supported.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>The rationality of such design decision is to simplify both tests and
|
|
|
|
|
|
a test framework and to avoid failures related to incompatible flags
|
|
|
|
|
|
combination till there is a good solution for that. However there are
|
|
|
|
|
|
cases when one wants to test a JVM with specific flags combination,
|
|
|
|
|
|
<code>_JAVA_OPTIONS</code> environment variable can be used to do that.
|
|
|
|
|
|
Flags from <code>_JAVA_OPTIONS</code> will be used in
|
|
|
|
|
|
<code>TEST_VM</code>, <code>TEST_OTHER_VM</code> and
|
|
|
|
|
|
<code>TEST_VM_ASSERT*</code> tests.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="test-specific-flags">Test-specific flags</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Passing flags to a tested JVM in <code>TEST_OTHER_VM</code> and
|
|
|
|
|
|
<code>TEST_VM_ASSERT*</code> should be possible, but is not implemented
|
|
|
|
|
|
yet.</p>
|
|
|
|
|
|
<p>Facility to pass test-specific flags is needed for system, regression
|
|
|
|
|
|
or other types of tests which require a fully initialized JVM in some
|
|
|
|
|
|
particular configuration, e.g. with Serial GC selected. There is no
|
|
|
|
|
|
support for such tests now, however, there is a plan to add that in
|
|
|
|
|
|
upcoming releases.</p>
|
|
|
|
|
|
<p>For now, if a test depends on flags values, it should have
|
|
|
|
|
|
<code>if (!<flag>) { return }</code> guards in the very beginning
|
|
|
|
|
|
and <code>@requires</code> comment similar to jtreg
|
|
|
|
|
|
<code>@requires</code> directive right before test macros. <a
|
|
|
|
|
|
href="https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/g1/test_g1IHOPControl.cpp"
|
|
|
|
|
|
class="uri">https://git.openjdk.org/jdk/blob/master/test/hotspot/gtest/gc/g1/test_g1IHOPControl.cpp</a>
|
|
|
|
|
|
ha an example of this temporary workaround. It is important to follow
|
|
|
|
|
|
that pattern as it allows us to easily find all such tests and update
|
|
|
|
|
|
them as soon as there is an implementation of flag passing facility.</p>
|
|
|
|
|
|
<p>In long-term, we expect jtreg to support GoogleTest tests as first
|
|
|
|
|
|
class citizens, that is to say, jtreg will parse <span class="citation"
|
|
|
|
|
|
data-cites="requires">@requires</span> comments and filter out
|
|
|
|
|
|
inapplicable tests.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h3 id="flag-restoring">Flag restoring</h3>
|
|
|
|
|
|
<p>Restore changed flags.</p>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>It is quite common for tests to configure JVM in a certain way
|
|
|
|
|
|
changing flags’ values. GoogleTest provides two ways to set up
|
|
|
|
|
|
environment before a test and restore it afterward: using either
|
|
|
|
|
|
constructor and destructor or <code>SetUp</code> and
|
|
|
|
|
|
<code>TearDown</code> functions. Both ways require to use a test fixture
|
|
|
|
|
|
class, which sometimes is too wordy. The simpler facilities like
|
|
|
|
|
|
<code>FLAG_GUARD</code> macro or <code>*FlagSetting</code> classes could
|
|
|
|
|
|
be used in such cases to restore/set values.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<p>Caveats:</p>
|
|
|
|
|
|
<ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><p>Changing a flag’s value could break the invariants between flags'
|
|
|
|
|
|
values and hence could lead to unexpected/unsupported JVM
|
|
|
|
|
|
state.</p></li>
|
|
|
|
|
|
<li><p><code>FLAG_SET_*</code> macros can change more than one flag (in
|
|
|
|
|
|
order to maintain invariants) so it is hard to predict what flags will
|
|
|
|
|
|
be changed and it makes restoring all changed flags a nontrivial task.
|
|
|
|
|
|
Thus in case one uses <code>FLAG_SET_*</code> macros, they should use
|
|
|
|
|
|
<code>TEST_OTHER_VM</code> test type.</p></li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul>
|
|
|
|
|
|
<h3 id="googletest-documentation">GoogleTest documentation</h3>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>In case you have any questions regarding GoogleTest itself, its
|
|
|
|
|
|
asserts, test declaration macros, other macros, etc, please consult its
|
|
|
|
|
|
documentation.</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<h2 id="todo">TODO</h2>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<p>Although this document provides guidelines on the most important
|
|
|
|
|
|
parts of test development using GTest, it still misses a few items:</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><p>Examples, esp for <a href="#access-to-non-public-members">access
|
|
|
|
|
|
to non-public members</a></p></li>
|
|
|
|
|
|
<li><p>test types: purpose, drawbacks, limitation</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
|
|
|
|
|
<li><code>TEST_VM</code></li>
|
|
|
|
|
|
<li><code>TEST_VM_F</code></li>
|
|
|
|
|
|
<li><code>TEST_OTHER_VM</code></li>
|
|
|
|
|
|
<li><code>TEST_VM_ASSERT</code></li>
|
|
|
|
|
|
<li><code>TEST_VM_ASSERT_MSG</code></li>
|
|
|
|
|
|
</ul></li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li><p>Miscellaneous</p>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
<ul>
|
|
|
|
|
|
<li>Test libraries
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li>where to place</li>
|
|
|
|
|
|
<li>how to write</li>
|
|
|
|
|
|
<li>how to use</li>
|
|
|
|
|
|
</ul></li>
|
|
|
|
|
|
<li>test your tests
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li>how to run tests in random order</li>
|
|
|
|
|
|
<li>how to run only specific tests</li>
|
|
|
|
|
|
<li>how to run each test separately</li>
|
2022-11-21 18:32:40 +00:00
|
|
|
|
<li>check that a test can find bugs it is supposed to by introducing
|
|
|
|
|
|
them</li>
|
2020-08-19 10:11:15 +00:00
|
|
|
|
</ul></li>
|
|
|
|
|
|
<li>mocks/stubs/dependency injection</li>
|
|
|
|
|
|
<li>setUp/tearDown
|
|
|
|
|
|
<ul>
|
|
|
|
|
|
<li>vs c-tor/d-tor</li>
|
|
|
|
|
|
<li>empty test to test them</li>
|
|
|
|
|
|
</ul></li>
|
|
|
|
|
|
<li>internal (declared in .cpp) struct/classes</li>
|
|
|
|
|
|
</ul></li>
|
|
|
|
|
|
</ul>
|
|
|
|
|
|
</body>
|
|
|
|
|
|
</html>
|
