8295663: Rephrase introduction to testing.md
Reviewed-by: ihse, erikj
This commit is contained in:
Ludvig Janiuk
Magnus Ihse Bursie
parent
7e85b41d37
commit
b6738c1831
@ -23,7 +23,8 @@
|
||||
</header>
|
||||
<nav id="TOC">
|
||||
<ul>
|
||||
<li><a href="#using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</a><ul>
|
||||
<li><a href="#overview">Overview</a></li>
|
||||
<li><a href="#running-tests-locally-with-make-test">Running tests locally with <code>make test</code></a><ul>
|
||||
<li><a href="#configuration">Configuration</a></li>
|
||||
</ul></li>
|
||||
<li><a href="#test-selection">Test selection</a><ul>
|
||||
@ -49,10 +50,12 @@
|
||||
<li><a href="#editing-this-document">Editing this document</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
<h2 id="using-make-test-the-run-test-framework">Using "make test" (the run-test framework)</h2>
|
||||
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
|
||||
<p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
|
||||
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases.</p>
|
||||
<h2 id="overview">Overview</h2>
|
||||
<p>The bulk of JDK tests use <a href="https://openjdk.org/jtreg/">jtreg</a>, a regression test framework and test runner built for the JDK's specific needs. Other test frameworks are also used. The different test frameworks can be executed directly, but there is also a set of make targets intended to simplify the interface, and figure out how to run your tests for you.</p>
|
||||
<h2 id="running-tests-locally-with-make-test">Running tests locally with <code>make test</code></h2>
|
||||
<p>This is the easiest way to get started. Assuming you've built the JDK locally, execute:</p>
|
||||
<pre><code>$ make test</code></pre>
|
||||
<p>This will run a default set of tests against the JDK, and present you with the results. <code>make test</code> is part of a family of test-related make targets which simplify running tests, because they invoke the various test frameworks for you. The "make test framework" is simple to start with, but more complex ad-hoc combination of tests is also possible. You can always invoke the test frameworks directly if you want even more control.</p>
|
||||
<p>Some example command-lines:</p>
|
||||
<pre><code>$ make test-tier1
|
||||
$ make test-jdk_lang JTREG="JOBS=8"
|
||||
@ -62,6 +65,8 @@ $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT_FACTO
|
||||
$ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
|
||||
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
|
||||
$ make exploded-test TEST=tier2</code></pre>
|
||||
<p>"tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a test selection argument which the make test framework will use to try to find the tests you want. It iterates over the available test frameworks, and if the test isn't present in one, it tries the next one. The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
|
||||
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code> are kept as aliases.</p>
|
||||
<h3 id="configuration">Configuration</h3>
|
||||
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=<path to jtreg home></code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
|
||||
<p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=<path to JMH jars></code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
|
||||
@ -81,8 +86,8 @@ $ make exploded-test TEST=tier2</code></pre>
|
||||
<p>A brief description of the tiered test groups:</p>
|
||||
<ul>
|
||||
<li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li>
|
||||
<li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components (for example, XML).</p></li>
|
||||
<li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests (<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li>
|
||||
<li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components(for example, XML).</p></li>
|
||||
<li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests(<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li>
|
||||
<li><p><code>tier4</code>: This test group includes every other test not covered by previous tiers. It includes, for example, <code>vmTestbase</code> suites for Hotspot, which run for many hours even on large machines. It also runs GUI tests, so the same <code>TEST_JOBS</code> and <code>JTREG_KEYWORDS</code> caveats apply.</p></li>
|
||||
</ul>
|
||||
<h3 id="jtreg">JTReg</h3>
|
||||
|
||||
115
doc/testing.md
115
doc/testing.md
@ -1,21 +1,26 @@
|
||||
% Testing the JDK
|
||||
|
||||
## Using "make test" (the run-test framework)
|
||||
## Overview
|
||||
|
||||
This new way of running tests is developer-centric. It assumes that you have
|
||||
built a JDK locally and want to test it. Running common test targets is simple,
|
||||
and more complex ad-hoc combination of tests is possible. The user interface is
|
||||
forgiving, and clearly report errors it cannot resolve.
|
||||
The bulk of JDK tests use [jtreg](https://openjdk.org/jtreg/), a regression
|
||||
test framework and test runner built for the JDK's specific needs. Other test
|
||||
frameworks are also used. The different test frameworks can be executed
|
||||
directly, but there is also a set of make targets intended to simplify the
|
||||
interface, and figure out how to run your tests for you.
|
||||
|
||||
The main target `test` uses the jdk-image as the tested product. There is
|
||||
also an alternate target `exploded-test` that uses the exploded image
|
||||
instead. Not all tests will run successfully on the exploded image, but using
|
||||
this target can greatly improve rebuild times for certain workflows.
|
||||
## Running tests locally with `make test`
|
||||
|
||||
Previously, `make test` was used to invoke an old system for running tests, and
|
||||
`make run-test` was used for the new test framework. For backward compatibility
|
||||
with scripts and muscle memory, `run-test` (and variants like
|
||||
`exploded-run-test` or `run-test-tier1`) are kept as aliases.
|
||||
This is the easiest way to get started. Assuming you've built the JDK locally,
|
||||
execute:
|
||||
|
||||
$ make test
|
||||
|
||||
This will run a default set of tests against the JDK, and present you with the
|
||||
results. `make test` is part of a family of test-related make targets which
|
||||
simplify running tests, because they invoke the various test frameworks for
|
||||
you. The "make test framework" is simple to start with, but more complex
|
||||
ad-hoc combination of tests is also possible. You can always invoke the test
|
||||
frameworks directly if you want even more control.
|
||||
|
||||
Some example command-lines:
|
||||
|
||||
@ -28,6 +33,20 @@ Some example command-lines:
|
||||
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
|
||||
$ make exploded-test TEST=tier2
|
||||
|
||||
"tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a
|
||||
test selection argument which the make test framework will use to try to
|
||||
find the tests you want. It iterates over the available test frameworks, and
|
||||
if the test isn't present in one, it tries the next one. The main target
|
||||
`test` uses the jdk-image as the tested product. There is also an alternate
|
||||
target `exploded-test` that uses the exploded image instead. Not all tests
|
||||
will run successfully on the exploded image, but using this target can
|
||||
greatly improve rebuild times for certain workflows.
|
||||
|
||||
Previously, `make test` was used to invoke an old system for running tests,
|
||||
and `make run-test` was used for the new test framework. For backward
|
||||
compatibility with scripts and muscle memory, `run-test` and variants like
|
||||
`exploded-run-test` or `run-test-tier1` are kept as aliases.
|
||||
|
||||
### Configuration
|
||||
|
||||
To be able to run JTReg tests, `configure` needs to know where to find the
|
||||
@ -91,49 +110,53 @@ if you want to shortcut the parser.
|
||||
|
||||
### Common Test Groups
|
||||
|
||||
Ideally, all tests are run for every change but this may not be practical due to the limited
|
||||
testing resources, the scope of the change, etc.
|
||||
Ideally, all tests are run for every change but this may not be practical due
|
||||
to the limited testing resources, the scope of the change, etc.
|
||||
|
||||
The source tree currently defines a few common test groups in the relevant `TEST.groups`
|
||||
files. There are test groups that cover a specific component, for example `hotspot_gc`.
|
||||
It is a good idea to look into `TEST.groups` files to get a sense what tests are relevant
|
||||
to a particular JDK component.
|
||||
The source tree currently defines a few common test groups in the relevant
|
||||
`TEST.groups` files. There are test groups that cover a specific component,
|
||||
for example `hotspot_gc`. It is a good idea to look into `TEST.groups` files
|
||||
to get a sense what tests are relevant to a particular JDK component.
|
||||
|
||||
Component-specific tests may miss some unintended consequences of a change, so other
|
||||
tests should also be run. Again, it might be impractical to run all tests, and therefore
|
||||
_tiered_ test groups exist. Tiered test groups are not component-specific, but rather cover
|
||||
the significant parts of the entire JDK.
|
||||
Component-specific tests may miss some unintended consequences of a change, so
|
||||
other tests should also be run. Again, it might be impractical to run all
|
||||
tests, and therefore
|
||||
_tiered_ test groups exist. Tiered test groups are not component-specific, but
|
||||
rather cover the significant parts of the entire JDK.
|
||||
|
||||
Multiple tiers allow balancing test coverage and testing costs. Lower test tiers are supposed to
|
||||
contain the simpler, quicker and more stable tests. Higher tiers are supposed to contain
|
||||
progressively more thorough, slower, and sometimes less stable tests, or the tests that require
|
||||
special configuration.
|
||||
Multiple tiers allow balancing test coverage and testing costs. Lower test
|
||||
tiers are supposed to contain the simpler, quicker and more stable tests.
|
||||
Higher tiers are supposed to contain progressively more thorough, slower, and
|
||||
sometimes less stable tests, or the tests that require special
|
||||
configuration.
|
||||
|
||||
Contributors are expected to run the tests for the areas that are changed, and the first N tiers
|
||||
they can afford to run, but at least tier1.
|
||||
Contributors are expected to run the tests for the areas that are changed, and
|
||||
the first N tiers they can afford to run, but at least tier1.
|
||||
|
||||
A brief description of the tiered test groups:
|
||||
|
||||
- `tier1`: This is the lowest test tier. Multiple developers run these tests every day.
|
||||
Because of the widespread use, the tests in `tier1` are carefully selected and optimized to run
|
||||
fast, and to run in the most stable manner. The test failures in `tier1` are usually followed up
|
||||
on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows,
|
||||
if enabled, run `tier1` tests.
|
||||
- `tier1`: This is the lowest test tier. Multiple developers run these tests
|
||||
every day. Because of the widespread use, the tests in `tier1` are
|
||||
carefully selected and optimized to run fast, and to run in the most stable
|
||||
manner. The test failures in `tier1` are usually followed up on quickly,
|
||||
either with fixes, or adding relevant tests to problem list. GitHub Actions
|
||||
workflows, if enabled, run `tier1` tests.
|
||||
|
||||
- `tier2`: This test group covers even more ground. These contain, among other things,
|
||||
tests that either run for too long to be at `tier1`, or may require special configuration,
|
||||
or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components
|
||||
(for example, XML).
|
||||
- `tier2`: This test group covers even more ground. These contain, among other
|
||||
things, tests that either run for too long to be at `tier1`, or may require
|
||||
special configuration, or tests that are less stable, or cover the broader
|
||||
range of non-core JVM and JDK features/components(for example, XML).
|
||||
|
||||
- `tier3`: This test group includes more stressful tests, the tests for corner cases
|
||||
not covered by previous tiers, plus the tests that require GUIs. As such, this suite
|
||||
should either be run with low concurrency (`TEST_JOBS=1`), or without headful tests
|
||||
(`JTREG_KEYWORDS=\!headful`), or both.
|
||||
- `tier3`: This test group includes more stressful tests, the tests for corner
|
||||
cases not covered by previous tiers, plus the tests that require GUIs. As
|
||||
such, this suite should either be run with low concurrency
|
||||
(`TEST_JOBS=1`), or without headful tests(`JTREG_KEYWORDS=\!headful`), or
|
||||
both.
|
||||
|
||||
- `tier4`: This test group includes every other test not covered by previous tiers. It includes,
|
||||
for example, `vmTestbase` suites for Hotspot, which run for many hours even on large
|
||||
machines. It also runs GUI tests, so the same `TEST_JOBS` and `JTREG_KEYWORDS` caveats
|
||||
apply.
|
||||
- `tier4`: This test group includes every other test not covered by previous
|
||||
tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run
|
||||
for many hours even on large machines. It also runs GUI tests, so the same
|
||||
`TEST_JOBS` and `JTREG_KEYWORDS` caveats apply.
|
||||
|
||||
### JTReg
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user