stan/jdk-24
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

8295663: Rephrase introduction to testing.md

Browse Source 
Reviewed-by: ihse, erikj
This commit is contained in:
Ludvig Janiuk 2022-11-07 23:12:21 +00:00 committed by Magnus Ihse Bursie
parent 7e85b41d37
commit b6738c1831
2 changed files with 81 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
doc/testing.html
View File

@ -23,7 +23,8 @@
</header> </header>
<nav id="TOC"> <nav id="TOC">
<ul> <ul>
<li><a href="#using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</a><ul> <li><a href="#overview">Overview</a></li>
<li><a href="#running-tests-locally-with-make-test">Running tests locally with <code>make test</code></a><ul>
<li><a href="#configuration">Configuration</a></li> <li><a href="#configuration">Configuration</a></li>
</ul></li> </ul></li>
<li><a href="#test-selection">Test selection</a><ul> <li><a href="#test-selection">Test selection</a><ul>
@ -49,10 +50,12 @@
<li><a href="#editing-this-document">Editing this document</a></li> <li><a href="#editing-this-document">Editing this document</a></li>
</ul> </ul>
</nav> </nav>
<h2 id="using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</h2> <h2 id="overview">Overview</h2>
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p> <p>The bulk of JDK tests use <a href="https://openjdk.org/jtreg/">jtreg</a>, a regression test framework and test runner built for the JDK's specific needs. Other test frameworks are also used. The different test frameworks can be executed directly, but there is also a set of make targets intended to simplify the interface, and figure out how to run your tests for you.</p>
<p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p> <h2 id="running-tests-locally-with-make-test">Running tests locally with <code>make test</code></h2>
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases.</p> <p>This is the easiest way to get started. Assuming you've built the JDK locally, execute:</p>
<pre><code>$ make test</code></pre>
<p>This will run a default set of tests against the JDK, and present you with the results. <code>make test</code> is part of a family of test-related make targets which simplify running tests, because they invoke the various test frameworks for you. The &quot;make test framework&quot; is simple to start with, but more complex ad-hoc combination of tests is also possible. You can always invoke the test frameworks directly if you want even more control.</p>
<p>Some example command-lines:</p> <p>Some example command-lines:</p>
<pre><code>$ make test-tier1 <pre><code>$ make test-tier1
$ make test-jdk_lang JTREG=&quot;JOBS=8&quot; $ make test-jdk_lang JTREG=&quot;JOBS=8&quot;
@ -62,6 +65,8 @@ $ make test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT_FACTO
$ make test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot; $ make test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
$ make test TEST=&quot;micro:java.lang.reflect&quot; MICRO=&quot;FORK=1;WARMUP_ITER=2&quot; $ make test TEST=&quot;micro:java.lang.reflect&quot; MICRO=&quot;FORK=1;WARMUP_ITER=2&quot;
$ make exploded-test TEST=tier2</code></pre> $ make exploded-test TEST=tier2</code></pre>
<p>&quot;tier1&quot; and &quot;tier2&quot; refer to tiered testing, see further down. &quot;TEST&quot; is a test selection argument which the make test framework will use to try to find the tests you want. It iterates over the available test frameworks, and if the test isn't present in one, it tries the next one. The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
<p>Previously, <code>make test</code> was used to invoke an old system for running tests, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code> are kept as aliases.</p>
<h3 id="configuration">Configuration</h3> <h3 id="configuration">Configuration</h3>
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
<p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=&lt;path to JMH jars&gt;</code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p> <p>To be able to run microbenchmarks, <code>configure</code> needs to know where to find the JMH dependency. Use <code>--with-jmh=&lt;path to JMH jars&gt;</code> to point to a directory containing the core JMH and transitive dependencies. The recommended dependencies can be retrieved by running <code>sh make/devkit/createJMHBundle.sh</code>, after which <code>--with-jmh=build/jmh/jars</code> should work.</p>
@ -81,8 +86,8 @@ $ make exploded-test TEST=tier2</code></pre>
<p>A brief description of the tiered test groups:</p> <p>A brief description of the tiered test groups:</p>
<ul> <ul>
<li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li> <li><p><code>tier1</code>: This is the lowest test tier. Multiple developers run these tests every day. Because of the widespread use, the tests in <code>tier1</code> are carefully selected and optimized to run fast, and to run in the most stable manner. The test failures in <code>tier1</code> are usually followed up on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, if enabled, run <code>tier1</code> tests.</p></li>
<li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components (for example, XML).</p></li> <li><p><code>tier2</code>: This test group covers even more ground. These contain, among other things, tests that either run for too long to be at <code>tier1</code>, or may require special configuration, or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components(for example, XML).</p></li>
<li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests (<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li> <li><p><code>tier3</code>: This test group includes more stressful tests, the tests for corner cases not covered by previous tiers, plus the tests that require GUIs. As such, this suite should either be run with low concurrency (<code>TEST_JOBS=1</code>), or without headful tests(<code>JTREG_KEYWORDS=\!headful</code>), or both.</p></li>
<li><p><code>tier4</code>: This test group includes every other test not covered by previous tiers. It includes, for example, <code>vmTestbase</code> suites for Hotspot, which run for many hours even on large machines. It also runs GUI tests, so the same <code>TEST_JOBS</code> and <code>JTREG_KEYWORDS</code> caveats apply.</p></li> <li><p><code>tier4</code>: This test group includes every other test not covered by previous tiers. It includes, for example, <code>vmTestbase</code> suites for Hotspot, which run for many hours even on large machines. It also runs GUI tests, so the same <code>TEST_JOBS</code> and <code>JTREG_KEYWORDS</code> caveats apply.</p></li>
</ul> </ul>
<h3 id="jtreg">JTReg</h3> <h3 id="jtreg">JTReg</h3>

115
doc/testing.md
View File

@ -1,21 +1,26 @@
% Testing the JDK % Testing the JDK
## Using "make test" (the run-test framework) ## Overview
This new way of running tests is developer-centric. It assumes that you have The bulk of JDK tests use [jtreg](https://openjdk.org/jtreg/), a regression
built a JDK locally and want to test it. Running common test targets is simple, test framework and test runner built for the JDK's specific needs. Other test
and more complex ad-hoc combination of tests is possible. The user interface is frameworks are also used. The different test frameworks can be executed
forgiving, and clearly report errors it cannot resolve. directly, but there is also a set of make targets intended to simplify the
interface, and figure out how to run your tests for you.
The main target `test` uses the jdk-image as the tested product. There is ## Running tests locally with `make test`
also an alternate target `exploded-test` that uses the exploded image
instead. Not all tests will run successfully on the exploded image, but using
this target can greatly improve rebuild times for certain workflows.
Previously, `make test` was used to invoke an old system for running tests, and This is the easiest way to get started. Assuming you've built the JDK locally,
`make run-test` was used for the new test framework. For backward compatibility execute:
with scripts and muscle memory, `run-test` (and variants like
`exploded-run-test` or `run-test-tier1`) are kept as aliases. $ make test
This will run a default set of tests against the JDK, and present you with the
results. `make test` is part of a family of test-related make targets which
simplify running tests, because they invoke the various test frameworks for
you. The "make test framework" is simple to start with, but more complex
ad-hoc combination of tests is also possible. You can always invoke the test
frameworks directly if you want even more control.
Some example command-lines: Some example command-lines:
@ -28,6 +33,20 @@ Some example command-lines:
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
$ make exploded-test TEST=tier2 $ make exploded-test TEST=tier2
"tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a
test selection argument which the make test framework will use to try to
find the tests you want. It iterates over the available test frameworks, and
if the test isn't present in one, it tries the next one. The main target
`test` uses the jdk-image as the tested product. There is also an alternate
target `exploded-test` that uses the exploded image instead. Not all tests
will run successfully on the exploded image, but using this target can
greatly improve rebuild times for certain workflows.
Previously, `make test` was used to invoke an old system for running tests,
and `make run-test` was used for the new test framework. For backward
compatibility with scripts and muscle memory, `run-test` and variants like
`exploded-run-test` or `run-test-tier1` are kept as aliases.
### Configuration ### Configuration
To be able to run JTReg tests, `configure` needs to know where to find the To be able to run JTReg tests, `configure` needs to know where to find the
@ -91,49 +110,53 @@ if you want to shortcut the parser.
### Common Test Groups ### Common Test Groups
Ideally, all tests are run for every change but this may not be practical due to the limited Ideally, all tests are run for every change but this may not be practical due
testing resources, the scope of the change, etc. to the limited testing resources, the scope of the change, etc.
The source tree currently defines a few common test groups in the relevant `TEST.groups` The source tree currently defines a few common test groups in the relevant
files. There are test groups that cover a specific component, for example `hotspot_gc`. `TEST.groups` files. There are test groups that cover a specific component,
It is a good idea to look into `TEST.groups` files to get a sense what tests are relevant for example `hotspot_gc`. It is a good idea to look into `TEST.groups` files
to a particular JDK component. to get a sense what tests are relevant to a particular JDK component.
Component-specific tests may miss some unintended consequences of a change, so other Component-specific tests may miss some unintended consequences of a change, so
tests should also be run. Again, it might be impractical to run all tests, and therefore other tests should also be run. Again, it might be impractical to run all
_tiered_ test groups exist. Tiered test groups are not component-specific, but rather cover tests, and therefore
the significant parts of the entire JDK. _tiered_ test groups exist. Tiered test groups are not component-specific, but
rather cover the significant parts of the entire JDK.
Multiple tiers allow balancing test coverage and testing costs. Lower test tiers are supposed to Multiple tiers allow balancing test coverage and testing costs. Lower test
contain the simpler, quicker and more stable tests. Higher tiers are supposed to contain tiers are supposed to contain the simpler, quicker and more stable tests.
progressively more thorough, slower, and sometimes less stable tests, or the tests that require Higher tiers are supposed to contain progressively more thorough, slower, and
special configuration. sometimes less stable tests, or the tests that require special
configuration.
Contributors are expected to run the tests for the areas that are changed, and the first N tiers Contributors are expected to run the tests for the areas that are changed, and
they can afford to run, but at least tier1. the first N tiers they can afford to run, but at least tier1.
A brief description of the tiered test groups: A brief description of the tiered test groups:
- `tier1`: This is the lowest test tier. Multiple developers run these tests every day. - `tier1`: This is the lowest test tier. Multiple developers run these tests
Because of the widespread use, the tests in `tier1` are carefully selected and optimized to run every day. Because of the widespread use, the tests in `tier1` are
fast, and to run in the most stable manner. The test failures in `tier1` are usually followed up carefully selected and optimized to run fast, and to run in the most stable
on quickly, either with fixes, or adding relevant tests to problem list. GitHub Actions workflows, manner. The test failures in `tier1` are usually followed up on quickly,
if enabled, run `tier1` tests. either with fixes, or adding relevant tests to problem list. GitHub Actions
workflows, if enabled, run `tier1` tests.
- `tier2`: This test group covers even more ground. These contain, among other things, - `tier2`: This test group covers even more ground. These contain, among other
tests that either run for too long to be at `tier1`, or may require special configuration, things, tests that either run for too long to be at `tier1`, or may require
or tests that are less stable, or cover the broader range of non-core JVM and JDK features/components special configuration, or tests that are less stable, or cover the broader
(for example, XML). range of non-core JVM and JDK features/components(for example, XML).
- `tier3`: This test group includes more stressful tests, the tests for corner cases - `tier3`: This test group includes more stressful tests, the tests for corner
not covered by previous tiers, plus the tests that require GUIs. As such, this suite cases not covered by previous tiers, plus the tests that require GUIs. As
should either be run with low concurrency (`TEST_JOBS=1`), or without headful tests such, this suite should either be run with low concurrency
(`JTREG_KEYWORDS=\!headful`), or both. (`TEST_JOBS=1`), or without headful tests(`JTREG_KEYWORDS=\!headful`), or
both.
- `tier4`: This test group includes every other test not covered by previous tiers. It includes, - `tier4`: This test group includes every other test not covered by previous
for example, `vmTestbase` suites for Hotspot, which run for many hours even on large tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run
machines. It also runs GUI tests, so the same `TEST_JOBS` and `JTREG_KEYWORDS` caveats for many hours even on large machines. It also runs GUI tests, so the same
apply. `TEST_JOBS` and `JTREG_KEYWORDS` caveats apply.
### JTReg ### JTReg