stan/jdk-24
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

8320334: Reflow markdown in testing.md

Browse Source 
Reviewed-by: iris
This commit is contained in:
Magnus Ihse Bursie 2023-11-17 18:09:56 +00:00
parent 9902d2eb17
commit 1fce70b666
1 changed files with 103 additions and 107 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

210
doc/testing.md
View File

@ -18,8 +18,8 @@ execute:
This will run a default set of tests against the JDK, and present you with the This will run a default set of tests against the JDK, and present you with the
results. `make test` is part of a family of test-related make targets which results. `make test` is part of a family of test-related make targets which
simplify running tests, because they invoke the various test frameworks for simplify running tests, because they invoke the various test frameworks for
you. The "make test framework" is simple to start with, but more complex you. The "make test framework" is simple to start with, but more complex ad-hoc
ad-hoc combination of tests is also possible. You can always invoke the test combination of tests is also possible. You can always invoke the test
frameworks directly if you want even more control. frameworks directly if you want even more control.
Some example command-lines: Some example command-lines:
@ -33,18 +33,18 @@ Some example command-lines:
$ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2" $ make test TEST="micro:java.lang.reflect" MICRO="FORK=1;WARMUP_ITER=2"
$ make exploded-test TEST=tier2 $ make exploded-test TEST=tier2
"tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a "tier1" and "tier2" refer to tiered testing, see further down. "TEST" is a test
test selection argument which the make test framework will use to try to selection argument which the make test framework will use to try to find the
find the tests you want. It iterates over the available test frameworks, and tests you want. It iterates over the available test frameworks, and if the
if the test isn't present in one, it tries the next one. The main target test isn't present in one, it tries the next one. The main target `test` uses
`test` uses the jdk-image as the tested product. There is also an alternate the jdk-image as the tested product. There is also an alternate target
target `exploded-test` that uses the exploded image instead. Not all tests `exploded-test` that uses the exploded image instead. Not all tests will run
will run successfully on the exploded image, but using this target can successfully on the exploded image, but using this target can greatly improve
greatly improve rebuild times for certain workflows. rebuild times for certain workflows.
Previously, `make test` was used to invoke an old system for running tests, Previously, `make test` was used to invoke an old system for running tests, and
and `make run-test` was used for the new test framework. For backward `make run-test` was used for the new test framework. For backward compatibility
compatibility with scripts and muscle memory, `run-test` and variants like with scripts and muscle memory, `run-test` and variants like
`exploded-run-test` or `run-test-tier1` are kept as aliases. `exploded-run-test` or `run-test-tier1` are kept as aliases.
### Configuration ### Configuration
@ -64,26 +64,27 @@ after which `--with-jmh=build/jmh/jars` should work.
When tests fail or timeout, jtreg runs its failure handler to capture necessary When tests fail or timeout, jtreg runs its failure handler to capture necessary
data from the system where the test was run. This data can then be used to data from the system where the test was run. This data can then be used to
analyze the test failures. Collecting this data involves running various commands analyze the test failures. Collecting this data involves running various
(which are listed in files residing in `test/failure_handler/src/share/conf`) commands (which are listed in files residing in
and some of these commands use `sudo`. If the system's `sudoers` file isn't `test/failure_handler/src/share/conf`) and some of these commands use `sudo`.
configured to allow running these commands, then it can result in password being If the system's `sudoers` file isn't configured to allow running these
prompted during the failure handler execution. Typically, when running locally, commands, then it can result in password being prompted during the failure
collecting this additional data isn't always necessary. To disable running the handler execution. Typically, when running locally, collecting this additional
failure handler, use `--enable-jtreg-failure-handler=no` when running `configure`. data isn't always necessary. To disable running the failure handler, use
If, however, you want to let the failure handler to run and don't want to be `--enable-jtreg-failure-handler=no` when running `configure`. If, however, you
prompted for sudo password, then you can configure your `sudoers` file want to let the failure handler to run and don't want to be prompted for sudo
appropriately. Please read the necessary documentation of your operating system password, then you can configure your `sudoers` file appropriately. Please read
to see how to do that; here we only show one possible way of doing that - edit the necessary documentation of your operating system to see how to do that;
the `/etc/sudoers.d/sudoers` file to include the following line: here we only show one possible way of doing that - edit the
`/etc/sudoers.d/sudoers` file to include the following line:
``` ```
johndoe ALL=(ALL) NOPASSWD: /sbin/dmesg johndoe ALL=(ALL) NOPASSWD: /sbin/dmesg
``` ```
This line configures `sudo` to _not_ prompt for password for the `/sbin/dmesg` This line configures `sudo` to _not_ prompt for password for the `/sbin/dmesg`
command (this is one of the commands that is listed in the files command (this is one of the commands that is listed in the files at
at `test/failure_handler/src/share/conf`), for the user `johndoe`. Here `johndoe` `test/failure_handler/src/share/conf`), for the user `johndoe`. Here `johndoe`
is the user account under which the jtreg tests are run. Replace the username is the user account under which the jtreg tests are run. Replace the username
with a relevant user account of your system. with a relevant user account of your system.
@ -114,42 +115,38 @@ Ideally, all tests are run for every change but this may not be practical due
to the limited testing resources, the scope of the change, etc. to the limited testing resources, the scope of the change, etc.
The source tree currently defines a few common test groups in the relevant The source tree currently defines a few common test groups in the relevant
`TEST.groups` files. There are test groups that cover a specific component, `TEST.groups` files. There are test groups that cover a specific component, for
for example `hotspot_gc`. It is a good idea to look into `TEST.groups` files example `hotspot_gc`. It is a good idea to look into `TEST.groups` files to get
to get a sense what tests are relevant to a particular JDK component. a sense what tests are relevant to a particular JDK component.
Component-specific tests may miss some unintended consequences of a change, so Component-specific tests may miss some unintended consequences of a change, so
other tests should also be run. Again, it might be impractical to run all other tests should also be run. Again, it might be impractical to run all
tests, and therefore tests, and therefore _tiered_ test groups exist. Tiered test groups are not
_tiered_ test groups exist. Tiered test groups are not component-specific, but component-specific, but rather cover the significant parts of the entire JDK.
rather cover the significant parts of the entire JDK.
Multiple tiers allow balancing test coverage and testing costs. Lower test Multiple tiers allow balancing test coverage and testing costs. Lower test
tiers are supposed to contain the simpler, quicker and more stable tests. tiers are supposed to contain the simpler, quicker and more stable tests.
Higher tiers are supposed to contain progressively more thorough, slower, and Higher tiers are supposed to contain progressively more thorough, slower, and
sometimes less stable tests, or the tests that require special sometimes less stable tests, or the tests that require special configuration.
configuration.
Contributors are expected to run the tests for the areas that are changed, and Contributors are expected to run the tests for the areas that are changed, and
the first N tiers they can afford to run, but at least tier1. the first N tiers they can afford to run, but at least tier1.
A brief description of the tiered test groups: A brief description of the tiered test groups:
- `tier1`: This is the most fundamental test tier. - `tier1`: This is the most fundamental test tier. Roughly speaking, a failure
Roughly speaking, a failure of a test in this tier has the potential of a test in this tier has the potential to indicate a problem that would
to indicate a problem that would affect many Java programs. Tests in affect many Java programs. Tests in `tier1` include tests of HotSpot, core
`tier1` include tests of HotSpot, core APIs in the `java.base` APIs in the `java.base` module, and the `javac` compiler. Multiple developers
module, and the `javac` compiler. Multiple developers run these run these tests every day. Because of the widespread use, the tests in
tests every day. Because of the widespread use, the tests in `tier1` `tier1` are carefully selected and optimized to run fast, and to run in the
are carefully selected and optimized to run fast, and to run in the most stable manner. As a guideline, nearly all individual tests in `tier1`
most stable manner. As a guideline, nearly all individual tests in are expected to run to completion in ten seconds or less when run on common
`tier1` are expected to run to completion in ten seconds or less configurations used for development. Long-running tests, even of core
when run on common configurations used for development. Long-running functionality, should occur in higher tiers or be covered in other kinds of
tests, even of core functionality, should occur in higher tiers or testing. The test failures in `tier1` are usually followed up on quickly,
be covered in other kinds of testing. The test failures in `tier1` either with fixes, or adding relevant tests to problem list. GitHub Actions
are usually followed up on quickly, either with fixes, or adding workflows, if enabled, run `tier1` tests.
relevant tests to problem list. GitHub Actions workflows, if
enabled, run `tier1` tests.
- `tier2`: This test group covers even more ground. These contain, among other - `tier2`: This test group covers even more ground. These contain, among other
things, tests that either run for too long to be at `tier1`, or may require things, tests that either run for too long to be at `tier1`, or may require
@ -158,9 +155,8 @@ A brief description of the tiered test groups:
- `tier3`: This test group includes more stressful tests, the tests for corner - `tier3`: This test group includes more stressful tests, the tests for corner
cases not covered by previous tiers, plus the tests that require GUIs. As cases not covered by previous tiers, plus the tests that require GUIs. As
such, this suite should either be run with low concurrency such, this suite should either be run with low concurrency (`TEST_JOBS=1`),
(`TEST_JOBS=1`), or without headful tests(`JTREG_KEYWORDS=\!headful`), or or without headful tests(`JTREG_KEYWORDS=\!headful`), or both.
both.
- `tier4`: This test group includes every other test not covered by previous - `tier4`: This test group includes every other test not covered by previous
tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run tiers. It includes, for example, `vmTestbase` suites for Hotspot, which run
@ -169,18 +165,17 @@ A brief description of the tiered test groups:
### JTReg ### JTReg
JTReg tests can be selected either by picking a JTReg test group, or a selection JTReg tests can be selected either by picking a JTReg test group, or a
of files or directories containing JTReg tests. Documentation can be found at selection of files or directories containing JTReg tests. Documentation can be
[https://openjdk.org/jtreg/](https://openjdk.org/jtreg/), note especially the found at [https://openjdk.org/jtreg/](https://openjdk.org/jtreg/), note
extensive [FAQ](https://openjdk.org/jtreg/faq.html). especially the extensive [FAQ](https://openjdk.org/jtreg/faq.html).
JTReg test groups can be specified either without a test root, e.g. `:tier1` JTReg test groups can be specified either without a test root, e.g. `:tier1`
(or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`, (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
`test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
root can be specified either as an absolute path, or a path relative to the root can be specified either as an absolute path, or a path relative to the JDK
JDK top directory, or the `test` directory. For simplicity, the hotspot top directory, or the `test` directory. For simplicity, the hotspot JTReg test
JTReg test root, which really is `hotspot/jtreg` can be abbreviated as root, which really is `hotspot/jtreg` can be abbreviated as just `hotspot`.
just `hotspot`.
When specified without a test root, all matching groups from all test roots When specified without a test root, all matching groups from all test roots
will be added. Otherwise, only the group from the specified test root will be will be added. Otherwise, only the group from the specified test root will be
@ -201,8 +196,8 @@ use a fully qualified test descriptor, add `jtreg:`, e.g.
### Gtest ### Gtest
**Note:** To be able to run the Gtest suite, you need to configure your build to **Note:** To be able to run the Gtest suite, you need to configure your build
be able to find a proper version of the gtest source. For details, see the to be able to find a proper version of the gtest source. For details, see the
section ["Running Tests" in the build section ["Running Tests" in the build
documentation](building.html#running-tests). documentation](building.html#running-tests).
@ -222,10 +217,10 @@ present, then `gtest:all` will be equivalent to `gtest:all/server`.
### Microbenchmarks ### Microbenchmarks
Which microbenchmarks to run is selected using a regular expression Which microbenchmarks to run is selected using a regular expression following
following the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This the `micro:` test descriptor, e.g., `micro:java.lang.reflect`. This delegates
delegates the test selection to JMH, meaning package name, class name and even the test selection to JMH, meaning package name, class name and even benchmark
benchmark method names can be used to select tests. method names can be used to select tests.
Using special characters like `|` in the regular expression is possible, but Using special characters like `|` in the regular expression is possible, but
needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`. needs to be escaped multiple times: `micro:ArrayCopy\\\\\|reflect`.
@ -302,11 +297,12 @@ normally eats `;`, the recommended usage is to write the assignment inside
qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved, qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
as in `JTREG="JAVA_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`. as in `JTREG="JAVA_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
(Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`. (Other ways are possible, e.g. using backslash:
Also, as a special technique, the string `%20` will be replaced with space for `JTREG=JOBS=1\;TIMEOUT_FACTOR=8`. Also, as a special technique, the string
certain options, e.g. `JTREG=JAVA_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. `%20` will be replaced with space for certain options, e.g.
This can be useful if you have layers of scripts and have trouble getting `JTREG=JAVA_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`. This can be useful if
proper quoting of command line arguments through.) you have layers of scripts and have trouble getting proper quoting of command
line arguments through.)
As far as possible, the names of the keywords have been standardized between As far as possible, the names of the keywords have been standardized between
test suites. test suites.
@ -319,7 +315,8 @@ control variable to use, then you can use the general TEST_OPTS control
variable. variable.
There are also some keywords that applies globally to the test runner system, There are also some keywords that applies globally to the test runner system,
not to any specific test suites. These are also available as TEST_OPTS keywords. not to any specific test suites. These are also available as TEST_OPTS
keywords.
#### JOBS #### JOBS
@ -373,8 +370,8 @@ The report is stored in
The test concurrency (`-concurrency`). The test concurrency (`-concurrency`).
Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
JOBS, except for Hotspot, where the default is *number of CPU cores/2*, JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
but never more than *memory size in GB/2*. never more than *memory size in GB/2*.
#### TIMEOUT_FACTOR #### TIMEOUT_FACTOR
@ -389,12 +386,12 @@ This is only valid if the failure handler is built.
#### JTREG_TEST_THREAD_FACTORY #### JTREG_TEST_THREAD_FACTORY
Sets the `-testThreadFactory` for JTReg. It should be the fully qualified classname Sets the `-testThreadFactory` for JTReg. It should be the fully qualified
of a class which implements `java.util.concurrent.ThreadFactory`. classname of a class which implements `java.util.concurrent.ThreadFactory`. One
One such implementation class, named Virtual, is currently part of the JDK build such implementation class, named Virtual, is currently part of the JDK build in
in the `test/jtreg_test_thread_factory/` directory. This class gets compiled during the `test/jtreg_test_thread_factory/` directory. This class gets compiled
the test image build. The implementation of the Virtual class creates a new virtual during the test image build. The implementation of the Virtual class creates a
thread for executing each test class. new virtual thread for executing each test class.
#### TEST_MODE #### TEST_MODE
@ -456,9 +453,8 @@ the tests to be run.
Use the problem lists to select tests instead of excluding them. Use the problem lists to select tests instead of excluding them.
Set to `true` or `false`. Set to `true` or `false`. If `true`, JTReg will use `-match:` option, otherwise
If `true`, JTReg will use `-match:` option, otherwise `-exclude:` will be used. `-exclude:` will be used. Default is `false`.
Default is `false`.
#### OPTIONS #### OPTIONS
@ -492,15 +488,13 @@ modules. If multiple modules are specified, they should be separated by space
#### RETRY_COUNT #### RETRY_COUNT
Retry failed tests up to a set number of times, until they pass. Retry failed tests up to a set number of times, until they pass. This allows to
This allows to pass the tests with intermittent failures. pass the tests with intermittent failures. Defaults to 0.
Defaults to 0.
#### REPEAT_COUNT #### REPEAT_COUNT
Repeat the tests up to a set number of times, stopping at first failure. Repeat the tests up to a set number of times, stopping at first failure. This
This helps to reproduce intermittent test failures. helps to reproduce intermittent test failures. Defaults to 0.
Defaults to 0.
#### REPORT #### REPORT
@ -551,7 +545,8 @@ Same as specifying `-wi <num>`.
#### WARMUP_TIME #### WARMUP_TIME
Amount of time to spend in each warmup iteration. Same as specifying `-w <num>`. Amount of time to spend in each warmup iteration. Same as specifying `-w
<num>`.
#### RESULTS_FORMAT #### RESULTS_FORMAT
@ -592,9 +587,9 @@ $ make test TEST="jtreg:test/hotspot/jtreg/containers/docker" \
If your locale is non-US, some tests are likely to fail. To work around this If your locale is non-US, some tests are likely to fail. To work around this
you can set the locale to US. On Unix platforms simply setting `LANG="en_US"` you can set the locale to US. On Unix platforms simply setting `LANG="en_US"`
in the environment before running tests should work. On Windows or MacOS, setting in the environment before running tests should work. On Windows or MacOS,
`JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for most, but setting `JTREG="VM_OPTIONS=-Duser.language=en -Duser.country=US"` helps for
not all test cases. most, but not all test cases.
For example: For example:
@ -660,29 +655,30 @@ Note: restart is required to make the settings take effect.
#### Robot API #### Robot API
Most automated Client UI tests use `Robot` API to control the UI. Usually, Most automated Client UI tests use `Robot` API to control the UI. Usually, the
the default operating system settings need to be adjusted for Robot default operating system settings need to be adjusted for Robot to work
to work correctly. The detailed steps how to access and update these settings correctly. The detailed steps how to access and update these settings for
for different platforms are provided below. different platforms are provided below.
##### macOS ##### macOS
`Robot` is not permitted to control your Mac by default since `Robot` is not permitted to control your Mac by default since macOS 10.15. To
macOS 10.15. To allow it, choose Apple menu -> System Settings, click allow it, choose Apple menu -> System Settings, click Privacy & Security; then
Privacy & Security; then click Accessibility and ensure the following apps are click Accessibility and ensure the following apps are allowed to control your
allowed to control your computer: *Java* and *Terminal*. If the tests are run computer: *Java* and *Terminal*. If the tests are run from an IDE, the IDE
from an IDE, the IDE should be granted this permission too. should be granted this permission too.
##### Windows ##### Windows
On Windows if Cygwin terminal is used to run the tests, there is a delay in On Windows if Cygwin terminal is used to run the tests, there is a delay in
focus transfer. Usually it causes automated UI test failure. To disable the focus transfer. Usually it causes automated UI test failure. To disable the
delay, type `regedit` in the Search and then select Registry Editor; navigate delay, type `regedit` in the Search and then select Registry Editor; navigate
to the following key: `HKEY_CURRENT_USER\Control Panel\Desktop`; make sure to the following key: `HKEY_CURRENT_USER\Control Panel\Desktop`; make sure the
the `ForegroundLockTimeout` value is set to 0. `ForegroundLockTimeout` value is set to 0.
Additional information about Client UI tests configuration for various operating Additional information about Client UI tests configuration for various
systems can be obtained at [Automated client GUI testing system set up operating systems can be obtained at [Automated client GUI testing system set
up
requirements](https://wiki.openjdk.org/display/ClientLibs/Automated+client+GUI+testing+system+set+up+requirements) requirements](https://wiki.openjdk.org/display/ClientLibs/Automated+client+GUI+testing+system+set+up+requirements)
## Editing this document ## Editing this document