stan/jdk-24
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

8210958: Rename "make run-test" to "make test"

Browse Source 
Reviewed-by: erikj
This commit is contained in:
Magnus Ihse Bursie 2018-10-31 09:30:24 +01:00
parent 80aa38305d
commit 53ecc07aaa
11 changed files with 175 additions and 126 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

36
doc/testing.html
View File

@ -18,12 +18,13 @@
</header> </header>
<nav id="TOC"> <nav id="TOC">
<ul> <ul>
<li><a href="#using-the-run-test-framework">Using the run-test framework</a><ul> <li><a href="#using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</a><ul>
<li><a href="#configuration">Configuration</a></li> <li><a href="#configuration">Configuration</a></li>
</ul></li> </ul></li>
<li><a href="#test-selection">Test selection</a><ul> <li><a href="#test-selection">Test selection</a><ul>
<li><a href="#jtreg">JTReg</a></li> <li><a href="#jtreg">JTReg</a></li>
<li><a href="#gtest">Gtest</a></li> <li><a href="#gtest">Gtest</a></li>
<li><a href="#special-tests">Special tests</a></li>
</ul></li> </ul></li>
<li><a href="#test-results-and-summary">Test results and summary</a></li> <li><a href="#test-results-and-summary">Test results and summary</a></li>
<li><a href="#test-suite-control">Test suite control</a><ul> <li><a href="#test-suite-control">Test suite control</a><ul>
@ -32,22 +33,23 @@
</ul></li> </ul></li>
</ul> </ul>
</nav> </nav>
<h2 id="using-the-run-test-framework">Using the run-test framework</h2> <h2 id="using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</h2>
<p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p> <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
<p>The main target &quot;run-test&quot; uses the jdk-image as the tested product. There is also an alternate target &quot;exploded-run-test&quot; that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p> <p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
<p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test &amp;&amp; make</code>.</p>
<p>Some example command-lines:</p> <p>Some example command-lines:</p>
<pre><code>$ make run-test-tier1 <pre><code>$ make test-tier1
$ make run-test-jdk_lang JTREG=&quot;JOBS=8&quot; $ make test-jdk_lang JTREG=&quot;JOBS=8&quot;
$ make run-test TEST=jdk_lang $ make test TEST=jdk_lang
$ make run-test-only TEST=&quot;gtest:LogTagSet gtest:LogTagSetDescriptions&quot; GTEST=&quot;REPEAT=-1&quot; $ make test-only TEST=&quot;gtest:LogTagSet gtest:LogTagSetDescriptions&quot; GTEST=&quot;REPEAT=-1&quot;
$ make run-test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug&quot; $ make test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug&quot;
$ make run-test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot; $ make test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
$ make exploded-run-test TEST=tier2</code></pre> $ make exploded-test TEST=tier2</code></pre>
<h3 id="configuration">Configuration</h3> <h3 id="configuration">Configuration</h3>
<p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p> <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
<h2 id="test-selection">Test selection</h2> <h2 id="test-selection">Test selection</h2>
<p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p> <p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST=&quot;tier1&quot;</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST=&quot;x&quot;</code> solution needs to be used.</p> <p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST=&quot;tier1&quot;</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST=&quot;x&quot;</code> solution needs to be used.</p>
<p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p> <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
<h3 id="jtreg">JTReg</h3> <h3 id="jtreg">JTReg</h3>
<p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p> <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
@ -59,6 +61,14 @@ $ make exploded-run-test TEST=tier2</code></pre>
<p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p> <p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
<p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p> <p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
<p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/&lt;variant&gt;</code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p> <p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/&lt;variant&gt;</code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
<h3 id="special-tests">Special tests</h3>
<p>A handful of odd tests that are not covered by any other testing framework are accessible using the <code>special:</code> test descriptor. Currently, this includes <code>hotspot-internal</code>, <code>failure-handler</code> and <code>make</code>.</p>
<ul>
<li><p>Hotspot legacy internal testing (run using <code>-XX:+ExecuteInternalVMTests</code>) is run using <code>special:hotspot-internal</code> or just <code>hotspot-internal</code> as test descriptor, and will only work on a debug JVM.</p></li>
<li><p>Failure handler testing is run using <code>special:failure-handler</code> or just <code>failure-handler</code> as test descriptor.</p></li>
<li><p>Tests for the build system, including both makefiles and related functionality, is run using <code>special:make</code> or just <code>make</code> as test descriptor. This is equivalent to <code>special:make:all</code>.</p>
<p>A specific make test can be run by supplying it as argument, e.g. <code>special:make:idea</code>. As a special syntax, this can also be expressed as <code>make-idea</code>, which allows for command lines as <code>make test-make-idea</code>.</p></li>
</ul>
<h2 id="test-results-and-summary">Test results and summary</h2> <h2 id="test-results-and-summary">Test results and summary</h2>
<p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p> <p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
<pre><code>============================== <pre><code>==============================
@ -72,7 +82,7 @@ Test summary
TEST FAILURE</code></pre> TEST FAILURE</code></pre>
<p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>&gt;&gt; ... &lt;&lt;</code> marker for easy identification.</p> <p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>&gt;&gt; ... &lt;&lt;</code> marker for easy identification.</p>
<p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p> <p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p>
<p>In case of test failures, <code>make run-test</code> will exit with a non-zero exit value.</p> <p>In case of test failures, <code>make test</code> will exit with a non-zero exit value.</p>
<p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p> <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p>
<p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p> <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p>
<h2 id="test-suite-control">Test suite control</h2> <h2 id="test-suite-control">Test suite control</h2>

61
doc/testing.md
View File

@ -1,26 +1,32 @@
% Testing the JDK % Testing the JDK
## Using the run-test framework ## Using "make test" (the run-test framework)
This new way of running tests is developer-centric. It assumes that you have This new way of running tests is developer-centric. It assumes that you have
built a JDK locally and want to test it. Running common test targets is simple, built a JDK locally and want to test it. Running common test targets is simple,
and more complex ad-hoc combination of tests is possible. The user interface is and more complex ad-hoc combination of tests is possible. The user interface is
forgiving, and clearly report errors it cannot resolve. forgiving, and clearly report errors it cannot resolve.
The main target "run-test" uses the jdk-image as the tested product. There is The main target `test` uses the jdk-image as the tested product. There is
also an alternate target "exploded-run-test" that uses the exploded image also an alternate target `exploded-test` that uses the exploded image
instead. Not all tests will run successfully on the exploded image, but using instead. Not all tests will run successfully on the exploded image, but using
this target can greatly improve rebuild times for certain workflows. this target can greatly improve rebuild times for certain workflows.
Previously, `make test` was used invoke an old system for running test, and
`make run-test` was used for the new test framework. For backward compatibility
with scripts and muscle memory, `run-test` (and variants like
`exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
can still be accessed for some time using `cd test && make`.
Some example command-lines: Some example command-lines:
$ make run-test-tier1 $ make test-tier1
$ make run-test-jdk_lang JTREG="JOBS=8" $ make test-jdk_lang JTREG="JOBS=8"
$ make run-test TEST=jdk_lang $ make test TEST=jdk_lang
$ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
$ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
$ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
$ make exploded-run-test TEST=tier2 $ make exploded-test TEST=tier2
### Configuration ### Configuration
@ -33,16 +39,16 @@ environment variable to point to the JTReg home before running `configure`.)
## Test selection ## Test selection
All functionality is available using the run-test make target. In this use All functionality is available using the `test` make target. In this use case,
case, the test or tests to be executed is controlled using the `TEST` variable. the test or tests to be executed is controlled using the `TEST` variable. To
To speed up subsequent test runs with no source code changes, run-test-only can speed up subsequent test runs with no source code changes, `test-only` can be
be used instead, which do not depend on the source and test image build. used instead, which do not depend on the source and test image build.
For some common top-level tests, direct make targets have been generated. This For some common top-level tests, direct make targets have been generated. This
includes all JTReg test groups, the hotspot gtest, and custom tests (if includes all JTReg test groups, the hotspot gtest, and custom tests (if
present). This means that `make run-test-tier1` is equivalent to `make run-test present). This means that `make test-tier1` is equivalent to `make test
TEST="tier1"`, but the latter is more tab-completion friendly. For more complex TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
test runs, the `run-test TEST="x"` solution needs to be used. test runs, the `test TEST="x"` solution needs to be used.
The test specifications given in `TEST` is parsed into fully qualified test The test specifications given in `TEST` is parsed into fully qualified test
descriptors, which clearly and unambigously show which tests will be run. As an descriptors, which clearly and unambigously show which tests will be run. As an
@ -98,6 +104,27 @@ is defined by adding `/<variant>` to the test descriptor, e.g.
variant present (e.g. server, client). So if you only have the server JVM variant present (e.g. server, client). So if you only have the server JVM
present, then `gtest:all` will be equivalent to `gtest:all/server`. present, then `gtest:all` will be equivalent to `gtest:all/server`.
### Special tests
A handful of odd tests that are not covered by any other testing framework are
accessible using the `special:` test descriptor. Currently, this includes
`hotspot-internal`, `failure-handler` and `make`.
* Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`)
is run using `special:hotspot-internal` or just `hotspot-internal` as test
descriptor, and will only work on a debug JVM.
* Failure handler testing is run using `special:failure-handler` or just
`failure-handler` as test descriptor.
* Tests for the build system, including both makefiles and related
functionality, is run using `special:make` or just `make` as test
descriptor. This is equivalent to `special:make:all`.
A specific make test can be run by supplying it as argument, e.g.
`special:make:idea`. As a special syntax, this can also be expressed as
`make-idea`, which allows for command lines as `make test-make-idea`.
## Test results and summary ## Test results and summary
At the end of the test run, a summary of all tests run will be presented. This At the end of the test run, a summary of all tests run will be presented. This
@ -123,7 +150,7 @@ the summary, ERROR is used as a catch-all for tests that neither passed nor are
classified as failed by the framework. This might indicate test framework classified as failed by the framework. This might indicate test framework
error, timeout or other problems. error, timeout or other problems.
In case of test failures, `make run-test` will exit with a non-zero exit value. In case of test failures, `make test` will exit with a non-zero exit value.
All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
where TEST_ID is a path-safe conversion from the fully qualified test where TEST_ID is a path-safe conversion from the fully qualified test

20
make/Help.gmk
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2012, 2017, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2012, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -55,10 +55,10 @@ help:
$(info $(_) make install # Install the generated images locally) $(info $(_) make install # Install the generated images locally)
$(info $(_) make reconfigure # Rerun configure with the same arguments as last time) $(info $(_) make reconfigure # Rerun configure with the same arguments as last time)
$(info $(_) make help # Give some help on using make) $(info $(_) make help # Give some help on using make)
$(info $(_) make test # Run tests, default is all tests (see TEST below)) $(info $(_) make check # Run basic testing (currently tier1))
$(info $(_) make run-test-<test> # Run test, e.g. run-test-tier1) $(info $(_) make test-<test> # Run test, e.g. test-tier1)
$(info $(_) make run-test TEST=<t> # Run test(s) given by TEST specification) $(info $(_) make test TEST=<t> # Run test(s) given by TEST specification)
$(info $(_) make exploded-run-test TEST=<t> # Run test(s) on the exploded image instead of) $(info $(_) make exploded-test TEST=<t> # Run test(s) on the exploded image instead of)
$(info $(_) # the full jdk image) $(info $(_) # the full jdk image)
$(info ) $(info )
$(info Targets for cleaning) $(info Targets for cleaning)
@ -99,10 +99,12 @@ help:
$(info $(_) TEST_JOBS=<n> # Run <n> parallel test jobs) $(info $(_) TEST_JOBS=<n> # Run <n> parallel test jobs)
$(info $(_) CONF_CHECK=<method> # What to do if spec file is out of date) $(info $(_) CONF_CHECK=<method> # What to do if spec file is out of date)
$(info $(_) # method is 'auto', 'ignore' or 'fail' (default)) $(info $(_) # method is 'auto', 'ignore' or 'fail' (default))
$(info $(_) make test TEST=<test> # Only run the given test or tests, e.g.) $(info $(_) TEST="test1 ..." # Use the given test descriptor(s) for testing, e.g.)
$(info $(_) # make test TEST="jdk_lang jdk_net") $(info $(_) # make test TEST="jdk_lang gtest:all")
$(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness for run-test) $(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness)
$(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness for run-test) $(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness)
$(info $(_) TEST_OPTS="OPT1=x;..." # Generic control of all test harnesses)
$(info $(_) TEST_VM_OPTS="ARG ..." # Same as setting TEST_OPTS to VM_OPTIONS="ARG ...")
$(info ) $(info )
$(if $(all_confs), $(info Available configurations in $(build_dir):) $(foreach var,$(all_confs),$(info * $(var))),\ $(if $(all_confs), $(info Available configurations in $(build_dir):) $(foreach var,$(all_confs),$(info * $(var))),\
$(info No configurations were found in $(build_dir).) $(info Run 'bash configure' to create a configuration.)) $(info No configurations were found in $(build_dir).) $(info Run 'bash configure' to create a configuration.))

101
make/Main.gmk
View File

@ -473,10 +473,10 @@ ALL_TARGETS += $(INTERIM_JMOD_TARGETS) interim-image generate-link-opt-data
# #
define DeclareRunTestRecipe define DeclareRunTestRecipe
run-test-$1: test-$1:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$1") +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$1")
exploded-run-test-$1: exploded-test-$1:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \ +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
TEST="$1" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR)) TEST="$1" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
@ -484,8 +484,8 @@ endef
# ALL_NAMED_TESTS is defined in FindTests.gmk # ALL_NAMED_TESTS is defined in FindTests.gmk
$(foreach t, $(ALL_NAMED_TESTS), $(eval $(call DeclareRunTestRecipe,$t))) $(foreach t, $(ALL_NAMED_TESTS), $(eval $(call DeclareRunTestRecipe,$t)))
ALL_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) ALL_TEST_TARGETS := $(addprefix test-, $(ALL_NAMED_TESTS))
ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS)) ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-test-, $(ALL_NAMED_TESTS))
ALL_TARGETS += $(ALL_TEST_TARGETS) $(ALL_EXPLODED_TEST_TARGETS) ALL_TARGETS += $(ALL_TEST_TARGETS) $(ALL_EXPLODED_TEST_TARGETS)
@ -520,13 +520,6 @@ test-image-hotspot-jtreg-graal:
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f JtregGraalUnit.gmk \ +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f JtregGraalUnit.gmk \
test-image-hotspot-jtreg-graal) test-image-hotspot-jtreg-graal)
run-test:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
exploded-run-test:
+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
ifeq ($(BUILD_GTEST), true) ifeq ($(BUILD_GTEST), true)
test-image-hotspot-gtest: test-image-hotspot-gtest:
+($(CD) $(TOPDIR)/make/hotspot/test && $(MAKE) $(MAKE_ARGS) -f GtestImage.gmk) +($(CD) $(TOPDIR)/make/hotspot/test && $(MAKE) $(MAKE_ARGS) -f GtestImage.gmk)
@ -541,11 +534,6 @@ ifeq ($(BUILD_FAILURE_HANDLER), true)
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \ +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
-f BuildFailureHandler.gmk build) -f BuildFailureHandler.gmk build)
# Runs the tests for the failure handler jtreg extension
test-failure-handler:
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
-f BuildFailureHandler.gmk test)
# Copies the failure handler jtreg extension into the test image # Copies the failure handler jtreg extension into the test image
test-image-failure-handler: test-image-failure-handler:
+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \ +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
@ -556,40 +544,19 @@ ALL_TARGETS += prepare-test-image build-test-hotspot-jtreg-native \
test-image-hotspot-jtreg-native build-test-jdk-jtreg-native \ test-image-hotspot-jtreg-native build-test-jdk-jtreg-native \
test-image-jdk-jtreg-native build-test-lib build-test-failure-handler \ test-image-jdk-jtreg-native build-test-lib build-test-failure-handler \
test-failure-handler test-image-failure-handler test-image-hotspot-gtest \ test-failure-handler test-image-failure-handler test-image-hotspot-gtest \
test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal \ test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal
run-test exploded-run-test
################################################################################ ################################################################################
# Run tests # Run tests
# Run tests specified by $(TEST), or the default test set.
test: test:
$(call RunTests, $(TEST), $(JDK_IMAGE_DIR)) +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
test-hotspot-jtreg: exploded-test:
$(call RunTests, "hotspot_all", $(JDK_IMAGE_DIR)) +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
test-hotspot-jtreg-native: ALL_TARGETS += test exploded-test
$(call RunTests, "hotspot_native_sanity", $(JDK_IMAGE_DIR))
test-hotspot-internal:
$(call RunTests, "hotspot_internal", $(JDK_OUTPUTDIR))
test-hotspot-gtest:
$(call RunTests, "hotspot_gtest", $(JDK_OUTPUTDIR))
test-jdk-jtreg-native:
$(call RunTests, "jdk_native_sanity", $(JDK_IMAGE_DIR))
test-make:
($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk $(TEST_TARGET))
test-compile-commands:
($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk test-compile-commands)
ALL_TARGETS += test test-hotspot-jtreg test-hotspot-jtreg-native \
test-hotspot-internal test-hotspot-gtest test-jdk-jtreg-native test-make \
test-compile-commands
################################################################################ ################################################################################
# Bundles # Bundles
@ -874,14 +841,18 @@ else
docs-zip: docs-jdk docs-zip: docs-jdk
# Tests
test: jdk-image test-image test: jdk-image test-image
run-test: jdk-image test-image exploded-test: exploded-image test-image
exploded-run-test: exploded-image test-image
test-make: clean-test-make compile-commands
test-make-compile-commands: compile-commands
# Declare dependency for all generated test targets # Declare dependency for all generated test targets
$(foreach t, $(ALL_TEST_TARGETS), $(eval $t: jdk-image test-image)) $(foreach t, $(filter-out test-make%, $(ALL_TEST_TARGETS)), $(eval $t: jdk-image test-image))
$(foreach t, $(ALL_EXPLODED_TEST_TARGETS), $(eval $t: exploded-image test-image)) $(foreach t, $(filter-out exploded-test-make%, $(ALL_EXPLODED_TEST_TARGETS)), $(eval $t: exploded-image test-image))
create-buildjdk-copy: jdk.jlink-java java.base-gendata \ create-buildjdk-copy: jdk.jlink-java java.base-gendata \
$(addsuffix -java, $(INTERIM_IMAGE_MODULES)) $(addsuffix -java, $(INTERIM_IMAGE_MODULES))
@ -890,16 +861,10 @@ else
interim-image: $(INTERIM_JMOD_TARGETS) interim-image: $(INTERIM_JMOD_TARGETS)
test-make: clean-test-make
test-compile-commands: compile-commands
build-test-lib: exploded-image-optimize build-test-lib: exploded-image-optimize
build-test-failure-handler: interim-langtools build-test-failure-handler: interim-langtools
test-failure-handler: build-test-failure-handler
test-image-failure-handler: build-test-failure-handler test-image-failure-handler: build-test-failure-handler
build-test-hotspot-jtreg-native: buildtools-jdk \ build-test-hotspot-jtreg-native: buildtools-jdk \
@ -917,12 +882,6 @@ else
test-image-hotspot-gtest: hotspot test-image-hotspot-gtest: hotspot
test-hotspot-internal: exploded-image
test-hotspot-jtreg: jdk-image test-image
test-hotspot-gtest: exploded-image test-image-hotspot-gtest
install: product-images install: product-images
product-bundles: product-images product-bundles: product-images
@ -1091,6 +1050,30 @@ all: all-images
ALL_TARGETS += default jdk images docs bundles all ALL_TARGETS += default jdk images docs bundles all
# Aliases used for running tests.
# Let "run-test" be an alias for "test"
$(foreach t, $(ALL_NAMED_TESTS), $(eval run-test-$t: test-$t))
$(foreach t, $(ALL_NAMED_TESTS), $(eval exploded-run-test-$t: exploded-test-$t))
RUN_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) \
$(addprefix exploded-run-test-, $(ALL_NAMED_TESTS))
run-test: test
exploded-run-test: exploded-test
# "make check" is a common idiom for running basic testing
check: test-tier1
# Keep some old names as aliases
test-hotspot-jtreg: test-hotspot_all
test-hotspot-jtreg-native: test-hotspot_native_sanity
test-hotspot-gtest: exploded-test-gtest
test-jdk-jtreg-native: test-jdk_native_sanity
ALL_TARGETS += $(RUN_TEST_TARGETS) run-test exploded-run-test check \
test-hotspot-jtreg test-hotspot-jtreg-native test-hotspot-gtest \
test-jdk-jtreg-native
################################################################################ ################################################################################
################################################################################ ################################################################################
# #

13
make/MainSupport.gmk
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2011, 2017, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2011, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -30,17 +30,6 @@
ifndef _MAINSUPPORT_GMK ifndef _MAINSUPPORT_GMK
_MAINSUPPORT_GMK := 1 _MAINSUPPORT_GMK := 1
# Run the tests specified by $1, with PRODUCT_HOME specified by $2
# JT_JAVA is picked up by the jtreg launcher and used to run Jtreg itself.
define RunTests
($(CD) $(TOPDIR)/test && $(MAKE) $(MAKE_ARGS) -j1 -k MAKEFLAGS= \
JT_HOME=$(JT_HOME) PRODUCT_HOME=$(strip $2) \
TEST_IMAGE_DIR=$(TEST_IMAGE_DIR) \
ALT_OUTPUTDIR=$(OUTPUTDIR) TEST_JOBS=$(TEST_JOBS) \
JT_JAVA=$(BOOT_JDK) JIB_JAR=$(JIB_JAR) \
JOBS=$(JOBS) $1) || true
endef
define CleanDocs define CleanDocs
@$(PRINTF) "Cleaning docs ..." @$(PRINTF) "Cleaning docs ..."
@$(PRINTF) "\n" $(LOG_DEBUG) @$(PRINTF) "\n" $(LOG_DEBUG)

26
make/RunTests.gmk
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2016, 2017, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2016, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -300,16 +300,32 @@ endef
# Helper function to determine if a test specification is a special test # Helper function to determine if a test specification is a special test
# #
# It is a special test if it is "special:" followed by a test name. # It is a special test if it is "special:" followed by a test name,
# if it is "make:" or "make-" followed by a make test, or any of the special test names
# as a single word.
define ParseSpecialTestSelection define ParseSpecialTestSelection
$(if $(filter special:%, $1), \ $(if $(filter special:%, $1), \
$1 \ $1 \
) \
$(if $(filter make%, $1), \
$(if $(filter make:%, $1), \
special:$(strip $1) \
) \
$(if $(filter make-%, $1), \
special:$(patsubst make-%,make:%, $1) \
) \
$(if $(filter make, $1), \
special:make:all \
)
) \
$(if $(filter hotspot-internal failure-handler, $1), \
special:$(strip $1) \
) )
endef endef
ifeq ($(TEST), ) ifeq ($(TEST), )
$(info No test selection given in TEST!) $(info No test selection given in TEST!)
$(info Please use e.g. 'run-test TEST=tier1' or 'run-test-tier1') $(info Please use e.g. 'make test TEST=tier1' or 'make test-tier1')
$(info See doc/testing.[md|html] for help) $(info See doc/testing.[md|html] for help)
$(error Cannot continue) $(error Cannot continue)
endif endif
@ -663,9 +679,13 @@ define SetupRunSpecialTestBody
$$(JDK_IMAGE_DIR)/bin/java -XX:+ExecuteInternalVMTests \ $$(JDK_IMAGE_DIR)/bin/java -XX:+ExecuteInternalVMTests \
-XX:+ShowMessageBoxOnError -version -XX:+ShowMessageBoxOnError -version
else ifeq ($$($1_TEST_NAME), failure-handler) else ifeq ($$($1_TEST_NAME), failure-handler)
ifeq ($(BUILD_FAILURE_HANDLER), true)
$1_TEST_COMMAND_LINE := \ $1_TEST_COMMAND_LINE := \
($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \ ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \
BuildFailureHandler.gmk test) BuildFailureHandler.gmk test)
else
$$(error Cannot test failure handler if it is not built)
endif
else ifeq ($$($1_TEST_NAME), make) else ifeq ($$($1_TEST_NAME), make)
$1_TEST_COMMAND_LINE := \ $1_TEST_COMMAND_LINE := \
($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f \ ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f \

11
make/common/FindTests.gmk
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2017, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -72,6 +72,15 @@ ALL_NAMED_TESTS += $(JTREG_TEST_GROUPS)
# Add Gtest # Add Gtest
ALL_NAMED_TESTS += gtest ALL_NAMED_TESTS += gtest
# Find make test targets
MAKE_TEST_TARGETS := $(shell $(MAKE) -s --no-print-directory $(MAKE_ARGS) \
SPEC=$(SPEC) -f $(TOPDIR)/test/make/TestMake.gmk print-targets)
ALL_NAMED_TESTS += $(addprefix make-, $(MAKE_TEST_TARGETS))
# Add special tests
ALL_NAMED_TESTS += hotspot-internal failure-handler make
################################################################################ ################################################################################
endif # _FIND_TESTS_GMK endif # _FIND_TESTS_GMK

2
make/conf/jib-profiles.js
View File

@ -759,7 +759,7 @@ var getJibProfilesProfiles = function (input, common, data) {
testedProfile + ".test" testedProfile + ".test"
], ],
src: "src.conf", src: "src.conf",
make_args: [ "run-test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ], make_args: [ "test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ],
environment: { environment: {
"BOOT_JDK": common.boot_jdk_home, "BOOT_JDK": common.boot_jdk_home,
"JDK_IMAGE_DIR": input.get(testedProfile + ".jdk", "home_path"), "JDK_IMAGE_DIR": input.get(testedProfile + ".jdk", "home_path"),

7
test/Makefile
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2010, 2015, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2010, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -27,11 +27,16 @@
# Makefile to run tests from multiple sibling directories # Makefile to run tests from multiple sibling directories
# #
$(info WARNING: This way of running tests ("cd test && make") is deprecated)
$(info Please use "make test TEST=..." instead. See doc/testing.md for details)
# Macro to run a test target in a subdir # Macro to run a test target in a subdir
define SUBDIR_TEST # subdirectory target define SUBDIR_TEST # subdirectory target
if [ -d $1 ] ; then \ if [ -d $1 ] ; then \
if [ -r $1/Makefile ] ; then \ if [ -r $1/Makefile ] ; then \
$(MAKE) --no-print-directory -k -C $1 $2 ; \ $(MAKE) --no-print-directory -k -C $1 $2 ; \
echo 'WARNING: This way of running tests ("cd test && make") is deprecated' ; \
echo 'Please use "make test TEST=..." instead. See doc/testing.md for details' ; \
else \ else \
echo "ERROR: File does not exist: $1/Makefile"; \ echo "ERROR: File does not exist: $1/Makefile"; \
exit 1; \ exit 1; \

14
test/make/TestMake.gmk
View File

@ -1,5 +1,5 @@
# #
# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved. # Copyright (c) 2014, 2018, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
# #
# This code is free software; you can redistribute it and/or modify it # This code is free software; you can redistribute it and/or modify it
@ -36,13 +36,17 @@ java-compilation:
copy-files: copy-files:
+$(MAKE) -f TestCopyFiles.gmk $(TEST_SUBTARGET) +$(MAKE) -f TestCopyFiles.gmk $(TEST_SUBTARGET)
test-idea: idea:
+$(MAKE) -f TestIdea.gmk $(TEST_SUBTARGET) +$(MAKE) -f TestIdea.gmk $(TEST_SUBTARGET)
test-compile-commands: compile-commands:
+$(MAKE) -f TestCompileCommands.gmk $(TEST_SUBTARGET) +$(MAKE) -f TestCompileCommands.gmk $(TEST_SUBTARGET)
TARGETS += make-base java-compilation copy-files idea compile-commands
all: make-base java-compilation copy-files test-idea all: $(TARGETS)
.PHONY: default all make-base java-compilation copy-files test-idea test-compile-commands print-targets:
$(ECHO) "$(TARGETS)"
.PHONY: default all $(TARGETS)

4
test/make/TestMakeBase.gmk
View File

@ -344,7 +344,7 @@ $(eval $(call assert-equals, \
KWBASE := APA=banan;GURKA=tomat;COUNT=1%202%203%204%205;SUM=1+2+3+4+5;MANY_WORDS=I have the best words. KWBASE := APA=banan;GURKA=tomat;COUNT=1%202%203%204%205;SUM=1+2+3+4+5;MANY_WORDS=I have the best words.
$(eval $(call ParseKeywordVariable, KWBASE, \ $(eval $(call ParseKeywordVariable, KWBASE, \
KEYWORDS := APA GURKA SUM, \ SINGLE_KEYWORDS := APA GURKA SUM, \
STRING_KEYWORDS := COUNT MANY_WORDS, \ STRING_KEYWORDS := COUNT MANY_WORDS, \
)) ))
@ -377,7 +377,7 @@ override KWBASE_WEIRD_GURKA := paprika
KWBASE_WEIRD := ;;APA=banan;;;GURKA=apelsin;APA=skansen;; KWBASE_WEIRD := ;;APA=banan;;;GURKA=apelsin;APA=skansen;;
$(eval $(call ParseKeywordVariable, KWBASE_WEIRD, \ $(eval $(call ParseKeywordVariable, KWBASE_WEIRD, \
KEYWORDS := APA GURKA SUM, \ SINGLE_KEYWORDS := APA GURKA SUM, \
STRING_KEYWORDS := COUNT, \ STRING_KEYWORDS := COUNT, \
)) ))