diff --git a/doc/testing.html b/doc/testing.html index 457061a7eb9..f66ef9b11e9 100644 --- a/doc/testing.html +++ b/doc/testing.html @@ -18,12 +18,13 @@ -

Using the run-test framework

+

Using "make test" (the run-test framework)

This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.

-

The main target "run-test" uses the jdk-image as the tested product. There is also an alternate target "exploded-run-test" that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.

+

The main target test uses the jdk-image as the tested product. There is also an alternate target exploded-test that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.

+

Previously, make test was used invoke an old system for running test, and make run-test was used for the new test framework. For backward compatibility with scripts and muscle memory, run-test (and variants like exploded-run-test or run-test-tier1) are kept as aliases. The old system can still be accessed for some time using cd test && make.

Some example command-lines:

-
$ make run-test-tier1
-$ make run-test-jdk_lang JTREG="JOBS=8"
-$ make run-test TEST=jdk_lang
-$ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
-$ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
-$ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
-$ make exploded-run-test TEST=tier2
+
$ make test-tier1
+$ make test-jdk_lang JTREG="JOBS=8"
+$ make test TEST=jdk_lang
+$ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
+$ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
+$ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
+$ make exploded-test TEST=tier2

Configuration

To be able to run JTReg tests, configure needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the --with-jtreg=<path to jtreg home> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing lib/jtreg.jar etc. (An alternative is to set the JT_HOME environment variable to point to the JTReg home before running configure.)

Test selection

-

All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the TEST variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.

-

For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that make run-test-tier1 is equivalent to make run-test TEST="tier1", but the latter is more tab-completion friendly. For more complex test runs, the run-test TEST="x" solution needs to be used.

+

All functionality is available using the test make target. In this use case, the test or tests to be executed is controlled using the TEST variable. To speed up subsequent test runs with no source code changes, test-only can be used instead, which do not depend on the source and test image build.

+

For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that make test-tier1 is equivalent to make test TEST="tier1", but the latter is more tab-completion friendly. For more complex test runs, the test TEST="x" solution needs to be used.

The test specifications given in TEST is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, :tier1 will expand to jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1. You can always submit a list of fully qualified test descriptors in the TEST variable if you want to shortcut the parser.

JTReg

JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.

@@ -59,6 +61,14 @@ $ make exploded-run-test TEST=tier2

Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just gtest, or as a fully qualified test descriptor gtest:all.

If you want, you can single out an individual test or a group of tests, for instance gtest:LogDecorations or gtest:LogDecorations.level_test_vm. This can be particularly useful if you want to run a shaky test repeatedly.

For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding /<variant> to the test descriptor, e.g. gtest:Log/client. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then gtest:all will be equivalent to gtest:all/server.

+

Special tests

+

A handful of odd tests that are not covered by any other testing framework are accessible using the special: test descriptor. Currently, this includes hotspot-internal, failure-handler and make.

+

Test results and summary

At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:

==============================
@@ -72,7 +82,7 @@ Test summary
 TEST FAILURE

Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the >> ... << marker for easy identification.

The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.

-

In case of test failures, make run-test will exit with a non-zero exit value.

+

In case of test failures, make test will exit with a non-zero exit value.

All tests have their result stored in build/$BUILD/test-results/$TEST_ID, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for jtreg:jdk/test:tier1 the TEST_ID is jtreg_jdk_test_tier1. This path is also printed in the log at the end of the test run.

Additional work data is stored in build/$BUILD/test-support/$TEST_ID. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.

Test suite control

diff --git a/doc/testing.md b/doc/testing.md index 7aa40a6c42b..fc7752274a5 100644 --- a/doc/testing.md +++ b/doc/testing.md @@ -1,26 +1,32 @@ % Testing the JDK -## Using the run-test framework +## Using "make test" (the run-test framework) This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve. -The main target "run-test" uses the jdk-image as the tested product. There is -also an alternate target "exploded-run-test" that uses the exploded image +The main target `test` uses the jdk-image as the tested product. There is +also an alternate target `exploded-test` that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows. +Previously, `make test` was used invoke an old system for running test, and +`make run-test` was used for the new test framework. For backward compatibility +with scripts and muscle memory, `run-test` (and variants like +`exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system +can still be accessed for some time using `cd test && make`. + Some example command-lines: - $ make run-test-tier1 - $ make run-test-jdk_lang JTREG="JOBS=8" - $ make run-test TEST=jdk_lang - $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" - $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" - $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" - $ make exploded-run-test TEST=tier2 + $ make test-tier1 + $ make test-jdk_lang JTREG="JOBS=8" + $ make test TEST=jdk_lang + $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1" + $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug" + $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java" + $ make exploded-test TEST=tier2 ### Configuration @@ -33,16 +39,16 @@ environment variable to point to the JTReg home before running `configure`.) ## Test selection -All functionality is available using the run-test make target. In this use -case, the test or tests to be executed is controlled using the `TEST` variable. -To speed up subsequent test runs with no source code changes, run-test-only can -be used instead, which do not depend on the source and test image build. +All functionality is available using the `test` make target. In this use case, +the test or tests to be executed is controlled using the `TEST` variable. To +speed up subsequent test runs with no source code changes, `test-only` can be +used instead, which do not depend on the source and test image build. For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if -present). This means that `make run-test-tier1` is equivalent to `make run-test +present). This means that `make test-tier1` is equivalent to `make test TEST="tier1"`, but the latter is more tab-completion friendly. For more complex -test runs, the `run-test TEST="x"` solution needs to be used. +test runs, the `test TEST="x"` solution needs to be used. The test specifications given in `TEST` is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an @@ -98,6 +104,27 @@ is defined by adding `/` to the test descriptor, e.g. variant present (e.g. server, client). So if you only have the server JVM present, then `gtest:all` will be equivalent to `gtest:all/server`. +### Special tests + +A handful of odd tests that are not covered by any other testing framework are +accessible using the `special:` test descriptor. Currently, this includes +`hotspot-internal`, `failure-handler` and `make`. + + * Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`) + is run using `special:hotspot-internal` or just `hotspot-internal` as test + descriptor, and will only work on a debug JVM. + + * Failure handler testing is run using `special:failure-handler` or just + `failure-handler` as test descriptor. + + * Tests for the build system, including both makefiles and related + functionality, is run using `special:make` or just `make` as test + descriptor. This is equivalent to `special:make:all`. + + A specific make test can be run by supplying it as argument, e.g. + `special:make:idea`. As a special syntax, this can also be expressed as + `make-idea`, which allows for command lines as `make test-make-idea`. + ## Test results and summary At the end of the test run, a summary of all tests run will be presented. This @@ -123,7 +150,7 @@ the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems. -In case of test failures, `make run-test` will exit with a non-zero exit value. +In case of test failures, `make test` will exit with a non-zero exit value. All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`, where TEST_ID is a path-safe conversion from the fully qualified test diff --git a/make/Help.gmk b/make/Help.gmk index e58f54862af..7fe15453249 100644 --- a/make/Help.gmk +++ b/make/Help.gmk @@ -1,5 +1,5 @@ # -# Copyright (c) 2012, 2017, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2012, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -55,10 +55,10 @@ help: $(info $(_) make install # Install the generated images locally) $(info $(_) make reconfigure # Rerun configure with the same arguments as last time) $(info $(_) make help # Give some help on using make) - $(info $(_) make test # Run tests, default is all tests (see TEST below)) - $(info $(_) make run-test- # Run test, e.g. run-test-tier1) - $(info $(_) make run-test TEST= # Run test(s) given by TEST specification) - $(info $(_) make exploded-run-test TEST= # Run test(s) on the exploded image instead of) + $(info $(_) make check # Run basic testing (currently tier1)) + $(info $(_) make test- # Run test, e.g. test-tier1) + $(info $(_) make test TEST= # Run test(s) given by TEST specification) + $(info $(_) make exploded-test TEST= # Run test(s) on the exploded image instead of) $(info $(_) # the full jdk image) $(info ) $(info Targets for cleaning) @@ -99,10 +99,12 @@ help: $(info $(_) TEST_JOBS= # Run parallel test jobs) $(info $(_) CONF_CHECK= # What to do if spec file is out of date) $(info $(_) # method is 'auto', 'ignore' or 'fail' (default)) - $(info $(_) make test TEST= # Only run the given test or tests, e.g.) - $(info $(_) # make test TEST="jdk_lang jdk_net") - $(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness for run-test) - $(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness for run-test) + $(info $(_) TEST="test1 ..." # Use the given test descriptor(s) for testing, e.g.) + $(info $(_) # make test TEST="jdk_lang gtest:all") + $(info $(_) JTREG="OPT1=x;OPT2=y" # Control the JTREG test harness) + $(info $(_) GTEST="OPT1=x;OPT2=y" # Control the GTEST test harness) + $(info $(_) TEST_OPTS="OPT1=x;..." # Generic control of all test harnesses) + $(info $(_) TEST_VM_OPTS="ARG ..." # Same as setting TEST_OPTS to VM_OPTIONS="ARG ...") $(info ) $(if $(all_confs), $(info Available configurations in $(build_dir):) $(foreach var,$(all_confs),$(info * $(var))),\ $(info No configurations were found in $(build_dir).) $(info Run 'bash configure' to create a configuration.)) diff --git a/make/Main.gmk b/make/Main.gmk index d16463910bc..ef60594ba0a 100644 --- a/make/Main.gmk +++ b/make/Main.gmk @@ -473,10 +473,10 @@ ALL_TARGETS += $(INTERIM_JMOD_TARGETS) interim-image generate-link-opt-data # define DeclareRunTestRecipe - run-test-$1: + test-$1: +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$1") - exploded-run-test-$1: + exploded-test-$1: +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \ TEST="$1" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR)) @@ -484,8 +484,8 @@ endef # ALL_NAMED_TESTS is defined in FindTests.gmk $(foreach t, $(ALL_NAMED_TESTS), $(eval $(call DeclareRunTestRecipe,$t))) -ALL_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) -ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS)) +ALL_TEST_TARGETS := $(addprefix test-, $(ALL_NAMED_TESTS)) +ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-test-, $(ALL_NAMED_TESTS)) ALL_TARGETS += $(ALL_TEST_TARGETS) $(ALL_EXPLODED_TEST_TARGETS) @@ -520,13 +520,6 @@ test-image-hotspot-jtreg-graal: +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f JtregGraalUnit.gmk \ test-image-hotspot-jtreg-graal) -run-test: - +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)") - -exploded-run-test: - +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \ - TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR)) - ifeq ($(BUILD_GTEST), true) test-image-hotspot-gtest: +($(CD) $(TOPDIR)/make/hotspot/test && $(MAKE) $(MAKE_ARGS) -f GtestImage.gmk) @@ -541,11 +534,6 @@ ifeq ($(BUILD_FAILURE_HANDLER), true) +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \ -f BuildFailureHandler.gmk build) - # Runs the tests for the failure handler jtreg extension - test-failure-handler: - +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \ - -f BuildFailureHandler.gmk test) - # Copies the failure handler jtreg extension into the test image test-image-failure-handler: +($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \ @@ -556,40 +544,19 @@ ALL_TARGETS += prepare-test-image build-test-hotspot-jtreg-native \ test-image-hotspot-jtreg-native build-test-jdk-jtreg-native \ test-image-jdk-jtreg-native build-test-lib build-test-failure-handler \ test-failure-handler test-image-failure-handler test-image-hotspot-gtest \ - test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal \ - run-test exploded-run-test + test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal ################################################################################ # Run tests -# Run tests specified by $(TEST), or the default test set. test: - $(call RunTests, $(TEST), $(JDK_IMAGE_DIR)) + +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)") -test-hotspot-jtreg: - $(call RunTests, "hotspot_all", $(JDK_IMAGE_DIR)) +exploded-test: + +($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \ + TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR)) -test-hotspot-jtreg-native: - $(call RunTests, "hotspot_native_sanity", $(JDK_IMAGE_DIR)) - -test-hotspot-internal: - $(call RunTests, "hotspot_internal", $(JDK_OUTPUTDIR)) - -test-hotspot-gtest: - $(call RunTests, "hotspot_gtest", $(JDK_OUTPUTDIR)) - -test-jdk-jtreg-native: - $(call RunTests, "jdk_native_sanity", $(JDK_IMAGE_DIR)) - -test-make: - ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk $(TEST_TARGET)) - -test-compile-commands: - ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk test-compile-commands) - -ALL_TARGETS += test test-hotspot-jtreg test-hotspot-jtreg-native \ - test-hotspot-internal test-hotspot-gtest test-jdk-jtreg-native test-make \ - test-compile-commands +ALL_TARGETS += test exploded-test ################################################################################ # Bundles @@ -874,14 +841,18 @@ else docs-zip: docs-jdk + # Tests test: jdk-image test-image - run-test: jdk-image test-image - exploded-run-test: exploded-image test-image + exploded-test: exploded-image test-image + + test-make: clean-test-make compile-commands + + test-make-compile-commands: compile-commands # Declare dependency for all generated test targets - $(foreach t, $(ALL_TEST_TARGETS), $(eval $t: jdk-image test-image)) - $(foreach t, $(ALL_EXPLODED_TEST_TARGETS), $(eval $t: exploded-image test-image)) + $(foreach t, $(filter-out test-make%, $(ALL_TEST_TARGETS)), $(eval $t: jdk-image test-image)) + $(foreach t, $(filter-out exploded-test-make%, $(ALL_EXPLODED_TEST_TARGETS)), $(eval $t: exploded-image test-image)) create-buildjdk-copy: jdk.jlink-java java.base-gendata \ $(addsuffix -java, $(INTERIM_IMAGE_MODULES)) @@ -890,16 +861,10 @@ else interim-image: $(INTERIM_JMOD_TARGETS) - test-make: clean-test-make - - test-compile-commands: compile-commands - build-test-lib: exploded-image-optimize build-test-failure-handler: interim-langtools - test-failure-handler: build-test-failure-handler - test-image-failure-handler: build-test-failure-handler build-test-hotspot-jtreg-native: buildtools-jdk \ @@ -917,12 +882,6 @@ else test-image-hotspot-gtest: hotspot - test-hotspot-internal: exploded-image - - test-hotspot-jtreg: jdk-image test-image - - test-hotspot-gtest: exploded-image test-image-hotspot-gtest - install: product-images product-bundles: product-images @@ -1091,6 +1050,30 @@ all: all-images ALL_TARGETS += default jdk images docs bundles all +# Aliases used for running tests. + +# Let "run-test" be an alias for "test" +$(foreach t, $(ALL_NAMED_TESTS), $(eval run-test-$t: test-$t)) +$(foreach t, $(ALL_NAMED_TESTS), $(eval exploded-run-test-$t: exploded-test-$t)) +RUN_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) \ + $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS)) + +run-test: test +exploded-run-test: exploded-test + +# "make check" is a common idiom for running basic testing +check: test-tier1 + +# Keep some old names as aliases +test-hotspot-jtreg: test-hotspot_all +test-hotspot-jtreg-native: test-hotspot_native_sanity +test-hotspot-gtest: exploded-test-gtest +test-jdk-jtreg-native: test-jdk_native_sanity + +ALL_TARGETS += $(RUN_TEST_TARGETS) run-test exploded-run-test check \ + test-hotspot-jtreg test-hotspot-jtreg-native test-hotspot-gtest \ + test-jdk-jtreg-native + ################################################################################ ################################################################################ # diff --git a/make/MainSupport.gmk b/make/MainSupport.gmk index c7710ecb2c2..d7796c509d5 100644 --- a/make/MainSupport.gmk +++ b/make/MainSupport.gmk @@ -1,5 +1,5 @@ # -# Copyright (c) 2011, 2017, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2011, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -30,17 +30,6 @@ ifndef _MAINSUPPORT_GMK _MAINSUPPORT_GMK := 1 -# Run the tests specified by $1, with PRODUCT_HOME specified by $2 -# JT_JAVA is picked up by the jtreg launcher and used to run Jtreg itself. -define RunTests - ($(CD) $(TOPDIR)/test && $(MAKE) $(MAKE_ARGS) -j1 -k MAKEFLAGS= \ - JT_HOME=$(JT_HOME) PRODUCT_HOME=$(strip $2) \ - TEST_IMAGE_DIR=$(TEST_IMAGE_DIR) \ - ALT_OUTPUTDIR=$(OUTPUTDIR) TEST_JOBS=$(TEST_JOBS) \ - JT_JAVA=$(BOOT_JDK) JIB_JAR=$(JIB_JAR) \ - JOBS=$(JOBS) $1) || true -endef - define CleanDocs @$(PRINTF) "Cleaning docs ..." @$(PRINTF) "\n" $(LOG_DEBUG) diff --git a/make/RunTests.gmk b/make/RunTests.gmk index cffa93aae37..7b661be775f 100644 --- a/make/RunTests.gmk +++ b/make/RunTests.gmk @@ -1,5 +1,5 @@ # -# Copyright (c) 2016, 2017, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2016, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -300,16 +300,32 @@ endef # Helper function to determine if a test specification is a special test # -# It is a special test if it is "special:" followed by a test name. +# It is a special test if it is "special:" followed by a test name, +# if it is "make:" or "make-" followed by a make test, or any of the special test names +# as a single word. define ParseSpecialTestSelection $(if $(filter special:%, $1), \ $1 \ + ) \ + $(if $(filter make%, $1), \ + $(if $(filter make:%, $1), \ + special:$(strip $1) \ + ) \ + $(if $(filter make-%, $1), \ + special:$(patsubst make-%,make:%, $1) \ + ) \ + $(if $(filter make, $1), \ + special:make:all \ + ) + ) \ + $(if $(filter hotspot-internal failure-handler, $1), \ + special:$(strip $1) \ ) endef ifeq ($(TEST), ) $(info No test selection given in TEST!) - $(info Please use e.g. 'run-test TEST=tier1' or 'run-test-tier1') + $(info Please use e.g. 'make test TEST=tier1' or 'make test-tier1') $(info See doc/testing.[md|html] for help) $(error Cannot continue) endif @@ -663,9 +679,13 @@ define SetupRunSpecialTestBody $$(JDK_IMAGE_DIR)/bin/java -XX:+ExecuteInternalVMTests \ -XX:+ShowMessageBoxOnError -version else ifeq ($$($1_TEST_NAME), failure-handler) - $1_TEST_COMMAND_LINE := \ - ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \ - BuildFailureHandler.gmk test) + ifeq ($(BUILD_FAILURE_HANDLER), true) + $1_TEST_COMMAND_LINE := \ + ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \ + BuildFailureHandler.gmk test) + else + $$(error Cannot test failure handler if it is not built) + endif else ifeq ($$($1_TEST_NAME), make) $1_TEST_COMMAND_LINE := \ ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f \ diff --git a/make/common/FindTests.gmk b/make/common/FindTests.gmk index 35a9e20f305..e84152579c9 100644 --- a/make/common/FindTests.gmk +++ b/make/common/FindTests.gmk @@ -1,5 +1,5 @@ # -# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2017, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -72,6 +72,15 @@ ALL_NAMED_TESTS += $(JTREG_TEST_GROUPS) # Add Gtest ALL_NAMED_TESTS += gtest +# Find make test targets +MAKE_TEST_TARGETS := $(shell $(MAKE) -s --no-print-directory $(MAKE_ARGS) \ + SPEC=$(SPEC) -f $(TOPDIR)/test/make/TestMake.gmk print-targets) + +ALL_NAMED_TESTS += $(addprefix make-, $(MAKE_TEST_TARGETS)) + +# Add special tests +ALL_NAMED_TESTS += hotspot-internal failure-handler make + ################################################################################ endif # _FIND_TESTS_GMK diff --git a/make/conf/jib-profiles.js b/make/conf/jib-profiles.js index 6147b41838e..295dae375e6 100644 --- a/make/conf/jib-profiles.js +++ b/make/conf/jib-profiles.js @@ -759,7 +759,7 @@ var getJibProfilesProfiles = function (input, common, data) { testedProfile + ".test" ], src: "src.conf", - make_args: [ "run-test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ], + make_args: [ "test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ], environment: { "BOOT_JDK": common.boot_jdk_home, "JDK_IMAGE_DIR": input.get(testedProfile + ".jdk", "home_path"), diff --git a/test/Makefile b/test/Makefile index 3b92b8d5081..cfff86fa7d6 100644 --- a/test/Makefile +++ b/test/Makefile @@ -1,5 +1,5 @@ # -# Copyright (c) 2010, 2015, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2010, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -27,11 +27,16 @@ # Makefile to run tests from multiple sibling directories # +$(info WARNING: This way of running tests ("cd test && make") is deprecated) +$(info Please use "make test TEST=..." instead. See doc/testing.md for details) + # Macro to run a test target in a subdir define SUBDIR_TEST # subdirectory target if [ -d $1 ] ; then \ if [ -r $1/Makefile ] ; then \ $(MAKE) --no-print-directory -k -C $1 $2 ; \ + echo 'WARNING: This way of running tests ("cd test && make") is deprecated' ; \ + echo 'Please use "make test TEST=..." instead. See doc/testing.md for details' ; \ else \ echo "ERROR: File does not exist: $1/Makefile"; \ exit 1; \ diff --git a/test/make/TestMake.gmk b/test/make/TestMake.gmk index f274f7c2ea2..e4681c5ec75 100644 --- a/test/make/TestMake.gmk +++ b/test/make/TestMake.gmk @@ -1,5 +1,5 @@ # -# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved. +# Copyright (c) 2014, 2018, Oracle and/or its affiliates. All rights reserved. # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. # # This code is free software; you can redistribute it and/or modify it @@ -36,13 +36,17 @@ java-compilation: copy-files: +$(MAKE) -f TestCopyFiles.gmk $(TEST_SUBTARGET) -test-idea: +idea: +$(MAKE) -f TestIdea.gmk $(TEST_SUBTARGET) -test-compile-commands: +compile-commands: +$(MAKE) -f TestCompileCommands.gmk $(TEST_SUBTARGET) +TARGETS += make-base java-compilation copy-files idea compile-commands -all: make-base java-compilation copy-files test-idea +all: $(TARGETS) -.PHONY: default all make-base java-compilation copy-files test-idea test-compile-commands +print-targets: + $(ECHO) "$(TARGETS)" + +.PHONY: default all $(TARGETS) diff --git a/test/make/TestMakeBase.gmk b/test/make/TestMakeBase.gmk index 26b6cdad9ce..1b6d1344b20 100644 --- a/test/make/TestMakeBase.gmk +++ b/test/make/TestMakeBase.gmk @@ -344,7 +344,7 @@ $(eval $(call assert-equals, \ KWBASE := APA=banan;GURKA=tomat;COUNT=1%202%203%204%205;SUM=1+2+3+4+5;MANY_WORDS=I have the best words. $(eval $(call ParseKeywordVariable, KWBASE, \ - KEYWORDS := APA GURKA SUM, \ + SINGLE_KEYWORDS := APA GURKA SUM, \ STRING_KEYWORDS := COUNT MANY_WORDS, \ )) @@ -377,7 +377,7 @@ override KWBASE_WEIRD_GURKA := paprika KWBASE_WEIRD := ;;APA=banan;;;GURKA=apelsin;APA=skansen;; $(eval $(call ParseKeywordVariable, KWBASE_WEIRD, \ - KEYWORDS := APA GURKA SUM, \ + SINGLE_KEYWORDS := APA GURKA SUM, \ STRING_KEYWORDS := COUNT, \ ))