stan/jdk-24
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

8282567: Improve source-date handling in build system

Browse Source 
Reviewed-by: erikj, sgehwolf
This commit is contained in:
Magnus Ihse Bursie 2022-03-07 19:40:34 +00:00
parent 3996782c5a
commit 5d5bf16b0a
8 changed files with 161 additions and 81 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

42
doc/building.html
View File

@ -890,30 +890,28 @@ spawn failed</code></pre>
<p>If you need general help or advice about developing for the JDK, you can also contact the Adoption Group. See the section on <a href="#contributing-to-openjdk">Contributing to OpenJDK</a> for more information.</p>
<h2 id="reproducible-builds">Reproducible Builds</h2>
<p>Build reproducibility is the property of getting exactly the same bits out when building, every time, independent on who builds the product, or where. This is for many reasons a harder goal than it initially appears, but it is an important goal, for security reasons and others. Please see <a href="https://reproducible-builds.org">Reproducible Builds</a> for more information about the background and reasons for reproducible builds.</p>
<p>Currently, it is not possible to build OpenJDK fully reproducibly, but getting there is an ongoing effort. There are some things you can do to minimize non-determinism and make a larger part of the build reproducible:</p>
<p>Currently, it is not possible to build OpenJDK fully reproducibly, but getting there is an ongoing effort.</p>
<p>An absolute prerequisite for building reproducible is to speficy a fixed build time, since time stamps are embedded in many file formats. This is done by setting the <code>SOURCE_DATE_EPOCH</code> environment variable, which is an <a href="https://reproducible-builds.org/docs/source-date-epoch/">industry standard</a>, that many tools, such as gcc, recognize, and use in place of the current time when generating output.</p>
<p>To generate reproducible builds, you must set <code>SOURCE_DATE_EPOCH</code> before running <code>configure</code>. The value in <code>SOURCE_DATE_EPOCH</code> will be stored in the configuration, and used by <code>make</code>. Setting <code>SOURCE_DATE_EPOCH</code> before running <code>make</code> will have no effect on the build.</p>
<p>You must also make sure your build does not rely on <code>configure</code>'s default adhoc version strings. Default adhoc version strings <code>OPT</code> segment include user name and source directory. You can either override just the <code>OPT</code> segment using <code>--with-version-opt=&lt;any fixed string&gt;</code>, or you can specify the entire version string using <code>--with-version-string=&lt;your version&gt;</code>.</p>
<p>This is a typical example of how to build the JDK in a reproducible way:</p>
<pre><code>export SOURCE_DATE_EPOCH=946684800
bash configure --with-version-opt=adhoc
make</code></pre>
<p>Note that regardless if you specify a source date for <code>configure</code> or not, the JDK build system will set <code>SOURCE_DATE_EPOCH</code> for all build tools when building. If <code>--with-source-date</code> has the value <code>updated</code> (which is the default unless <code>SOURCE_DATE_EPOCH</code> is found by in the environment by <code>configure</code>), the source date value will be determined at build time.</p>
<p>There are several aspects of reproducible builds that can be individually adjusted by <code>configure</code> arguments. If any of these are given, they will override the value derived from <code>SOURCE_DATE_EPOCH</code>. These arguments are:</p>
<ul>
<li>Turn on build system support for reproducible builds</li>
<li><p><code>--with-source-date</code></p>
<p>This option controls how the JDK build sets <code>SOURCE_DATE_EPOCH</code> when building. It can be set to a value describing a date, either an epoch based timestamp as an integer, or a valid ISO-8601 date.</p>
<p>It can also be set to one of the special values <code>current</code>, <code>updated</code> or <code>version</code>. <code>current</code> means that the time of running <code>configure</code> will be used. <code>version</code> will use the nominal release date for the current JDK version. <code>updated</code>, which means that <code>SOURCE_DATE_EPOCH</code> will be set to the current time each time you are running <code>make</code>. All choices, except for <code>updated</code>, will set a fixed value for the source date timestamp.</p>
<p>When <code>SOURCE_DATE_EPOCH</code> is set, the default value for <code>--with-source-date</code> will be the value given by <code>SOURCE_DATE_EPOCH</code>. Otherwise, the default value is <code>updated</code>.</p></li>
<li><p><code>--with-hotspot-build-time</code></p>
<p>This option controls the build time string that will be included in the hotspot library (<code>libjvm.so</code> or <code>jvm.dll</code>). When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), the default value for <code>--with-hotspot-build-time</code> will be an ISO 8601 representation of that time stamp. Otherwise the default value will be the current time when building hotspot.</p></li>
<li><p><code>--with-copyright-year</code></p>
<p>This option controls the copyright year in some generated text files. When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), the default value for <code>--with-copyright-year</code> will be the year of that time stamp. Otherwise the default is the current year at the time of running configure. This can be overridden by <code>--with-copyright-year=&lt;year&gt;</code>.</p></li>
<li><p><code>--enable-reproducible-build</code></p>
<p>This option controls some additional behavior needed to make the build reproducible. When the source date is fixed (e.g. by setting <code>SOURCE_DATE_EPOCH</code>), this flag will be turned on by default. Otherwise, the value is determined by heuristics. If it is explicitly turned off, the build might not be reproducible.</p></li>
</ul>
<p>Add the flag <code>--enable-reproducible-build</code> to your <code>configure</code> command line. This will turn on support for reproducible builds where it could otherwise be lacking.</p>
<ul>
<li>Do not rely on <code>configure</code>'s default adhoc version strings</li>
</ul>
<p>Default adhoc version strings OPT segment include user name, source directory and timestamp. You can either override just the OPT segment using <code>--with-version-opt=&lt;any fixed string&gt;</code>, or you can specify the entire version string using <code>--with-version-string=&lt;your version&gt;</code>.</p>
<ul>
<li>Specify how the build sets <code>SOURCE_DATE_EPOCH</code></li>
</ul>
<p>The JDK build system will set the <code>SOURCE_DATE_EPOCH</code> environment variable during building, depending on the value of the <code>--with-source-date</code> option for <code>configure</code>. The default value is <code>updated</code>, which means that <code>SOURCE_DATE_EPOCH</code> will be set to the current time each time you are running <code>make</code>.</p>
<p>The <a href="https://reproducible-builds.org/docs/source-date-epoch/"><code>SOURCE_DATE_EPOCH</code> environment variable</a> is an industry standard, that many tools, such as gcc, recognize, and use in place of the current time when generating output.</p>
<p>For reproducible builds, you need to set this to a fixed value. You can use the special value <code>version</code> which will use the nominal release date for the current JDK version, or a value describing a date, either an epoch based timestamp as an integer, or a valid ISO-8601 date.</p>
<p><strong>Hint:</strong> If your build environment already sets <code>SOURCE_DATE_EPOCH</code>, you can propagate this using <code>--with-source-date=$SOURCE_DATE_EPOCH</code>.</p>
<ul>
<li>Specify a hotspot build time</li>
</ul>
<p>Set a fixed hotspot build time. This will be included in the hotspot library (<code>libjvm.so</code> or <code>jvm.dll</code>) and defaults to the current time when building hotspot. Use <code>--with-hotspot-build-time=&lt;any fixed string&gt;</code> for reproducible builds. It's a string so you don't need to format it specifically, so e.g. <code>n/a</code> will do. Another solution is to use the <code>SOURCE_DATE_EPOCH</code> variable, e.g. <code>--with-hotspot-build-time=$(date --date=@$SOURCE_DATE_EPOCH)</code>.</p>
<ul>
<li>Copyright year</li>
</ul>
<p>The copyright year in some generated text files are normally set to the current year. This can be overridden by <code>--with-copyright-year=&lt;year&gt;</code>. For fully reproducible builds, this needs to be set to a fixed value.</p>
<h2 id="hints-and-suggestions-for-advanced-users">Hints and Suggestions for Advanced Users</h2>
<h3 id="bash-completion">Bash Completion</h3>
<p>The <code>configure</code> and <code>make</code> commands tries to play nice with bash command-line completion (using <code>&lt;tab&gt;</code> or <code>&lt;tab&gt;&lt;tab&gt;</code>). To use this functionality, make sure you enable completion in your <code>~/.bashrc</code> (see instructions for bash in your operating system).</p>

102
doc/building.md
View File

@ -1527,57 +1527,85 @@ https://reproducible-builds.org) for more information about the background and
reasons for reproducible builds.
Currently, it is not possible to build OpenJDK fully reproducibly, but getting
there is an ongoing effort. There are some things you can do to minimize
non-determinism and make a larger part of the build reproducible:
there is an ongoing effort.
* Turn on build system support for reproducible builds
An absolute prerequisite for building reproducible is to speficy a fixed build
time, since time stamps are embedded in many file formats. This is done by
setting the `SOURCE_DATE_EPOCH` environment variable, which is an [industry
standard]( https://reproducible-builds.org/docs/source-date-epoch/), that many
tools, such as gcc, recognize, and use in place of the current time when
generating output.
Add the flag `--enable-reproducible-build` to your `configure` command line.
This will turn on support for reproducible builds where it could otherwise be
lacking.
To generate reproducible builds, you must set `SOURCE_DATE_EPOCH` before running
`configure`. The value in `SOURCE_DATE_EPOCH` will be stored in the
configuration, and used by `make`. Setting `SOURCE_DATE_EPOCH` before running
`make` will have no effect on the build.
* Do not rely on `configure`'s default adhoc version strings
Default adhoc version strings OPT segment include user name, source directory
and timestamp. You can either override just the OPT segment using
You must also make sure your build does not rely on `configure`'s default adhoc
version strings. Default adhoc version strings `OPT` segment include user name
and source directory. You can either override just the `OPT` segment using
`--with-version-opt=<any fixed string>`, or you can specify the entire version
string using `--with-version-string=<your version>`.
* Specify how the build sets `SOURCE_DATE_EPOCH`
This is a typical example of how to build the JDK in a reproducible way:
The JDK build system will set the `SOURCE_DATE_EPOCH` environment variable
during building, depending on the value of the `--with-source-date` option for
`configure`. The default value is `updated`, which means that
`SOURCE_DATE_EPOCH` will be set to the current time each time you are running
`make`.
```
export SOURCE_DATE_EPOCH=946684800
bash configure --with-version-opt=adhoc
make
```
The [`SOURCE_DATE_EPOCH` environment variable](
https://reproducible-builds.org/docs/source-date-epoch/) is an industry
standard, that many tools, such as gcc, recognize, and use in place of the
current time when generating output.
Note that regardless if you specify a source date for `configure` or not, the
JDK build system will set `SOURCE_DATE_EPOCH` for all build tools when building.
If `--with-source-date` has the value `updated` (which is the default unless
`SOURCE_DATE_EPOCH` is found by in the environment by `configure`), the source
date value will be determined at build time.
For reproducible builds, you need to set this to a fixed value. You can use the
special value `version` which will use the nominal release date for the current
JDK version, or a value describing a date, either an epoch based timestamp as an
integer, or a valid ISO-8601 date.
There are several aspects of reproducible builds that can be individually
adjusted by `configure` arguments. If any of these are given, they will override
the value derived from `SOURCE_DATE_EPOCH`. These arguments are:
**Hint:** If your build environment already sets `SOURCE_DATE_EPOCH`, you can
propagate this using `--with-source-date=$SOURCE_DATE_EPOCH`.
* `--with-source-date`
* Specify a hotspot build time
This option controls how the JDK build sets `SOURCE_DATE_EPOCH` when
building. It can be set to a value describing a date, either an epoch based
timestamp as an integer, or a valid ISO-8601 date.
Set a fixed hotspot build time. This will be included in the hotspot library
(`libjvm.so` or `jvm.dll`) and defaults to the current time when building
hotspot. Use `--with-hotspot-build-time=<any fixed string>` for reproducible
builds. It's a string so you don't need to format it specifically, so e.g. `n/a`
will do. Another solution is to use the `SOURCE_DATE_EPOCH` variable, e.g.
`--with-hotspot-build-time=$(date --date=@$SOURCE_DATE_EPOCH)`.
It can also be set to one of the special values `current`, `updated` or
`version`. `current` means that the time of running `configure` will be
used. `version` will use the nominal release date for the current JDK
version. `updated`, which means that `SOURCE_DATE_EPOCH` will be set to the
current time each time you are running `make`. All choices, except for
`updated`, will set a fixed value for the source date timestamp.
* Copyright year
When `SOURCE_DATE_EPOCH` is set, the default value for `--with-source-date`
will be the value given by `SOURCE_DATE_EPOCH`. Otherwise, the default value
is `updated`.
The copyright year in some generated text files are normally set to the current
year. This can be overridden by `--with-copyright-year=<year>`. For fully
reproducible builds, this needs to be set to a fixed value.
* `--with-hotspot-build-time`
This option controls the build time string that will be included in the
hotspot library (`libjvm.so` or `jvm.dll`). When the source date is fixed
(e.g. by setting `SOURCE_DATE_EPOCH`), the default value for
`--with-hotspot-build-time` will be an ISO 8601 representation of that time
stamp. Otherwise the default value will be the current time when building
hotspot.
* `--with-copyright-year`
This option controls the copyright year in some generated text files. When
the source date is fixed (e.g. by setting `SOURCE_DATE_EPOCH`), the default
value for `--with-copyright-year` will be the year of that time stamp.
Otherwise the default is the current year at the time of running configure.
This can be overridden by `--with-copyright-year=<year>`.
* `--enable-reproducible-build`
This option controls some additional behavior needed to make the build
reproducible. When the source date is fixed (e.g. by setting
`SOURCE_DATE_EPOCH`), this flag will be turned on by default. Otherwise, the
value is determined by heuristics. If it is explicitly turned off, the build
might not be reproducible.
## Hints and Suggestions for Advanced Users

23
make/InitSupport.gmk
View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2011, 2021, Oracle and/or its affiliates. All rights reserved.
# Copyright (c) 2011, 2022, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@ -310,17 +310,16 @@ else # $(HAS_SPEC)=true
# level of reproducible builds
define SetupReproducibleBuild
ifeq ($$(SOURCE_DATE), updated)
SOURCE_DATE := $$(shell $$(DATE) +"%s")
endif
export SOURCE_DATE_EPOCH := $$(SOURCE_DATE)
ifeq ($$(IS_GNU_DATE), yes)
export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) --utc \
--date="@$$(SOURCE_DATE_EPOCH)" \
+"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null)
else
export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) -u \
-j -f "%s" "$$(SOURCE_DATE_EPOCH)" \
+"%Y-%m-%dT%H:%M:%SZ" 2> /dev/null)
# For static values of SOURCE_DATE (not "updated"), these are set in spec.gmk
export SOURCE_DATE_EPOCH := $$(shell $$(DATE) +"%s")
ifeq ($$(IS_GNU_DATE), yes)
export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) --utc \
--date="@$$(SOURCE_DATE_EPOCH)" +"$$(ISO_8601_FORMAT_STRING)" \
2> /dev/null)
else
export SOURCE_DATE_ISO_8601 := $$(shell $$(DATE) -u -j -f "%s" \
"$$(SOURCE_DATE_EPOCH)" +"$$(ISO_8601_FORMAT_STRING)" 2> /dev/null)
endif
endif
endef

2
make/autoconf/configure.ac
View File

@ -152,6 +152,7 @@ BOOTJDK_SETUP_DOCS_REFERENCE_JDK
#
###############################################################################
JDKOPT_SETUP_REPRODUCIBLE_BUILD
JDKOPT_SETUP_JDK_OPTIONS
###############################################################################
@ -207,7 +208,6 @@ PLATFORM_SETUP_OPENJDK_TARGET_BITS
PLATFORM_SETUP_OPENJDK_TARGET_ENDIANNESS
# Configure flags for the tools. Need to know if we should build reproducible.
JDKOPT_SETUP_REPRODUCIBLE_BUILD
FLAGS_SETUP_FLAGS
# Setup debug symbols (need objcopy from the toolchain for that)

7
make/autoconf/help.m4
View File

@ -296,6 +296,13 @@ AC_DEFUN_ONCE([HELP_PRINT_SUMMARY_AND_WARNINGS],
printf "* OpenJDK target: OS: $OPENJDK_TARGET_OS, CPU architecture: $OPENJDK_TARGET_CPU_ARCH, address length: $OPENJDK_TARGET_CPU_BITS\n"
printf "* Version string: $VERSION_STRING ($VERSION_SHORT)\n"
if test "x$SOURCE_DATE" != xupdated; then
source_date_info="$SOURCE_DATE ($SOURCE_DATE_ISO_8601)"
else
source_date_info="Determined at build time"
fi
printf "* Source date: $source_date_info\n"
printf "\n"
printf "Tools summary:\n"
if test "x$OPENJDK_BUILD_OS" = "xwindows"; then

18
make/autoconf/hotspot.m4
View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2011, 2020, Oracle and/or its affiliates. All rights reserved.
# Copyright (c) 2011, 2022, Oracle and/or its affiliates. All rights reserved.
# DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
#
# This code is free software; you can redistribute it and/or modify it
@ -114,12 +114,26 @@ AC_DEFUN_ONCE([HOTSPOT_SETUP_MISC],
HOTSPOT_TARGET_CPU_ARCH=zero
fi
AC_ARG_WITH([hotspot-build-time], [AS_HELP_STRING([--with-hotspot-build-time],
[timestamp to use in hotspot version string, empty for on-the-fly @<:@empty@:>@])])
[timestamp to use in hotspot version string, empty means determined at build time @<:@source-date/empty@:>@])])
AC_MSG_CHECKING([what hotspot build time to use])
if test "x$with_hotspot_build_time" != x; then
HOTSPOT_BUILD_TIME="$with_hotspot_build_time"
AC_MSG_RESULT([$HOTSPOT_BUILD_TIME (from --with-hotspot-build-time)])
else
if test "x$SOURCE_DATE" = xupdated; then
HOTSPOT_BUILD_TIME=""
AC_MSG_RESULT([determined at build time (default)])
else
# If we have a fixed value for SOURCE_DATE, use it as default
HOTSPOT_BUILD_TIME="$SOURCE_DATE_ISO_8601"
AC_MSG_RESULT([$HOTSPOT_BUILD_TIME (from --with-source-date)])
fi
fi
AC_SUBST(HOTSPOT_BUILD_TIME)

41
make/autoconf/jdk-options.m4
View File

@ -211,16 +211,16 @@ AC_DEFUN_ONCE([JDKOPT_SETUP_JDK_OPTIONS],
# Setup default copyright year. Mostly overridden when building close to a new year.
AC_ARG_WITH(copyright-year, [AS_HELP_STRING([--with-copyright-year],
[Set copyright year value for build @<:@current year@:>@])])
[Set copyright year value for build @<:@current year/source-date@:>@])])
if test "x$with_copyright_year" = xyes; then
AC_MSG_ERROR([Copyright year must have a value])
elif test "x$with_copyright_year" != x; then
COPYRIGHT_YEAR="$with_copyright_year"
elif test "x$SOURCE_DATE_EPOCH" != x; then
elif test "x$SOURCE_DATE" != xupdated; then
if test "x$IS_GNU_DATE" = xyes; then
COPYRIGHT_YEAR=`date --date=@$SOURCE_DATE_EPOCH +%Y`
COPYRIGHT_YEAR=`$DATE --date=@$SOURCE_DATE +%Y`
else
COPYRIGHT_YEAR=`date -j -f %s $SOURCE_DATE_EPOCH +%Y`
COPYRIGHT_YEAR=`$DATE -j -f %s $SOURCE_DATE +%Y`
fi
else
COPYRIGHT_YEAR=`$DATE +'%Y'`
@ -662,15 +662,28 @@ AC_DEFUN([JDKOPT_ALLOW_ABSOLUTE_PATHS_IN_OUTPUT],
AC_DEFUN_ONCE([JDKOPT_SETUP_REPRODUCIBLE_BUILD],
[
AC_ARG_WITH([source-date], [AS_HELP_STRING([--with-source-date],
[how to set SOURCE_DATE_EPOCH ('updated', 'current', 'version' a timestamp or an ISO-8601 date) @<:@updated@:>@])],
[how to set SOURCE_DATE_EPOCH ('updated', 'current', 'version' a timestamp or an ISO-8601 date) @<:@updated/value of SOURCE_DATE_EPOCH@:>@])],
[with_source_date_present=true], [with_source_date_present=false])
if test "x$SOURCE_DATE_EPOCH" != x && test "x$with_source_date" != x; then
AC_MSG_WARN([--with-source-date will override SOURCE_DATE_EPOCH])
fi
AC_MSG_CHECKING([what source date to use])
if test "x$with_source_date" = xyes; then
AC_MSG_ERROR([--with-source-date must have a value])
elif test "x$with_source_date" = xupdated || test "x$with_source_date" = x; then
# Tell the makefiles to update at each build
elif test "x$with_source_date" = x; then
if test "x$SOURCE_DATE_EPOCH" != x; then
SOURCE_DATE=$SOURCE_DATE_EPOCH
with_source_date_present=true
AC_MSG_RESULT([$SOURCE_DATE, from SOURCE_DATE_EPOCH])
else
# Tell the makefiles to update at each build
SOURCE_DATE=updated
AC_MSG_RESULT([determined at build time (default)])
fi
elif test "x$with_source_date" = xupdated; then
SOURCE_DATE=updated
AC_MSG_RESULT([determined at build time, from 'updated'])
elif test "x$with_source_date" = xcurrent; then
@ -702,6 +715,18 @@ AC_DEFUN_ONCE([JDKOPT_SETUP_REPRODUCIBLE_BUILD],
fi
fi
ISO_8601_FORMAT_STRING="%Y-%m-%dT%H:%M:%SZ"
if test "x$SOURCE_DATE" != xupdated; then
# If we have a fixed value for SOURCE_DATE, we need to set SOURCE_DATE_EPOCH
# for the rest of configure.
SOURCE_DATE_EPOCH="$SOURCE_DATE"
if test "x$IS_GNU_DATE" = xyes; then
SOURCE_DATE_ISO_8601=`$DATE --utc --date="@$SOURCE_DATE" +"$ISO_8601_FORMAT_STRING" 2> /dev/null`
else
SOURCE_DATE_ISO_8601=`$DATE -u -j -f "%s" "$SOURCE_DATE" +"$ISO_8601_FORMAT_STRING" 2> /dev/null`
fi
fi
REPRODUCIBLE_BUILD_DEFAULT=$with_source_date_present
if test "x$OPENJDK_BUILD_OS" = xwindows && \
@ -726,4 +751,6 @@ AC_DEFUN_ONCE([JDKOPT_SETUP_REPRODUCIBLE_BUILD],
AC_SUBST(SOURCE_DATE)
AC_SUBST(ENABLE_REPRODUCIBLE_BUILD)
AC_SUBST(ISO_8601_FORMAT_STRING)
AC_SUBST(SOURCE_DATE_ISO_8601)
])

7
make/autoconf/spec.gmk.in
View File

@ -130,6 +130,13 @@ RELEASE_FILE_LIBC:=@RELEASE_FILE_LIBC@
SOURCE_DATE := @SOURCE_DATE@
ENABLE_REPRODUCIBLE_BUILD := @ENABLE_REPRODUCIBLE_BUILD@
ISO_8601_FORMAT_STRING := @ISO_8601_FORMAT_STRING@
ifneq ($(SOURCE_DATE), updated)
# For "updated" source date value, these are set in InitSupport.gmk
export SOURCE_DATE_EPOCH := $(SOURCE_DATE)
SOURCE_DATE_ISO_8601 := @SOURCE_DATE_ISO_8601@
endif
LIBM:=@LIBM@
LIBDL:=@LIBDL@