8213698: Improve devkit creation and add support for linux/ppc64/ppc64le/s390x

Reviewed-by: erikj, ihse
This commit is contained in:
Volker Simonis 2018-11-22 17:22:40 +01:00
parent c047c5cfcf
commit faff794a88
4 changed files with 168 additions and 53 deletions

View File

@ -69,6 +69,7 @@
</ul></li>
<li><a href="#running-tests">Running Tests</a></li>
<li><a href="#cross-compiling">Cross-compiling</a><ul>
<li><a href="#cross-compiling-the-easy-way-with-openjdk-devkits">Cross compiling the easy way with OpenJDK devkits</a></li>
<li><a href="#boot-jdk-and-build-jdk">Boot JDK and Build JDK</a></li>
<li><a href="#specifying-the-target-platform">Specifying the Target Platform</a></li>
<li><a href="#toolchain-considerations">Toolchain Considerations</a></li>
@ -570,6 +571,47 @@ CC: Sun C++ 5.13 SunOS_i386 151846-10 2015/10/30</code></pre>
<p>This requires a more complex setup and build procedure. This section assumes you are familiar with cross-compiling in general, and will only deal with the particularities of cross-compiling the JDK. If you are new to cross-compiling, please see the <a href="https://en.wikipedia.org/wiki/Cross_compiler#External_links">external links at Wikipedia</a> for a good start on reading materials.</p>
<p>Cross-compiling the JDK requires you to be able to build both for the build platform and for the target platform. The reason for the former is that we need to build and execute tools during the build process, both native tools and Java tools.</p>
<p>If all you want to do is to compile a 32-bit version, for the same OS, on a 64-bit machine, consider using <code>--with-target-bits=32</code> instead of doing a full-blown cross-compilation. (While this surely is possible, it's a lot more work and will take much longer to build.)</p>
<h3 id="cross-compiling-the-easy-way-with-openjdk-devkits">Cross compiling the easy way with OpenJDK devkits</h3>
<p>The OpenJDK build system provides out-of-the box support for creating and using so called devkits. A <code>devkit</code> is basically a collection of a cross-compiling toolchain and a sysroot environment which can easily be used together with the <code>--with-devkit</code> configure option to cross compile the OpenJDK. On Linux/x86_64, the following command:</p>
<pre><code>bash configure --with-devkit=&lt;devkit-path&gt; --openjdk-target=ppc64-linux-gnu &amp;&amp; make</code></pre>
<p>will configure and build OpenJDK for Linux/ppc64 assuming that <code>&lt;devkit-path&gt;</code> points to a Linux/x86_64 to Linux/ppc64 devkit.</p>
<p>Devkits can be created from the <code>make/devkit</code> directory by executing:</p>
<pre><code>make [ TARGETS=&quot;&lt;TARGET_TRIPLET&gt;+&quot; ] [ BASE_OS=&lt;OS&gt; ] [ BASE_OS_VERSION=&lt;VER&gt; ]</code></pre>
<p>where <code>TARGETS</code> contains one or more <code>TARGET_TRIPLET</code>s of the form described in <a href="https://sourceware.org/autobook/autobook/autobook_17.html">section 3.4 of the GNU Autobook</a>. If no targets are given, a native toolchain for the current platform will be created. Currently, at least the following targets are known to work:</p>
<table>
<thead>
<tr class="header">
<th style="text-align: left;">Supported devkit targets</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">x86_64-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">aarch64-linux-gnu</td>
</tr>
<tr class="odd">
<td style="text-align: left;">arm-linux-gnueabihf</td>
</tr>
<tr class="even">
<td style="text-align: left;">ppc64-linux-gnu</td>
</tr>
<tr class="odd">
<td style="text-align: left;">ppc64le-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">s390x-linux-gnu</td>
</tr>
</tbody>
</table>
<p><code>BASE_OS</code> must be one of &quot;OEL6&quot; for Oracle Enterprise Linux 6 or &quot;Fedora&quot; (if not specified &quot;OEL6&quot; will be the default). If the base OS is &quot;Fedora&quot; the corresponding Fedora release can be specified with the help of the <code>BASE_OS_VERSION</code> option (with &quot;27&quot; as default version). If the build is successful, the new devkits can be found in the <code>build/devkit/result</code> subdirectory:</p>
<pre><code>cd make/devkit
make TARGETS=&quot;ppc64le-linux-gnu aarch64-linux-gnu&quot; BASE_OS=Fedora BASE_OS_VERSION=21
ls -1 ../../build/devkit/result/
x86_64-linux-gnu-to-aarch64-linux-gnu
x86_64-linux-gnu-to-ppc64le-linux-gnu</code></pre>
<p>Notice that devkits are not only useful for targeting different build platforms. Because they contain the full build dependencies for a system (i.e. compiler and root file system), they can easily be used to build well-known, reliable and reproducible build environments. You can for example create and use a devkit with GCC 7.3 and a Fedora 12 sysroot environment (with glibc 2.11) on Ubuntu 14.04 (which doesn't have GCC 7.3 by default) to produce OpenJDK binaries which will run on all Linux systems with runtime libraries newer than the ones from Fedora 12 (e.g. Ubuntu 16.04, SLES 11 or RHEL 6).</p>
<h3 id="boot-jdk-and-build-jdk">Boot JDK and Build JDK</h3>
<p>When cross-compiling, make sure you use a boot JDK that runs on the <em>build</em> system, and not on the <em>target</em> system.</p>
<p>To be able to build, we need a &quot;Build JDK&quot;, which is a JDK built from the current sources (that is, the same as the end result of the entire build process), but able to run on the <em>build</em> system, and not the <em>target</em> system. (In contrast, the Boot JDK should be from an older release, e.g. JDK 8 when building JDK 9.)</p>
@ -662,8 +704,8 @@ ls build/linux-aarch64-normal-server-release/</code></pre></li>
<th style="text-align: left;">Target</th>
<th style="text-align: left;"><code>CC</code></th>
<th style="text-align: left;"><code>CXX</code></th>
<th><code>--arch=...</code></th>
<th><code>--openjdk-target=...</code></th>
<th style="text-align: left;"><code>--arch=...</code></th>
<th style="text-align: left;"><code>--openjdk-target=...</code></th>
</tr>
</thead>
<tbody>
@ -671,36 +713,36 @@ ls build/linux-aarch64-normal-server-release/</code></pre></li>
<td style="text-align: left;">x86</td>
<td style="text-align: left;">default</td>
<td style="text-align: left;">default</td>
<td>i386</td>
<td>i386-linux-gnu</td>
<td style="text-align: left;">i386</td>
<td style="text-align: left;">i386-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">armhf</td>
<td style="text-align: left;">gcc-arm-linux-gnueabihf</td>
<td style="text-align: left;">g++-arm-linux-gnueabihf</td>
<td>armhf</td>
<td>arm-linux-gnueabihf</td>
<td style="text-align: left;">armhf</td>
<td style="text-align: left;">arm-linux-gnueabihf</td>
</tr>
<tr class="odd">
<td style="text-align: left;">aarch64</td>
<td style="text-align: left;">gcc-aarch64-linux-gnu</td>
<td style="text-align: left;">g++-aarch64-linux-gnu</td>
<td>arm64</td>
<td>aarch64-linux-gnu</td>
<td style="text-align: left;">arm64</td>
<td style="text-align: left;">aarch64-linux-gnu</td>
</tr>
<tr class="even">
<td style="text-align: left;">ppc64el</td>
<td style="text-align: left;">gcc-powerpc64le-linux-gnu</td>
<td style="text-align: left;">g++-powerpc64le-linux-gnu</td>
<td>ppc64el</td>
<td>powerpc64le-linux-gnu</td>
<td style="text-align: left;">ppc64el</td>
<td style="text-align: left;">powerpc64le-linux-gnu</td>
</tr>
<tr class="odd">
<td style="text-align: left;">s390x</td>
<td style="text-align: left;">gcc-s390x-linux-gnu</td>
<td style="text-align: left;">g++-s390x-linux-gnu</td>
<td>s390x</td>
<td>s390x-linux-gnu</td>
<td style="text-align: left;">s390x</td>
<td style="text-align: left;">s390x-linux-gnu</td>
</tr>
</tbody>
</table>
@ -813,7 +855,12 @@ cannot create ... Permission denied
spawn failed</code></pre>
<p>This can be a sign of a Cygwin problem. See the information about solving problems in the <a href="#cygwin">Cygwin</a> section. Rebooting the computer might help temporarily.</p>
<h3 id="getting-help">Getting Help</h3>
<p>If none of the suggestions in this document helps you, or if you find what you believe is a bug in the build system, please contact the Build Group by sending a mail to <a href="mailto:build-dev@openjdk.java.net">build-dev@openjdk.java.net</a>. Please include the relevant parts of the configure and/or build log.</p>
<p>If none of the suggestions in this document helps you, or if you find what you believe is a bug in the build system, please contact the Build Group by sending a mail to <script type="text/javascript">
<!--
h='&#x6f;&#112;&#x65;&#110;&#106;&#100;&#x6b;&#46;&#106;&#x61;&#118;&#x61;&#46;&#110;&#x65;&#116;';a='&#64;';n='&#98;&#x75;&#x69;&#108;&#100;&#x2d;&#100;&#x65;&#118;';e=n+a+h;
document.write('<a h'+'ref'+'="ma'+'ilto'+':'+e+'" clas'+'s="em' + 'ail">'+e+'<\/'+'a'+'>');
// -->
</script><noscript>&#98;&#x75;&#x69;&#108;&#100;&#x2d;&#100;&#x65;&#118;&#32;&#x61;&#116;&#32;&#x6f;&#112;&#x65;&#110;&#106;&#100;&#x6b;&#32;&#100;&#x6f;&#116;&#32;&#106;&#x61;&#118;&#x61;&#32;&#100;&#x6f;&#116;&#32;&#110;&#x65;&#116;</noscript>. Please include the relevant parts of the configure and/or build log.</p>
<p>If you need general help or advice about developing for the JDK, you can also contact the Adoption Group. See the section on <a href="#contributing-to-openjdk">Contributing to OpenJDK</a> for more information.</p>
<h2 id="hints-and-suggestions-for-advanced-users">Hints and Suggestions for Advanced Users</h2>
<h3 id="setting-up-a-repository-for-pushing-changes-defpath">Setting Up a Repository for Pushing Changes (defpath)</h3>

View File

@ -884,6 +884,64 @@ If all you want to do is to compile a 32-bit version, for the same OS, on a
full-blown cross-compilation. (While this surely is possible, it's a lot more
work and will take much longer to build.)
### Cross compiling the easy way with OpenJDK devkits
The OpenJDK build system provides out-of-the box support for creating and using
so called devkits. A `devkit` is basically a collection of a cross-compiling
toolchain and a sysroot environment which can easily be used together with the
`--with-devkit` configure option to cross compile the OpenJDK. On Linux/x86_64,
the following command:
```
bash configure --with-devkit=<devkit-path> --openjdk-target=ppc64-linux-gnu && make
```
will configure and build OpenJDK for Linux/ppc64 assuming that `<devkit-path>`
points to a Linux/x86_64 to Linux/ppc64 devkit.
Devkits can be created from the `make/devkit` directory by executing:
```
make [ TARGETS="<TARGET_TRIPLET>+" ] [ BASE_OS=<OS> ] [ BASE_OS_VERSION=<VER> ]
```
where `TARGETS` contains one or more `TARGET_TRIPLET`s of the form
described in [section 3.4 of the GNU Autobook](
https://sourceware.org/autobook/autobook/autobook_17.html). If no
targets are given, a native toolchain for the current platform will be
created. Currently, at least the following targets are known to work:
Supported devkit targets
------------------------
x86_64-linux-gnu
aarch64-linux-gnu
arm-linux-gnueabihf
ppc64-linux-gnu
ppc64le-linux-gnu
s390x-linux-gnu
`BASE_OS` must be one of "OEL6" for Oracle Enterprise Linux 6 or
"Fedora" (if not specified "OEL6" will be the default). If the base OS
is "Fedora" the corresponding Fedora release can be specified with the
help of the `BASE_OS_VERSION` option (with "27" as default version).
If the build is successful, the new devkits can be found in the
`build/devkit/result` subdirectory:
```
cd make/devkit
make TARGETS="ppc64le-linux-gnu aarch64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=21
ls -1 ../../build/devkit/result/
x86_64-linux-gnu-to-aarch64-linux-gnu
x86_64-linux-gnu-to-ppc64le-linux-gnu
```
Notice that devkits are not only useful for targeting different build
platforms. Because they contain the full build dependencies for a
system (i.e. compiler and root file system), they can easily be used
to build well-known, reliable and reproducible build environments. You
can for example create and use a devkit with GCC 7.3 and a Fedora 12
sysroot environment (with glibc 2.11) on Ubuntu 14.04 (which doesn't
have GCC 7.3 by default) to produce OpenJDK binaries which will run on
all Linux systems with runtime libraries newer than the ones from
Fedora 12 (e.g. Ubuntu 16.04, SLES 11 or RHEL 6).
### Boot JDK and Build JDK
When cross-compiling, make sure you use a boot JDK that runs on the *build*

View File

@ -41,9 +41,12 @@
# To build the full set of crosstools for additional platforms, use a command
# line looking like this:
#
# make cross_compile_target="aarch64-linux-gnu" BASE_OS=Fedora27
# make TARGETS="aarch64-linux-gnu" BASE_OS=Fedora
# or
# make cross_compile_target="arm-linux-gnueabihf" BASE_OS=Fedora27
# make TARGETS="arm-linux-gnueabihf ppc64-linux-gnu" BASE_OS=Fedora BASE_OS_VERSION=17
#
# to build several devkits for a specific OS version at once.
# You can find the final results under ../../build/devkit/result/<host>-to-<target>
#
# This is the makefile which iterates over all host and target platforms.
#
@ -52,18 +55,18 @@ os := $(shell uname -o)
cpu := $(shell uname -p)
# Figure out what platform this is building on.
me := $(cpu)-$(if $(findstring Linux,$(os)),unknown-linux-gnu)
me := $(cpu)-$(if $(findstring Linux,$(os)),linux-gnu)
$(info Building on platform $(me))
#
# By default just build for the current platform, which is assumed to be Linux
#
ifeq ($(cross_compile_target), )
ifeq ($(TARGETS), )
platforms := $(me)
host_platforms := $(platforms)
else
platforms := $(cross_compile_target)
platforms := $(TARGETS)
host_platforms := $(me)
endif
target_platforms := $(platforms)
@ -79,37 +82,37 @@ endif
OUTPUT_ROOT = $(abspath ../../build/devkit)
RESULT = $(OUTPUT_ROOT)/result
submakevars = HOST=$@ BUILD=$(me) \
RESULT=$(RESULT) PREFIX=$(RESULT)/$@ \
OUTPUT_ROOT=$(OUTPUT_ROOT)
submakevars = HOST=$@ BUILD=$(me) RESULT=$(RESULT) OUTPUT_ROOT=$(OUTPUT_ROOT)
$(host_platforms) :
@echo 'Building compilers for $@'
@echo 'Targets: $(target_platforms)'
for p in $(filter $@, $(target_platforms)) $(filter-out $@, $(target_platforms)); do \
$(MAKE) -f Tools.gmk download-rpms $(submakevars) TARGET=$$p && \
$(MAKE) -f Tools.gmk download-rpms $(submakevars) \
TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
$(MAKE) -f Tools.gmk all $(submakevars) \
TARGET=$$p || exit 1 ; \
TARGET=$$p PREFIX=$(RESULT)/$@-to-$$p && \
$(MAKE) -f Tools.gmk ccache $(submakevars) \
TARGET=$@ PREFIX=$(RESULT)/$@-to-$$p BUILDDIR=$(OUTPUT_ROOT)/$@/$$p || exit 1 ; \
done
@echo 'Building ccache program for $@'
$(MAKE) -f Tools.gmk ccache $(submakevars) TARGET=$@
@echo 'All done"'
today := $(shell date +%Y%m%d)
define Mktar
$(1)_tar = $$(RESULT)/sdk-$(1)-$$(today).tar.gz
$$($(1)_tar) : PLATFORM = $(1)
TARFILES += $$($(1)_tar)
$$($(1)_tar) : $(1) $$(shell find $$(RESULT)/$(1))
$(1)-to-$(2)_tar = $$(RESULT)/sdk-$(1)-to-$(2)-$$(today).tar.gz
$$($(1)-to-$(2)_tar) : PLATFORM = $(1)-to-$(2)
TARFILES += $$($(1)-to-$(2)_tar)
$$($(1)-to-$(2)_tar) : $$(shell find $$(RESULT)/$(1)-to-$(2) -type f)
endef
$(foreach p,$(host_platforms),$(eval $(call Mktar,$(p))))
$(foreach p,$(host_platforms),$(foreach t,$(target_platforms),$(eval $(call Mktar,$(p),$(t)))))
tars : all $(TARFILES)
onlytars : $(TARFILES)
%.tar.gz :
@echo 'Creating compiler package $@'
cd $(RESULT)/$(PLATFORM) && tar -czf $@ *
cd $(RESULT) && tar -czf $@ $(PLATFORM)/*
touch $@
clean :

View File

@ -52,16 +52,25 @@ endif
$(info ARCH=$(ARCH))
ifeq ($(BASE_OS), OEL6)
OEL_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
BASE_URL := http://yum.oracle.com/repo/OracleLinux/OL6/4/base/$(ARCH)/
LINUX_VERSION := OEL6.4
else ifeq ($(BASE_OS), Fedora27)
ifeq ($(ARCH), aarch64)
FEDORA_TYPE=fedora-secondary
else
FEDORA_TYPE=fedora/linux
else ifeq ($(BASE_OS), Fedora)
DEFAULT_OS_VERSION := 27
ifeq ($(BASE_OS_VERSION), )
BASE_OS_VERSION := $(DEFAULT_OS_VERSION)
endif
OEL_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/27/Everything/$(ARCH)/os/Packages/
LINUX_VERSION := Fedora 27
ifeq ($(filter x86_64 armhfp, $(ARCH)), )
FEDORA_TYPE := fedora-secondary
else
FEDORA_TYPE := fedora/linux
endif
ARCHIVED := $(shell [ $(BASE_OS_VERSION) -lt $(DEFAULT_OS_VERSION) ] && echo true)
ifeq ($(ARCHIVED),true)
BASE_URL := https://archives.fedoraproject.org/pub/archive/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
else
BASE_URL := https://dl.fedoraproject.org/pub/$(FEDORA_TYPE)/releases/$(BASE_OS_VERSION)/Everything/$(ARCH)/os/Packages/
endif
LINUX_VERSION := Fedora_$(BASE_OS_VERSION)
else
$(error Unknown base OS $(BASE_OS))
endif
@ -137,13 +146,11 @@ else
endif
# Define directories
RESULT := $(OUTPUT_ROOT)/result
BUILDDIR := $(OUTPUT_ROOT)/$(HOST)/$(TARGET)
PREFIX := $(RESULT)/$(HOST)
TARGETDIR := $(PREFIX)/$(TARGET)
SYSROOT := $(TARGETDIR)/sysroot
DOWNLOAD := $(OUTPUT_ROOT)/download
DOWNLOAD_RPMS := $(DOWNLOAD)/rpms
DOWNLOAD_RPMS := $(DOWNLOAD)/rpms/$(TARGET)-$(LINUX_VERSION)
SRCDIR := $(OUTPUT_ROOT)/src
# Marker file for unpacking rpms
@ -159,7 +166,7 @@ download-rpms:
# Only run this if rpm dir is empty.
ifeq ($(wildcard $(DOWNLOAD_RPMS)/*.rpm), )
cd $(DOWNLOAD_RPMS) && \
wget -r -np -nd $(patsubst %, -A "*%*.rpm", $(RPM_LIST)) $(OEL_URL)
wget -r -np -nd $(patsubst %, -A "*%*.rpm", $(RPM_LIST)) $(BASE_URL)
endif
##########################################################################################
@ -190,8 +197,8 @@ $(foreach p,GCC BINUTILS CCACHE MPFR GMP MPC GDB,$(eval $(call Download,$(p))))
##########################################################################################
# Unpack RPMS
RPM_ARCHS := $(ARCH) noarch
ifeq ($(ARCH),x86_64)
RPM_ARCHS := x86_64 noarch
ifeq ($(BUILD),$(HOST))
ifeq ($(TARGET),$(HOST))
# When building the native compiler for x86_64, enable mixed mode.
@ -199,11 +206,9 @@ ifeq ($(ARCH),x86_64)
endif
endif
else ifeq ($(ARCH),i686)
RPM_ARCHS := i386 i686 noarch
RPM_ARCHS += i386
else ifeq ($(ARCH), armhfp)
RPM_ARCHS := $(ARCH) armv7hl noarch
else
RPM_ARCHS := $(ARCH) noarch
RPM_ARCHS += armv7hl
endif
RPM_FILE_LIST := $(sort $(foreach a, $(RPM_ARCHS), \
@ -277,7 +282,7 @@ CONFIG = --target=$(TARGET) \
--host=$(HOST) --build=$(BUILD) \
--prefix=$(PREFIX)
PATHEXT = $(RESULT)/$(BUILD)/bin:
PATHEXT = $(PREFIX)/bin:
PATHPRE = PATH=$(PATHEXT)$(PATH)
NUM_CORES := $(shell cat /proc/cpuinfo | grep -c processor)
@ -427,6 +432,11 @@ ifeq ($(ARCH), armhfp)
$(BUILDDIR)/$(gcc_ver)/Makefile : CONFIG += --with-float=hard
endif
ifneq ($(filter ppc64 ppc64le s390x, $(ARCH)), )
# We only support 64-bit on these platforms anyway
CONFIG += --disable-multilib
endif
# Want:
# c,c++
# shared libs
@ -552,7 +562,7 @@ $(TARGETDIR)/%.done : $(BUILDDIR)/%/Makefile
##########################################################################################
$(PREFIX)/devkit.info: FRC
$(PREFIX)/devkit.info:
@echo 'Creating devkit.info in the root of the kit'
rm -f $@
touch $@
@ -611,7 +621,4 @@ all : binutils gcc bfdlib $(PREFIX)/devkit.info $(missing-links) $(SYSROOT_LINKS
# this is only built for host. so separate.
ccache : $(ccache)
# Force target
FRC:
.PHONY : gcc all binutils bfdlib link_libs rpms libs sysroot