diff options
| -rw-r--r-- | llvm/docs/Advanced_Builds.rst | 174 | ||||
| -rw-r--r-- | llvm/docs/CMake.rst | 35 | ||||
| -rw-r--r-- | llvm/docs/index.rst | 1 |
3 files changed, 210 insertions, 0 deletions
diff --git a/llvm/docs/Advanced_Builds.rst b/llvm/docs/Advanced_Builds.rst new file mode 100644 index 00000000000..126f9d0f2fb --- /dev/null +++ b/llvm/docs/Advanced_Builds.rst @@ -0,0 +1,174 @@ +============================= +Advanced Build Configurations +============================= + +.. contents:: + :local: + +Introduction +============ + +`CMake <http://www.cmake.org/>`_ is a cross-platform build-generator tool. CMake +does not build the project, it generates the files needed by your build tool +(GNU make, Visual Studio, etc.) for building LLVM. + +If **you are a new contributor**, please start with the :doc:`GettingStarted` or +:doc:`CMake` pages. This page is intended for users doing more complex builds. + +Many of the examples below are written assuming specific CMake Generators. +Unless otherwise explicitly called out these commands should work with any CMake +generator. + +Bootstrap Builds +================ + +The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a +high level a multi-stage build is a chain of builds that pass data from one +stage into the next. The most common and simple version of this is a traditional +bootstrap build. + +In a simple two-stage bootstrap build, we build clang using the system compiler, +then use that just-built clang to build clang again. In CMake this simplest form +of a bootstrap build can be configured with a single option, +CLANG_ENABLE_BOOTSTRAP. + +.. code-block:: console + + $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On <path to source> + $ ninja stage2 + +This command itself isn’t terribly useful because it assumes default +configurations for each stage. The next series of examples utilize CMake cache +scripts to provide more complex options. + +The clang build system refers to builds as stages. A stage1 build is a standard +build using the compiler installed on the host, and a stage2 build is built +using the stage1 compiler. This nomenclature holds up to more stages too. In +general a stage*n* build is built using the output from stage*n-1*. + +Apple Clang Builds (A More Complex Bootstrap) +============================================= + +Apple’s Clang builds are a slightly more complicated example of the simple +bootstrapping scenario. Apple Clang is built using a 2-stage build. + +The stage1 compiler is a host-only compiler with some options set. The stage1 +compiler is a balance of optimization vs build time because it is a throwaway. +The stage2 compiler is the fully optimized compiler intended to ship to users. + +Setting up these compilers requires a lot of options. To simplify the +configuration the Apple Clang build settings are contained in CMake Cache files. +You can build an Apple Clang compiler using the following commands: + +.. code-block:: console + + $ make -G Ninja -C <path to clang>/cmake/caches/Apple-stage1.cmake <path to source> + $ ninja stage2-distribution + +This CMake invocation configures the stage1 host compiler, and sets +CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the +stage2 configuration step. + +When you build the stage2-distribution target it builds the minimal stage1 +compiler and required tools, then configures and builds the stage2 compiler +based on the settings in Apple-stage2.cmake. + +This pattern of using cache scripts to set complex settings, and specifically to +make later stage builds include cache scripts is common in our more advanced +build configurations. + +Multi-stage PGO +=============== + +Profile-Guided Optimizations (PGO) is a really great way to optimize the code +clang generates. Our multi-stage PGO builds are a workflow for generating PGO +profiles that can be used to optimize clang. + +At a high level, the way PGO works is that you build an instrumented compiler, +then you run the instrumented compiler against sample source files. While the +instrumented compiler runs it will output a bunch of files containing +performance counters (.profraw files). After generating all the profraw files +you use llvm-profdata to merge the files into a single profdata file that you +can feed into the LLVM_PROFDATA_FILE option. + +Our PGO.cmake cache script automates that whole process. You can use it by +running: + +.. code-block:: console + + $ make -G Ninja -C <path_to_clang>/cmake/caches/PGO.cmake <source dir> + $ ninja stage2-instrumented-generate-profdata + +If you let that run for a few hours or so, it will place a profdata file in your +build directory. This takes a really long time because it builds clang twice, +and you *must* have compiler-rt in your build tree. + +This process uses any source files under the perf-training directory as training +data as long as the source files are marked up with LIT-style RUN lines. + +After it finishes you can use “find . -name clang.profdata” to find it, but it +should be at a path something like: + +.. code-block:: console + + <build dir>/tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata + +You can feed that file into the LLVM_PROFDATA_FILE option when you build your +optimized compiler. + +The PGO came cache has a slightly different stage naming scheme than other +multi-stage builds. It generates three stages; stage1, stage2-instrumented, and +stage2. Both of the stage2 builds are built using the stage1 compiler. + +The PGO came cache generates the following additional targets: + +**stage2-instrumented** + Builds a stage1 x86 compiler, runtime, and required tools (llvm-config, + llvm-profdata) then uses that compiler to build an instrumented stage2 compiler. + +**stage2-instrumented-generate-profdata** + Depends on “stage2-instrumented” and will use the instrumented compiler to + generate profdata based on the training files in <clang>/utils/perf-training + +**stage2** + Depends of “stage2-instrumented-generate-profdata” and will use the stage1 + compiler with the stage2 profdata to build a PGO-optimized compiler. + +**stage2-check-llvm** + Depends on stage2 and runs check-llvm using the stage2 compiler. + +**stage2-check-clang** + Depends on stage2 and runs check-clang using the stage2 compiler. + +**stage2-check-all** + Depends on stage2 and runs check-all using the stage2 compiler. + +**stage2-test-suite** + Depends on stage2 and runs the test-suite using the stage3 compiler (requires + in-tree test-suite). + +3-Stage Non-Determinism +======================= + +In the ancient lore of compilers non-determinism is like the multi-headed hydra. +Whenever it's head pops up, terror and chaos ensue. + +Historically one of the tests to verify that a compiler was deterministic would +be a three stage build. The idea of a three stage build is you take your sources +and build a compiler (stage1), then use that compiler to rebuild the sources +(stage2), then you use that compiler to rebuild the sources a third time +(stage3) with an identical configuration to the stage2 build. At the end of +this, you have a stage2 and stage3 compiler that should be bit-for-bit +identical. + +You can perform one of these 3-stage builds with LLVM & clang using the +following commands: + +.. code-block:: console + + $ cmake -G Ninja -C <path_to_clang>/cmake/caches/3-stage.cmake <source dir> + $ ninja stage3 + +After the build you can compare the stage2 & stage3 compilers. We have a bot +setup `here <http://lab.llvm.org:8011/builders/clang-3stage-ubuntu>`_ that runs +this build and compare configuration. diff --git a/llvm/docs/CMake.rst b/llvm/docs/CMake.rst index 879e4627e41..34493385207 100644 --- a/llvm/docs/CMake.rst +++ b/llvm/docs/CMake.rst @@ -474,6 +474,41 @@ LLVM-specific variables If you want to build LLVM as a shared library, you should use the ``LLVM_BUILD_LLVM_DYLIB`` option. +CMake Caches +============ + +Recently LLVM and Clang have been adding some more complicated build system +features. Utilizing these new features often involves a complicated chain of +CMake variables passed on the command line. Clang provides a collection of CMake +cache scripts to make these features more approachable. + +CMake cache files are utilized using CMake's -C flag: + +.. code-block:: console + + $ cmake -C <path to cache file> <path to sources> + +CMake cache scripts are processed in an isolated scope, only cached variables +remain set when the main configuration runs. CMake cached variables do not reset +variables that are already set unless the FORCE option is specified. + +A few notes about CMake Caches: + +- Order of command line arguments is important + + - -D arguments specified before -C are set before the cache is processed and + can be read inside the cache file + - -D arguments specified after -C are set after the cache is processed and + are unset inside the cache file + +- All -D arguments will override cache file settings +- CMAKE_TOOLCHAIN_FILE is evaluated after both the cache file and the command + line arguments +- It is recommended that all -D options should be specified *before* -C + +For more information about some of the advanced build configurations supported +via Cache files see :doc:`Advanced_Builds`. + Executing the test suite ======================== diff --git a/llvm/docs/index.rst b/llvm/docs/index.rst index a077155630c..4b6d0466f27 100644 --- a/llvm/docs/index.rst +++ b/llvm/docs/index.rst @@ -65,6 +65,7 @@ representation. :hidden: CMake + Advanced_Builds HowToBuildOnARM HowToCrossCompileLLVM CommandGuide/index |
