Added Dockerfiles to build clang from sources.

Reviewers: klimek, chandlerc, mehdi_amini

Reviewed By: klimek, mehdi_amini

Subscribers: mehdi_amini, jlebar, llvm-commits

Differential Revision: https://reviews.llvm.org/D34197

llvm-svn: 306810

1 files changed, 205 insertions, 0 deletions

diff --git a/llvm/docs/Docker.rst b/llvm/docs/Docker.rst
new file mode 100644
index 00000000000..d873e1ebeeb
--- /dev/null
+++ b/llvm/docs/Docker.rst
@@ -0,0 +1,205 @@
+=========================================
+A guide to Dockerfiles for building LLVM
+=========================================
+
+Introduction
+============
+You can find a number of sources to build docker images with LLVM components in
+``llvm/utils/docker``. They can be used by anyone who wants to build the docker
+images for their own use, or as a starting point for someone who wants to write
+their own Dockerfiles.
+
+We currently provide Dockerfiles with ``debian8`` and ``nvidia-cuda`` base images.
+We also provide an ``example`` image, which contains placeholders that one would need
+to fill out in order to produce Dockerfiles for a new docker image.
+
+Why?
+----
+Docker images provide a way to produce binary distributions of
+software inside a controlled environment. Having Dockerfiles to builds docker images
+inside LLVM repo makes them much more discoverable than putting them into any other
+place.
+
+Docker basics
+-------------
+If you've never heard about Docker before, you might find this section helpful
+to get a very basic explanation of it.
+`Docker <https://www.docker.com/>`_ is a popular solution for running programs in
+an isolated and reproducible environment, especially to maintain releases for
+software deployed to large distributed fleets.
+It uses linux kernel namespaces and cgroups to provide a lightweight isolation
+inside currently running linux kernel.
+A single active instance of dockerized environment is called a *docker
+container*.
+A snapshot of a docker container filesystem is called a *docker image*.
+One can start a container from a prebuilt docker image.
+
+Docker images are built from a so-called *Dockerfile*, a source file written in
+a specialized language that defines instructions to be used when build
+the docker image (see `official
+documentation <https://docs.docker.com/engine/reference/builder/>`_ for more
+details). A minimal Dockerfile typically contains a base image and a number
+of RUN commands that have to be executed to build the image. When building a new
+image, docker will first download your base image, mount its filesystem as
+read-only and then add a writable overlay on top of it to keep track of all
+filesystem modifications, performed while building your image. When the build
+process is finished, a diff between your image's final filesystem state and the
+base image's filesystem is stored in the resulting image.
+
+Overview
+========
+The ``llvm/utils/docker`` folder contains Dockerfiles and simple bash scripts to
+serve as a basis for anyone who wants to create their own Docker image with
+LLVM components, compiled from sources. The sources are checked out from the
+upstream svn repository when building the image.
+
+Inside each subfolder we host Dockerfiles for two images:
+
+- ``build/`` image is used to compile LLVM, it installs a system compiler and all
+  build dependencies of LLVM. After the build process is finished, the build
+  image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
+- ``release/`` image usually only contains LLVM components, compiled by the
+  ``build/`` image, and also libstdc++ and binutils to make image minimally
+  useful for C++ development. The assumption is that you usually want clang to
+  be one of the provided components.
+
+To build both of those images, use ``build_docker_image.sh`` script.
+It will checkout LLVM sources and build clang in the ``build`` container, copy results
+of the build to the local filesystem and then build the ``release`` container using
+those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
+checkout, and arguments for CMake invocation.
+
+If you want to write your own docker image, start with an ``example/`` subfolder.
+It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
+you need to take in order to make your Dockerfiles functional.
+
+Usage
+=====
+The ``llvm/utils/build_docker_image.sh`` script provides a rather high degree of
+control on how to run the build. It allows you to specify the projects to
+checkout from svn and provide a list of CMake arguments to use during when
+building LLVM inside docker container.
+
+Here's a very simple example of getting a docker image with clang binary,
+compiled by the system compiler in the debian8 image:
+
+.. code-block:: bash
+
+    ./llvm/utils/docker/build_docker_image.sh \
+	--source debian8 \
+	--docker-repository clang-debian8 --docker-tag "staging" \
+	-- \
+	-p clang -i install-clang -i install-clang-headers \
+	-- \
+	-DCMAKE_BUILD_TYPE=Release
+
+Note there are two levels of ``--`` indirection. First one separates
+``build_docker_image.sh`` arguments from ``llvm/utils/build_install_llvm.sh``
+arguments. Second one separates CMake arguments from ``build_install_llvm.sh``
+arguments. Note that build like that doesn't use a 2-stage build process that
+you probably want for clang. Running a 2-stage build is a little more intricate,
+this command will do that:
+
+.. code-block:: bash
+
+    # Run a 2-stage build.
+    #   LLVM_TARGETS_TO_BUILD=Native is to reduce stage1 compile time.
+    #   Options, starting with BOOTSTRAP_* are passed to stage2 cmake invocation.
+    ./build_docker_image.sh \
+	--source debian8 \
+	--docker-repository clang-debian8 --docker-tag "staging" \
+	-- \
+	-p clang -i stage2-install-clang -i stage2-install-clang-headers \
+	-- \
+	-DLLVM_TARGETS_TO_BUILD=Native -DCMAKE_BUILD_TYPE=Release \
+	-DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
+	-DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
+	
+This will produce two images, a release image ``clang-debian8:staging`` and a
+build image ``clang-debian8-build:staging`` from the latest upstream revision.
+After the image is built you can run bash inside a container based on your
+image like this:
+
+.. code-block:: bash
+
+    docker run -ti clang-debian8:staging bash
+
+Now you can run bash commands as you normally would:
+
+.. code-block:: bash
+
+    root@80f351b51825:/# clang -v
+    clang version 5.0.0 (trunk 305064)
+    Target: x86_64-unknown-linux-gnu
+    Thread model: posix
+    InstalledDir: /bin
+    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
+    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
+    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
+    Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.2
+    Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
+    Candidate multilib: .;@m64
+    Selected multilib: .;@m64
+
+
+Which image should I choose?
+============================
+We currently provide two images: debian8-based and nvidia-cuda-based. They
+differ in the base image that they use, i.e. they have a different set of
+preinstalled binaries. Debian8 is very minimal, nvidia-cuda is larger, but has
+preinstalled CUDA libraries and allows to access a GPU, installed on your
+machine.
+
+If you need a minimal linux distribution with only clang and libstdc++ included,
+you should try debian8-based image.
+
+If you want to use CUDA libraries and have access to a GPU on your machine,
+you should choose nvidia-cuda-based image and use `nvidia-docker
+<https://github.com/NVIDIA/nvidia-docker>`_ to run your docker containers. Note
+that you don't need nvidia-docker to build the images, but you need it in order
+to have an access to GPU from a docker container that is running the built
+image.
+
+If you have a different use-case, you could create your own image based on
+``example/`` folder.
+
+Any docker image can be built and run using only the docker binary, i.e. you can
+run debian8 build on Fedora or any other Linux distribution. You don't need to
+install CMake, compilers or any other clang dependencies. It is all handled
+during the build process inside Docker's isolated environment.
+
+Stable build
+============
+If you want a somewhat recent and somewhat stable build, use the
+``branches/google/stable`` branch, i.e. the following command will produce a
+debian8-based image using the latest ``google/stable`` sources for you:
+
+.. code-block:: bash
+
+    ./llvm/utils/docker/build_docker_image.sh \
+	-s debian8 --d clang-debian8 -t "staging" \
+	-- \
+	--branch branches/google/stable \
+	-p clang -i install-clang -i install-clang-headers \
+	-- \
+	-DCMAKE_BUILD_TYPE=Release
+
+
+Minimizing docker image size
+============================
+Due to Docker restrictions we use two images (i.e., build and release folders)
+for the release image to be as small as possible. It's much easier to achieve
+that using two images, because Docker would store a filesystem layer for each
+command in the  Dockerfile, i.e. if you install some packages in one command,
+then remove  those in a separate command, the size of the resulting image will
+still be proportinal to the size of an image with installed packages.
+Therefore, we strive to provide a very simple release image which only copies
+compiled clang and does not do anything else.
+
+Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
+image, i.e. remove the parts that were actually deleted. That is an easier way
+to produce the smallest images possible by using just a single image. We do not
+use it because as of today the flag is in experimental stage and not everyone
+may have the latest docker version available. When the flag is out of
+experimental stage, we should investigate replacing two images approach with
+just a single image, built using ``--squash`` flag.