diff options
author | Patrick Williams <patrick@stwcx.xyz> | 2016-08-17 14:31:25 -0500 |
---|---|---|
committer | Patrick Williams <patrick@stwcx.xyz> | 2016-08-22 16:43:26 +0000 |
commit | 60f9d69e016b11c468c98ea75ba0a60c44afbbc4 (patch) | |
tree | ecb49581a9e41a37943c22cd9ef3f63451b20ee7 /import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml | |
parent | e18c61205e0234b03697129c20cc69c9b3940efc (diff) | |
download | blackbird-openbmc-60f9d69e016b11c468c98ea75ba0a60c44afbbc4.tar.gz blackbird-openbmc-60f9d69e016b11c468c98ea75ba0a60c44afbbc4.zip |
yocto-poky: Move to import-layers subdir
We are going to import additional layers, so create a subdir to
hold all of the layers that we import with git-subtree.
Change-Id: I6f732153a22be8ca663035c518837e3cc5ec0799
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Diffstat (limited to 'import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml')
-rw-r--r-- | import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml | 472 |
1 files changed, 472 insertions, 0 deletions
diff --git a/import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml b/import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml new file mode 100644 index 000000000..1de114826 --- /dev/null +++ b/import-layers/yocto-poky/documentation/ref-manual/ref-bitbake.xml @@ -0,0 +1,472 @@ +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" +"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" +[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > + +<chapter id='ref-bitbake'> + + <title>BitBake</title> + + <para> + BitBake is a program written in Python that interprets the + <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> used by + the OpenEmbedded build system. + At some point, developers wonder what actually happens when you enter: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para> + + <para> + This chapter provides an overview of what happens behind the scenes from BitBake's perspective. + </para> + + <note> + BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships. + As such, it has no real knowledge of what the tasks being executed actually do. + BitBake just considers a list of tasks with dependencies and handles + <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> + consisting of variables in a certain format that get passed to the tasks. + </note> + + <section id='ref-bitbake-parsing'> + <title>Parsing</title> + + <para> + BitBake parses configuration files, classes, and <filename>.bb</filename> files. + </para> + + <para> + The first thing BitBake does is look for the <filename>bitbake.conf</filename> file. + This file resides in the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink> + within the <filename>meta/conf/</filename> directory. + BitBake finds it by examining its + <link linkend='var-BBPATH'><filename>BBPATH</filename></link> environment + variable and looking for the <filename>meta/conf/</filename> + directory. + </para> + + <para> + The <filename>bitbake.conf</filename> file lists other configuration + files to include from a <filename>conf/</filename> + directory below the directories listed in <filename>BBPATH</filename>. + In general, the most important configuration file from a user's perspective + is <filename>local.conf</filename>, which contains a user's customized + settings for the OpenEmbedded build environment. + Other notable configuration files are the distribution + configuration file (set by the + <filename><link linkend='var-DISTRO'>DISTRO</link></filename> variable) + and the machine configuration file + (set by the + <filename><link linkend='var-MACHINE'>MACHINE</link></filename> variable). + The <filename>DISTRO</filename> and <filename>MACHINE</filename> BitBake environment + variables are both usually set in + the <filename>local.conf</filename> file. + Valid distribution + configuration files are available in the <filename>meta/conf/distro/</filename> directory + and valid machine configuration + files in the <filename>meta/conf/machine/</filename> directory. + Within the <filename>meta/conf/machine/include/</filename> + directory are various <filename>tune-*.inc</filename> configuration files that provide common + "tuning" settings specific to and shared between particular architectures and machines. + </para> + + <para> + After the parsing of the configuration files, some standard classes are included. + The <filename>base.bbclass</filename> file is always included. + Other classes that are specified in the configuration using the + <filename><link linkend='var-INHERIT'>INHERIT</link></filename> + variable are also included. + Class files are searched for in a <filename>classes</filename> subdirectory + under the paths in <filename>BBPATH</filename> in the same way as + configuration files. + </para> + + <para> + After classes are included, the variable + <filename><link linkend='var-BBFILES'>BBFILES</link></filename> + is set, usually in + <filename>local.conf</filename>, and defines the list of places to search for + <filename>.bb</filename> files. + By default, the <filename>BBFILES</filename> variable specifies the + <filename>meta/recipes-*/</filename> directory within Poky. + Adding extra content to <filename>BBFILES</filename> is best achieved through the use of + BitBake layers as described in the + "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and + Creating Layers</ulink>" section of the Yocto Project Development Manual. + </para> + + <para> + BitBake parses each <filename>.bb</filename> file in <filename>BBFILES</filename> and + stores the values of various variables. + In summary, for each <filename>.bb</filename> + file the configuration plus the base class of variables are set, followed + by the data in the <filename>.bb</filename> file + itself, followed by any inherit commands that + <filename>.bb</filename> file might contain. + </para> + + <para> + Because parsing <filename>.bb</filename> files is a time + consuming process, a cache is kept to speed up subsequent parsing. + This cache is invalid if the timestamp of the <filename>.bb</filename> + file itself changes, or if the timestamps of any of the include, + configuration files or class files on which the + <filename>.bb</filename> file depends change. + </para> + + <note> + <para> + You need to be aware of how BitBake parses curly braces. + If a recipe uses a closing curly brace within the function and + the character has no leading spaces, BitBake produces a parsing + error. + If you use a pair of curly brace in a shell function, the + closing curly brace must not be located at the start of the line + without leading spaces. + </para> + + <para> + Here is an example that causes BitBake to produce a parsing + error: + <literallayout class='monospaced'> + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ###### The following "}" at the start of the line causes a parsing error ###### + } + EOF + } + </literallayout> + Writing the recipe this way avoids the error: + <literallayout class='monospaced'> + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ######The following "}" with a leading space at the start of the line avoids the error ###### + } + EOF + } + </literallayout> + </para> + </note> + </section> + + <section id='ref-bitbake-providers'> + <title>Preferences and Providers</title> + + <para> + Once all the <filename>.bb</filename> files have been + parsed, BitBake starts to build the target (<filename>core-image-sato</filename> + in the previous section's example) and looks for providers of that target. + Once a provider is selected, BitBake resolves all the dependencies for + the target. + In the case of <filename>core-image-sato</filename>, it would lead to + <filename>packagegroup-core-x11-sato</filename>, + which in turn leads to recipes like <filename>matchbox-terminal</filename>, + <filename>pcmanfm</filename> and <filename>gthumb</filename>. + These recipes in turn depend on <filename>glibc</filename> and the toolchain. + </para> + + <para> + Sometimes a target might have multiple providers. + A common example is "virtual/kernel", which is provided by each kernel package. + Each machine often selects the best kernel provider by using a line similar to the + following in the machine configuration file: + </para> + + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" + </literallayout> + + <para> + The default <filename><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></filename> + is the provider with the same name as the target. + </para> + + <para> + Understanding how providers are chosen is made complicated by the fact + that multiple versions might exist. + BitBake defaults to the highest version of a provider. + Version comparisons are made using the same method as Debian. + You can use the + <filename><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></filename> + variable to specify a particular version (usually in the distro configuration). + You can influence the order by using the + <filename><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></filename> + variable. + By default, files have a preference of "0". + Setting the <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the + package unlikely to be used unless it is explicitly referenced. + Setting the <filename>DEFAULT_PREFERENCE</filename> to "1" makes it likely the package is used. + <filename>PREFERRED_VERSION</filename> overrides any <filename>DEFAULT_PREFERENCE</filename> setting. + <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer and more experimental package + versions until they have undergone sufficient testing to be considered stable. + </para> + + <para> + In summary, BitBake has created a list of providers, which is prioritized, for each target. + </para> + </section> + + <section id='ref-bitbake-dependencies'> + <title>Dependencies</title> + + <para> + Each target BitBake builds consists of multiple tasks such as + <filename>fetch</filename>, <filename>unpack</filename>, + <filename>patch</filename>, <filename>configure</filename>, + and <filename>compile</filename>. + For best performance on multi-core systems, BitBake considers each task as an independent + entity with its own set of dependencies. + </para> + + <para> + Dependencies are defined through several variables. + You can find information about variables BitBake uses in the BitBake documentation, + which is found in the <filename>bitbake/doc/manual</filename> directory within the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. + At a basic level, it is sufficient to know that BitBake uses the + <filename><link linkend='var-DEPENDS'>DEPENDS</link></filename> and + <filename><link linkend='var-RDEPENDS'>RDEPENDS</link></filename> variables when + calculating dependencies. + </para> + </section> + + <section id='ref-bitbake-tasklist'> + <title>The Task List</title> + + <para> + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly what tasks it needs to run and in what + order it needs to run them. + The build now starts with BitBake forking off threads up to the limit set in the + <filename><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></filename> variable. + BitBake continues to fork threads as long as there are tasks ready to run, + those tasks have all their dependencies met, and the thread threshold has not been + exceeded. + </para> + + <para> + It is worth noting that you can greatly speed up the build time by properly setting + the <filename>BB_NUMBER_THREADS</filename> variable. + See the + "<ulink url='&YOCTO_DOCS_QS_URL;#qs-building-images'>Building Images</ulink>" + section in the Yocto Project Quick Start for more information. + </para> + + <para> + As each task completes, a timestamp is written to the directory specified by the + <filename><link linkend='var-STAMP'>STAMP</link></filename> variable. + On subsequent runs, BitBake looks within the <filename>build/tmp/stamps</filename> + directory and does not rerun + tasks that are already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per + <filename>.bb</filename> file basis. + So, for example, if the configure stamp has a timestamp greater than the + compile timestamp for a given target, then the compile task would rerun. + Running the compile task again, however, has no effect on other providers + that depend on that target. + This behavior could change or become configurable in future versions of BitBake. + </para> + + <note> + Some tasks are marked as "nostamp" tasks. + No timestamp file is created when these tasks are run. + Consequently, "nostamp" tasks are always rerun. + </note> + </section> + + <section id='ref-bitbake-runtask'> + <title>Running a Task</title> + + <para> + Tasks can either be a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script. + The generated shell script contains all the exported variables, and the shell functions + with all variables expanded. + Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>. + Looking at the expanded shell functions in the run file and the output in the log files + is a useful debugging technique. + </para> + + <para> + For Python tasks, BitBake executes the task internally and logs information to the + controlling terminal. + Future versions of BitBake will write the functions to files similar to the way + shell tasks are handled. + Logging will be handled in a way similar to shell tasks as well. + </para> + + <para> + Once all the tasks have been completed BitBake exits. + </para> + + <para> + When running a task, BitBake tightly controls the execution environment + of the build tasks to make sure unwanted contamination from the build machine + cannot influence the build. + Consequently, if you do want something to get passed into the build + task's environment, you must take a few steps: + <orderedlist> + <listitem><para>Tell BitBake to load what you want from the environment + into the data store. + You can do so through the <filename>BB_ENV_EXTRAWHITE</filename> + variable. + For example, assume you want to prevent the build system from + accessing your <filename>$HOME/.ccache</filename> directory. + The following command tells BitBake to load + <filename>CCACHE_DIR</filename> from the environment into the data + store: + <literallayout class='monospaced'> + export BB_ENV_EXTRAWHITE="$BB_ENV_EXTRAWHITE CCACHE_DIR" + </literallayout></para></listitem> + <listitem><para>Tell BitBake to export what you have loaded into the + environment store to the task environment of every running task. + Loading something from the environment into the data store + (previous step) only makes it available in the datastore. + To export it to the task environment of every running task, + use a command similar to the following in your + <filename>local.conf</filename> or distro configuration file: + <literallayout class='monospaced'> + export CCACHE_DIR + </literallayout></para></listitem> + </orderedlist> + </para> + + <note> + A side effect of the previous steps is that BitBake records the variable + as a dependency of the build process in things like the shared state + checksums. + If doing so results in unnecessary rebuilds of tasks, you can whitelist the + variable so that the shared state code ignores the dependency when it creates + checksums. + For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename> + example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section. + </note> + </section> + + <section id='ref-bitbake-commandline'> + <title>BitBake Command Line</title> + + <para> + Following is the BitBake help output: + </para> + + <screen> +$ bitbake --help +Usage: bitbake [options] [recipename/target ...] + + Executes the specified task (default is 'build') for a given set of target recipes (.bb files). + It is assumed there is a conf/bblayers.conf available in cwd or in BBPATH which + will provide the layer, BBFILES and other configuration information. + +Options: + --version show program's version number and exit + -h, --help show this help message and exit + -b BUILDFILE, --buildfile=BUILDFILE + Execute tasks from a specific .bb recipe directly. + WARNING: Does not handle any dependencies from other + recipes. + -k, --continue Continue as much as possible after an error. While the + target that failed and anything depending on it cannot + be built, as much as possible will be built before + stopping. + -a, --tryaltconfigs Continue with builds by trying to use alternative + providers where possible. + -f, --force Force the specified targets/task to run (invalidating + any existing stamp file). + -c CMD, --cmd=CMD Specify the task to execute. The exact options + available depend on the metadata. Some examples might + be 'compile' or 'populate_sysroot' or 'listtasks' may + give a list of the tasks available. + -C INVALIDATE_STAMP, --clear-stamp=INVALIDATE_STAMP + Invalidate the stamp for the specified task such as + 'compile' and then run the default task for the + specified target(s). + -r PREFILE, --read=PREFILE + Read the specified file before bitbake.conf. + -R POSTFILE, --postread=POSTFILE + Read the specified file after bitbake.conf. + -v, --verbose Output more log message data to the terminal. + -D, --debug Increase the debug level. You can specify this more + than once. + -n, --dry-run Don't execute, just go through the motions. + -S, --dump-signatures + Don't execute, just dump out the signature + construction information. + -p, --parse-only Quit after parsing the BB recipes. + -s, --show-versions Show current and preferred versions of all recipes. + -e, --environment Show the global or per-package environment complete + with information about where variables were + set/changed. + -g, --graphviz Save dependency tree information for the specified + targets in the dot syntax. + -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED + Assume these dependencies don't exist and are already + provided (equivalent to ASSUME_PROVIDED). Useful to + make dependency graphs more appealing + -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS + Show debug logging for the specified logging domains + -P, --profile Profile the command and save reports. + -u UI, --ui=UI The user interface to use (e.g. knotty and depexp). + -t SERVERTYPE, --servertype=SERVERTYPE + Choose which server to use, process or xmlrpc. + --revisions-changed Set the exit code depending on whether upstream + floating revisions have changed or not. + --server-only Run bitbake without a UI, only starting a server + (cooker) process. + -B BIND, --bind=BIND The name/address for the bitbake server to bind to. + --no-setscene Do not run any setscene tasks. sstate will be ignored + and everything needed, built. + --remote-server=REMOTE_SERVER + Connect to the specified server. + -m, --kill-server Terminate the remote server. + --observe-only Connect to a server as an observing-only client. + </screen> + </section> + + <section id='ref-bitbake-fetchers'> + <title>Fetchers</title> + + <para> + BitBake also contains a set of "fetcher" modules that allow + retrieval of source code from various types of sources. + For example, BitBake can get source code from a disk with the metadata, from websites, + from remote shell accounts, or from Source Code Management (SCM) systems + like <filename>cvs/subversion/git</filename>. + </para> + + <para> + Fetchers are usually triggered by entries in + <filename><link linkend='var-SRC_URI'>SRC_URI</link></filename>. + You can find information about the options and formats of entries for specific + fetchers in the BitBake manual located in the + <filename>bitbake/doc/manual</filename> directory of the + <ulink url='&YOCTO_DOCS_DEV_URL;#source-directory'>Source Directory</ulink>. + </para> + + <para> + One useful feature for certain Source Code Manager (SCM) fetchers is the ability to + "auto-update" when the upstream SCM changes version. + Since this ability requires certain functionality from the SCM, not all + systems support it. + Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update". + This feature works using the <filename><link linkend='var-SRCREV'>SRCREV</link></filename> + variable. + See the + "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-appdev-srcrev'>Using an External SCM</ulink>" section + in the Yocto Project Development Manual for more information. + </para> + + </section> + +</chapter> +<!-- +vim: expandtab tw=80 ts=4 spell spelllang=en_gb +--> |