diff options
Diffstat (limited to 'import-layers/yocto-poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml')
| -rw-r--r-- | import-layers/yocto-poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml | 931 |
1 files changed, 931 insertions, 0 deletions
diff --git a/import-layers/yocto-poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/import-layers/yocto-poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml new file mode 100644 index 000000000..b1b72e0aa --- /dev/null +++ b/import-layers/yocto-poky/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml @@ -0,0 +1,931 @@ +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" +"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"> + +<chapter id="bitbake-user-manual-execution"> + <title>Execution</title> + + <para> + The primary purpose for running BitBake is to produce some kind + of output such as a single installable package, a kernel, a software + development kit, or even a full, board-specific bootable Linux image, + complete with bootloader, kernel, and root filesystem. + Of course, you can execute the <filename>bitbake</filename> + command with options that cause it to execute single tasks, + compile single recipe files, capture or clear data, or simply + return information about the execution environment. + </para> + + <para> + This chapter describes BitBake's execution process from start + to finish when you use it to create an image. + The execution process is launched using the following command + form: + <literallayout class='monospaced'> + $ bitbake <replaceable>target</replaceable> + </literallayout> + For information on the BitBake command and its options, + see + "<link linkend='bitbake-user-manual-command'>The BitBake Command</link>" + section. + <note> + <para> + Prior to executing BitBake, you should take advantage of available + parallel thread execution on your build host by setting the + <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> + variable in your project's <filename>local.conf</filename> + configuration file. + </para> + + <para> + A common method to determine this value for your build host is to run + the following: + <literallayout class='monospaced'> + $ grep processor /proc/cpuinfo + </literallayout> + This command returns the number of processors, which takes into + account hyper-threading. + Thus, a quad-core build host with hyper-threading most likely + shows eight processors, which is the value you would then assign to + <filename>BB_NUMBER_THREADS</filename>. + </para> + + <para> + A possibly simpler solution is that some Linux distributions + (e.g. Debian and Ubuntu) provide the <filename>ncpus</filename> command. + </para> + </note> + </para> + + <section id='parsing-the-base-configuration-metadata'> + <title>Parsing the Base Configuration Metadata</title> + + <para> + The first thing BitBake does is parse base configuration + metadata. + Base configuration metadata consists of your project's + <filename>bblayers.conf</filename> file to determine what + layers BitBake needs to recognize, all necessary + <filename>layer.conf</filename> files (one from each layer), + and <filename>bitbake.conf</filename>. + The data itself is of various types: + <itemizedlist> + <listitem><para><emphasis>Recipes:</emphasis> + Details about particular pieces of software. + </para></listitem> + <listitem><para><emphasis>Class Data:</emphasis> + An abstraction of common build information + (e.g. how to build a Linux kernel). + </para></listitem> + <listitem><para><emphasis>Configuration Data:</emphasis> + Machine-specific settings, policy decisions, + and so forth. + Configuration data acts as the glue to bind everything + together.</para></listitem> + </itemizedlist> + </para> + + <para> + The <filename>layer.conf</filename> files are used to + construct key variables such as + <link linkend='var-BBPATH'><filename>BBPATH</filename></link> + and + <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. + <filename>BBPATH</filename> is used to search for + configuration and class files under the + <filename>conf</filename> and <filename>classes</filename> + directories, respectively. + <filename>BBFILES</filename> is used to locate both recipe + and recipe append files + (<filename>.bb</filename> and <filename>.bbappend</filename>). + If there is no <filename>bblayers.conf</filename> file, + it is assumed the user has set the <filename>BBPATH</filename> + and <filename>BBFILES</filename> directly in the environment. + </para> + + <para> + Next, the <filename>bitbake.conf</filename> file is located + using the <filename>BBPATH</filename> variable that was + just constructed. + The <filename>bitbake.conf</filename> file may also include other + configuration files using the + <filename>include</filename> or + <filename>require</filename> directives. + </para> + + <para> + Prior to parsing configuration files, Bitbake looks + at certain variables, including: + <itemizedlist> + <listitem><para> + <link linkend='var-BB_ENV_WHITELIST'><filename>BB_ENV_WHITELIST</filename></link> + </para></listitem> + <listitem><para> + <link linkend='var-BB_ENV_EXTRAWHITE'><filename>BB_ENV_EXTRAWHITE</filename></link> + </para></listitem> + <listitem><para> + <link linkend='var-BB_PRESERVE_ENV'><filename>BB_PRESERVE_ENV</filename></link> + </para></listitem> + <listitem><para> + <link linkend='var-BB_ORIGENV'><filename>BB_ORIGENV</filename></link> + </para></listitem> + <listitem><para> + <link linkend='var-BITBAKE_UI'><filename>BITBAKE_UI</filename></link> + </para></listitem> + </itemizedlist> + The first four variables in this list relate to how BitBake treats shell + environment variables during task execution. + By default, BitBake cleans the environment variables and provides tight + control over the shell execution environment. + However, through the use of these first four variables, you can + apply your control regarding the + environment variables allowed to be used by BitBake in the shell + during execution of tasks. + See the + "<link linkend='passing-information-into-the-build-task-environment'>Passing Information Into the Build Task Environment</link>" + section and the information about these variables in the + variable glossary for more information on how they work and + on how to use them. + </para> + + <para> + The base configuration metadata is global + and therefore affects all recipes and tasks that are executed. + </para> + + <para> + BitBake first searches the current working directory for an + optional <filename>conf/bblayers.conf</filename> configuration file. + This file is expected to contain a + <link linkend='var-BBLAYERS'><filename>BBLAYERS</filename></link> + variable that is a space-delimited list of 'layer' directories. + Recall that if BitBake cannot find a <filename>bblayers.conf</filename> + file, then it is assumed the user has set the <filename>BBPATH</filename> + and <filename>BBFILES</filename> variables directly in the environment. + </para> + + <para> + For each directory (layer) in this list, a <filename>conf/layer.conf</filename> + file is located and parsed with the + <link linkend='var-LAYERDIR'><filename>LAYERDIR</filename></link> + variable being set to the directory where the layer was found. + The idea is these files automatically set up + <link linkend='var-BBPATH'><filename>BBPATH</filename></link> + and other variables correctly for a given build directory. + </para> + + <para> + BitBake then expects to find the <filename>conf/bitbake.conf</filename> + file somewhere in the user-specified <filename>BBPATH</filename>. + That configuration file generally has include directives to pull + in any other metadata such as files specific to the architecture, + the machine, the local environment, and so forth. + </para> + + <para> + Only variable definitions and include directives are allowed + in BitBake <filename>.conf</filename> files. + Some variables directly influence BitBake's behavior. + These variables might have been set from the environment + depending on the environment variables previously + mentioned or set in the configuration files. + The + "<link linkend='ref-variables-glos'>Variables Glossary</link>" + chapter presents a full list of variables. + </para> + + <para> + After parsing configuration files, BitBake uses its rudimentary + inheritance mechanism, which is through class files, to inherit + some standard classes. + BitBake parses a class when the inherit directive responsible + for getting that class is encountered. + </para> + + <para> + The <filename>base.bbclass</filename> file is always included. + Other classes that are specified in the configuration using the + <link linkend='var-INHERIT'><filename>INHERIT</filename></link> + variable are also included. + BitBake searches for class files in a + <filename>classes</filename> subdirectory under + the paths in <filename>BBPATH</filename> in the same way as + configuration files. + </para> + + <para> + A good way to get an idea of the configuration files and + the class files used in your execution environment is to + run the following BitBake command: + <literallayout class='monospaced'> + $ bitbake -e > mybb.log + </literallayout> + Examining the top of the <filename>mybb.log</filename> + shows you the many configuration files and class files + used in your execution environment. + </para> + + <note> + <para> + You need to be aware of how BitBake parses curly braces. + If a recipe uses a closing curly brace within the function and + the character has no leading spaces, BitBake produces a parsing + error. + If you use a pair of curly braces in a shell function, the + closing curly brace must not be located at the start of the line + without leading spaces. + </para> + + <para> + Here is an example that causes BitBake to produce a parsing + error: + <literallayout class='monospaced'> + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ###### The following "}" at the start of the line causes a parsing error ###### + } + EOF + } + </literallayout> + Writing the recipe this way avoids the error: + <literallayout class='monospaced'> + fakeroot create_shar() { + cat << "EOF" > ${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.sh + usage() + { + echo "test" + ######The following "}" with a leading space at the start of the line avoids the error ###### + } + EOF + } + </literallayout> + </para> + </note> + </section> + + <section id='locating-and-parsing-recipes'> + <title>Locating and Parsing Recipes</title> + + <para> + During the configuration phase, BitBake will have set + <link linkend='var-BBFILES'><filename>BBFILES</filename></link>. + BitBake now uses it to construct a list of recipes to parse, + along with any append files (<filename>.bbappend</filename>) + to apply. + <filename>BBFILES</filename> is a space-separated list of + available files and supports wildcards. + An example would be: + <literallayout class='monospaced'> + BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" + </literallayout> + BitBake parses each recipe and append file located + with <filename>BBFILES</filename> and stores the values of + various variables into the datastore. + <note> + Append files are applied in the order they are encountered in + <filename>BBFILES</filename>. + </note> + For each file, a fresh copy of the base configuration is + made, then the recipe is parsed line by line. + Any inherit statements cause BitBake to find and + then parse class files (<filename>.bbclass</filename>) + using + <link linkend='var-BBPATH'><filename>BBPATH</filename></link> + as the search path. + Finally, BitBake parses in order any append files found in + <filename>BBFILES</filename>. + </para> + + <para> + One common convention is to use the recipe filename to define + pieces of metadata. + For example, in <filename>bitbake.conf</filename> the recipe + name and version are used to set the variables + <link linkend='var-PN'><filename>PN</filename></link> and + <link linkend='var-PV'><filename>PV</filename></link>: + <literallayout class='monospaced'> + PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" + PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" + </literallayout> + In this example, a recipe called "something_1.2.3.bb" would set + <filename>PN</filename> to "something" and + <filename>PV</filename> to "1.2.3". + </para> + + <para> + By the time parsing is complete for a recipe, BitBake + has a list of tasks that the recipe defines and a set of + data consisting of keys and values as well as + dependency information about the tasks. + </para> + + <para> + BitBake does not need all of this information. + It only needs a small subset of the information to make + decisions about the recipe. + Consequently, BitBake caches the values in which it is + interested and does not store the rest of the information. + Experience has shown it is faster to re-parse the metadata than to + try and write it out to the disk and then reload it. + </para> + + <para> + Where possible, subsequent BitBake commands reuse this cache of + recipe information. + The validity of this cache is determined by first computing a + checksum of the base configuration data (see + <link linkend='var-BB_HASHCONFIG_WHITELIST'><filename>BB_HASHCONFIG_WHITELIST</filename></link>) + and then checking if the checksum matches. + If that checksum matches what is in the cache and the recipe + and class files have not changed, Bitbake is able to use + the cache. + BitBake then reloads the cached information about the recipe + instead of reparsing it from scratch. + </para> + + <para> + Recipe file collections exist to allow the user to + have multiple repositories of + <filename>.bb</filename> files that contain the same + exact package. + For example, one could easily use them to make one's + own local copy of an upstream repository, but with + custom modifications that one does not want upstream. + Here is an example: + <literallayout class='monospaced'> + BBFILES = "/stuff/openembedded/*/*.bb /stuff/openembedded.modified/*/*.bb" + BBFILE_COLLECTIONS = "upstream local" + BBFILE_PATTERN_upstream = "^/stuff/openembedded/" + BBFILE_PATTERN_local = "^/stuff/openembedded.modified/" + BBFILE_PRIORITY_upstream = "5" + BBFILE_PRIORITY_local = "10" + </literallayout> + <note> + The layers mechanism is now the preferred method of collecting + code. + While the collections code remains, its main use is to set layer + priorities and to deal with overlap (conflicts) between layers. + </note> + </para> + </section> + + <section id='bb-bitbake-providers'> + <title>Providers</title> + + <para> + Assuming BitBake has been instructed to execute a target + and that all the recipe files have been parsed, BitBake + starts to figure out how to build the target. + BitBake looks through the <filename>PROVIDES</filename> list + for each of the recipes. + A <filename>PROVIDES</filename> list is the list of names by which + the recipe can be known. + Each recipe's <filename>PROVIDES</filename> list is created + implicitly through the recipe's + <link linkend='var-PN'><filename>PN</filename></link> variable + and explicitly through the recipe's + <link linkend='var-PROVIDES'><filename>PROVIDES</filename></link> + variable, which is optional. + </para> + + <para> + When a recipe uses <filename>PROVIDES</filename>, that recipe's + functionality can be found under an alternative name or names other + than the implicit <filename>PN</filename> name. + As an example, suppose a recipe named <filename>keyboard_1.0.bb</filename> + contained the following: + <literallayout class='monospaced'> + PROVIDES += "fullkeyboard" + </literallayout> + The <filename>PROVIDES</filename> list for this recipe becomes + "keyboard", which is implicit, and "fullkeyboard", which is explicit. + Consequently, the functionality found in + <filename>keyboard_1.0.bb</filename> can be found under two + different names. + </para> + </section> + + <section id='bb-bitbake-preferences'> + <title>Preferences</title> + + <para> + The <filename>PROVIDES</filename> list is only part of the solution + for figuring out a target's recipes. + Because targets might have multiple providers, BitBake needs + to prioritize providers by determining provider preferences. + </para> + + <para> + A common example in which a target has multiple providers + is "virtual/kernel", which is on the + <filename>PROVIDES</filename> list for each kernel recipe. + Each machine often selects the best kernel provider by using a + line similar to the following in the machine configuration file: + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel = "linux-yocto" + </literallayout> + The default + <link linkend='var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></link> + is the provider with the same name as the target. + Bitbake iterates through each target it needs to build and + resolves them and their dependencies using this process. + </para> + + <para> + Understanding how providers are chosen is made complicated by the fact + that multiple versions might exist for a given provider. + BitBake defaults to the highest version of a provider. + Version comparisons are made using the same method as Debian. + You can use the + <link linkend='var-PREFERRED_VERSION'><filename>PREFERRED_VERSION</filename></link> + variable to specify a particular version. + You can influence the order by using the + <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> + variable. + </para> + + <para> + By default, files have a preference of "0". + Setting <filename>DEFAULT_PREFERENCE</filename> to "-1" makes the + recipe unlikely to be used unless it is explicitly referenced. + Setting <filename>DEFAULT_PREFERENCE</filename> to "1" makes it + likely the recipe is used. + <filename>PREFERRED_VERSION</filename> overrides any + <filename>DEFAULT_PREFERENCE</filename> setting. + <filename>DEFAULT_PREFERENCE</filename> is often used to mark newer + and more experimental recipe versions until they have undergone + sufficient testing to be considered stable. + </para> + + <para> + When there are multiple “versions” of a given recipe, + BitBake defaults to selecting the most recent + version, unless otherwise specified. + If the recipe in question has a + <link linkend='var-DEFAULT_PREFERENCE'><filename>DEFAULT_PREFERENCE</filename></link> + set lower than the other recipes (default is 0), then + it will not be selected. + This allows the person or persons maintaining + the repository of recipe files to specify + their preference for the default selected version. + Additionally, the user can specify their preferred version. + </para> + + <para> + If the first recipe is named <filename>a_1.1.bb</filename>, then the + <link linkend='var-PN'><filename>PN</filename></link> variable + will be set to “a”, and the + <link linkend='var-PV'><filename>PV</filename></link> + variable will be set to 1.1. + </para> + + <para> + Thus, if a recipe named <filename>a_1.2.bb</filename> exists, BitBake + will choose 1.2 by default. + However, if you define the following variable in a + <filename>.conf</filename> file that BitBake parses, you + can change that preference: + <literallayout class='monospaced'> + PREFERRED_VERSION_a = "1.1" + </literallayout> + </para> + + <note> + <para> + It is common for a recipe to provide two versions -- a stable, + numbered (and preferred) version, and a version that is + automatically checked out from a source code repository that + is considered more "bleeding edge" but can be selected only + explicitly. + </para> + + <para> + For example, in the OpenEmbedded codebase, there is a standard, + versioned recipe file for BusyBox, + <filename>busybox_1.22.1.bb</filename>, + but there is also a Git-based version, + <filename>busybox_git.bb</filename>, which explicitly contains the line + <literallayout class='monospaced'> + DEFAULT_PREFERENCE = "-1" + </literallayout> + to ensure that the numbered, stable version is always preferred + unless the developer selects otherwise. + </para> + </note> + </section> + + <section id='bb-bitbake-dependencies'> + <title>Dependencies</title> + + <para> + Each target BitBake builds consists of multiple tasks such as + <filename>fetch</filename>, <filename>unpack</filename>, + <filename>patch</filename>, <filename>configure</filename>, + and <filename>compile</filename>. + For best performance on multi-core systems, BitBake considers each + task as an independent + entity with its own set of dependencies. + </para> + + <para> + Dependencies are defined through several variables. + You can find information about variables BitBake uses in + the <link linkend='ref-variables-glos'>Variables Glossary</link> + near the end of this manual. + At a basic level, it is sufficient to know that BitBake uses the + <link linkend='var-DEPENDS'><filename>DEPENDS</filename></link> and + <link linkend='var-RDEPENDS'><filename>RDEPENDS</filename></link> variables when + calculating dependencies. + </para> + + <para> + For more information on how BitBake handles dependencies, see the + "<link linkend='dependencies'>Dependencies</link>" section. + </para> + </section> + + <section id='ref-bitbake-tasklist'> + <title>The Task List</title> + + <para> + Based on the generated list of providers and the dependency information, + BitBake can now calculate exactly what tasks it needs to run and in what + order it needs to run them. + The + "<link linkend='executing-tasks'>Executing Tasks</link>" section has more + information on how BitBake chooses which task to execute next. + </para> + + <para> + The build now starts with BitBake forking off threads up to the limit set in the + <link linkend='var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></link> + variable. + BitBake continues to fork threads as long as there are tasks ready to run, + those tasks have all their dependencies met, and the thread threshold has not been + exceeded. + </para> + + <para> + It is worth noting that you can greatly speed up the build time by properly setting + the <filename>BB_NUMBER_THREADS</filename> variable. + </para> + + <para> + As each task completes, a timestamp is written to the directory specified by the + <link linkend='var-STAMP'><filename>STAMP</filename></link> variable. + On subsequent runs, BitBake looks in the build directory within + <filename>tmp/stamps</filename> and does not rerun + tasks that are already completed unless a timestamp is found to be invalid. + Currently, invalid timestamps are only considered on a per + recipe file basis. + So, for example, if the configure stamp has a timestamp greater than the + compile timestamp for a given target, then the compile task would rerun. + Running the compile task again, however, has no effect on other providers + that depend on that target. + </para> + + <para> + The exact format of the stamps is partly configurable. + In modern versions of BitBake, a hash is appended to the + stamp so that if the configuration changes, the stamp becomes + invalid and the task is automatically rerun. + This hash, or signature used, is governed by the signature policy + that is configured (see the + "<link linkend='checksums'>Checksums (Signatures)</link>" + section for information). + It is also possible to append extra metadata to the stamp using + the "stamp-extra-info" task flag. + For example, OpenEmbedded uses this flag to make some tasks machine-specific. + </para> + + <note> + Some tasks are marked as "nostamp" tasks. + No timestamp file is created when these tasks are run. + Consequently, "nostamp" tasks are always rerun. + </note> + + <para> + For more information on tasks, see the + "<link linkend='tasks'>Tasks</link>" section. + </para> + </section> + + <section id='executing-tasks'> + <title>Executing Tasks</title> + + <para> + Tasks can be either a shell task or a Python task. + For shell tasks, BitBake writes a shell script to + <filename>${</filename><link linkend='var-T'><filename>T</filename></link><filename>}/run.do_taskname.pid</filename> + and then executes the script. + The generated shell script contains all the exported variables, + and the shell functions with all variables expanded. + Output from the shell script goes to the file + <filename>${T}/log.do_taskname.pid</filename>. + Looking at the expanded shell functions in the run file and + the output in the log files is a useful debugging technique. + </para> + + <para> + For Python tasks, BitBake executes the task internally and logs + information to the controlling terminal. + Future versions of BitBake will write the functions to files + similar to the way shell tasks are handled. + Logging will be handled in a way similar to shell tasks as well. + </para> + + <para> + The order in which BitBake runs the tasks is controlled by its + task scheduler. + It is possible to configure the scheduler and define custom + implementations for specific use cases. + For more information, see these variables that control the + behavior: + <itemizedlist> + <listitem><para> + <link linkend='var-BB_SCHEDULER'><filename>BB_SCHEDULER</filename></link> + </para></listitem> + <listitem><para> + <link linkend='var-BB_SCHEDULERS'><filename>BB_SCHEDULERS</filename></link> + </para></listitem> + </itemizedlist> + It is possible to have functions run before and after a task's main + function. + This is done using the "prefuncs" and "postfuncs" flags of the task + that lists the functions to run. + </para> + </section> + + <section id='checksums'> + <title>Checksums (Signatures)</title> + + <para> + A checksum is a unique signature of a task's inputs. + The signature of a task can be used to determine if a task + needs to be run. + Because it is a change in a task's inputs that triggers running + the task, BitBake needs to detect all the inputs to a given task. + For shell tasks, this turns out to be fairly easy because + BitBake generates a "run" shell script for each task and + it is possible to create a checksum that gives you a good idea of when + the task's data changes. + </para> + + <para> + To complicate the problem, some things should not be included in + the checksum. + First, there is the actual specific build path of a given task - + the working directory. + It does not matter if the working directory changes because it should not + affect the output for target packages. + The simplistic approach for excluding the working directory is to set + it to some fixed value and create the checksum for the "run" script. + BitBake goes one step better and uses the + <link linkend='var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></link> + variable to define a list of variables that should never be included + when generating the signatures. + </para> + + <para> + Another problem results from the "run" scripts containing functions that + might or might not get called. + The incremental build solution contains code that figures out dependencies + between shell functions. + This code is used to prune the "run" scripts down to the minimum set, + thereby alleviating this problem and making the "run" scripts much more + readable as a bonus. + </para> + + <para> + So far we have solutions for shell scripts. + What about Python tasks? + The same approach applies even though these tasks are more difficult. + The process needs to figure out what variables a Python function accesses + and what functions it calls. + Again, the incremental build solution contains code that first figures out + the variable and function dependencies, and then creates a checksum for the data + used as the input to the task. + </para> + + <para> + Like the working directory case, situations exist where dependencies + should be ignored. + For these cases, you can instruct the build process to ignore a dependency + by using a line like the following: + <literallayout class='monospaced'> + PACKAGE_ARCHS[vardepsexclude] = "MACHINE" + </literallayout> + This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not + depend on the value of <filename>MACHINE</filename>, even if it does reference it. + </para> + + <para> + Equally, there are cases where we need to add dependencies BitBake + is not able to find. + You can accomplish this by using a line like the following: + <literallayout class='monospaced'> + PACKAGE_ARCHS[vardeps] = "MACHINE" + </literallayout> + This example explicitly adds the <filename>MACHINE</filename> variable as a + dependency for <filename>PACKAGE_ARCHS</filename>. + </para> + + <para> + Consider a case with in-line Python, for example, where BitBake is not + able to figure out dependencies. + When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake + produces output when it discovers something for which it cannot figure out + dependencies. + </para> + + <para> + Thus far, this section has limited discussion to the direct inputs into a task. + Information based on direct inputs is referred to as the "basehash" in the + code. + However, there is still the question of a task's indirect inputs - the + things that were already built and present in the build directory. + The checksum (or signature) for a particular task needs to add the hashes + of all the tasks on which the particular task depends. + Choosing which dependencies to add is a policy decision. + However, the effect is to generate a master checksum that combines the basehash + and the hashes of the task's dependencies. + </para> + + <para> + At the code level, there are a variety of ways both the basehash and the + dependent task hashes can be influenced. + Within the BitBake configuration file, we can give BitBake some extra information + to help it construct the basehash. + The following statement effectively results in a list of global variable + dependency excludes - variables never included in any checksum. + This example uses variables from OpenEmbedded to help illustrate + the concept: + <literallayout class='monospaced'> + BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ + SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ + USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ + PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ + CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" + </literallayout> + The previous example excludes the work directory, which is part of + <filename>TMPDIR</filename>. + </para> + + <para> + The rules for deciding which hashes of dependent tasks to include through + dependency chains are more complex and are generally accomplished with a + Python function. + The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples + of this and also illustrates how you can insert your own policy into the system + if so desired. + This file defines the two basic signature generators OpenEmbedded Core + uses: "OEBasic" and "OEBasicHash". + By default, there is a dummy "noop" signature handler enabled in BitBake. + This means that behavior is unchanged from previous versions. + <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default + through this setting in the <filename>bitbake.conf</filename> file: + <literallayout class='monospaced'> + BB_SIGNATURE_HANDLER ?= "OEBasicHash" + </literallayout> + The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the + "OEBasic" version but adds the task hash to the stamp files. + This results in any metadata change that changes the task hash, automatically + causing the task to be run again. + This removes the need to bump + <link linkend='var-PR'><filename>PR</filename></link> + values, and changes to metadata automatically ripple across the build. + </para> + + <para> + It is also worth noting that the end result of these signature generators is to + make some dependency and hash information available to the build. + This information includes: + <itemizedlist> + <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: + The base hashes for each task in the recipe. + </para></listitem> + <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: + The base hashes for each dependent task. + </para></listitem> + <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: + The task dependencies for each task. + </para></listitem> + <listitem><para><filename>BB_TASKHASH</filename>: + The hash of the currently running task. + </para></listitem> + </itemizedlist> + </para> + + <para> + It is worth noting that BitBake's "-S" option lets you + debug Bitbake's processing of signatures. + The options passed to -S allow different debugging modes + to be used, either using BitBake's own debug functions + or possibly those defined in the metadata/signature handler + itself. + The simplest parameter to pass is "none", which causes a + set of signature information to be written out into + <filename>STAMP_DIR</filename> + corresponding to the targets specified. + The other currently available parameter is "printdiff", + which causes BitBake to try to establish the closest + signature match it can (e.g. in the sstate cache) and then + run <filename>bitbake-diffsigs</filename> over the matches + to determine the stamps and delta where these two + stamp trees diverge. + <note> + It is likely that future versions of BitBake will + provide other signature handlers triggered through + additional "-S" parameters. + </note> + </para> + + <para> + You can find more information on checksum metadata in the + "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" + section. + </para> + </section> + + <section id='setscene'> + <title>Setscene</title> + + <para> + The setscene process enables BitBake to handle "pre-built" artifacts. + The ability to handle and reuse these artifacts allows BitBake + the luxury of not having to build something from scratch every time. + Instead, BitBake can use, when possible, existing build artifacts. + </para> + + <para> + BitBake needs to have reliable data indicating whether or not an + artifact is compatible. + Signatures, described in the previous section, provide an ideal + way of representing whether an artifact is compatible. + If a signature is the same, an object can be reused. + </para> + + <para> + If an object can be reused, the problem then becomes how to + replace a given task or set of tasks with the pre-built artifact. + BitBake solves the problem with the "setscene" process. + </para> + + <para> + When BitBake is asked to build a given target, before building anything, + it first asks whether cached information is available for any of the + targets it's building, or any of the intermediate targets. + If cached information is available, BitBake uses this information instead of + running the main tasks. + </para> + + <para> + BitBake first calls the function defined by the + <link linkend='var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></link> + variable with a list of tasks and corresponding + hashes it wants to build. + This function is designed to be fast and returns a list + of the tasks for which it believes in can obtain artifacts. + </para> + + <para> + Next, for each of the tasks that were returned as possibilities, + BitBake executes a setscene version of the task that the possible + artifact covers. + Setscene versions of a task have the string "_setscene" appended to the + task name. + So, for example, the task with the name <filename>xxx</filename> has + a setscene task named <filename>xxx_setscene</filename>. + The setscene version of the task executes and provides the necessary + artifacts returning either success or failure. + </para> + + <para> + As previously mentioned, an artifact can cover more than one task. + For example, it is pointless to obtain a compiler if you + already have the compiled binary. + To handle this, BitBake calls the + <link linkend='var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></link> + function for each successful setscene task to know whether or not it needs + to obtain the dependencies of that task. + </para> + + <para> + Finally, after all the setscene tasks have executed, BitBake calls the + function listed in + <link linkend='var-BB_SETSCENE_VERIFY_FUNCTION'><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></link> + with the list of tasks BitBake thinks has been "covered". + The metadata can then ensure that this list is correct and can + inform BitBake that it wants specific tasks to be run regardless + of the setscene result. + </para> + + <para> + You can find more information on setscene metadata in the + "<link linkend='task-checksums-and-setscene'>Task Checksums and Setscene</link>" + section. + </para> + </section> +</chapter> |
