diff options
Diffstat (limited to 'poky/documentation/dev-manual/dev-manual-common-tasks.xml')
| -rw-r--r-- | poky/documentation/dev-manual/dev-manual-common-tasks.xml | 15326 |
1 files changed, 15326 insertions, 0 deletions
diff --git a/poky/documentation/dev-manual/dev-manual-common-tasks.xml b/poky/documentation/dev-manual/dev-manual-common-tasks.xml new file mode 100644 index 000000000..fe1bfba6c --- /dev/null +++ b/poky/documentation/dev-manual/dev-manual-common-tasks.xml @@ -0,0 +1,15326 @@ +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" +"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" +[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > + +<chapter id='extendpoky'> + +<title>Common Tasks</title> + <para> + This chapter describes fundamental procedures such as creating layers, + adding new software packages, extending or customizing images, + porting work to new hardware (adding a new machine), and so forth. + You will find that the procedures documented here occur often in the + development cycle using the Yocto Project. + </para> + + <section id="understanding-and-creating-layers"> + <title>Understanding and Creating Layers</title> + + <para> + The OpenEmbedded build system supports organizing + <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> into + multiple layers. + Layers allow you to isolate different types of customizations from + each other. + For introductory information on the Yocto Project Layer Model, + see the + "<ulink url='&YOCTO_DOCS_OM_URL;#the-yocto-project-layer-model'>The Yocto Project Layer Model</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + + <section id='creating-your-own-layer'> + <title>Creating Your Own Layer</title> + + <para> + It is very easy to create your own layers to use with the + OpenEmbedded build system. + The Yocto Project ships with tools that speed up creating + layers. + This section describes the steps you perform by hand to create + layers so that you can better understand them. + For information about the layer-creation tools, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'>Creating a New BSP Layer Using the <filename>bitbake-layers</filename> Script</ulink>" + section in the Yocto Project Board Support Package (BSP) + Developer's Guide and the + "<link linkend='creating-a-general-layer-using-the-bitbake-layers-script'>Creating a General Layer Using the <filename>bitbake-layers</filename> Script</link>" + section further down in this manual. + </para> + + <para> + Follow these general steps to create your layer without using + tools: + <orderedlist> + <listitem><para> + <emphasis>Check Existing Layers:</emphasis> + Before creating a new layer, you should be sure someone + has not already created a layer containing the Metadata + you need. + You can see the + <ulink url='http://layers.openembedded.org/layerindex/layers/'>OpenEmbedded Metadata Index</ulink> + for a list of layers from the OpenEmbedded community + that can be used in the Yocto Project. + You could find a layer that is identical or close to + what you need. + </para></listitem> + <listitem><para> + <emphasis>Create a Directory:</emphasis> + Create the directory for your layer. + When you create the layer, be sure to create the + directory in an area not associated with the + Yocto Project + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. the cloned <filename>poky</filename> repository). + </para> + + <para>While not strictly required, prepend the name of + the directory with the string "meta-". + For example: + <literallayout class='monospaced'> + meta-mylayer + meta-GUI_xyz + meta-mymachine + </literallayout> + With rare exceptions, a layer's name follows this + form: + <literallayout class='monospaced'> + meta-<replaceable>root_name</replaceable> + </literallayout> + Following this layer naming convention can + save you trouble later when tools, components, or + variables "assume" your layer name begins with "meta-". + A notable example is in configuration files as + shown in the following step where layer names without + the "meta-" string are appended + to several variables used in the configuration. + </para></listitem> + <listitem><para id='dev-layer-config-file-description'> + <emphasis>Create a Layer Configuration File:</emphasis> + Inside your new layer folder, you need to create a + <filename>conf/layer.conf</filename> file. + It is easiest to take an existing layer configuration + file and copy that to your layer's + <filename>conf</filename> directory and then modify the + file as needed.</para> + + <para>The + <filename>meta-yocto-bsp/conf/layer.conf</filename> file + in the Yocto Project + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta-yocto-bsp/conf'>Source Repositories</ulink> + demonstrates the required syntax. + For your layer, you need to replace "yoctobsp" with + a unique identifier for your layer (e.g. "machinexyz" + for a layer named "meta-machinexyz"): + <literallayout class='monospaced'> + # We have a conf and classes directory, add to BBPATH + BBPATH .= ":${LAYERDIR}" + + # We have recipes-* directories, add to BBFILES + BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + + BBFILE_COLLECTIONS += "yoctobsp" + BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/" + BBFILE_PRIORITY_yoctobsp = "5" + LAYERVERSION_yoctobsp = "4" + LAYERSERIES_COMPAT_yoctobsp = "&DISTRO_NAME_NO_CAP;" + </literallayout> + Following is an explanation of the layer configuration + file: + <itemizedlist> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>: + Adds the layer's root directory to BitBake's + search path. + Through the use of the + <filename>BBPATH</filename> variable, BitBake + locates class files + (<filename>.bbclass</filename>), + configuration files, and files that are + included with <filename>include</filename> and + <filename>require</filename> statements. + For these cases, BitBake uses the first file + that matches the name found in + <filename>BBPATH</filename>. + This is similar to the way the + <filename>PATH</filename> variable is used for + binaries. + It is recommended, therefore, that you use + unique class and configuration filenames in + your custom layer. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'><filename>BBFILES</filename></ulink>: + Defines the location for all recipes in the + layer. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_COLLECTIONS'><filename>BBFILE_COLLECTIONS</filename></ulink>: + Establishes the current layer through a + unique identifier that is used throughout the + OpenEmbedded build system to refer to the layer. + In this example, the identifier "yoctobsp" is + the representation for the container layer + named "meta-yocto-bsp". + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PATTERN'><filename>BBFILE_PATTERN</filename></ulink>: + Expands immediately during parsing to + provide the directory of the layer. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink>: + Establishes a priority to use for + recipes in the layer when the OpenEmbedded build + finds recipes of the same name in different + layers. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERVERSION'><filename>LAYERVERSION</filename></ulink>: + Establishes a version number for the layer. + You can use this version number to specify this + exact version of the layer as a dependency when + using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDEPENDS'><filename>LAYERDEPENDS</filename></ulink> + variable. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERSERIES_COMPAT'><filename>LAYERSERIES_COMPAT</filename></ulink>: + Lists the + <ulink url='&YOCTO_WIKI_URL;/wiki/Releases'>Yocto Project</ulink> + releases for which the current version is + compatible. + This variable is a good way to indicate how + up-to-date your particular layer is. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Add Content:</emphasis> + Depending on the type of layer, add the content. + If the layer adds support for a machine, add the machine + configuration in a <filename>conf/machine/</filename> + file within the layer. + If the layer adds distro policy, add the distro + configuration in a <filename>conf/distro/</filename> + file within the layer. + If the layer introduces new recipes, put the recipes + you need in <filename>recipes-*</filename> + subdirectories within the layer. + <note> + For an explanation of layer hierarchy that + is compliant with the Yocto Project, see + the + "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-filelayout'>Example Filesystem Layout</ulink>" + section in the Yocto Project Board + Support Package (BSP) Developer's Guide. + </note> + </para></listitem> + <listitem><para> + <emphasis>Optionally Test for Compatibility:</emphasis> + If you want permission to use the Yocto Project + Compatibility logo with your layer or application that + uses your layer, perform the steps to apply for + compatibility. + See the + "<link linkend='making-sure-your-layer-is-compatible-with-yocto-project'>Making Sure Your Layer is Compatible With Yocto Project</link>" + section for more information. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='best-practices-to-follow-when-creating-layers'> + <title>Following Best Practices When Creating Layers</title> + + <para> + To create layers that are easier to maintain and that will + not impact builds for other machines, you should consider the + information in the following list: + <itemizedlist> + <listitem><para> + <emphasis>Avoid "Overlaying" Entire Recipes from Other Layers in Your Configuration:</emphasis> + In other words, do not copy an entire recipe into your + layer and then modify it. + Rather, use an append file + (<filename>.bbappend</filename>) to override only those + parts of the original recipe you need to modify. + </para></listitem> + <listitem><para> + <emphasis>Avoid Duplicating Include Files:</emphasis> + Use append files (<filename>.bbappend</filename>) + for each recipe that uses an include file. + Or, if you are introducing a new recipe that requires + the included file, use the path relative to the + original layer directory to refer to the file. + For example, use + <filename>require recipes-core/</filename><replaceable>package</replaceable><filename>/</filename><replaceable>file</replaceable><filename>.inc</filename> + instead of + <filename>require </filename><replaceable>file</replaceable><filename>.inc</filename>. + If you're finding you have to overlay the include file, + it could indicate a deficiency in the include file in + the layer to which it originally belongs. + If this is the case, you should try to address that + deficiency instead of overlaying the include file. + For example, you could address this by getting the + maintainer of the include file to add a variable or + variables to make it easy to override the parts needing + to be overridden. + </para></listitem> + <listitem><para> + <emphasis>Structure Your Layers:</emphasis> + Proper use of overrides within append files and + placement of machine-specific files within your layer + can ensure that a build is not using the wrong Metadata + and negatively impacting a build for a different + machine. + Following are some examples: + <itemizedlist> + <listitem><para> + <emphasis>Modify Variables to Support a + Different Machine:</emphasis> + Suppose you have a layer named + <filename>meta-one</filename> that adds support + for building machine "one". + To do so, you use an append file named + <filename>base-files.bbappend</filename> and + create a dependency on "foo" by altering the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable: + <literallayout class='monospaced'> + DEPENDS = "foo" + </literallayout> + The dependency is created during any build that + includes the layer + <filename>meta-one</filename>. + However, you might not want this dependency + for all machines. + For example, suppose you are building for + machine "two" but your + <filename>bblayers.conf</filename> file has the + <filename>meta-one</filename> layer included. + During the build, the + <filename>base-files</filename> for machine + "two" will also have the dependency on + <filename>foo</filename>.</para> + <para>To make sure your changes apply only when + building machine "one", use a machine override + with the <filename>DEPENDS</filename> statement: + <literallayout class='monospaced'> + DEPENDS_one = "foo" + </literallayout> + You should follow the same strategy when using + <filename>_append</filename> and + <filename>_prepend</filename> operations: + <literallayout class='monospaced'> + DEPENDS_append_one = " foo" + DEPENDS_prepend_one = "foo " + </literallayout> + As an actual example, here's a line from the recipe + for gnutls, which adds dependencies on + "argp-standalone" when building with the musl C + library: + <literallayout class='monospaced'> + DEPENDS_append_libc-musl = " argp-standalone" + </literallayout> + <note> + Avoiding "+=" and "=+" and using + machine-specific + <filename>_append</filename> + and <filename>_prepend</filename> operations + is recommended as well. + </note> + </para></listitem> + <listitem><para> + <emphasis>Place Machine-Specific Files in + Machine-Specific Locations:</emphasis> + When you have a base recipe, such as + <filename>base-files.bb</filename>, that + contains a + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statement to a file, you can use an append file + to cause the build to use your own version of + the file. + For example, an append file in your layer at + <filename>meta-one/recipes-core/base-files/base-files.bbappend</filename> + could extend + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + using + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + as follows: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:" + </literallayout> + The build for machine "one" will pick up your + machine-specific file as long as you have the + file in + <filename>meta-one/recipes-core/base-files/base-files/</filename>. + However, if you are building for a different + machine and the + <filename>bblayers.conf</filename> file includes + the <filename>meta-one</filename> layer and + the location of your machine-specific file is + the first location where that file is found + according to <filename>FILESPATH</filename>, + builds for all machines will also use that + machine-specific file.</para> + <para>You can make sure that a machine-specific + file is used for a particular machine by putting + the file in a subdirectory specific to the + machine. + For example, rather than placing the file in + <filename>meta-one/recipes-core/base-files/base-files/</filename> + as shown above, put it in + <filename>meta-one/recipes-core/base-files/base-files/one/</filename>. + Not only does this make sure the file is used + only when building for machine "one", but the + build process locates the file more quickly.</para> + <para>In summary, you need to place all files + referenced from <filename>SRC_URI</filename> + in a machine-specific subdirectory within the + layer in order to restrict those files to + machine-specific builds. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Perform Steps to Apply for Yocto Project Compatibility:</emphasis> + If you want permission to use the + Yocto Project Compatibility logo with your layer + or application that uses your layer, perform the + steps to apply for compatibility. + See the + "<link linkend='making-sure-your-layer-is-compatible-with-yocto-project'>Making Sure Your Layer is Compatible With Yocto Project</link>" + section for more information. + </para></listitem> + <listitem><para> + <emphasis>Follow the Layer Naming Convention:</emphasis> + Store custom layers in a Git repository that use the + <filename>meta-<replaceable>layer_name</replaceable></filename> + format. + </para></listitem> + <listitem><para> + <emphasis>Group Your Layers Locally:</emphasis> + Clone your repository alongside other cloned + <filename>meta</filename> directories from the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='making-sure-your-layer-is-compatible-with-yocto-project'> + <title>Making Sure Your Layer is Compatible With Yocto Project</title> + + <para> + When you create a layer used with the Yocto Project, it is + advantageous to make sure that the layer interacts well with + existing Yocto Project layers (i.e. the layer is compatible + with the Yocto Project). + Ensuring compatibility makes the layer easy to be consumed + by others in the Yocto Project community and could allow you + permission to use the Yocto Project Compatible Logo. + <note> + Only Yocto Project member organizations are permitted to + use the Yocto Project Compatible Logo. + The logo is not available for general use. + For information on how to become a Yocto Project member + organization, see the + <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>. + </note> + </para> + + <para> + The Yocto Project Compatibility Program consists of a layer + application process that requests permission to use the Yocto + Project Compatibility Logo for your layer and application. + The process consists of two parts: + <orderedlist> + <listitem><para> + Successfully passing a script + (<filename>yocto-check-layer</filename>) that + when run against your layer, tests it against + constraints based on experiences of how layers have + worked in the real world and where pitfalls have been + found. + Getting a "PASS" result from the script is required for + successful compatibility registration. + </para></listitem> + <listitem><para> + Completion of an application acceptance form, which + you can find at + <ulink url='https://www.yoctoproject.org/webform/yocto-project-compatible-registration'></ulink>. + </para></listitem> + </orderedlist> + </para> + + <para> + To be granted permission to use the logo, you need to satisfy + the following: + <itemizedlist> + <listitem><para> + Be able to check the box indicating that you + got a "PASS" when running the script against your + layer. + </para></listitem> + <listitem><para> + Answer "Yes" to the questions on the form or have an + acceptable explanation for any questions answered "No". + </para></listitem> + <listitem><para> + Be a Yocto Project Member Organization. + </para></listitem> + </itemizedlist> + </para> + + <para> + The remainder of this section presents information on the + registration form and on the + <filename>yocto-check-layer</filename> script. + </para> + + <section id='yocto-project-compatible-program-application'> + <title>Yocto Project Compatible Program Application</title> + + <para> + Use the form to apply for your layer's approval. + Upon successful application, you can use the Yocto + Project Compatibility Logo with your layer and the + application that uses your layer. + </para> + + <para> + To access the form, use this link: + <ulink url='https://www.yoctoproject.org/webform/yocto-project-compatible-registration'></ulink>. + Follow the instructions on the form to complete your + application. + </para> + + <para> + The application consists of the following sections: + <itemizedlist> + <listitem><para> + <emphasis>Contact Information:</emphasis> + Provide your contact information as the fields + require. + Along with your information, provide the + released versions of the Yocto Project for which + your layer is compatible. + </para></listitem> + <listitem><para> + <emphasis>Acceptance Criteria:</emphasis> + Provide "Yes" or "No" answers for each of the + items in the checklist. + Space exists at the bottom of the form for any + explanations for items for which you answered "No". + </para></listitem> + <listitem><para> + <emphasis>Recommendations:</emphasis> + Provide answers for the questions regarding Linux + kernel use and build success. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='yocto-check-layer-script'> + <title><filename>yocto-check-layer</filename> Script</title> + + <para> + The <filename>yocto-check-layer</filename> script + provides you a way to assess how compatible your layer is + with the Yocto Project. + You should run this script prior to using the form to + apply for compatibility as described in the previous + section. + You need to achieve a "PASS" result in order to have + your application form successfully processed. + </para> + + <para> + The script divides tests into three areas: COMMON, BSP, + and DISTRO. + For example, given a distribution layer (DISTRO), the + layer must pass both the COMMON and DISTRO related tests. + Furthermore, if your layer is a BSP layer, the layer must + pass the COMMON and BSP set of tests. + </para> + + <para> + To execute the script, enter the following commands from + your build directory: + <literallayout class='monospaced'> + $ source oe-init-build-env + $ yocto-check-layer <replaceable>your_layer_directory</replaceable> + </literallayout> + Be sure to provide the actual directory for your layer + as part of the command. + </para> + + <para> + Entering the command causes the script to determine the + type of layer and then to execute a set of specific + tests against the layer. + The following list overviews the test: + <itemizedlist> + <listitem><para> + <filename>common.test_readme</filename>: + Tests if a <filename>README</filename> file + exists in the layer and the file is not empty. + </para></listitem> + <listitem><para> + <filename>common.test_parse</filename>: + Tests to make sure that BitBake can parse the + files without error (i.e. + <filename>bitbake -p</filename>). + </para></listitem> + <listitem><para> + <filename>common.test_show_environment</filename>: + Tests that the global or per-recipe environment + is in order without errors (i.e. + <filename>bitbake -e</filename>). + </para></listitem> + <listitem><para> + <filename>common.test_signatures</filename>: + Tests to be sure that BSP and DISTRO layers do not + come with recipes that change signatures. + </para></listitem> + <listitem><para> + <filename>bsp.test_bsp_defines_machines</filename>: + Tests if a BSP layer has machine configurations. + </para></listitem> + <listitem><para> + <filename>bsp.test_bsp_no_set_machine</filename>: + Tests to ensure a BSP layer does not set the + machine when the layer is added. + </para></listitem> + <listitem><para> + <filename>distro.test_distro_defines_distros</filename>: + Tests if a DISTRO layer has distro configurations. + </para></listitem> + <listitem><para> + <filename>distro.test_distro_no_set_distro</filename>: + Tests to ensure a DISTRO layer does not set the + distribution when the layer is added. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='enabling-your-layer'> + <title>Enabling Your Layer</title> + + <para> + Before the OpenEmbedded build system can use your new layer, + you need to enable it. + To enable your layer, simply add your layer's path to the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBLAYERS'>BBLAYERS</ulink></filename> + variable in your <filename>conf/bblayers.conf</filename> file, + which is found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + The following example shows how to enable a layer named + <filename>meta-mylayer</filename>: + <literallayout class='monospaced'> + # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf + # changes incompatibly + POKY_BBLAYERS_CONF_VERSION = "2" + + BBPATH = "${TOPDIR}" + BBFILES ?= "" + + BBLAYERS ?= " \ + /home/<replaceable>user</replaceable>/poky/meta \ + /home/<replaceable>user</replaceable>/poky/meta-poky \ + /home/<replaceable>user</replaceable>/poky/meta-yocto-bsp \ + /home/<replaceable>user</replaceable>/poky/meta-mylayer \ + " + </literallayout> + </para> + + <para> + BitBake parses each <filename>conf/layer.conf</filename> file + from the top down as specified in the + <filename>BBLAYERS</filename> variable + within the <filename>conf/bblayers.conf</filename> file. + During the processing of each + <filename>conf/layer.conf</filename> file, BitBake adds the + recipes, classes and configurations contained within the + particular layer to the source directory. + </para> + </section> + + <section id='using-bbappend-files'> + <title>Using .bbappend Files in Your Layer</title> + + <para> + A recipe that appends Metadata to another recipe is called a + BitBake append file. + A BitBake append file uses the <filename>.bbappend</filename> + file type suffix, while the corresponding recipe to which + Metadata is being appended uses the <filename>.bb</filename> + file type suffix. + </para> + + <para> + You can use a <filename>.bbappend</filename> file in your + layer to make additions or changes to the content of another + layer's recipe without having to copy the other layer's + recipe into your layer. + Your <filename>.bbappend</filename> file resides in your layer, + while the main <filename>.bb</filename> recipe file to + which you are appending Metadata resides in a different layer. + </para> + + <para> + Being able to append information to an existing recipe not only + avoids duplication, but also automatically applies recipe + changes from a different layer into your layer. + If you were copying recipes, you would have to manually merge + changes as they occur. + </para> + + <para> + When you create an append file, you must use the same root + name as the corresponding recipe file. + For example, the append file + <filename>someapp_&DISTRO;.bbappend</filename> must apply to + <filename>someapp_&DISTRO;.bb</filename>. + This means the original recipe and append file names are + version number-specific. + If the corresponding recipe is renamed to update to a newer + version, you must also rename and possibly update + the corresponding <filename>.bbappend</filename> as well. + During the build process, BitBake displays an error on starting + if it detects a <filename>.bbappend</filename> file that does + not have a corresponding recipe with a matching name. + See the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></ulink> + variable for information on how to handle this error. + </para> + + <para> + As an example, consider the main formfactor recipe and a + corresponding formfactor append file both from the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + Here is the main formfactor recipe, which is named + <filename>formfactor_0.0.bb</filename> and located in the + "meta" layer at + <filename>meta/recipes-bsp/formfactor</filename>: + <literallayout class='monospaced'> + SUMMARY = "Device formfactor information" + SECTION = "base" + LICENSE = "MIT" + LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + PR = "r45" + + SRC_URI = "file://config file://machconfig" + S = "${WORKDIR}" + + PACKAGE_ARCH = "${MACHINE_ARCH}" + INHIBIT_DEFAULT_DEPS = "1" + + do_install() { + # Install file only if it has contents + install -d ${D}${sysconfdir}/formfactor/ + install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/ + if [ -s "${S}/machconfig" ]; then + install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/ + fi + } </literallayout> + In the main recipe, note the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable, which tells the OpenEmbedded build system where to + find files during the build. + </para> + + <para> + Following is the append file, which is named + <filename>formfactor_0.0.bbappend</filename> and is from the + Raspberry Pi BSP Layer named + <filename>meta-raspberrypi</filename>. + The file is in the layer at + <filename>recipes-bsp/formfactor</filename>: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" + </literallayout> + </para> + + <para> + By default, the build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + variable to locate files. + This append file extends the locations by setting the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + variable. + Setting this variable in the <filename>.bbappend</filename> + file is the most reliable and recommended method for adding + directories to the search path used by the build system + to find files. + </para> + + <para> + The statement in this example extends the directories to + include + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-THISDIR'><filename>THISDIR</filename></ulink><filename>}/${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename>, + which resolves to a directory named + <filename>formfactor</filename> in the same directory + in which the append file resides (i.e. + <filename>meta-raspberrypi/recipes-bsp/formfactor</filename>. + This implies that you must have the supporting directory + structure set up that will contain any files or patches you + will be including from the layer. + </para> + + <para> + Using the immediate expansion assignment operator + <filename>:=</filename> is important because of the reference + to <filename>THISDIR</filename>. + The trailing colon character is important as it ensures that + items in the list remain colon-separated. + <note> + <para> + BitBake automatically defines the + <filename>THISDIR</filename> variable. + You should never set this variable yourself. + Using "_prepend" as part of the + <filename>FILESEXTRAPATHS</filename> ensures your path + will be searched prior to other paths in the final + list. + </para> + + <para> + Also, not all append files add extra files. + Many append files simply exist to add build options + (e.g. <filename>systemd</filename>). + For these cases, your append file would not even + use the <filename>FILESEXTRAPATHS</filename> statement. + </para> + </note> + </para> + </section> + + <section id='prioritizing-your-layer'> + <title>Prioritizing Your Layer</title> + + <para> + Each layer is assigned a priority value. + Priority values control which layer takes precedence if there + are recipe files with the same name in multiple layers. + For these cases, the recipe file from the layer with a higher + priority number takes precedence. + Priority values also affect the order in which multiple + <filename>.bbappend</filename> files for the same recipe are + applied. + You can either specify the priority manually, or allow the + build system to calculate it based on the layer's dependencies. + </para> + + <para> + To specify the layer's priority manually, use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink> + variable and append the layer's root name: + <literallayout class='monospaced'> + BBFILE_PRIORITY_mylayer = "1" + </literallayout> + </para> + + <note> + <para>It is possible for a recipe with a lower version number + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + in a layer that has a higher priority to take precedence.</para> + <para>Also, the layer priority does not currently affect the + precedence order of <filename>.conf</filename> + or <filename>.bbclass</filename> files. + Future versions of BitBake might address this.</para> + </note> + </section> + + <section id='managing-layers'> + <title>Managing Layers</title> + + <para> + You can use the BitBake layer management tool + <filename>bitbake-layers</filename> to provide a view + into the structure of recipes across a multi-layer project. + Being able to generate output that reports on configured layers + with their paths and priorities and on + <filename>.bbappend</filename> files and their applicable + recipes can help to reveal potential problems. + </para> + + <para> + For help on the BitBake layer management tool, use the + following command: + <literallayout class='monospaced'> + $ bitbake-layers --help + NOTE: Starting bitbake server... + usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ... + + BitBake layers utility + + optional arguments: + -d, --debug Enable debug output + -q, --quiet Print only errors + -F, --force Force add without recipe parse verification + --color COLOR Colorize output (where COLOR is auto, always, never) + -h, --help show this help message and exit + + subcommands: + <subcommand> + show-layers show current configured layers. + show-overlayed list overlayed recipes (where the same recipe exists + in another layer) + show-recipes list available recipes, showing the layer they are + provided by + show-appends list bbappend files and recipe files they apply to + show-cross-depends Show dependencies between recipes that cross layer + boundaries. + add-layer Add one or more layers to bblayers.conf. + remove-layer Remove one or more layers from bblayers.conf. + flatten flatten layer configuration into a separate output + directory. + layerindex-fetch Fetches a layer from a layer index along with its + dependent layers, and adds them to conf/bblayers.conf. + layerindex-show-depends + Find layer dependencies from layer index. + create-layer Create a basic layer + + Use bitbake-layers <subcommand> --help to get help on a specific command + </literallayout> + </para> + + <para> + The following list describes the available commands: + <itemizedlist> + <listitem><para> + <emphasis><filename>help:</filename></emphasis> + Displays general help or help on a specified command. + </para></listitem> + <listitem><para> + <emphasis><filename>show-layers:</filename></emphasis> + Shows the current configured layers. + </para></listitem> + <listitem><para> + <emphasis><filename>show-overlayed:</filename></emphasis> + Lists overlayed recipes. + A recipe is overlayed when a recipe with the same name + exists in another layer that has a higher layer + priority. + </para></listitem> + <listitem><para> + <emphasis><filename>show-recipes:</filename></emphasis> + Lists available recipes and the layers that provide them. + </para></listitem> + <listitem><para> + <emphasis><filename>show-appends:</filename></emphasis> + Lists <filename>.bbappend</filename> files and the + recipe files to which they apply. + </para></listitem> + <listitem><para> + <emphasis><filename>show-cross-depends:</filename></emphasis> + Lists dependency relationships between recipes that + cross layer boundaries. + </para></listitem> + <listitem><para> + <emphasis><filename>add-layer:</filename></emphasis> + Adds a layer to <filename>bblayers.conf</filename>. + </para></listitem> + <listitem><para> + <emphasis><filename>remove-layer:</filename></emphasis> + Removes a layer from <filename>bblayers.conf</filename> + </para></listitem> + <listitem><para> + <emphasis><filename>flatten:</filename></emphasis> + Flattens the layer configuration into a separate output + directory. + Flattening your layer configuration builds a "flattened" + directory that contains the contents of all layers, + with any overlayed recipes removed and any + <filename>.bbappend</filename> files appended to the + corresponding recipes. + You might have to perform some manual cleanup of the + flattened layer as follows: + <itemizedlist> + <listitem><para> + Non-recipe files (such as patches) + are overwritten. + The flatten command shows a warning for these + files. + </para></listitem> + <listitem><para> + Anything beyond the normal layer + setup has been added to the + <filename>layer.conf</filename> file. + Only the lowest priority layer's + <filename>layer.conf</filename> is used. + </para></listitem> + <listitem><para> + Overridden and appended items from + <filename>.bbappend</filename> files need to be + cleaned up. + The contents of each + <filename>.bbappend</filename> end up in the + flattened recipe. + However, if there are appended or changed + variable values, you need to tidy these up + yourself. + Consider the following example. + Here, the <filename>bitbake-layers</filename> + command adds the line + <filename>#### bbappended ...</filename> so that + you know where the following lines originate: + <literallayout class='monospaced'> + ... + DESCRIPTION = "A useful utility" + ... + EXTRA_OECONF = "--enable-something" + ... + + #### bbappended from meta-anotherlayer #### + + DESCRIPTION = "Customized utility" + EXTRA_OECONF += "--enable-somethingelse" + </literallayout> + Ideally, you would tidy up these utilities as + follows: + <literallayout class='monospaced'> + ... + DESCRIPTION = "Customized utility" + ... + EXTRA_OECONF = "--enable-something --enable-somethingelse" + ... + </literallayout> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis><filename>layerindex-fetch</filename>:</emphasis> + Fetches a layer from a layer index, along with its + dependent layers, and adds the layers to the + <filename>conf/bblayers.conf</filename> file. + </para></listitem> + <listitem><para> + <emphasis><filename>layerindex-show-depends</filename>:</emphasis> + Finds layer dependencies from the layer index. + </para></listitem> + <listitem><para> + <emphasis><filename>create-layer</filename>:</emphasis> + Creates a basic layer. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='creating-a-general-layer-using-the-bitbake-layers-script'> + <title>Creating a General Layer Using the <filename>bitbake-layers</filename> Script</title> + + <para> + The <filename>bitbake-layers</filename> script with the + <filename>create-layer</filename> subcommand simplifies + creating a new general layer. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + For information on BSP layers, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" + section in the Yocto Project Board Specific (BSP) + Developer's Guide. + </para></listitem> + <listitem><para> + In order to use a layer with the OpenEmbedded + build system, you need to add the layer to your + <filename>bblayers.conf</filename> configuration + file. + See the + "<link linkend='adding-a-layer-using-the-bitbake-layers-script'>Adding a Layer Using the <filename>bitbake-layers</filename> Script</link>" + section for more information. + </para></listitem> + </itemizedlist> + </note> + The default mode of the script's operation with this + subcommand is to create a layer with the following: + <itemizedlist> + <listitem><para>A layer priority of 6. + </para></listitem> + <listitem><para>A <filename>conf</filename> + subdirectory that contains a + <filename>layer.conf</filename> file. + </para></listitem> + <listitem><para> + A <filename>recipes-example</filename> subdirectory + that contains a further subdirectory named + <filename>example</filename>, which contains + an <filename>example.bb</filename> recipe file. + </para></listitem> + <listitem><para>A <filename >COPYING.MIT</filename>, + which is the license statement for the layer. + The script assumes you want to use the MIT license, + which is typical for most layers, for the contents of + the layer itself. + </para></listitem> + <listitem><para> + A <filename>README</filename> file, which is a file + describing the contents of your new layer. + </para></listitem> + </itemizedlist> + </para> + + <para> + In its simplest form, you can use the following command form + to create a layer. + The command creates a layer whose name corresponds to + <replaceable>your_layer_name</replaceable> in the current + directory: + <literallayout class='monospaced'> + $ bitbake-layers create-layer <replaceable>your_layer_name</replaceable> + </literallayout> + As an example, the following command creates a layer named + <filename>meta-scottrif</filename> in your home directory: + <literallayout class='monospaced'> + $ cd /usr/home + $ bitbake-layers create-layer meta-scottrif + NOTE: Starting bitbake server... + Add your new layer with 'bitbake-layers add-layer meta-scottrif' + </literallayout> + </para> + + <para> + If you want to set the priority of the layer to other than the + default value of "6", you can either use the + <filename>‐‐priority</filename> option or you can + edit the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink> + value in the <filename>conf/layer.conf</filename> after the + script creates it. + Furthermore, if you want to give the example recipe file + some name other than the default, you can + use the + <filename>‐‐example-recipe-name</filename> option. + </para> + + <para> + The easiest way to see how the + <filename>bitbake-layers create-layer</filename> command + works is to experiment with the script. + You can also read the usage information by entering the + following: + <literallayout class='monospaced'> + $ bitbake-layers create-layer --help + NOTE: Starting bitbake server... + usage: bitbake-layers create-layer [-h] [--priority PRIORITY] + [--example-recipe-name EXAMPLERECIPE] + layerdir + + Create a basic layer + + positional arguments: + layerdir Layer directory to create + + optional arguments: + -h, --help show this help message and exit + --priority PRIORITY, -p PRIORITY + Layer directory to create + --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE + Filename of the example recipe + </literallayout> + </para> + </section> + + <section id='adding-a-layer-using-the-bitbake-layers-script'> + <title>Adding a Layer Using the <filename>bitbake-layers</filename> Script</title> + + <para> + Once you create your general layer, you must add it to your + <filename>bblayers.conf</filename> file. + Adding the layer to this configuration file makes the + OpenEmbedded build system aware of your layer so that it can + search it for metadata. + </para> + + <para> + Add your layer by using the + <filename>bitbake-layers add-layer</filename> command: + <literallayout class='monospaced'> + $ bitbake-layers add-layer <replaceable>your_layer_name</replaceable> + </literallayout> + Here is an example that adds a layer named + <filename>meta-scottrif</filename> to the configuration file. + Following the command that adds the layer is another + <filename>bitbake-layers</filename> command that shows the + layers that are in your <filename>bblayers.conf</filename> + file: + <literallayout class='monospaced'> + $ bitbake-layers add-layer meta-scottrif + NOTE: Starting bitbake server... + Parsing recipes: 100% |##########################################################| Time: 0:00:49 + Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors. + $ bitbake-layers show-layers + NOTE: Starting bitbake server... + layer path priority + ========================================================================== + meta /home/scottrif/poky/meta 5 + meta-poky /home/scottrif/poky/meta-poky 5 + meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5 + workspace /home/scottrif/poky/build/workspace 99 + meta-scottrif /home/scottrif/poky/build/meta-scottrif 6 + </literallayout> + Adding the layer to this file enables the build system to + locate the layer during the build. + <note> + During a build, the OpenEmbedded build system looks in + the layers from the top of the list down to the bottom + in that order. + </note> + </para> + </section> + </section> + + <section id='usingpoky-extend-customimage'> + <title>Customizing Images</title> + + <para> + You can customize images to satisfy particular requirements. + This section describes several methods and provides guidelines for each. + </para> + + <section id='usingpoky-extend-customimage-localconf'> + <title>Customizing Images Using <filename>local.conf</filename></title> + + <para> + Probably the easiest way to customize an image is to add a + package by way of the <filename>local.conf</filename> + configuration file. + Because it is limited to local use, this method generally only + allows you to add packages and is not as flexible as creating + your own customized image. + When you add packages using local variables this way, you need + to realize that these variable changes are in effect for every + build and consequently affect all images, which might not + be what you require. + </para> + + <para> + To add a package to your image using the local configuration + file, use the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename> + variable with the <filename>_append</filename> operator: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " strace" + </literallayout> + Use of the syntax is important - specifically, the space between + the quote and the package name, which is + <filename>strace</filename> in this example. + This space is required since the <filename>_append</filename> + operator does not add the space. + </para> + + <para> + Furthermore, you must use <filename>_append</filename> instead + of the <filename>+=</filename> operator if you want to avoid + ordering issues. + The reason for this is because doing so unconditionally appends + to the variable and avoids ordering problems due to the + variable being set in image recipes and + <filename>.bbclass</filename> files with operators like + <filename>?=</filename>. + Using <filename>_append</filename> ensures the operation takes + affect. + </para> + + <para> + As shown in its simplest use, + <filename>IMAGE_INSTALL_append</filename> affects all images. + It is possible to extend the syntax so that the variable + applies to a specific image only. + Here is an example: + <literallayout class='monospaced'> + IMAGE_INSTALL_append_pn-core-image-minimal = " strace" + </literallayout> + This example adds <filename>strace</filename> to the + <filename>core-image-minimal</filename> image only. + </para> + + <para> + You can add packages using a similar approach through the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'>CORE_IMAGE_EXTRA_INSTALL</ulink></filename> + variable. + If you use this variable, only + <filename>core-image-*</filename> images are affected. + </para> + </section> + + <section id='usingpoky-extend-customimage-imagefeatures'> + <title>Customizing Images Using Custom <filename>IMAGE_FEATURES</filename> and + <filename>EXTRA_IMAGE_FEATURES</filename></title> + + <para> + Another method for customizing your image is to enable or + disable high-level image features by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> + variables. + Although the functions for both variables are nearly equivalent, + best practices dictate using <filename>IMAGE_FEATURES</filename> + from within a recipe and using + <filename>EXTRA_IMAGE_FEATURES</filename> from within + your <filename>local.conf</filename> file, which is found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + </para> + + <para> + To understand how these features work, the best reference is + <filename>meta/classes/core-image.bbclass</filename>. + This class lists out the available + <filename>IMAGE_FEATURES</filename> of which most map to + package groups while some, such as + <filename>debug-tweaks</filename> and + <filename>read-only-rootfs</filename>, resolve as general + configuration settings. + </para> + + <para> + In summary, the file looks at the contents of the + <filename>IMAGE_FEATURES</filename> variable and then maps + or configures the feature accordingly. + Based on this information, the build system automatically + adds the appropriate packages or configurations to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink> + variable. + Effectively, you are enabling extra features by extending the + class or creating a custom class for use with specialized image + <filename>.bb</filename> files. + </para> + + <para> + Use the <filename>EXTRA_IMAGE_FEATURES</filename> variable + from within your local configuration file. + Using a separate area from which to enable features with + this variable helps you avoid overwriting the features in the + image recipe that are enabled with + <filename>IMAGE_FEATURES</filename>. + The value of <filename>EXTRA_IMAGE_FEATURES</filename> is added + to <filename>IMAGE_FEATURES</filename> within + <filename>meta/conf/bitbake.conf</filename>. + </para> + + <para> + To illustrate how you can use these variables to modify your + image, consider an example that selects the SSH server. + The Yocto Project ships with two SSH servers you can use + with your images: Dropbear and OpenSSH. + Dropbear is a minimal SSH server appropriate for + resource-constrained environments, while OpenSSH is a + well-known standard SSH server implementation. + By default, the <filename>core-image-sato</filename> image + is configured to use Dropbear. + The <filename>core-image-full-cmdline</filename> and + <filename>core-image-lsb</filename> images both + include OpenSSH. + The <filename>core-image-minimal</filename> image does not + contain an SSH server. + </para> + + <para> + You can customize your image and change these defaults. + Edit the <filename>IMAGE_FEATURES</filename> variable + in your recipe or use the + <filename>EXTRA_IMAGE_FEATURES</filename> in your + <filename>local.conf</filename> file so that it configures the + image you are working with to include + <filename>ssh-server-dropbear</filename> or + <filename>ssh-server-openssh</filename>. + </para> + + <note> + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + section in the Yocto Project Reference Manual for a complete + list of image features that ship with the Yocto Project. + </note> + </section> + + <section id='usingpoky-extend-customimage-custombb'> + <title>Customizing Images Using Custom .bb Files</title> + + <para> + You can also customize an image by creating a custom recipe + that defines additional software as part of the image. + The following example shows the form for the two lines you need: + <literallayout class='monospaced'> + IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2" + + inherit core-image + </literallayout> + </para> + + <para> + Defining the software using a custom recipe gives you total + control over the contents of the image. + It is important to use the correct names of packages in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename> + variable. + You must use the OpenEmbedded notation and not the Debian notation for the names + (e.g. <filename>glibc-dev</filename> instead of <filename>libc6-dev</filename>). + </para> + + <para> + The other method for creating a custom image is to base it on an existing image. + For example, if you want to create an image based on <filename>core-image-sato</filename> + but add the additional package <filename>strace</filename> to the image, + copy the <filename>meta/recipes-sato/images/core-image-sato.bb</filename> to a + new <filename>.bb</filename> and add the following line to the end of the copy: + <literallayout class='monospaced'> + IMAGE_INSTALL += "strace" + </literallayout> + </para> + </section> + + <section id='usingpoky-extend-customimage-customtasks'> + <title>Customizing Images Using Custom Package Groups</title> + + <para> + For complex custom images, the best approach for customizing + an image is to create a custom package group recipe that is + used to build the image or images. + A good example of a package group recipe is + <filename>meta/recipes-core/packagegroups/packagegroup-base.bb</filename>. + </para> + + <para> + If you examine that recipe, you see that the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'>PACKAGES</ulink></filename> + variable lists the package group packages to produce. + The <filename>inherit packagegroup</filename> statement + sets appropriate default values and automatically adds + <filename>-dev</filename>, <filename>-dbg</filename>, and + <filename>-ptest</filename> complementary packages for each + package specified in the <filename>PACKAGES</filename> + statement. + <note> + The <filename>inherit packages</filename> should be + located near the top of the recipe, certainly before + the <filename>PACKAGES</filename> statement. + </note> + </para> + + <para> + For each package you specify in <filename>PACKAGES</filename>, + you can use + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'>RDEPENDS</ulink></filename> + and + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'>RRECOMMENDS</ulink></filename> + entries to provide a list of packages the parent task package + should contain. + You can see examples of these further down in the + <filename>packagegroup-base.bb</filename> recipe. + </para> + + <para> + Here is a short, fabricated example showing the same basic + pieces: + <literallayout class='monospaced'> + DESCRIPTION = "My Custom Package Groups" + + inherit packagegroup + + PACKAGES = "\ + packagegroup-custom-apps \ + packagegroup-custom-tools \ + " + + RDEPENDS_packagegroup-custom-apps = "\ + dropbear \ + portmap \ + psplash" + + RDEPENDS_packagegroup-custom-tools = "\ + oprofile \ + oprofileui-server \ + lttng-tools" + + RRECOMMENDS_packagegroup-custom-tools = "\ + kernel-module-oprofile" + </literallayout> + </para> + + <para> + In the previous example, two package group packages are created with their dependencies and their + recommended package dependencies listed: <filename>packagegroup-custom-apps</filename>, and + <filename>packagegroup-custom-tools</filename>. + To build an image using these package group packages, you need to add + <filename>packagegroup-custom-apps</filename> and/or + <filename>packagegroup-custom-tools</filename> to + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename>. + For other forms of image dependencies see the other areas of this section. + </para> + </section> + + <section id='usingpoky-extend-customimage-image-name'> + <title>Customizing an Image Hostname</title> + + <para> + By default, the configured hostname (i.e. + <filename>/etc/hostname</filename>) in an image is the + same as the machine name. + For example, if + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + equals "qemux86", the configured hostname written to + <filename>/etc/hostname</filename> is "qemux86". + </para> + + <para> + You can customize this name by altering the value of the + "hostname" variable in the + <filename>base-files</filename> recipe using either + an append file or a configuration file. + Use the following in an append file: + <literallayout class='monospaced'> + hostname="myhostname" + </literallayout> + Use the following in a configuration file: + <literallayout class='monospaced'> + hostname_pn-base-files = "myhostname" + </literallayout> + </para> + + <para> + Changing the default value of the variable "hostname" can be + useful in certain situations. + For example, suppose you need to do extensive testing on an + image and you would like to easily identify the image + under test from existing images with typical default + hostnames. + In this situation, you could change the default hostname to + "testme", which results in all the images using the name + "testme". + Once testing is complete and you do not need to rebuild the + image for test any longer, you can easily reset the default + hostname. + </para> + + <para> + Another point of interest is that if you unset the variable, + the image will have no default hostname in the filesystem. + Here is an example that unsets the variable in a + configuration file: + <literallayout class='monospaced'> + hostname_pn-base-files = "" + </literallayout> + Having no default hostname in the filesystem is suitable for + environments that use dynamic hostnames such as virtual + machines. + </para> + </section> + </section> + + <section id='new-recipe-writing-a-new-recipe'> + <title>Writing a New Recipe</title> + + <para> + Recipes (<filename>.bb</filename> files) are fundamental components + in the Yocto Project environment. + Each software component built by the OpenEmbedded build system + requires a recipe to define the component. + This section describes how to create, write, and test a new + recipe. + <note> + For information on variables that are useful for recipes and + for information about recipe naming issues, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-varlocality-recipe-required'>Required</ulink>" + section of the Yocto Project Reference Manual. + </note> + </para> + + <section id='new-recipe-overview'> + <title>Overview</title> + + <para> + The following figure shows the basic process for creating a + new recipe. + The remainder of the section provides details for the steps. + <imagedata fileref="figures/recipe-workflow.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> + </section> + + <section id='new-recipe-locate-or-automatically-create-a-base-recipe'> + <title>Locate or Automatically Create a Base Recipe</title> + + <para> + You can always write a recipe from scratch. + However, three choices exist that can help you quickly get a + start on a new recipe: + <itemizedlist> + <listitem><para> + <emphasis><filename>devtool add</filename>:</emphasis> + A command that assists in creating a recipe and + an environment conducive to development. + </para></listitem> + <listitem><para> + <emphasis><filename>recipetool create</filename>:</emphasis> + A command provided by the Yocto Project that automates + creation of a base recipe based on the source + files. + </para></listitem> + <listitem><para> + <emphasis>Existing Recipes:</emphasis> + Location and modification of an existing recipe that is + similar in function to the recipe you need. + </para></listitem> + </itemizedlist> + <note> + For information on recipe syntax, see the + "<link linkend='recipe-syntax'>Recipe Syntax</link>" + section. + </note> + </para> + + <section id='new-recipe-creating-the-base-recipe-using-devtool'> + <title>Creating the Base Recipe Using <filename>devtool add</filename></title> + + <para> + The <filename>devtool add</filename> command uses the same + logic for auto-creating the recipe as + <filename>recipetool create</filename>, which is listed + below. + Additionally, however, <filename>devtool add</filename> + sets up an environment that makes it easy for you to + patch the source and to make changes to the recipe as + is often necessary when adding a recipe to build a new + piece of software to be included in a build. + </para> + + <para> + You can find a complete description of the + <filename>devtool add</filename> command in the + "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-a-closer-look-at-devtool-add'>A Closer Look at <filename>devtool</filename> add</ulink>" + section in the Yocto Project Application Development + and the Extensible Software Development Kit (eSDK) manual. + </para> + </section> + + <section id='new-recipe-creating-the-base-recipe-using-recipetool'> + <title>Creating the Base Recipe Using <filename>recipetool create</filename></title> + + <para> + <filename>recipetool create</filename> automates creation + of a base recipe given a set of source code files. + As long as you can extract or point to the source files, + the tool will construct a recipe and automatically + configure all pre-build information into the recipe. + For example, suppose you have an application that builds + using Autotools. + Creating the base recipe using + <filename>recipetool</filename> results in a recipe + that has the pre-build dependencies, license requirements, + and checksums configured. + </para> + + <para> + To run the tool, you just need to be in your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + and have sourced the build environment setup script + (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>oe-init-build-env</filename></ulink>). + To get help on the tool, use the following command: + <literallayout class='monospaced'> + $ recipetool -h + NOTE: Starting bitbake server... + usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ... + + OpenEmbedded recipe tool + + options: + -d, --debug Enable debug output + -q, --quiet Print only errors + --color COLOR Colorize output (where COLOR is auto, always, never) + -h, --help show this help message and exit + + subcommands: + create Create a new recipe + newappend Create a bbappend for the specified target in the specified + layer + setvar Set a variable within a recipe + appendfile Create/update a bbappend to replace a target file + appendsrcfiles Create/update a bbappend to add or replace source files + appendsrcfile Create/update a bbappend to add or replace a source file + Use recipetool <subcommand> --help to get help on a specific command + </literallayout> + </para> + + <para> + Running + <filename>recipetool create -o</filename> <replaceable>OUTFILE</replaceable> + creates the base recipe and locates it properly in the + layer that contains your source files. + Following are some syntax examples: + </para> + + <para> + Use this syntax to generate a recipe based on + <replaceable>source</replaceable>. + Once generated, the recipe resides in the existing source + code layer: + <literallayout class='monospaced'> + recipetool create -o <replaceable>OUTFILE</replaceable> <replaceable>source</replaceable> + </literallayout> + Use this syntax to generate a recipe using code that you + extract from <replaceable>source</replaceable>. + The extracted code is placed in its own layer defined + by <replaceable>EXTERNALSRC</replaceable>. + <literallayout class='monospaced'> + recipetool create -o <replaceable>OUTFILE</replaceable> -x <replaceable>EXTERNALSRC</replaceable> <replaceable>source</replaceable> + </literallayout> + Use this syntax to generate a recipe based on + <replaceable>source</replaceable>. + The options direct <filename>recipetool</filename> to + generate debugging information. + Once generated, the recipe resides in the existing source + code layer: + <literallayout class='monospaced'> + recipetool create -d -o <replaceable>OUTFILE</replaceable> <replaceable>source</replaceable> + </literallayout> + </para> + </section> + + <section id='new-recipe-locating-and-using-a-similar-recipe'> + <title>Locating and Using a Similar Recipe</title> + + <para> + Before writing a recipe from scratch, it is often useful to + discover whether someone else has already written one that + meets (or comes close to meeting) your needs. + The Yocto Project and OpenEmbedded communities maintain many + recipes that might be candidates for what you are doing. + You can find a good central index of these recipes in the + <ulink url='http://layers.openembedded.org'>OpenEmbedded Layer Index</ulink>. + </para> + + <para> + Working from an existing recipe or a skeleton recipe is the + best way to get started. + Here are some points on both methods: + <itemizedlist> + <listitem><para><emphasis>Locate and modify a recipe that + is close to what you want to do:</emphasis> + This method works when you are familiar with the + current recipe space. + The method does not work so well for those new to + the Yocto Project or writing recipes.</para> + <para>Some risks associated with this method are + using a recipe that has areas totally unrelated to + what you are trying to accomplish with your recipe, + not recognizing areas of the recipe that you might + have to add from scratch, and so forth. + All these risks stem from unfamiliarity with the + existing recipe space.</para></listitem> + <listitem><para><emphasis>Use and modify the following + skeleton recipe:</emphasis> + If for some reason you do not want to use + <filename>recipetool</filename> and you cannot + find an existing recipe that is close to meeting + your needs, you can use the following structure to + provide the fundamental areas of a new recipe. + <literallayout class='monospaced'> + DESCRIPTION = "" + HOMEPAGE = "" + LICENSE = "" + SECTION = "" + DEPENDS = "" + LIC_FILES_CHKSUM = "" + + SRC_URI = "" + </literallayout> + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='new-recipe-storing-and-naming-the-recipe'> + <title>Storing and Naming the Recipe</title> + + <para> + Once you have your base recipe, you should put it in your + own layer and name it appropriately. + Locating it correctly ensures that the OpenEmbedded build + system can find it when you use BitBake to process the + recipe. + </para> + + <itemizedlist> + <listitem><para><emphasis>Storing Your Recipe:</emphasis> + The OpenEmbedded build system locates your recipe + through the layer's <filename>conf/layer.conf</filename> + file and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'><filename>BBFILES</filename></ulink> + variable. + This variable sets up a path from which the build system can + locate recipes. + Here is the typical use: + <literallayout class='monospaced'> + BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + </literallayout> + Consequently, you need to be sure you locate your new recipe + inside your layer such that it can be found.</para> + <para>You can find more information on how layers are + structured in the + "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" + section.</para></listitem> + <listitem><para><emphasis>Naming Your Recipe:</emphasis> + When you name your recipe, you need to follow this naming + convention: + <literallayout class='monospaced'> + <replaceable>basename</replaceable>_<replaceable>version</replaceable>.bb + </literallayout> + Use lower-cased characters and do not include the reserved + suffixes <filename>-native</filename>, + <filename>-cross</filename>, <filename>-initial</filename>, + or <filename>-dev</filename> casually (i.e. do not use them + as part of your recipe name unless the string applies). + Here are some examples: + <literallayout class='monospaced'> + cups_1.7.0.bb + gawk_4.0.2.bb + irssi_0.8.16-rc1.bb + </literallayout></para></listitem> + </itemizedlist> + </section> + + <section id='new-recipe-running-a-build-on-the-recipe'> + <title>Running a Build on the Recipe</title> + + <para> + Creating a new recipe is usually an iterative process that + requires using BitBake to process the recipe multiple times in + order to progressively discover and add information to the + recipe file. + </para> + + <para> + Assuming you have sourced the build environment setup script (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>) + and you are in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>, + use BitBake to process your recipe. + All you need to provide is the + <filename><replaceable>basename</replaceable></filename> of the recipe as described + in the previous section: + <literallayout class='monospaced'> + $ bitbake <replaceable>basename</replaceable> + </literallayout> + + </para> + + <para> + During the build, the OpenEmbedded build system creates a + temporary work directory for each recipe + (<filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>) + where it keeps extracted source files, log files, intermediate + compilation and packaging files, and so forth. + </para> + + <para> + The path to the per-recipe temporary work directory depends + on the context in which it is being built. + The quickest way to find this path is to have BitBake return it + by running the following: + <literallayout class='monospaced'> + $ bitbake -e <replaceable>basename</replaceable> | grep ^WORKDIR= + </literallayout> + As an example, assume a Source Directory top-level folder named + <filename>poky</filename>, a default Build Directory at + <filename>poky/build</filename>, and a + <filename>qemux86-poky-linux</filename> machine target system. + Furthermore, suppose your recipe is named + <filename>foo_1.3.0.bb</filename>. + In this case, the work directory the build system uses to + build the package would be as follows: + <literallayout class='monospaced'> + poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0 + </literallayout> + Inside this directory you can find sub-directories such as + <filename>image</filename>, <filename>packages-split</filename>, + and <filename>temp</filename>. + After the build, you can examine these to determine how well + the build went. + <note> + You can find log files for each task in the recipe's + <filename>temp</filename> directory (e.g. + <filename>poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp</filename>). + Log files are named <filename>log.<replaceable>taskname</replaceable></filename> + (e.g. <filename>log.do_configure</filename>, + <filename>log.do_fetch</filename>, and + <filename>log.do_compile</filename>). + </note> + </para> + + <para> + You can find more information about the build process in + "<ulink url='&YOCTO_DOCS_OM_URL;#overview-development-environment'>The Yocto Project Development Environment</ulink>" + chapter of the Yocto Project Overview and Concepts Manual. + </para> + </section> + + <section id='new-recipe-fetching-code'> + <title>Fetching Code</title> + + <para> + The first thing your recipe must do is specify how to fetch + the source files. + Fetching is controlled mainly through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable. + Your recipe must have a <filename>SRC_URI</filename> variable + that points to where the source is located. + For a graphical representation of source locations, see the + "<ulink url='&YOCTO_DOCS_OM_URL;#sources-dev-environment'>Sources</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + + <para> + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> + task uses the prefix of each entry in the + <filename>SRC_URI</filename> variable value to determine which + fetcher to use to get your source files. + It is the <filename>SRC_URI</filename> variable that triggers + the fetcher. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + task uses the variable after source is fetched to apply + patches. + The OpenEmbedded build system uses + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESOVERRIDES'><filename>FILESOVERRIDES</filename></ulink> + for scanning directory locations for local files in + <filename>SRC_URI</filename>. + </para> + + <para> + The <filename>SRC_URI</filename> variable in your recipe must + define each unique location for your source files. + It is good practice to not hard-code pathnames in an URL used + in <filename>SRC_URI</filename>. + Rather than hard-code these paths, use + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink><filename>}</filename>, + which causes the fetch process to use the version specified in + the recipe filename. + Specifying the version in this manner means that upgrading the + recipe to a future version is as simple as renaming the recipe + to match the new version. + </para> + + <para> + Here is a simple example from the + <filename>meta/recipes-devtools/cdrtools/cdrtools-native_3.01a20.bb</filename> + recipe where the source comes from a single tarball. + Notice the use of the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + variable: + <literallayout class='monospaced'> + SRC_URI = "ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-${PV}.tar.bz2" + </literallayout> + </para> + + <para> + Files mentioned in <filename>SRC_URI</filename> whose names end + in a typical archive extension (e.g. <filename>.tar</filename>, + <filename>.tar.gz</filename>, <filename>.tar.bz2</filename>, + <filename>.zip</filename>, and so forth), are automatically + extracted during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> + task. + For another example that specifies these types of files, see + the + "<link linkend='new-recipe-autotooled-package'>Autotooled Package</link>" + section. + </para> + + <para> + Another way of specifying source is from an SCM. + For Git repositories, you must specify + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + and you should specify + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + to include the revision with + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCPV'><filename>SRCPV</filename></ulink>. + Here is an example from the recipe + <filename>meta/recipes-kernel/blktrace/blktrace_git.bb</filename>: + <literallayout class='monospaced'> + SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385" + + PR = "r6" + PV = "1.0.5+git${SRCPV}" + + SRC_URI = "git://git.kernel.dk/blktrace.git \ + file://ldflags.patch" + </literallayout> + </para> + + <para> + If your <filename>SRC_URI</filename> statement includes + URLs pointing to individual files fetched from a remote server + other than a version control system, BitBake attempts to + verify the files against checksums defined in your recipe to + ensure they have not been tampered with or otherwise modified + since the recipe was written. + Two checksums are used: + <filename>SRC_URI[md5sum]</filename> and + <filename>SRC_URI[sha256sum]</filename>. + </para> + + <para> + If your <filename>SRC_URI</filename> variable points to + more than a single URL (excluding SCM URLs), you need to + provide the <filename>md5</filename> and + <filename>sha256</filename> checksums for each URL. + For these cases, you provide a name for each URL as part of + the <filename>SRC_URI</filename> and then reference that name + in the subsequent checksum statements. + Here is an example: + <literallayout class='monospaced'> + SRC_URI = "${DEBIAN_MIRROR}/main/a/apmd/apmd_3.2.2.orig.tar.gz;name=tarball \ + ${DEBIAN_MIRROR}/main/a/apmd/apmd_${PV}.diff.gz;name=patch" + + SRC_URI[tarball.md5sum] = "b1e6309e8331e0f4e6efd311c2d97fa8" + SRC_URI[tarball.sha256sum] = "7f7d9f60b7766b852881d40b8ff91d8e39fccb0d1d913102a5c75a2dbb52332d" + + SRC_URI[patch.md5sum] = "57e1b689264ea80f78353519eece0c92" + SRC_URI[patch.sha256sum] = "7905ff96be93d725544d0040e425c42f9c05580db3c272f11cff75b9aa89d430" + </literallayout> + </para> + + <para> + Proper values for <filename>md5</filename> and + <filename>sha256</filename> checksums might be available + with other signatures on the download page for the upstream + source (e.g. <filename>md5</filename>, + <filename>sha1</filename>, <filename>sha256</filename>, + <filename>GPG</filename>, and so forth). + Because the OpenEmbedded build system only deals with + <filename>sha256sum</filename> and <filename>md5sum</filename>, + you should verify all the signatures you find by hand. + </para> + + <para> + If no <filename>SRC_URI</filename> checksums are specified + when you attempt to build the recipe, or you provide an + incorrect checksum, the build will produce an error for each + missing or incorrect checksum. + As part of the error message, the build system provides + the checksum string corresponding to the fetched file. + Once you have the correct checksums, you can copy and paste + them into your recipe and then run the build again to continue. + <note> + As mentioned, if the upstream source provides signatures + for verifying the downloaded source code, you should + verify those manually before setting the checksum values + in the recipe and continuing with the build. + </note> + </para> + + <para> + This final example is a bit more complicated and is from the + <filename>meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb</filename> + recipe. + The example's <filename>SRC_URI</filename> statement identifies + multiple files as the source files for the recipe: a tarball, a + patch file, a desktop file, and an icon. + <literallayout class='monospaced'> + SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \ + file://xwc.patch \ + file://rxvt.desktop \ + file://rxvt.png" + </literallayout> + </para> + + <para> + When you specify local files using the + <filename>file://</filename> URI protocol, the build system + fetches files from the local machine. + The path is relative to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + variable and searches specific directories in a certain order: + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BP'><filename>BP</filename></ulink><filename>}</filename>, + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink><filename>}</filename>, + and <filename>files</filename>. + The directories are assumed to be subdirectories of the + directory in which the recipe or append file resides. + For another example that specifies these types of files, see the + "<link linkend='new-recipe-single-c-file-package-hello-world'>Single .c File Package (Hello World!)</link>" + section. + </para> + + <para> + The previous example also specifies a patch file. + Patch files are files whose names usually end in + <filename>.patch</filename> or <filename>.diff</filename> but + can end with compressed suffixes such as + <filename>diff.gz</filename> and + <filename>patch.bz2</filename>, for example. + The build system automatically applies patches as described + in the + "<link linkend='new-recipe-patching-code'>Patching Code</link>" section. + </para> + </section> + + <section id='new-recipe-unpacking-code'> + <title>Unpacking Code</title> + + <para> + During the build, the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> + task unpacks the source with + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename> + pointing to where it is unpacked. + </para> + + <para> + If you are fetching your source files from an upstream source + archived tarball and the tarball's internal structure matches + the common convention of a top-level subdirectory named + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink><filename>}-${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink><filename>}</filename>, + then you do not need to set <filename>S</filename>. + However, if <filename>SRC_URI</filename> specifies to fetch + source from an archive that does not use this convention, + or from an SCM like Git or Subversion, your recipe needs to + define <filename>S</filename>. + </para> + + <para> + If processing your recipe using BitBake successfully unpacks + the source files, you need to be sure that the directory + pointed to by <filename>${S}</filename> matches the structure + of the source. + </para> + </section> + + <section id='new-recipe-patching-code'> + <title>Patching Code</title> + + <para> + Sometimes it is necessary to patch code after it has been + fetched. + Any files mentioned in <filename>SRC_URI</filename> whose + names end in <filename>.patch</filename> or + <filename>.diff</filename> or compressed versions of these + suffixes (e.g. <filename>diff.gz</filename> are treated as + patches. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + task automatically applies these patches. + </para> + + <para> + The build system should be able to apply patches with the "-p1" + option (i.e. one directory level in the path will be stripped + off). + If your patch needs to have more directory levels stripped off, + specify the number of levels using the "striplevel" option in + the <filename>SRC_URI</filename> entry for the patch. + Alternatively, if your patch needs to be applied in a specific + subdirectory that is not specified in the patch file, use the + "patchdir" option in the entry. + </para> + + <para> + As with all local files referenced in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + using <filename>file://</filename>, you should place + patch files in a directory next to the recipe either + named the same as the base name of the recipe + (<ulink url='&YOCTO_DOCS_REF_URL;#var-BP'><filename>BP</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>) + or "files". + </para> + </section> + + <section id='new-recipe-licensing'> + <title>Licensing</title> + + <para> + Your recipe needs to have both the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + variables: + <itemizedlist> + <listitem><para><emphasis><filename>LICENSE</filename>:</emphasis> + This variable specifies the license for the software. + If you do not know the license under which the software + you are building is distributed, you should go to the + source code and look for that information. + Typical files containing this information include + <filename>COPYING</filename>, + <filename>LICENSE</filename>, and + <filename>README</filename> files. + You could also find the information near the top of + a source file. + For example, given a piece of software licensed under + the GNU General Public License version 2, you would + set <filename>LICENSE</filename> as follows: + <literallayout class='monospaced'> + LICENSE = "GPLv2" + </literallayout></para> + <para>The licenses you specify within + <filename>LICENSE</filename> can have any name as long + as you do not use spaces, since spaces are used as + separators between license names. + For standard licenses, use the names of the files in + <filename>meta/files/common-licenses/</filename> + or the <filename>SPDXLICENSEMAP</filename> flag names + defined in <filename>meta/conf/licenses.conf</filename>. + </para></listitem> + <listitem><para><emphasis><filename>LIC_FILES_CHKSUM</filename>:</emphasis> + The OpenEmbedded build system uses this variable to + make sure the license text has not changed. + If it has, the build produces an error and it affords + you the chance to figure it out and correct the problem. + </para> + <para>You need to specify all applicable licensing + files for the software. + At the end of the configuration step, the build process + will compare the checksums of the files to be sure + the text has not changed. + Any differences result in an error with the message + containing the current checksum. + For more explanation and examples of how to set the + <filename>LIC_FILES_CHKSUM</filename> variable, see the + "<link link='usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</link>" + section.</para> + + <para>To determine the correct checksum string, you + can list the appropriate files in the + <filename>LIC_FILES_CHKSUM</filename> variable with + incorrect md5 strings, attempt to build the software, + and then note the resulting error messages that will + report the correct md5 strings. + See the + "<link linkend='new-recipe-fetching-code'>Fetching Code</link>" + section for additional information. + </para> + + <para> + Here is an example that assumes the software has a + <filename>COPYING</filename> file: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://COPYING;md5=xxx" + </literallayout> + When you try to build the software, the build system + will produce an error and give you the correct string + that you can substitute into the recipe file for a + subsequent build. + </para></listitem> + </itemizedlist> + </para> + +<!-- + + <para> + For trying this out I created a new recipe named + <filename>htop_1.0.2.bb</filename> and put it in + <filename>poky/meta/recipes-extended/htop</filename>. + There are two license type statements in my very simple + recipe: + <literallayout class='monospaced'> + LICENSE = "" + + LIC_FILES_CHKSUM = "" + + SRC_URI[md5sum] = "" + SRC_URI[sha256sum] = "" + </literallayout> + Evidently, you need to run a <filename>bitbake -c cleanall htop</filename>. + Next, you delete or comment out the two <filename>SRC_URI</filename> + lines at the end and then attempt to build the software with + <filename>bitbake htop</filename>. + Doing so causes BitBake to report some errors and and give + you the actual strings you need for the last two + <filename>SRC_URI</filename> lines. + Prior to this, you have to dig around in the home page of the + source for <filename>htop</filename> and determine that the + software is released under GPLv2. + You can provide that in the <filename>LICENSE</filename> + statement. + Now you edit your recipe to have those two strings for + the <filename>SRC_URI</filename> statements: + <literallayout class='monospaced'> + LICENSE = "GPLv2" + + LIC_FILES_CHKSUM = "" + + SRC_URI = "${SOURCEFORGE_MIRROR}/htop/htop-${PV}.tar.gz" + SRC_URI[md5sum] = "0d01cca8df3349c74569cefebbd9919e" + SRC_URI[sha256sum] = "ee60657b044ece0df096c053060df7abf3cce3a568ab34d260049e6a37ccd8a1" + </literallayout> + At this point, you can build the software again using the + <filename>bitbake htop</filename> command. + There is just a set of errors now associated with the + empty <filename>LIC_FILES_CHKSUM</filename> variable now. + </para> +--> + + </section> + + <section id='new-dependencies'> + <title>Dependencies</title> + + <para> + Most software packages have a short list of other packages + that they require, which are called dependencies. + These dependencies fall into two main categories: build-time + dependencies, which are required when the software is built; + and runtime dependencies, which are required to be installed + on the target in order for the software to run. + </para> + + <para> + Within a recipe, you specify build-time dependencies using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable. + Although nuances exist, items specified in + <filename>DEPENDS</filename> should be names of other recipes. + It is important that you specify all build-time dependencies + explicitly. + If you do not, due to the parallel nature of BitBake's + execution, you can end up with a race condition where the + dependency is present for one task of a recipe (e.g. + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>) + and then gone when the next task runs (e.g. + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>). + </para> + + <para> + Another consideration is that configure scripts might + automatically check for optional dependencies and enable + corresponding functionality if those dependencies are found. + This behavior means that to ensure deterministic results and + thus avoid more race conditions, you need to either explicitly + specify these dependencies as well, or tell the configure + script explicitly to disable the functionality. + If you wish to make a recipe that is more generally useful + (e.g. publish the recipe in a layer for others to use), + instead of hard-disabling the functionality, you can use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG'><filename>PACKAGECONFIG</filename></ulink> + variable to allow functionality and the corresponding + dependencies to be enabled and disabled easily by other + users of the recipe. + </para> + + <para> + Similar to build-time dependencies, you specify runtime + dependencies through a variable - + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + which is package-specific. + All variables that are package-specific need to have the name + of the package added to the end as an override. + Since the main package for a recipe has the same name as the + recipe, and the recipe's name can be found through the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename> + variable, then you specify the dependencies for the main + package by setting <filename>RDEPENDS_${PN}</filename>. + If the package were named <filename>${PN}-tools</filename>, + then you would set <filename>RDEPENDS_${PN}-tools</filename>, + and so forth. + </para> + + <para> + Some runtime dependencies will be set automatically at + packaging time. + These dependencies include any shared library dependencies + (i.e. if a package "example" contains "libexample" and + another package "mypackage" contains a binary that links to + "libexample" then the OpenEmbedded build system will + automatically add a runtime dependency to "mypackage" on + "example"). + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" + section in the Yocto Project Overview and Concepts Manual for + further details. + </para> + </section> + + <section id='new-recipe-configuring-the-recipe'> + <title>Configuring the Recipe</title> + + <para> + Most software provides some means of setting build-time + configuration options before compilation. + Typically, setting these options is accomplished by running a + configure script with some options, or by modifying a build + configuration file. + <note> + As of Yocto Project Release 1.7, some of the core recipes + that package binary configuration scripts now disable the + scripts due to the scripts previously requiring error-prone + path substitution. + The OpenEmbedded build system uses + <filename>pkg-config</filename> now, which is much more + robust. + You can find a list of the <filename>*-config</filename> + scripts that are disabled list in the + "<ulink url='&YOCTO_DOCS_REF_URL;#migration-1.7-binary-configuration-scripts-disabled'>Binary Configuration Scripts Disabled</ulink>" + section in the Yocto Project Reference Manual. + </note> + </para> + + <para> + A major part of build-time configuration is about checking for + build-time dependencies and possibly enabling optional + functionality as a result. + You need to specify any build-time dependencies for the + software you are building in your recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + value, in terms of other recipes that satisfy those + dependencies. + You can often find build-time or runtime + dependencies described in the software's documentation. + </para> + + <para> + The following list provides configuration items of note based + on how your software is built: + <itemizedlist> + <listitem><para><emphasis>Autotools:</emphasis> + If your source files have a + <filename>configure.ac</filename> file, then your + software is built using Autotools. + If this is the case, you just need to worry about + modifying the configuration.</para> + <para>When using Autotools, your recipe needs to inherit + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + class and your recipe does not have to contain a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task. + However, you might still want to make some adjustments. + For example, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink> + to pass any needed configure options that are specific + to the recipe.</para></listitem> + <listitem><para><emphasis>CMake:</emphasis> + If your source files have a + <filename>CMakeLists.txt</filename> file, then your + software is built using CMake. + If this is the case, you just need to worry about + modifying the configuration.</para> + <para>When you use CMake, your recipe needs to inherit + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-cmake'><filename>cmake</filename></ulink> + class and your recipe does not have to contain a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task. + You can make some adjustments by setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECMAKE'><filename>EXTRA_OECMAKE</filename></ulink> + to pass any needed configure options that are specific + to the recipe.</para></listitem> + <listitem><para><emphasis>Other:</emphasis> + If your source files do not have a + <filename>configure.ac</filename> or + <filename>CMakeLists.txt</filename> file, then your + software is built using some method other than Autotools + or CMake. + If this is the case, you normally need to provide a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task in your recipe + unless, of course, there is nothing to configure. + </para> + <para>Even if your software is not being built by + Autotools or CMake, you still might not need to deal + with any configuration issues. + You need to determine if configuration is even a required step. + You might need to modify a Makefile or some configuration file + used for the build to specify necessary build options. + Or, perhaps you might need to run a provided, custom + configure script with the appropriate options.</para> + <para>For the case involving a custom configure + script, you would run + <filename>./configure --help</filename> and look for + the options you need to set.</para></listitem> + </itemizedlist> + </para> + + <para> + Once configuration succeeds, it is always good practice to + look at the <filename>log.do_configure</filename> file to + ensure that the appropriate options have been enabled and no + additional build-time dependencies need to be added to + <filename>DEPENDS</filename>. + For example, if the configure script reports that it found + something not mentioned in <filename>DEPENDS</filename>, or + that it did not find something that it needed for some + desired optional functionality, then you would need to add + those to <filename>DEPENDS</filename>. + Looking at the log might also reveal items being checked for, + enabled, or both that you do not want, or items not being found + that are in <filename>DEPENDS</filename>, in which case + you would need to look at passing extra options to the + configure script as needed. + For reference information on configure options specific to the + software you are building, you can consult the output of the + <filename>./configure --help</filename> command within + <filename>${S}</filename> or consult the software's upstream + documentation. + </para> + </section> + + <section id='new-recipe-using-headers-to-interface-with-devices'> + <title>Using Headers to Interface with Devices</title> + + <para> + If your recipe builds an application that needs to + communicate with some device or needs an API into a custom + kernel, you will need to provide appropriate header files. + Under no circumstances should you ever modify the existing + <filename>meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc</filename> + file. + These headers are used to build <filename>libc</filename> and + must not be compromised with custom or machine-specific + header information. + If you customize <filename>libc</filename> through modified + headers all other applications that use + <filename>libc</filename> thus become affected. + <note><title>Warning</title> + Never copy and customize the <filename>libc</filename> + header file (i.e. + <filename>meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc</filename>). + </note> + The correct way to interface to a device or custom kernel is + to use a separate package that provides the additional headers + for the driver or other unique interfaces. + When doing so, your application also becomes responsible for + creating a dependency on that specific provider. + </para> + + <para> + Consider the following: + <itemizedlist> + <listitem><para> + Never modify + <filename>linux-libc-headers.inc</filename>. + Consider that file to be part of the + <filename>libc</filename> system, and not something + you use to access the kernel directly. + You should access <filename>libc</filename> through + specific <filename>libc</filename> calls. + </para></listitem> + <listitem><para> + Applications that must talk directly to devices + should either provide necessary headers themselves, + or establish a dependency on a special headers package + that is specific to that driver. + </para></listitem> + </itemizedlist> + </para> + + <para> + For example, suppose you want to modify an existing header + that adds I/O control or network support. + If the modifications are used by a small number programs, + providing a unique version of a header is easy and has little + impact. + When doing so, bear in mind the guidelines in the previous + list. + <note> + If for some reason your changes need to modify the behavior + of the <filename>libc</filename>, and subsequently all + other applications on the system, use a + <filename>.bbappend</filename> to modify the + <filename>linux-kernel-headers.inc</filename> file. + However, take care to not make the changes + machine specific. + </note> + </para> + + <para> + Consider a case where your kernel is older and you need + an older <filename>libc</filename> ABI. + The headers installed by your recipe should still be a + standard mainline kernel, not your own custom one. + </para> + + <para> + When you use custom kernel headers you need to get them from + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_KERNEL_DIR'><filename>STAGING_KERNEL_DIR</filename></ulink>, + which is the directory with kernel headers that are + required to build out-of-tree modules. + Your recipe will also need the following: + <literallayout class='monospaced'> + do_configure[depends] += "virtual/kernel:do_shared_workdir" + </literallayout> + </para> + </section> + + <section id='new-recipe-compilation'> + <title>Compilation</title> + + <para> + During a build, the <filename>do_compile</filename> task + happens after source is fetched, unpacked, and configured. + If the recipe passes through <filename>do_compile</filename> + successfully, nothing needs to be done. + </para> + + <para> + However, if the compile step fails, you need to diagnose the + failure. + Here are some common issues that cause failures. + <note> + For cases where improper paths are detected for + configuration files or for when libraries/headers cannot + be found, be sure you are using the more robust + <filename>pkg-config</filename>. + See the note in section + "<link linkend='new-recipe-configuring-the-recipe'>Configuring the Recipe</link>" + for additional information. + </note> + <itemizedlist> + <listitem><para><emphasis>Parallel build failures:</emphasis> + These failures manifest themselves as intermittent + errors, or errors reporting that a file or directory + that should be created by some other part of the build + process could not be found. + This type of failure can occur even if, upon inspection, + the file or directory does exist after the build has + failed, because that part of the build process happened + in the wrong order.</para> + <para>To fix the problem, you need to either satisfy + the missing dependency in the Makefile or whatever + script produced the Makefile, or (as a workaround) + set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + to an empty string: + <literallayout class='monospaced'> + PARALLEL_MAKE = "" + </literallayout></para> + <para> + For information on parallel Makefile issues, see the + "<link linkend='debugging-parallel-make-races'>Debugging Parallel Make Races</link>" + section. + </para></listitem> + <listitem><para><emphasis>Improper host path usage:</emphasis> + This failure applies to recipes building for the target + or <filename>nativesdk</filename> only. + The failure occurs when the compilation process uses + improper headers, libraries, or other files from the + host system when cross-compiling for the target. + </para> + <para>To fix the problem, examine the + <filename>log.do_compile</filename> file to identify + the host paths being used (e.g. + <filename>/usr/include</filename>, + <filename>/usr/lib</filename>, and so forth) and then + either add configure options, apply a patch, or do both. + </para></listitem> + <listitem><para><emphasis>Failure to find required + libraries/headers:</emphasis> + If a build-time dependency is missing because it has + not been declared in + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + or because the dependency exists but the path used by + the build process to find the file is incorrect and the + configure step did not detect it, the compilation + process could fail. + For either of these failures, the compilation process + notes that files could not be found. + In these cases, you need to go back and add additional + options to the configure script as well as possibly + add additional build-time dependencies to + <filename>DEPENDS</filename>.</para> + <para>Occasionally, it is necessary to apply a patch + to the source to ensure the correct paths are used. + If you need to specify paths to find files staged + into the sysroot from other recipes, use the variables + that the OpenEmbedded build system provides + (e.g. + <filename>STAGING_BINDIR</filename>, + <filename>STAGING_INCDIR</filename>, + <filename>STAGING_DATADIR</filename>, and so forth). +<!-- + (e.g. + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_BINDIR'><filename>STAGING_BINDIR</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_INCDIR'><filename>STAGING_INCDIR</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DATADIR'><filename>STAGING_DATADIR</filename></ulink>, + and so forth). +--> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-installing'> + <title>Installing</title> + + <para> + During <filename>do_install</filename>, the task copies the + built files along with their hierarchy to locations that + would mirror their locations on the target device. + The installation process copies files from the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename>, + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink><filename>}</filename>, + and + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename> + directories to the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename> + directory to create the structure as it should appear on the + target system. + </para> + + <para> + How your software is built affects what you must do to be + sure your software is installed correctly. + The following list describes what you must do for installation + depending on the type of build system used by the software + being built: + <itemizedlist> + <listitem><para><emphasis>Autotools and CMake:</emphasis> + If the software your recipe is building uses Autotools + or CMake, the OpenEmbedded build + system understands how to install the software. + Consequently, you do not have to have a + <filename>do_install</filename> task as part of your + recipe. + You just need to make sure the install portion of the + build completes with no issues. + However, if you wish to install additional files not + already being installed by + <filename>make install</filename>, you should do this + using a <filename>do_install_append</filename> function + using the install command as described in + the "Manual" bulleted item later in this list. + </para></listitem> + <listitem><para><emphasis>Other (using + <filename>make install</filename>):</emphasis> + You need to define a + <filename>do_install</filename> function in your + recipe. + The function should call + <filename>oe_runmake install</filename> and will likely + need to pass in the destination directory as well. + How you pass that path is dependent on how the + <filename>Makefile</filename> being run is written + (e.g. <filename>DESTDIR=${D}</filename>, + <filename>PREFIX=${D}</filename>, + <filename>INSTALLROOT=${D}</filename>, and so forth). + </para> + <para>For an example recipe using + <filename>make install</filename>, see the + "<link linkend='new-recipe-makefile-based-package'>Makefile-Based Package</link>" + section.</para></listitem> + <listitem><para><emphasis>Manual:</emphasis> + You need to define a + <filename>do_install</filename> function in your + recipe. + The function must first use + <filename>install -d</filename> to create the + directories under + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename>. + Once the directories exist, your function can use + <filename>install</filename> to manually install the + built software into the directories.</para> + <para>You can find more information on + <filename>install</filename> at + <ulink url='http://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html'></ulink>. + </para></listitem> + </itemizedlist> + </para> + + <para> + For the scenarios that do not use Autotools or + CMake, you need to track the installation + and diagnose and fix any issues until everything installs + correctly. + You need to look in the default location of + <filename>${D}</filename>, which is + <filename>${WORKDIR}/image</filename>, to be sure your + files have been installed correctly. + </para> + + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + During the installation process, you might need to + modify some of the installed files to suit the target + layout. + For example, you might need to replace hard-coded paths + in an initscript with values of variables provided by + the build system, such as replacing + <filename>/usr/bin/</filename> with + <filename>${bindir}</filename>. + If you do perform such modifications during + <filename>do_install</filename>, be sure to modify the + destination file after copying rather than before + copying. + Modifying after copying ensures that the build system + can re-execute <filename>do_install</filename> if + needed. + </para></listitem> + <listitem><para> + <filename>oe_runmake install</filename>, which can be + run directly or can be run indirectly by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-cmake'><filename>cmake</filename></ulink> + classes, runs <filename>make install</filename> in + parallel. + Sometimes, a Makefile can have missing dependencies + between targets that can result in race conditions. + If you experience intermittent failures during + <filename>do_install</filename>, you might be able to + work around them by disabling parallel Makefile + installs by adding the following to the recipe: + <literallayout class='monospaced'> + PARALLEL_MAKEINST = "" + </literallayout> + See + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> + for additional information. + </para></listitem> + </itemizedlist> + </note> + </section> + + <section id='new-recipe-enabling-system-services'> + <title>Enabling System Services</title> + + <para> + If you want to install a service, which is a process that + usually starts on boot and runs in the background, then + you must include some additional definitions in your recipe. + </para> + + <para> + If you are adding services and the service initialization + script or the service file itself is not installed, you must + provide for that installation in your recipe using a + <filename>do_install_append</filename> function. + If your recipe already has a <filename>do_install</filename> + function, update the function near its end rather than + adding an additional <filename>do_install_append</filename> + function. + </para> + + <para> + When you create the installation for your services, you need + to accomplish what is normally done by + <filename>make install</filename>. + In other words, make sure your installation arranges the output + similar to how it is arranged on the target system. + </para> + + <para> + The OpenEmbedded build system provides support for starting + services two different ways: + <itemizedlist> + <listitem><para><emphasis>SysVinit:</emphasis> + SysVinit is a system and service manager that + manages the init system used to control the very basic + functions of your system. + The init program is the first program + started by the Linux kernel when the system boots. + Init then controls the startup, running and shutdown + of all other programs.</para> + <para>To enable a service using SysVinit, your recipe + needs to inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-update-rc.d'><filename>update-rc.d</filename></ulink> + class. + The class helps facilitate safely installing the + package on the target.</para> + <para>You will need to set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_PACKAGES'><filename>INITSCRIPT_PACKAGES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_NAME'><filename>INITSCRIPT_NAME</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_PARAMS'><filename>INITSCRIPT_PARAMS</filename></ulink> + variables within your recipe.</para></listitem> + <listitem><para><emphasis>systemd:</emphasis> + System Management Daemon (systemd) was designed to + replace SysVinit and to provide + enhanced management of services. + For more information on systemd, see the systemd + homepage at + <ulink url='http://freedesktop.org/wiki/Software/systemd/'></ulink>. + </para> + <para>To enable a service using systemd, your recipe + needs to inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-systemd'><filename>systemd</filename></ulink> + class. + See the <filename>systemd.bbclass</filename> file + located in your + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + section for more information. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-packaging'> + <title>Packaging</title> + + <para> + Successful packaging is a combination of automated processes + performed by the OpenEmbedded build system and some + specific steps you need to take. + The following list describes the process: + <itemizedlist> + <listitem><para><emphasis>Splitting Files</emphasis>: + The <filename>do_package</filename> task splits the + files produced by the recipe into logical components. + Even software that produces a single binary might + still have debug symbols, documentation, and other + logical components that should be split out. + The <filename>do_package</filename> task ensures + that files are split up and packaged correctly. + </para></listitem> + <listitem><para><emphasis>Running QA Checks</emphasis>: + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> + class adds a step to + the package generation process so that output quality + assurance checks are generated by the OpenEmbedded + build system. + This step performs a range of checks to be sure the + build's output is free of common problems that show + up during runtime. + For information on these checks, see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> + class and the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-qa-checks'>QA Error and Warning Messages</ulink>" + chapter in the Yocto Project Reference Manual. + </para></listitem> + <listitem><para><emphasis>Hand-Checking Your Packages</emphasis>: + After you build your software, you need to be sure + your packages are correct. + Examine the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/packages-split</filename> + directory and make sure files are where you expect + them to be. + If you discover problems, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>, + <filename>do_install(_append)</filename>, and so forth as + needed. + </para></listitem> + <listitem><para><emphasis>Splitting an Application into Multiple Packages</emphasis>: + If you need to split an application into several + packages, see the + "<link linkend='splitting-an-application-into-multiple-packages'>Splitting an Application into Multiple Packages</link>" + section for an example. + </para></listitem> + <listitem><para><emphasis>Installing a Post-Installation Script</emphasis>: + For an example showing how to install a + post-installation script, see the + "<link linkend='new-recipe-post-installation-scripts'>Post-Installation Scripts</link>" + section. + </para></listitem> + <listitem><para><emphasis>Marking Package Architecture</emphasis>: + Depending on what your recipe is building and how it + is configured, it might be important to mark the + packages produced as being specific to a particular + machine, or to mark them as not being specific to + a particular machine or architecture at all.</para> + <para>By default, packages apply to any machine with the + same architecture as the target machine. + When a recipe produces packages that are + machine-specific (e.g. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + value is passed into the configure script or a patch + is applied only for a particular machine), you should + mark them as such by adding the following to the + recipe: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${MACHINE_ARCH}" + </literallayout></para> + <para>On the other hand, if the recipe produces packages + that do not contain anything specific to the target + machine or architecture at all (e.g. recipes + that simply package script files or configuration + files), you should use the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> + class to do this for you by adding this to your + recipe: + <literallayout class='monospaced'> + inherit allarch + </literallayout> + Ensuring that the package architecture is correct is + not critical while you are doing the first few builds + of your recipe. + However, it is important in order + to ensure that your recipe rebuilds (or does not + rebuild) appropriately in response to changes in + configuration, and to ensure that you get the + appropriate packages installed on the target machine, + particularly if you run separate builds for more + than one target machine. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-sharing-files-between-recipes'> + <title>Sharing Files Between Recipes</title> + + <para> + Recipes often need to use files provided by other recipes on + the build host. + For example, an application linking to a common library needs + access to the library itself and its associated headers. + The way this access is accomplished is by populating a sysroot + with files. + Each recipe has two sysroots in its work directory, one for + target files + (<filename>recipe-sysroot</filename>) and one for files that + are native to the build host + (<filename>recipe-sysroot-native</filename>). + <note> + You could find the term "staging" used within the Yocto + project regarding files populating sysroots (e.g. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DIR'><filename>STAGING_DIR</filename></ulink> + variable). + </note> + </para> + + <para> + Recipes should never populate the sysroot directly (i.e. write + files into sysroot). + Instead, files should be installed into standard locations + during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task within the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename> + directory. + The reason for this limitation is that almost all files that + populate the sysroot are cataloged in manifests in order to + ensure the files can be removed later when a recipe is either + modified or removed. + Thus, the sysroot is able to remain free from stale files. + </para> + + <para> + A subset of the files installed by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task are used by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> + task as defined by the the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SYSROOT_DIRS'><filename>SYSROOT_DIRS</filename></ulink> + variable to automatically populate the sysroot. + It is possible to modify the list of directories that populate + the sysroot. + The following example shows how you could add the + <filename>/opt</filename> directory to the list of + directories within a recipe: + <literallayout class='monospaced'> + SYSROOT_DIRS += "/opt" + </literallayout> + </para> + + <para> + For a more complete description of the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> + task and its associated functions, see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-staging'><filename>staging</filename></ulink> + class. + </para> + </section> + + <section id='metadata-virtual-providers'> + <title>Using Virtual Providers</title> + + <para> + Prior to a build, if you know that several different recipes + provide the same functionality, you can use a virtual provider + (i.e. <filename>virtual/*</filename>) as a placeholder for the + actual provider. + The actual provider is determined at build-time. + </para> + + <para> + A common scenario where a virtual provider is used would be + for the kernel recipe. + Suppose you have three kernel recipes whose + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> + values map to <filename>kernel-big</filename>, + <filename>kernel-mid</filename>, and + <filename>kernel-small</filename>. + Furthermore, each of these recipes in some way uses a + <ulink url='&YOCTO_DOCS_REF_URL;#var-PROVIDES'><filename>PROVIDES</filename></ulink> + statement that essentially identifies itself as being able + to provide <filename>virtual/kernel</filename>. + Here is one way through the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-kernel'><filename>kernel</filename></ulink> + class: + <literallayout class='monospaced'> + PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }" + </literallayout> + Any recipe that inherits the <filename>kernel</filename> class + is going to utilize a <filename>PROVIDES</filename> statement + that identifies that recipe as being able to provide the + <filename>virtual/kernel</filename> item. + </para> + + <para> + Now comes the time to actually build an image and you need a + kernel recipe, but which one? + You can configure your build to call out the kernel recipe + you want by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PREFERRED_PROVIDER'><filename>PREFERRED_PROVIDER</filename></ulink> + variable. + As an example, consider the + <ulink url='https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/conf/machine/include/x86-base.inc'><filename>x86-base.inc</filename></ulink> + include file, which is a machine + (i.e. <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>) + configuration file. + This include file is the reason all x86-based machines use the + <filename>linux-yocto</filename> kernel. + Here are the relevant lines from the include file: + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto" + PREFERRED_VERSION_linux-yocto ??= "4.15%" + </literallayout> + </para> + + <para> + When you use a virtual provider, you do not have to + "hard code" a recipe name as a build dependency. + You can use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable to state the build is dependent on + <filename>virtual/kernel</filename> for example: + <literallayout class='monospaced'> + DEPENDS = "virtual/kernel" + </literallayout> + During the build, the OpenEmbedded build system picks + the correct recipe needed for the + <filename>virtual/kernel</filename> dependency based on the + <filename>PREFERRED_PROVIDER</filename> variable. + If you want to use the small kernel mentioned at the beginning + of this section, configure your build as follows: + <literallayout class='monospaced'> + PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small" + </literallayout> + <note> + Any recipe that + <ulink url='&YOCTO_DOCS_REF_URL;#var-PROVIDES'><filename>PROVIDES</filename></ulink> + a <filename>virtual/*</filename> item that is ultimately + not selected through + <filename>PREFERRED_PROVIDER</filename> does not get built. + Preventing these recipes from building is usually the + desired behavior since this mechanism's purpose is to + select between mutually exclusive alternative providers. + </note> + </para> + + <para> + The following lists specific examples of virtual providers: + <itemizedlist> + <listitem><para> + <filename>virtual/kernel</filename>: + Provides the name of the kernel recipe to use when + building a kernel image. + </para></listitem> + <listitem><para> + <filename>virtual/bootloader</filename>: + Provides the name of the bootloader to use when + building an image. + </para></listitem> + <listitem><para> + <filename>virtual/mesa</filename>: + Provides <filename>gbm.pc</filename>. + </para></listitem> + <listitem><para> + <filename>virtual/egl</filename>: + Provides <filename>egl.pc</filename> and possibly + <filename>wayland-egl.pc</filename>. + </para></listitem> + <listitem><para> + <filename>virtual/libgl</filename>: + Provides <filename>gl.pc</filename> (i.e. libGL). + </para></listitem> + <listitem><para> + <filename>virtual/libgles1</filename>: + Provides <filename>glesv1_cm.pc</filename> + (i.e. libGLESv1_CM). + </para></listitem> + <listitem><para> + <filename>virtual/libgles2</filename>: + Provides <filename>glesv2.pc</filename> + (i.e. libGLESv2). + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='properly-versioning-pre-release-recipes'> + <title>Properly Versioning Pre-Release Recipes</title> + + <para> + Sometimes the name of a recipe can lead to versioning + problems when the recipe is upgraded to a final release. + For example, consider the + <filename>irssi_0.8.16-rc1.bb</filename> recipe file in + the list of example recipes in the + "<link linkend='new-recipe-storing-and-naming-the-recipe'>Storing and Naming the Recipe</link>" + section. + This recipe is at a release candidate stage (i.e. + "rc1"). + When the recipe is released, the recipe filename becomes + <filename>irssi_0.8.16.bb</filename>. + The version change from <filename>0.8.16-rc1</filename> + to <filename>0.8.16</filename> is seen as a decrease by the + build system and package managers, so the resulting packages + will not correctly trigger an upgrade. + </para> + + <para> + In order to ensure the versions compare properly, the + recommended convention is to set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + within the recipe to + "<replaceable>previous_version</replaceable>+<replaceable>current_version</replaceable>". + You can use an additional variable so that you can use the + current version elsewhere. + Here is an example: + <literallayout class='monospaced'> + REALPV = "0.8.16-rc1" + PV = "0.8.15+${REALPV}" + </literallayout> + </para> + </section> + + <section id='new-recipe-post-installation-scripts'> + <title>Post-Installation Scripts</title> + + <para> + Post-installation scripts run immediately after installing + a package on the target or during image creation when a + package is included in an image. + To add a post-installation script to a package, add a + <filename>pkg_postinst_</filename><replaceable>PACKAGENAME</replaceable><filename>()</filename> function to + the recipe file (<filename>.bb</filename>) and replace + <replaceable>PACKAGENAME</replaceable> with the name of the package + you want to attach to the <filename>postinst</filename> + script. + To apply the post-installation script to the main package + for the recipe, which is usually what is required, specify + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename> + in place of <replaceable>PACKAGENAME</replaceable>. + </para> + + <para> + A post-installation function has the following structure: + <literallayout class='monospaced'> + pkg_postinst_<replaceable>PACKAGENAME</replaceable>() { + # Commands to carry out + } + </literallayout> + </para> + + <para> + The script defined in the post-installation function is + called when the root filesystem is created. + If the script succeeds, the package is marked as installed. + If the script fails, the package is marked as unpacked and + the script is executed when the image boots again. + <note> + Any RPM post-installation script that runs on the target + should return a 0 exit code. + RPM does not allow non-zero exit codes for these scripts, + and the RPM package manager will cause the package to fail + installation on the target. + </note> + </para> + + <para> + Sometimes it is necessary for the execution of a + post-installation script to be delayed until the first boot. + For example, the script might need to be executed on the + device itself. + To delay script execution until boot time, use the following + structure in the post-installation script: + <literallayout class='monospaced'> + pkg_postinst_<replaceable>PACKAGENAME</replaceable>() { + if [ x"$D" = "x" ]; then + # Actions to carry out on the device go here + else + exit 1 + fi + } + </literallayout> + </para> + + <para> + The previous example delays execution until the image boots + again because the environment variable <filename>D</filename> + points to the directory containing the image when + the root filesystem is created at build time but is unset + when executed on the first boot. + </para> + + <para> + If you have recipes that use <filename>pkg_postinst</filename> + scripts and they require the use of non-standard native + tools that have dependencies during rootfs construction, you + need to use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_WRITE_DEPS'><filename>PACKAGE_WRITE_DEPS</filename></ulink> + variable in your recipe to list these tools. + If you do not use this variable, the tools might be missing and + execution of the post-installation script is deferred until + first boot. + Deferring the script to first boot is undesirable and for + read-only rootfs impossible. + </para> + + <note> + Equivalent support for pre-install, pre-uninstall, and + post-uninstall scripts exist by way of + <filename>pkg_preinst</filename>, + <filename>pkg_prerm</filename>, and + <filename>pkg_postrm</filename>, respectively. + These scrips work in exactly the same way as does + <filename>pkg_postinst</filename> with the exception that they + run at different times. + Also, because of when they run, they are not applicable to + being run at image creation time like + <filename>pkg_postinst</filename>. + </note> + </section> + + <section id='new-recipe-testing'> + <title>Testing</title> + + <para> + The final step for completing your recipe is to be sure that + the software you built runs correctly. + To accomplish runtime testing, add the build's output + packages to your image and test them on the target. + </para> + + <para> + For information on how to customize your image by adding + specific packages, see the + "<link linkend='usingpoky-extend-customimage'>Customizing Images</link>" + section. + </para> + </section> + + <section id='new-recipe-testing-examples'> + <title>Examples</title> + + <para> + To help summarize how to write a recipe, this section provides + some examples given various scenarios: + <itemizedlist> + <listitem><para>Recipes that use local files</para></listitem> + <listitem><para>Using an Autotooled package</para></listitem> + <listitem><para>Using a Makefile-based package</para></listitem> + <listitem><para>Splitting an application into multiple packages</para></listitem> + <listitem><para>Adding binaries to an image</para></listitem> + </itemizedlist> + </para> + + <section id='new-recipe-single-c-file-package-hello-world'> + <title>Single .c File Package (Hello World!)</title> + + <para> + Building an application from a single file that is stored + locally (e.g. under <filename>files</filename>) requires + a recipe that has the file listed in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename> + variable. + Additionally, you need to manually write the + <filename>do_compile</filename> and + <filename>do_install</filename> tasks. + The <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'>S</ulink></filename> + variable defines the directory containing the source code, + which is set to + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> + in this case - the directory BitBake uses for the build. + <literallayout class='monospaced'> + SUMMARY = "Simple helloworld application" + SECTION = "examples" + LICENSE = "MIT" + LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + + SRC_URI = "file://helloworld.c" + + S = "${WORKDIR}" + + do_compile() { + ${CC} helloworld.c -o helloworld + } + + do_install() { + install -d ${D}${bindir} + install -m 0755 helloworld ${D}${bindir} + } + </literallayout> + </para> + + <para> + By default, the <filename>helloworld</filename>, + <filename>helloworld-dbg</filename>, and + <filename>helloworld-dev</filename> packages are built. + For information on how to customize the packaging process, + see the + "<link linkend='splitting-an-application-into-multiple-packages'>Splitting an Application into Multiple Packages</link>" + section. + </para> + </section> + + <section id='new-recipe-autotooled-package'> + <title>Autotooled Package</title> + <para> + Applications that use Autotools such as <filename>autoconf</filename> and + <filename>automake</filename> require a recipe that has a source archive listed in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename> and + also inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + class, which contains the definitions of all the steps + needed to build an Autotool-based application. + The result of the build is automatically packaged. + And, if the application uses NLS for localization, packages with local information are + generated (one package per language). + Following is one example: (<filename>hello_2.3.bb</filename>) + <literallayout class='monospaced'> + SUMMARY = "GNU Helloworld application" + SECTION = "examples" + LICENSE = "GPLv2+" + LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe" + + SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz" + + inherit autotools gettext + </literallayout> + </para> + + <para> + The variable + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</ulink></filename> + is used to track source license changes as described in the + "<link linkend='usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</link>" + section in the Yocto Project Overview and Concepts Manual. + You can quickly create Autotool-based recipes in a manner + similar to the previous example. + </para> + </section> + + <section id='new-recipe-makefile-based-package'> + <title>Makefile-Based Package</title> + + <para> + Applications that use GNU <filename>make</filename> also require a recipe that has + the source archive listed in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename>. + You do not need to add a <filename>do_compile</filename> step since by default BitBake + starts the <filename>make</filename> command to compile the application. + If you need additional <filename>make</filename> options, you should store them in the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OEMAKE'><filename>EXTRA_OEMAKE</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink> + variables. + BitBake passes these options into the GNU <filename>make</filename> invocation. + Note that a <filename>do_install</filename> task is still required. + Otherwise, BitBake runs an empty <filename>do_install</filename> task by default. + </para> + + <para> + Some applications might require extra parameters to be passed to the compiler. + For example, the application might need an additional header path. + You can accomplish this by adding to the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-CFLAGS'>CFLAGS</ulink></filename> variable. + The following example shows this: + <literallayout class='monospaced'> + CFLAGS_prepend = "-I ${S}/include " + </literallayout> + </para> + + <para> + In the following example, <filename>mtd-utils</filename> is a makefile-based package: + <literallayout class='monospaced'> + SUMMARY = "Tools for managing memory technology devices" + SECTION = "base" + DEPENDS = "zlib lzo e2fsprogs util-linux" + HOMEPAGE = "http://www.linux-mtd.infradead.org/" + LICENSE = "GPLv2+" + LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \ + file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c" + + # Use the latest version at 26 Oct, 2013 + SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b" + SRC_URI = "git://git.infradead.org/mtd-utils.git \ + file://add-exclusion-to-mkfs-jffs2-git-2.patch \ + " + + PV = "1.5.1+git${SRCPV}" + + S = "${WORKDIR}/git" + + EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'" + + do_install () { + oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir} + } + + PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc" + + FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool" + FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*" + FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image" + + PARALLEL_MAKE = "" + + BBCLASSEXTEND = "native" + </literallayout> + </para> + </section> + + <section id='splitting-an-application-into-multiple-packages'> + <title>Splitting an Application into Multiple Packages</title> + + <para> + You can use the variables + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'>PACKAGES</ulink></filename> and + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'>FILES</ulink></filename> + to split an application into multiple packages. + </para> + + <para> + Following is an example that uses the <filename>libxpm</filename> recipe. + By default, this recipe generates a single package that contains the library along + with a few binaries. + You can modify the recipe to split the binaries into separate packages: + <literallayout class='monospaced'> + require xorg-lib-common.inc + + SUMMARY = "Xpm: X Pixmap extension library" + LICENSE = "BSD" + LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7" + DEPENDS += "libxext libsm libxt" + PE = "1" + + XORG_PN = "libXpm" + + PACKAGES =+ "sxpm cxpm" + FILES_cxpm = "${bindir}/cxpm" + FILES_sxpm = "${bindir}/sxpm" + </literallayout> + </para> + + <para> + In the previous example, we want to ship the <filename>sxpm</filename> + and <filename>cxpm</filename> binaries in separate packages. + Since <filename>bindir</filename> would be packaged into the main + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'>PN</ulink></filename> + package by default, we prepend the <filename>PACKAGES</filename> + variable so additional package names are added to the start of list. + This results in the extra <filename>FILES_*</filename> + variables then containing information that define which files and + directories go into which packages. + Files included by earlier packages are skipped by latter packages. + Thus, the main <filename>PN</filename> package + does not include the above listed files. + </para> + </section> + + <section id='packaging-externally-produced-binaries'> + <title>Packaging Externally Produced Binaries</title> + + <para> + Sometimes, you need to add pre-compiled binaries to an + image. + For example, suppose that binaries for proprietary code + exist, which are created by a particular division of a + company. + Your part of the company needs to use those binaries as + part of an image that you are building using the + OpenEmbedded build system. + Since you only have the binaries and not the source code, + you cannot use a typical recipe that expects to fetch the + source specified in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + and then compile it. + </para> + + <para> + One method is to package the binaries and then install them + as part of the image. + Generally, it is not a good idea to package binaries + since, among other things, it can hinder the ability to + reproduce builds and could lead to compatibility problems + with ABI in the future. + However, sometimes you have no choice. + </para> + + <para> + The easiest solution is to create a recipe that uses + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-bin-package'><filename>bin_package</filename></ulink> + class and to be sure that you are using default locations + for build artifacts. + In most cases, the <filename>bin_package</filename> class + handles "skipping" the configure and compile steps as well + as sets things up to grab packages from the appropriate + area. + In particular, this class sets <filename>noexec</filename> + on both the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + tasks, sets + <filename>FILES_${PN}</filename> to "/" so that it picks + up all files, and sets up a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task, which effectively copies all files from + <filename>${S}</filename> to <filename>${D}</filename>. + The <filename>bin_package</filename> class works well when + the files extracted into <filename>${S}</filename> are + already laid out in the way they should be laid out + on the target. + For more information on these variables, see the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> + variables in the Yocto Project Reference Manual's variable + glossary. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + Using + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + is a good idea even for components distributed + in binary form, and is often necessary for + shared libraries. + For a shared library, listing the library + dependencies in + <filename>DEPENDS</filename> makes sure that + the libraries are available in the staging + sysroot when other recipes link against the + library, which might be necessary for + successful linking. + </para></listitem> + <listitem><para> + Using <filename>DEPENDS</filename> also + allows runtime dependencies between packages + to be added automatically. + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#automatically-added-runtime-dependencies'>Automatically Added Runtime Dependencies</ulink>" + section in the Yocto Project Overview and + Concepts Manual for more information. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + If you cannot use the <filename>bin_package</filename> + class, you need to be sure you are doing the following: + <itemizedlist> + <listitem><para> + Create a recipe where the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + tasks do nothing: + It is usually sufficient to just not define these + tasks in the recipe, because the default + implementations do nothing unless a Makefile is + found in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename>. + </para> + + <para>If + <filename>${S}</filename> might contain a Makefile, + or if you inherit some class that replaces + <filename>do_configure</filename> and + <filename>do_compile</filename> with custom + versions, then you can use the + <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>noexec</filename></ulink><filename>]</filename> + flag to turn the tasks into no-ops, as follows: + <literallayout class='monospaced'> + do_configure[noexec] = "1" + do_compile[noexec] = "1" + </literallayout> + Unlike + <ulink url='&YOCTO_DOCS_BB_URL;#deleting-a-task'><filename>deleting the tasks</filename></ulink>, + using the flag preserves the dependency chain from + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>, <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + tasks to the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task. + </para></listitem> + <listitem><para>Make sure your + <filename>do_install</filename> task installs the + binaries appropriately. + </para></listitem> + <listitem><para>Ensure that you set up + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> + (usually + <filename>FILES_${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename>) + to point to the files you have installed, which of + course depends on where you have installed them + and whether those files are in different locations + than the defaults. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id="following-recipe-style-guidelines"> + <title>Following Recipe Style Guidelines</title> + + <para> + When writing recipes, it is good to conform to existing + style guidelines. + The + <ulink url='http://www.openembedded.org/wiki/Styleguide'>OpenEmbedded Styleguide</ulink> + wiki page provides rough guidelines for preferred recipe style. + </para> + + <para> + It is common for existing recipes to deviate a bit from this + style. + However, aiming for at least a consistent style is a good idea. + Some practices, such as omitting spaces around + <filename>=</filename> operators in assignments or ordering + recipe components in an erratic way, are widely seen as poor + style. + </para> + </section> + + <section id='recipe-syntax'> + <title>Recipe Syntax</title> + + <para> + Understanding recipe file syntax is important for writing + recipes. + The following list overviews the basic items that make up a + BitBake recipe file. + For more complete BitBake syntax descriptions, see the + "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" + chapter of the BitBake User Manual. + <itemizedlist> + <listitem><para> + <emphasis>Variable Assignments and Manipulations:</emphasis> + Variable assignments allow a value to be assigned to a + variable. + The assignment can be static text or might include + the contents of other variables. + In addition to the assignment, appending and prepending + operations are also supported.</para> + + <para>The following example shows some of the ways + you can use variables in recipes: + <literallayout class='monospaced'> + S = "${WORKDIR}/postfix-${PV}" + CFLAGS += "-DNO_ASM" + SRC_URI_append = " file://fixup.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Functions:</emphasis> + Functions provide a series of actions to be performed. + You usually use functions to override the default + implementation of a task function or to complement + a default function (i.e. append or prepend to an + existing function). + Standard functions use <filename>sh</filename> shell + syntax, although access to OpenEmbedded variables and + internal methods are also available.</para> + + <para>The following is an example function from the + <filename>sed</filename> recipe: + <literallayout class='monospaced'> + do_install () { + autotools_do_install + install -d ${D}${base_bindir} + mv ${D}${bindir}/sed ${D}${base_bindir}/sed + rmdir ${D}${bindir}/ + } + </literallayout> + It is also possible to implement new functions that + are called between existing tasks as long as the + new functions are not replacing or complementing the + default functions. + You can implement functions in Python + instead of shell. + Both of these options are not seen in the majority of + recipes. + </para></listitem> + <listitem><para><emphasis>Keywords:</emphasis> + BitBake recipes use only a few keywords. + You use keywords to include common + functions (<filename>inherit</filename>), load parts + of a recipe from other files + (<filename>include</filename> and + <filename>require</filename>) and export variables + to the environment (<filename>export</filename>). + </para> + + <para>The following example shows the use of some of + these keywords: + <literallayout class='monospaced'> + export POSTCONF = "${STAGING_BINDIR}/postconf" + inherit autoconf + require otherfile.inc + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Comments (#):</emphasis> + Any lines that begin with the hash character + (<filename>#</filename>) are treated as comment lines + and are ignored: + <literallayout class='monospaced'> + # This is a comment + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + This next list summarizes the most important and most commonly + used parts of the recipe syntax. + For more information on these parts of the syntax, you can + reference the + <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink> + chapter in the BitBake User Manual. + <itemizedlist> + <listitem><para> + <emphasis>Line Continuation (\):</emphasis> + Use the backward slash (<filename>\</filename>) + character to split a statement over multiple lines. + Place the slash character at the end of the line that + is to be continued on the next line: + <literallayout class='monospaced'> + VAR = "A really long \ + line" + </literallayout> + <note> + You cannot have any characters including spaces + or tabs after the slash character. + </note> + </para></listitem> + <listitem><para> + <emphasis>Using Variables (${<replaceable>VARNAME</replaceable>}):</emphasis> + Use the <filename>${<replaceable>VARNAME</replaceable>}</filename> + syntax to access the contents of a variable: + <literallayout class='monospaced'> + SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz" + </literallayout> + <note> + It is important to understand that the value of a + variable expressed in this form does not get + substituted automatically. + The expansion of these expressions happens + on-demand later (e.g. usually when a function that + makes reference to the variable executes). + This behavior ensures that the values are most + appropriate for the context in which they are + finally used. + On the rare occasion that you do need the variable + expression to be expanded immediately, you can use + the <filename>:=</filename> operator instead of + <filename>=</filename> when you make the + assignment, but this is not generally needed. + </note> + </para></listitem> + <listitem><para> + <emphasis>Quote All Assignments ("<replaceable>value</replaceable>"):</emphasis> + Use double quotes around values in all variable + assignments (e.g. + <filename>"<replaceable>value</replaceable>"</filename>). + Following is an example: + <literallayout class='monospaced'> + VAR1 = "${OTHERVAR}" + VAR2 = "The version is ${PV}" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Conditional Assignment (?=):</emphasis> + Conditional assignment is used to assign a + value to a variable, but only when the variable is + currently unset. + Use the question mark followed by the equal sign + (<filename>?=</filename>) to make a "soft" assignment + used for conditional assignment. + Typically, "soft" assignments are used in the + <filename>local.conf</filename> file for variables + that are allowed to come through from the external + environment. + </para> + + <para>Here is an example where + <filename>VAR1</filename> is set to "New value" if + it is currently empty. + However, if <filename>VAR1</filename> has already been + set, it remains unchanged: + <literallayout class='monospaced'> + VAR1 ?= "New value" + </literallayout> + In this next example, <filename>VAR1</filename> + is left with the value "Original value": + <literallayout class='monospaced'> + VAR1 = "Original value" + VAR1 ?= "New value" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Appending (+=):</emphasis> + Use the plus character followed by the equals sign + (<filename>+=</filename>) to append values to existing + variables. + <note> + This operator adds a space between the existing + content of the variable and the new content. + </note></para> + + <para>Here is an example: + <literallayout class='monospaced'> + SRC_URI += "file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Prepending (=+):</emphasis> + Use the equals sign followed by the plus character + (<filename>=+</filename>) to prepend values to existing + variables. + <note> + This operator adds a space between the new content + and the existing content of the variable. + </note></para> + + <para>Here is an example: + <literallayout class='monospaced'> + VAR =+ "Starts" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Appending (_append):</emphasis> + Use the <filename>_append</filename> operator to + append values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + + <para>The following example shows the space being + explicitly added to the start to ensure the appended + value is not merged with the existing value: + <literallayout class='monospaced'> + SRC_URI_append = " file://fix-makefile.patch" + </literallayout> + You can also use the <filename>_append</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + SRC_URI_append_sh4 = " file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Prepending (_prepend):</emphasis> + Use the <filename>_prepend</filename> operator to + prepend values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + + <para>The following example shows the space being + explicitly added to the end to ensure the prepended + value is not merged with the existing value: + <literallayout class='monospaced'> + CFLAGS_prepend = "-I${S}/myincludes " + </literallayout> + You can also use the <filename>_prepend</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + CFLAGS_prepend_sh4 = "-I${S}/myincludes " + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Overrides:</emphasis> + You can use overrides to set a value conditionally, + typically based on how the recipe is being built. + For example, to set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-KBRANCH'><filename>KBRANCH</filename></ulink> + variable's value to "standard/base" for any target + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + except for qemuarm where it should be set to + "standard/arm-versatile-926ejs", you would do the + following: + <literallayout class='monospaced'> + KBRANCH = "standard/base" + KBRANCH_qemuarm = "standard/arm-versatile-926ejs" + </literallayout> + Overrides are also used to separate alternate values + of a variable in other situations. + For example, when setting variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + that are specific to individual packages produced by + a recipe, you should always use an override that + specifies the name of the package. + </para></listitem> + <listitem><para> + <emphasis>Indentation:</emphasis> + Use spaces for indentation rather than than tabs. + For shell functions, both currently work. + However, it is a policy decision of the Yocto Project + to use tabs in shell functions. + Realize that some layers have a policy to use spaces + for all indentation. + </para></listitem> + <listitem><para> + <emphasis>Using Python for Complex Operations:</emphasis> + For more advanced processing, it is possible to use + Python code during variable assignments (e.g. + search and replacement on a variable).</para> + + <para>You indicate Python code using the + <filename>${@<replaceable>python_code</replaceable>}</filename> + syntax for the variable assignment: + <literallayout class='monospaced'> + SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Shell Function Syntax:</emphasis> + Write shell functions as if you were writing a shell + script when you describe a list of actions to take. + You should ensure that your script works with a generic + <filename>sh</filename> and that it does not require + any <filename>bash</filename> or other shell-specific + functionality. + The same considerations apply to various system + utilities (e.g. <filename>sed</filename>, + <filename>grep</filename>, <filename>awk</filename>, + and so forth) that you might wish to use. + If in doubt, you should check with multiple + implementations - including those from BusyBox. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id="platdev-newmachine"> + <title>Adding a New Machine</title> + + <para> + Adding a new machine to the Yocto Project is a straightforward + process. + This section describes how to add machines that are similar + to those that the Yocto Project already supports. + <note> + Although well within the capabilities of the Yocto Project, + adding a totally new architecture might require + changes to <filename>gcc/glibc</filename> and to the site + information, which is beyond the scope of this manual. + </note> + </para> + + <para> + For a complete example that shows how to add a new machine, + see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'>Creating a New BSP Layer Using the <filename>bitbake-layers</filename> Script</ulink>" + section in the Yocto Project Board Support Package (BSP) + Developer's Guide. + </para> + + <section id="platdev-newmachine-conffile"> + <title>Adding the Machine Configuration File</title> + + <para> + To add a new machine, you need to add a new machine + configuration file to the layer's + <filename>conf/machine</filename> directory. + This configuration file provides details about the device + you are adding. + </para> + + <para> + The OpenEmbedded build system uses the root name of the + machine configuration file to reference the new machine. + For example, given a machine configuration file named + <filename>crownbay.conf</filename>, the build system + recognizes the machine as "crownbay". + </para> + + <para> + The most important variables you must set in your machine + configuration file or include from a lower-level configuration + file are as follows: + <itemizedlist> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_ARCH'>TARGET_ARCH</ulink></filename> + (e.g. "arm")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</ulink>_virtual/kernel</filename> + </para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'>MACHINE_FEATURES</ulink></filename> + (e.g. "apm screen wifi")</para></listitem> + </itemizedlist> + </para> + + <para> + You might also need these variables: + <itemizedlist> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLES'>SERIAL_CONSOLES</ulink></filename> + (e.g. "115200;ttyS0 115200;ttyS1")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'>KERNEL_IMAGETYPE</ulink></filename> + (e.g. "zImage")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'>IMAGE_FSTYPES</ulink></filename> + (e.g. "tar.gz jffs2")</para></listitem> + </itemizedlist> + </para> + + <para> + You can find full details on these variables in the reference + section. + You can leverage existing machine <filename>.conf</filename> + files from <filename>meta-yocto-bsp/conf/machine/</filename>. + </para> + </section> + + <section id="platdev-newmachine-kernel"> + <title>Adding a Kernel for the Machine</title> + + <para> + The OpenEmbedded build system needs to be able to build a kernel + for the machine. + You need to either create a new kernel recipe for this machine, + or extend an existing kernel recipe. + You can find several kernel recipe examples in the + Source Directory at + <filename>meta/recipes-kernel/linux</filename> + that you can use as references. + </para> + + <para> + If you are creating a new kernel recipe, normal recipe-writing + rules apply for setting up a + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename>. + Thus, you need to specify any necessary patches and set + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'>S</ulink></filename> + to point at the source code. + You need to create a <filename>do_configure</filename> task that + configures the unpacked kernel with a + <filename>defconfig</filename> file. + You can do this by using a <filename>make defconfig</filename> + command or, more commonly, by copying in a suitable + <filename>defconfig</filename> file and then running + <filename>make oldconfig</filename>. + By making use of <filename>inherit kernel</filename> and + potentially some of the <filename>linux-*.inc</filename> files, + most other functionality is centralized and the defaults of the + class normally work well. + </para> + + <para> + If you are extending an existing kernel recipe, it is usually + a matter of adding a suitable <filename>defconfig</filename> + file. + The file needs to be added into a location similar to + <filename>defconfig</filename> files used for other machines + in a given kernel recipe. + A possible way to do this is by listing the file in the + <filename>SRC_URI</filename> and adding the machine to the + expression in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-COMPATIBLE_MACHINE'>COMPATIBLE_MACHINE</ulink></filename>: + <literallayout class='monospaced'> + COMPATIBLE_MACHINE = '(qemux86|qemumips)' + </literallayout> + For more information on <filename>defconfig</filename> files, + see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>" + section in the Yocto Project Linux Kernel Development Manual. + </para> + </section> + + <section id="platdev-newmachine-formfactor"> + <title>Adding a Formfactor Configuration File</title> + + <para> + A formfactor configuration file provides information about the + target hardware for which the image is being built and information that + the build system cannot obtain from other sources such as the kernel. + Some examples of information contained in a formfactor configuration file include + framebuffer orientation, whether or not the system has a keyboard, + the positioning of the keyboard in relation to the screen, and + the screen resolution. + </para> + + <para> + The build system uses reasonable defaults in most cases. + However, if customization is + necessary, you need to create a <filename>machconfig</filename> file + in the <filename>meta/recipes-bsp/formfactor/files</filename> + directory. + This directory contains directories for specific machines such as + <filename>qemuarm</filename> and <filename>qemux86</filename>. + For information about the settings available and the defaults, see the + <filename>meta/recipes-bsp/formfactor/files/config</filename> file found in the + same area. + </para> + + <para> + Following is an example for "qemuarm" machine: + <literallayout class='monospaced'> + HAVE_TOUCHSCREEN=1 + HAVE_KEYBOARD=1 + + DISPLAY_CAN_ROTATE=0 + DISPLAY_ORIENTATION=0 + #DISPLAY_WIDTH_PIXELS=640 + #DISPLAY_HEIGHT_PIXELS=480 + #DISPLAY_BPP=16 + DISPLAY_DPI=150 + DISPLAY_SUBPIXEL_ORDER=vrgb + </literallayout> + </para> + </section> + </section> + + <section id='gs-upgrading-recipes'> + <title>Upgrading Recipes</title> + + <para> + Over time, upstream developers publish new versions for software + built by layer recipes. + It is recommended to keep recipes up-to-date with upstream + version releases. + You can use the Automated Upgrade Helper (AUH) to set up + automatic version upgrades. + Alternatively, you can use <filename>devtool upgrade</filename> + to set up semi-automatic version upgrades. + Finally, you can even manually upgrade a recipe by editing the + recipe itself. + </para> + + <section id='gs-using-the-auto-upgrade-helper'> + <title>Using the Auto Upgrade Helper (AUH)</title> + + <para> + The AUH utility works in conjunction with the + OpenEmbedded build system in order to automatically generate + upgrades for recipes based on new versions being + published upstream. + Use AUH when you want to create a service that performs the + upgrades automatically and optionally sends you an email with + the results. + </para> + + <para> + AUH allows you to update several recipes with a single use. + You can also optionally perform build and integration tests + using images with the results saved to your hard drive and + emails of results optionally sent to recipe maintainers. + Finally, AUH creates Git commits with appropriate commit + messages in the layer's tree for the changes made to recipes. + <note> + Conditions do exist when you should not use AUH to upgrade + recipes and you should instead use either + <filename>devtool upgrade</filename> or upgrade your + recipes manually: + <itemizedlist> + <listitem><para> + When AUH cannot complete the upgrade sequence. + This situation usually results because custom + patches carried by the recipe cannot be + automatically rebased to the new version. + In this case, <filename>devtool upgrade</filename> + allows you to manually resolve conflicts. + </para></listitem> + <listitem><para> + When for any reason you want fuller control over + the upgrade process. + For example, when you want special arrangements + for testing. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The following steps describe how to set up the AUH utility: + <orderedlist> + <listitem><para> + <emphasis>Be Sure the Development Host is Set Up:</emphasis> + You need to be sure that your development host is + set up to use the Yocto Project. + For information on how to set up your host, see the + "<link linkend='setting-up-the-development-host-to-use-the-yocto-project'>Preparing the Build Host</link>" + section. + </para></listitem> + <listitem><para> + <emphasis>Make Sure Git is Configured:</emphasis> + The AUH utility requires Git to be configured because + AUH uses Git to save upgrades. + Thus, you must have Git user and email configured. + The following command shows your configurations: + <literallayout class='monospaced'> + $ git config --list + </literallayout> + If you do not have the user and email configured, you + can use the following commands to do so: + <literallayout class='monospaced'> + $ git config --global user.name <replaceable>some_name</replaceable> + $ git config --global user.email <replaceable>username</replaceable>@<replaceable>domain</replaceable>.com + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Clone the AUH Repository:</emphasis> + To use AUH, you must clone the repository onto your + development host. + The following command uses Git to create a local + copy of the repository on your system: + <literallayout class='monospaced'> + $ git clone git://git.yoctoproject.org/auto-upgrade-helper + Cloning into 'auto-upgrade-helper'... + remote: Counting objects: 768, done. + remote: Compressing objects: 100% (300/300), done. + remote: Total 768 (delta 499), reused 703 (delta 434) + Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done. + Resolving deltas: 100% (499/499), done. + Checking connectivity... done. + </literallayout> + AUH is not part of the + <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OpenEmbedded-Core (OE-Core)</ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#poky'>Poky</ulink> + repositories. + </para></listitem> + <listitem><para> + <emphasis>Create a Dedicated Build Directory:</emphasis> + Run the + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>oe-init-build-env</filename></ulink> + script to create a fresh build directory that you + use exclusively for running the AUH utility: + <literallayout class='monospaced'> + $ cd ~/poky + $ source oe-init-build-env <replaceable>your_AUH_build_directory</replaceable> + </literallayout> + Re-using an existing build directory and its + configurations is not recommended as existing settings + could cause AUH to fail or behave undesirably. + </para></listitem> + <listitem><para> + <emphasis>Make Configurations in Your Local Configuration File:</emphasis> + Several settings need to exist in the + <filename>local.conf</filename> file in the build + directory you just created for AUH. + Make these following configurations: + <itemizedlist> + <listitem><para> + Enable "distrodata" as follows: + <literallayout class='monospaced'> + INHERIT =+ "distrodata" + </literallayout> + </para></listitem> + <listitem><para> + If you want to enable + <ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-build-output-quality'>Build History</ulink>, + which is optional, you need the following + lines in the + <filename>conf/local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT =+ "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + With this configuration and a successful + upgrade, a build history "diff" file appears in + the + <filename>upgrade-helper/work/recipe/buildhistory-diff.txt</filename> + file found in your build directory. + </para></listitem> + <listitem><para> + If you want to enable testing through the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> + class, which is optional, you need to have the + following set in your + <filename>conf/local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT += "testimage" + </literallayout> + <note> + If your distro does not enable by default + ptest, which Poky does, you need the + following in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " ptest" + </literallayout> + </note> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Optionally Start a vncserver:</emphasis> + If you are running in a server without an X11 session, + you need to start a vncserver: + <literallayout class='monospaced'> + $ vncserver :1 + $ export DISPLAY=:1 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Create and Edit an AUH Configuration File:</emphasis> + You need to have the + <filename>upgrade-helper/upgrade-helper.conf</filename> + configuration file in your build directory. + You can find a sample configuration file in the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/'>AUH source repository</ulink>. + </para> + + <para>Read through the sample file and make + configurations as needed. + For example, if you enabled build history in your + <filename>local.conf</filename> as described earlier, + you must enable it in + <filename>upgrade-helper.conf</filename>.</para> + + <para>Also, if you are using the default + <filename>maintainers.inc</filename> file supplied + with Poky and located in + <filename>meta-yocto</filename> and you do not set a + "maintainers_whitelist" or "global_maintainer_override" + in the <filename>upgrade-helper.conf</filename> + configuration, and you specify "-e all" on the + AUH command-line, the utility automatically sends out + emails to all the default maintainers. + Please avoid this. + </para></listitem> + </orderedlist> + </para> + + <para> + This next set of examples describes how to use the AUH: + <itemizedlist> + <listitem><para> + <emphasis>Upgrading a Specific Recipe:</emphasis> + To upgrade a specific recipe, use the following + form: + <literallayout class='monospaced'> + $ upgrade-helper.py <replaceable>recipe_name</replaceable> + </literallayout> + For example, this command upgrades the + <filename>xmodmap</filename> recipe: + <literallayout class='monospaced'> + $ upgrade-helper.py xmodmap + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading a Specific Recipe to a Particular Version:</emphasis> + To upgrade a specific recipe to a particular version, + use the following form: + <literallayout class='monospaced'> + $ upgrade-helper.py <replaceable>recipe_name</replaceable> -t <replaceable>version</replaceable> + </literallayout> + For example, this command upgrades the + <filename>xmodmap</filename> recipe to version + 1.2.3: + <literallayout class='monospaced'> + $ upgrade-helper.py xmodmap -t 1.2.3 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading all Recipes to the Latest Versions and Suppressing Email Notifications:</emphasis> + To upgrade all recipes to their most recent versions + and suppress the email notifications, use the following + command: + <literallayout class='monospaced'> + $ upgrade-helper.py all + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Upgrading all Recipes to the Latest Versions and Send Email Notifications:</emphasis> + To upgrade all recipes to their most recent versions + and send email messages to maintainers for each + attempted recipe as well as a status email, use the + following command: + <literallayout class='monospaced'> + $ upgrade-helper.py -e all + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you have run the AUH utility, you can find the results + in the AUH build directory: + <literallayout class='monospaced'> + ${BUILDDIR}/upgrade-helper/<replaceable>timestamp</replaceable> + </literallayout> + The AUH utility also creates recipe update commits from + successful upgrade attempts in the layer tree. + </para> + + <para> + You can easily set up to run the AUH utility on a regular + basis by using a cron job. + See the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/auto-upgrade-helper/tree/weeklyjob.sh'><filename>weeklyjob.sh</filename></ulink> + file distributed with the utility for an example. + </para> + </section> + + <section id='gs-using-devtool-upgrade'> + <title>Using <filename>devtool upgrade</filename></title> + + <para> + As mentioned earlier, an alternative method for upgrading + recipes to newer versions is to use + <ulink url='&YOCTO_DOCS_REF_URL;#ref-devtool-reference'><filename>devtool upgrade</filename></ulink>. + You can read about <filename>devtool upgrade</filename> in + general in the + "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-devtool-use-devtool-upgrade-to-create-a-version-of-the-recipe-that-supports-a-newer-version-of-the-software'>Use <filename>devtool upgrade</filename> to Create a Version of the Recipe that Supports a Newer Version of the Software</ulink>" + section in the Yocto Project Application Development and the + Extensible Software Development Kit (eSDK) Manual. + </para> + + <para> + To see all the command-line options available with + <filename>devtool upgrade</filename>, use the following help + command: + <literallayout class='monospaced'> + $ devtool upgrade -h + </literallayout> + </para> + + <para> + If you want to find out what version a recipe is currently at + upstream without any attempt to upgrade your local version of + the recipe, you can use the following command: + <literallayout class='monospaced'> + $ devtool latest-version <replaceable>recipe_name</replaceable> + </literallayout> + </para> + + <para> + As mentioned in the previous section describing AUH, + <filename>devtool upgrade</filename> works in a + less-automated manner than AUH. + Specifically, <filename>devtool upgrade</filename> only + works on a single recipe that you name on the command line, + cannot perform build and integration testing using images, + and does not automatically generate commits for changes in + the source tree. + Despite all these "limitations", + <filename>devtool upgrade</filename> updates the recipe file + to the new upstream version and attempts to rebase custom + patches contained by the recipe as needed. + <note> + AUH uses much of <filename>devtool upgrade</filename> + behind the scenes making AUH somewhat of a "wrapper" + application for <filename>devtool upgrade</filename>. + </note> + </para> + + <para> + A typical scenario involves having used Git to clone an + upstream repository that you use during build operations. + Because you are (or have) built the recipe in the past, the + layer is likely added to your configuration already. + If for some reason, the layer is not added, you could add + it easily using the + <ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-bitbake-layers-script'><filename>bitbake-layers</filename></ulink> + script. + For example, suppose you use the <filename>nano.bb</filename> + recipe from the <filename>meta-oe</filename> layer in the + <filename>meta-openembedded</filename> repository. + For this example, assume that the layer has been cloned into + following area: + <literallayout class='monospaced'> + /home/scottrif/meta-openembedded + </literallayout> + The following command from your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + adds the layer to your build configuration (i.e. + <filename>${BUILDDIR}/conf/bblayers.conf</filename>): + <literallayout class='monospaced'> + $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe + NOTE: Starting bitbake server... + Parsing recipes: 100% |##########################################| Time: 0:00:55 + Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. + Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00 + Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00 + Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00 + Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00 + </literallayout> + For this example, assume that the <filename>nano.bb</filename> + recipe that is upstream has a 2.9.3 version number. + However, the version in the local repository is 2.7.4. + The following command from your build directory automatically + upgrades the recipe for you: + <note> + Using the <filename>-V</filename> option is not necessary. + Omitting the version number causes + <filename>devtool upgrade</filename> to upgrade the recipe + to the most recent version. + </note> + <literallayout class='monospaced'> + $ devtool upgrade nano -V 2.9.3 + NOTE: Starting bitbake server... + NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace + Parsing recipes: 100% |##########################################| Time: 0:00:46 + Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Extracting current version source... + NOTE: Resolving any missing task queue dependencies + . + . + . + NOTE: Executing SetScene Tasks + NOTE: Executing RunQueue Tasks + NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded. + Adding changed files: 100% |#####################################| Time: 0:00:00 + NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano + NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb + </literallayout> + Continuing with this example, you can use + <filename>devtool build</filename> to build the newly upgraded + recipe: + <literallayout class='monospaced'> + $ devtool build nano + NOTE: Starting bitbake server... + Loading cache: 100% |################################################################################################| Time: 0:00:01 + Loaded 2040 entries from dependency cache. + Parsing recipes: 100% |##############################################################################################| Time: 0:00:00 + Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Resolving any missing task queue dependencies + . + . + . + NOTE: Executing SetScene Tasks + NOTE: Executing RunQueue Tasks + NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano + NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded. + </literallayout> + Within the <filename>devtool upgrade</filename> workflow, + opportunity exists to deploy and test your rebuilt software. + For this example, however, running + <filename>devtool finish</filename> cleans up the workspace + once the source in your workspace is clean. + This usually means using Git to stage and submit commits + for the changes generated by the upgrade process. + </para> + + <para> + Once the tree is clean, you can clean things up in this + example with the following command from the + <filename>${BUILDDIR}/workspace/sources/nano</filename> + directory: + <literallayout class='monospaced'> + $ devtool finish nano meta-oe + NOTE: Starting bitbake server... + Loading cache: 100% |################################################################################################| Time: 0:00:00 + Loaded 2040 entries from dependency cache. + Parsing recipes: 100% |##############################################################################################| Time: 0:00:01 + Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors. + NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch + NOTE: Updating recipe nano_2.9.3.bb + NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb + NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano + NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually + </literallayout> + Using the <filename>devtool finish</filename> command cleans + up the workspace and creates a patch file based on your + commits. + The tool puts all patch files back into the source directory + in a sub-directory named <filename>nano</filename> in this + case. + </para> + </section> + + <section id='dev-manually-upgrading-a-recipe'> + <title>Manually Upgrading a Recipe</title> + + <para> + If for some reason you choose not to upgrade recipes using the + <link linkend='gs-using-the-auto-upgrade-helper'>Auto Upgrade Helper (AUH)</link> + or by using + <link linkend='gs-using-devtool-upgrade'><filename>devtool upgrade</filename></link>, + you can manually edit the recipe files to upgrade the versions. + <note><title>Caution</title> + Manually updating multiple recipes scales poorly and + involves many steps. + The recommendation to upgrade recipe versions is through + AUH or <filename>devtool upgrade</filename>, both of which + automate some steps and provide guidance for others needed + for the manual process. + </note> + </para> + + <para> + To manually upgrade recipe versions, follow these general steps: + <orderedlist> + <listitem><para> + <emphasis>Change the Version:</emphasis> + Rename the recipe such that the version (i.e. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + part of the recipe name) changes appropriately. + If the version is not part of the recipe name, change + the value as it is set for <filename>PV</filename> + within the recipe itself. + </para></listitem> + <listitem><para> + <emphasis>Update <filename>SRCREV</filename> if Needed:</emphasis> + If the source code your recipe builds is fetched from + Git or some other version control system, update + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + to point to the commit hash that matches the new + version. + </para></listitem> + <listitem><para> + <emphasis>Build the Software:</emphasis> + Try to build the recipe using BitBake. + Typical build failures include the following: + <itemizedlist> + <listitem><para> + License statements were updated for the new + version. + For this case, you need to review any changes + to the license and update the values of + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + as needed. + <note> + License changes are often inconsequential. + For example, the license text's copyright + year might have changed. + </note> + </para></listitem> + <listitem><para> + Custom patches carried by the older version of + the recipe might fail to apply to the new + version. + For these cases, you need to review the + failures. + Patches might not be necessary for the new + version of the software if the upgraded version + has fixed those issues. + If a patch is necessary and failing, you need + to rebase it into the new version. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Optionally Attempt to Build for Several Architectures:</emphasis> + Once you successfully build the new software for a + given architecture, you could test the build for + other architectures by changing the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + variable and rebuilding the software. + This optional step is especially important if the + recipe is to be released publicly. + </para></listitem> + <listitem><para> + <emphasis>Check the Upstream Change Log or Release Notes:</emphasis> + Checking both these reveals if new features exist that + could break backwards-compatibility. + If so, you need to take steps to mitigate or eliminate + that situation. + </para></listitem> + <listitem><para> + <emphasis>Optionally Create a Bootable Image and Test:</emphasis> + If you want, you can test the new software by booting + it onto actual hardware. + </para></listitem> + <listitem><para> + <emphasis>Create a Commit with the Change in the Layer Repository:</emphasis> + After all builds work and any testing is successful, + you can create commits for any changes in the layer + holding your upgraded recipe. + </para></listitem> + </orderedlist> + </para> + </section> + </section> + + <section id='finding-the-temporary-source-code'> + <title>Finding Temporary Source Code</title> + + <para> + You might find it helpful during development to modify the + temporary source code used by recipes to build packages. + For example, suppose you are developing a patch and you need to + experiment a bit to figure out your solution. + After you have initially built the package, you can iteratively + tweak the source code, which is located in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>, + and then you can force a re-compile and quickly test your altered + code. + Once you settle on a solution, you can then preserve your changes + in the form of patches. + </para> + + <para> + During a build, the unpacked temporary source code used by recipes + to build packages is available in the Build Directory as + defined by the + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> + variable. + Below is the default value for the <filename>S</filename> variable + as defined in the + <filename>meta/conf/bitbake.conf</filename> configuration file + in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>: + <literallayout class='monospaced'> + S = "${WORKDIR}/${BP}" + </literallayout> + You should be aware that many recipes override the + <filename>S</filename> variable. + For example, recipes that fetch their source from Git usually set + <filename>S</filename> to <filename>${WORKDIR}/git</filename>. + <note> + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-BP'><filename>BP</filename></ulink> + represents the base recipe name, which consists of the name + and version: + <literallayout class='monospaced'> + BP = "${BPN}-${PV}" + </literallayout> + </note> + </para> + + <para> + The path to the work directory for the recipe + (<ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>) + is defined as follows: + <literallayout class='monospaced'> + ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR} + </literallayout> + The actual directory depends on several things: + <itemizedlist> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>: + The top-level build output directory. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-MULTIMACH_TARGET_SYS'><filename>MULTIMACH_TARGET_SYS</filename></ulink>: + The target system identifier. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>: + The recipe name. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTENDPE'><filename>EXTENDPE</filename></ulink>: + The epoch - (if + <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink> + is not specified, which is usually the case for most + recipes, then <filename>EXTENDPE</filename> is blank). + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: + The recipe version. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>: + The recipe revision. + </para></listitem> + </itemizedlist> + </para> + + <para> + As an example, assume a Source Directory top-level folder + named <filename>poky</filename>, a default Build Directory at + <filename>poky/build</filename>, and a + <filename>qemux86-poky-linux</filename> machine target + system. + Furthermore, suppose your recipe is named + <filename>foo_1.3.0.bb</filename>. + In this case, the work directory the build system uses to + build the package would be as follows: + <literallayout class='monospaced'> + poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0 + </literallayout> + </para> + </section> + + <section id="using-a-quilt-workflow"> + <title>Using Quilt in Your Workflow</title> + + <para> + <ulink url='http://savannah.nongnu.org/projects/quilt'>Quilt</ulink> + is a powerful tool that allows you to capture source code changes + without having a clean source tree. + This section outlines the typical workflow you can use to modify + source code, test changes, and then preserve the changes in the + form of a patch all using Quilt. + <note><title>Tip</title> + With regard to preserving changes to source files, if you + clean a recipe or have <filename>rm_work</filename> enabled, + the + <ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'><filename>devtool</filename> workflow</ulink> + as described in the Yocto Project Application Development + and the Extensible Software Development Kit (eSDK) manual + is a safer development flow than the flow that uses Quilt. + </note> + </para> + + <para> + Follow these general steps: + <orderedlist> + <listitem><para> + <emphasis>Find the Source Code:</emphasis> + Temporary source code used by the OpenEmbedded build system + is kept in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + See the + "<link linkend='finding-the-temporary-source-code'>Finding Temporary Source Code</link>" + section to learn how to locate the directory that has the + temporary source code for a particular package. + </para></listitem> + <listitem><para> + <emphasis>Change Your Working Directory:</emphasis> + You need to be in the directory that has the temporary + source code. + That directory is defined by the + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> + variable.</para></listitem> + <listitem><para> + <emphasis>Create a New Patch:</emphasis> + Before modifying source code, you need to create a new + patch. + To create a new patch file, use + <filename>quilt new</filename> as below: + <literallayout class='monospaced'> + $ quilt new my_changes.patch + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Notify Quilt and Add Files:</emphasis> + After creating the patch, you need to notify Quilt about + the files you plan to edit. + You notify Quilt by adding the files to the patch you + just created: + <literallayout class='monospaced'> + $ quilt add file1.c file2.c file3.c + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Edit the Files:</emphasis> + Make your changes in the source code to the files you added + to the patch. + </para></listitem> + <listitem><para> + <emphasis>Test Your Changes:</emphasis> + Once you have modified the source code, the easiest way to + test your changes is by calling the + <filename>do_compile</filename> task as shown in the + following example: + <literallayout class='monospaced'> + $ bitbake -c compile -f <replaceable>package</replaceable> + </literallayout> + The <filename>-f</filename> or <filename>--force</filename> + option forces the specified task to execute. + If you find problems with your code, you can just keep + editing and re-testing iteratively until things work + as expected. + <note> + All the modifications you make to the temporary + source code disappear once you run the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-clean'><filename>do_clean</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-cleanall'><filename>do_cleanall</filename></ulink> + tasks using BitBake (i.e. + <filename>bitbake -c clean <replaceable>package</replaceable></filename> + and + <filename>bitbake -c cleanall <replaceable>package</replaceable></filename>). + Modifications will also disappear if you use the + <filename>rm_work</filename> feature as described + in the + "<ulink url='&YOCTO_DOCS_DEV_URL;#dev-saving-memory-during-a-build'>Conserving Disk Space During Builds</ulink>" + section. + </note> + </para></listitem> + <listitem><para> + <emphasis>Generate the Patch:</emphasis> + Once your changes work as expected, you need to use Quilt + to generate the final patch that contains all your + modifications. + <literallayout class='monospaced'> + $ quilt refresh + </literallayout> + At this point, the <filename>my_changes.patch</filename> + file has all your edits made to the + <filename>file1.c</filename>, <filename>file2.c</filename>, + and <filename>file3.c</filename> files.</para> + + <para>You can find the resulting patch file in the + <filename>patches/</filename> subdirectory of the source + (<filename>S</filename>) directory. + </para></listitem> + <listitem><para> + <emphasis>Copy the Patch File:</emphasis> + For simplicity, copy the patch file into a directory + named <filename>files</filename>, which you can create + in the same directory that holds the recipe + (<filename>.bb</filename>) file or the append + (<filename>.bbappend</filename>) file. + Placing the patch here guarantees that the OpenEmbedded + build system will find the patch. + Next, add the patch into the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename> + of the recipe. + Here is an example: + <literallayout class='monospaced'> + SRC_URI += "file://my_changes.patch" + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id="platdev-appdev-devshell"> + <title>Using a Development Shell</title> + + <para> + When debugging certain commands or even when just editing packages, + <filename>devshell</filename> can be a useful tool. + When you invoke <filename>devshell</filename>, all tasks up to and + including + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + are run for the specified target. + Then, a new terminal is opened and you are placed in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename>, + the source directory. + In the new terminal, all the OpenEmbedded build-related environment variables are + still defined so you can use commands such as <filename>configure</filename> and + <filename>make</filename>. + The commands execute just as if the OpenEmbedded build system were executing them. + Consequently, working this way can be helpful when debugging a build or preparing + software to be used with the OpenEmbedded build system. + </para> + + <para> + Following is an example that uses <filename>devshell</filename> on a target named + <filename>matchbox-desktop</filename>: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c devshell + </literallayout> + </para> + + <para> + This command spawns a terminal with a shell prompt within the OpenEmbedded build environment. + The <ulink url='&YOCTO_DOCS_REF_URL;#var-OE_TERMINAL'><filename>OE_TERMINAL</filename></ulink> + variable controls what type of shell is opened. + </para> + + <para> + For spawned terminals, the following occurs: + <itemizedlist> + <listitem><para>The <filename>PATH</filename> variable includes the + cross-toolchain.</para></listitem> + <listitem><para>The <filename>pkgconfig</filename> variables find the correct + <filename>.pc</filename> files.</para></listitem> + <listitem><para>The <filename>configure</filename> command finds the + Yocto Project site files as well as any other necessary files.</para></listitem> + </itemizedlist> + </para> + + <para> + Within this environment, you can run configure or compile + commands as if they were being run by + the OpenEmbedded build system itself. + As noted earlier, the working directory also automatically changes to the + Source Directory (<ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>). + </para> + + <para> + To manually run a specific task using <filename>devshell</filename>, + run the corresponding <filename>run.*</filename> script in + the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/temp</filename> + directory (e.g., + <filename>run.do_configure.</filename><replaceable>pid</replaceable>). + If a task's script does not exist, which would be the case if the task was + skipped by way of the sstate cache, you can create the task by first running + it outside of the <filename>devshell</filename>: + <literallayout class='monospaced'> + $ bitbake -c <replaceable>task</replaceable> + </literallayout> + <note><title>Notes</title> + <itemizedlist> + <listitem><para>Execution of a task's <filename>run.*</filename> + script and BitBake's execution of a task are identical. + In other words, running the script re-runs the task + just as it would be run using the + <filename>bitbake -c</filename> command. + </para></listitem> + <listitem><para>Any <filename>run.*</filename> file that does not + have a <filename>.pid</filename> extension is a + symbolic link (symlink) to the most recent version of that + file. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + Remember, that the <filename>devshell</filename> is a mechanism that allows + you to get into the BitBake task execution environment. + And as such, all commands must be called just as BitBake would call them. + That means you need to provide the appropriate options for + cross-compilation and so forth as applicable. + </para> + + <para> + When you are finished using <filename>devshell</filename>, exit the shell + or close the terminal window. + </para> + + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + It is worth remembering that when using <filename>devshell</filename> + you need to use the full compiler name such as <filename>arm-poky-linux-gnueabi-gcc</filename> + instead of just using <filename>gcc</filename>. + The same applies to other applications such as <filename>binutils</filename>, + <filename>libtool</filename> and so forth. + BitBake sets up environment variables such as <filename>CC</filename> + to assist applications, such as <filename>make</filename> to find the correct tools. + </para></listitem> + <listitem><para> + It is also worth noting that <filename>devshell</filename> still works over + X11 forwarding and similar situations. + </para></listitem> + </itemizedlist> + </note> + </section> + + <section id="platdev-appdev-devpyshell"> + <title>Using a Development Python Shell</title> + + <para> + Similar to working within a development shell as described in + the previous section, you can also spawn and work within an + interactive Python development shell. + When debugging certain commands or even when just editing packages, + <filename>devpyshell</filename> can be a useful tool. + When you invoke <filename>devpyshell</filename>, all tasks up to and + including + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + are run for the specified target. + Then a new terminal is opened. + Additionally, key Python objects and code are available in the same + way they are to BitBake tasks, in particular, the data store 'd'. + So, commands such as the following are useful when exploring the data + store and running functions: + <literallayout class='monospaced'> + pydevshell> d.getVar("STAGING_DIR", True) + '/media/build1/poky/build/tmp/sysroots' + pydevshell> d.getVar("STAGING_DIR", False) + '${TMPDIR}/sysroots' + pydevshell> d.setVar("FOO", "bar") + pydevshell> d.getVar("FOO", True) + 'bar' + pydevshell> d.delVar("FOO") + pydevshell> d.getVar("FOO", True) + pydevshell> bb.build.exec_func("do_unpack", d) + pydevshell> + </literallayout> + The commands execute just as if the OpenEmbedded build system were executing them. + Consequently, working this way can be helpful when debugging a build or preparing + software to be used with the OpenEmbedded build system. + </para> + + <para> + Following is an example that uses <filename>devpyshell</filename> on a target named + <filename>matchbox-desktop</filename>: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c devpyshell + </literallayout> + </para> + + <para> + This command spawns a terminal and places you in an interactive + Python interpreter within the OpenEmbedded build environment. + The <ulink url='&YOCTO_DOCS_REF_URL;#var-OE_TERMINAL'><filename>OE_TERMINAL</filename></ulink> + variable controls what type of shell is opened. + </para> + + <para> + When you are finished using <filename>devpyshell</filename>, you + can exit the shell either by using Ctrl+d or closing the terminal + window. + </para> + </section> + + <section id='dev-building'> + <title>Building</title> + + <para> + This section describes various build procedures. + For example, the steps needed for a simple build, a target that + uses multiple configurations, building an image for more than + one machine, and so forth. + </para> + + <section id='dev-building-a-simple-image'> + <title>Building a Simple Image</title> + + <para> + In the development environment, you need to build an image + whenever you change hardware support, add or change system + libraries, or add or change services that have dependencies. + Several methods exist that allow you to build an image within + the Yocto Project. + This section presents the basic steps you need to build a + simple image using BitBake from a build host running Linux. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + For information on how to build an image using + <ulink url='&YOCTO_DOCS_REF_URL;#toaster-term'>Toaster</ulink>, + see the + <ulink url='&YOCTO_DOCS_TOAST_URL;'>Toaster User Manual</ulink>. + </para></listitem> + <listitem><para> + For information on how to use + <filename>devtool</filename> to build images, see + the + "<ulink url='&YOCTO_DOCS_SDK_URL;#using-devtool-in-your-sdk-workflow'>Using <filename>devtool</filename> in Your SDK Workflow</ulink>" + section in the Yocto Project Application + Development and the Extensible Software Development + Kit (eSDK) manual. + </para></listitem> + <listitem><para> + For a quick example on how to build an image using + the OpenEmbedded build system, see the + <ulink url='&YOCTO_DOCS_BRIEF_URL;'>Yocto Project Quick Build</ulink> + document. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The build process creates an entire Linux distribution from + source and places it in your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + under <filename>tmp/deploy/images</filename>. + For detailed information on the build process using BitBake, + see the + "<ulink url='&YOCTO_DOCS_OM_URL;#images-dev-environment'>Images</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + + <para> + The following figure and list overviews the build process: + <imagedata fileref="figures/bitbake-build-flow.png" width="7in" depth="4in" align="center" scalefit="1" /> + <orderedlist> + <listitem><para> + <emphasis>Set up Your Host Development System to Support + Development Using the Yocto Project</emphasis>: + See the + "<link linkend='dev-manual-start'>Setting Up to Use the Yocto Project</link>" + section for options on how to get a build host ready to + use the Yocto Project. + </para></listitem> + <listitem><para> + <emphasis>Initialize the Build Environment:</emphasis> + Initialize the build environment by sourcing the build + environment script (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>): + <literallayout class='monospaced'> + $ source &OE_INIT_FILE; [<replaceable>build_dir</replaceable>] + </literallayout></para> + + <para>When you use the initialization script, the + OpenEmbedded build system uses + <filename>build</filename> as the default Build + Directory in your current work directory. + You can use a <replaceable>build_dir</replaceable> + argument with the script to specify a different build + directory. + <note><title>Tip</title> + A common practice is to use a different Build + Directory for different targets. + For example, <filename>~/build/x86</filename> for a + <filename>qemux86</filename> target, and + <filename>~/build/arm</filename> for a + <filename>qemuarm</filename> target. + </note> + </para></listitem> + <listitem><para> + <emphasis>Make Sure Your <filename>local.conf</filename> + File is Correct:</emphasis> + Ensure the <filename>conf/local.conf</filename> + configuration file, which is found in the Build + Directory, is set up how you want it. + This file defines many aspects of the build environment + including the target machine architecture through the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'>MACHINE</ulink></filename> variable, + the packaging format used during the build + (<ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>), + and a centralized tarball download directory through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> variable. + </para></listitem> + <listitem><para> + <emphasis>Build the Image:</emphasis> + Build the image using the <filename>bitbake</filename> + command: + <literallayout class='monospaced'> + $ bitbake <replaceable>target</replaceable> + </literallayout> + <note> + For information on BitBake, see the + <ulink url='&YOCTO_DOCS_BB_URL;'>BitBake User Manual</ulink>. + </note> + The <replaceable>target</replaceable> is the name of the + recipe you want to build. + Common targets are the images in + <filename>meta/recipes-core/images</filename>, + <filename>meta/recipes-sato/images</filename>, and so + forth all found in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + Or, the target can be the name of a recipe for a + specific piece of software such as BusyBox. + For more details about the images the OpenEmbedded build + system supports, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + chapter in the Yocto Project Reference Manual.</para> + + <para>As an example, the following command builds the + <filename>core-image-minimal</filename> image: + <literallayout class='monospaced'> + $ bitbake core-image-minimal + </literallayout> + Once an image has been built, it often needs to be + installed. + The images and kernels built by the OpenEmbedded + build system are placed in the Build Directory in + <filename class="directory">tmp/deploy/images</filename>. + For information on how to run pre-built images such as + <filename>qemux86</filename> and <filename>qemuarm</filename>, + see the + <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink> + manual. + For information about how to install these images, + see the documentation for your particular board or + machine. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='platdev-building-targets-with-multiple-configurations'> + <title>Building Targets with Multiple Configurations</title> + + <para> + Bitbake also has functionality that allows you to build + multiple targets at the same time, where each target uses + a different configuration. + </para> + + <para> + In order to accomplish this, you setup each of the configurations + you need to use in parallel by placing the configuration files in + your current build directory alongside the usual + <filename>local.conf</filename> file. + </para> + + <para> + Follow these guidelines to create an environment that supports + multiple configurations: + <itemizedlist> + <listitem><para> + <emphasis>Create Configuration Files</emphasis>: + You need to create a single configuration file for each + configuration for which you want to add support. + These files would contain lines such as the following: + <literallayout class='monospaced'> + MACHINE = "A" + </literallayout> + The files would contain any other variables that can + be set and built in the same directory. + <note> + You can change the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + to not conflict. + </note></para> + + <para> + Furthermore, the configuration file must be located in the + current build directory in a directory named + <filename>multiconfig</filename> under the build's + <filename>conf</filename> directory where + <filename>local.conf</filename> resides. + The reason for this restriction is because the + <filename>BBPATH</filename> variable is not constructed + until the layers are parsed. + Consequently, using the configuration file as a + pre-configuration file is not possible unless it is + located in the current working directory. + </para></listitem> + <listitem><para> + <emphasis>Add the BitBake Multi-Config Variable to you Local Configuration File</emphasis>: + Use the + <filename>BBMULTICONFIG</filename> + variable in your <filename>conf/local.conf</filename> + configuration file to specify each separate configuration. + For example, the following line tells BitBake it should load + <filename>conf/multiconfig/configA.conf</filename>, + <filename>conf/multiconfig/configB.conf</filename>, and + <filename>conf/multiconfig/configC.conf</filename>. + <literallayout class='monospaced'> + BBMULTICONFIG = "configA configB configC" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Launch BitBake</emphasis>: + Use the following BitBake command form to launch the + build: + <literallayout class='monospaced'> + $ bitbake [multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable> [[[multiconfig:<replaceable>multiconfigname</replaceable>:]<replaceable>target</replaceable>] ... ] + </literallayout> + Following is an example that supports building a minimal + image for configuration A alongside a standard + <filename>core-image-sato</filename>, which takes its + configuration from <filename>local.conf</filename>: + <literallayout class='monospaced'> + $ bitbake multiconfig:configA:core-image-minimal core-image-sato + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + Support for multiple configurations in this current release of + the Yocto Project (&DISTRO_NAME; &DISTRO;) has some known issues: + <itemizedlist> + <listitem><para> + No inter-multi-configuration dependencies exist. + </para></listitem> + <listitem><para> + Shared State (sstate) optimizations do not exist. + Consequently, if the build uses the same object twice + in, for example, two different + <filename>TMPDIR</filename> directories, the build + will either load from an existing sstate cache at the + start or build the object twice. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='building-an-initramfs-image'> + <title>Building an Initial RAM Filesystem (initramfs) Image</title> + + <para> + An initial RAM filesystem (initramfs) image provides a temporary + root filesystem used for early system initialization (e.g. + loading of modules needed to locate and mount the "real" root + filesystem). + <note> + The initramfs image is the successor of initial RAM disk + (initrd). + It is a "copy in and out" (cpio) archive of the initial + filesystem that gets loaded into memory during the Linux + startup process. + Because Linux uses the contents of the archive during + initialization, the initramfs image needs to contain all of the + device drivers and tools needed to mount the final root + filesystem. + </note> + </para> + + <para> + Follow these steps to create an initramfs image: + <orderedlist> + <listitem><para> + <emphasis>Create the initramfs Image Recipe:</emphasis> + You can reference the + <filename>core-image-minimal-initramfs.bb</filename> + recipe found in the <filename>meta/recipes-core</filename> + directory of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + as an example from which to work. + </para></listitem> + <listitem><para> + <emphasis>Decide if You Need to Bundle the initramfs Image + Into the Kernel Image:</emphasis> + If you want the initramfs image that is built to be + bundled in with the kernel image, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> + variable to "1" in your <filename>local.conf</filename> + configuration file and set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE'><filename>INITRAMFS_IMAGE</filename></ulink> + variable in the recipe that builds the kernel image. + <note><title>Tip</title> + It is recommended that you do bundle the initramfs + image with the kernel image to avoid circular + dependencies between the kernel recipe and the + initramfs recipe should the initramfs image + include kernel modules. + </note> + Setting the <filename>INITRAMFS_IMAGE_BUNDLE</filename> + flag causes the initramfs image to be unpacked + into the <filename>${B}/usr/</filename> directory. + The unpacked initramfs image is then passed to the kernel's + <filename>Makefile</filename> using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-CONFIG_INITRAMFS_SOURCE'><filename>CONFIG_INITRAMFS_SOURCE</filename></ulink> + variable, allowing the initramfs image to be built into + the kernel normally. + <note> + If you choose to not bundle the initramfs image with + the kernel image, you are essentially using an + <ulink url='https://en.wikipedia.org/wiki/Initrd'>Initial RAM Disk (initrd)</ulink>. + Creating an initrd is handled primarily through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRD_IMAGE'><filename>INITRD_IMAGE</filename></ulink>, + <filename>INITRD_LIVE</filename>, and + <filename>INITRD_IMAGE_LIVE</filename> variables. + For more information, see the + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/meta/classes/image-live.bbclass'><filename>image-live.bbclass</filename></ulink> + file. + </note> + </para></listitem> + <listitem><para> + <emphasis>Optionally Add Items to the initramfs Image + Through the initramfs Image Recipe:</emphasis> + If you add items to the initramfs image by way of its + recipe, you should use + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink> + rather than + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>. + <filename>PACKAGE_INSTALL</filename> gives more direct + control of what is added to the image as compared to + the defaults you might not necessarily want that are + set by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image'><filename>image</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-core-image'><filename>core-image</filename></ulink> + classes. + </para></listitem> + <listitem><para> + <emphasis>Build the Kernel Image and the initramfs + Image:</emphasis> + Build your kernel image using BitBake. + Because the initramfs image recipe is a dependency of the + kernel image, the initramfs image is built as well and + bundled with the kernel image if you used the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITRAMFS_IMAGE_BUNDLE'><filename>INITRAMFS_IMAGE_BUNDLE</filename></ulink> + variable described earlier. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='building-a-tiny-system'> + <title>Building a Tiny System</title> + + <para> + Very small distributions have some significant advantages such + as requiring less on-die or in-package memory (cheaper), better + performance through efficient cache usage, lower power requirements + due to less memory, faster boot times, and reduced development + overhead. + Some real-world examples where a very small distribution gives + you distinct advantages are digital cameras, medical devices, + and small headless systems. + </para> + + <para> + This section presents information that shows you how you can + trim your distribution to even smaller sizes than the + <filename>poky-tiny</filename> distribution, which is around + 5 Mbytes, that can be built out-of-the-box using the Yocto Project. + </para> + + <section id='tiny-system-overview'> + <title>Overview</title> + + <para> + The following list presents the overall steps you need to + consider and perform to create distributions with smaller + root filesystems, achieve faster boot times, maintain your critical + functionality, and avoid initial RAM disks: + <itemizedlist> + <listitem><para> + <link linkend='goals-and-guiding-principles'>Determine your goals and guiding principles.</link> + </para></listitem> + <listitem><para> + <link linkend='understand-what-gives-your-image-size'>Understand what contributes to your image size.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-root-filesystem'>Reduce the size of the root filesystem.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-kernel'>Reduce the size of the kernel.</link> + </para></listitem> + <listitem><para> + <link linkend='remove-package-management-requirements'>Eliminate packaging requirements.</link> + </para></listitem> + <listitem><para> + <link linkend='look-for-other-ways-to-minimize-size'>Look for other ways to minimize size.</link> + </para></listitem> + <listitem><para> + <link linkend='iterate-on-the-process'>Iterate on the process.</link> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='goals-and-guiding-principles'> + <title>Goals and Guiding Principles</title> + + <para> + Before you can reach your destination, you need to know + where you are going. + Here is an example list that you can use as a guide when + creating very small distributions: + <itemizedlist> + <listitem><para>Determine how much space you need + (e.g. a kernel that is 1 Mbyte or less and + a root filesystem that is 3 Mbytes or less). + </para></listitem> + <listitem><para>Find the areas that are currently + taking 90% of the space and concentrate on reducing + those areas. + </para></listitem> + <listitem><para>Do not create any difficult "hacks" + to achieve your goals.</para></listitem> + <listitem><para>Leverage the device-specific + options.</para></listitem> + <listitem><para>Work in a separate layer so that you + keep changes isolated. + For information on how to create layers, see + the "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" section. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='understand-what-gives-your-image-size'> + <title>Understand What Contributes to Your Image Size</title> + + <para> + It is easiest to have something to start with when creating + your own distribution. + You can use the Yocto Project out-of-the-box to create the + <filename>poky-tiny</filename> distribution. + Ultimately, you will want to make changes in your own + distribution that are likely modeled after + <filename>poky-tiny</filename>. + <note> + To use <filename>poky-tiny</filename> in your build, + set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable in your + <filename>local.conf</filename> file to "poky-tiny" + as described in the + "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>" + section. + </note> + </para> + + <para> + Understanding some memory concepts will help you reduce the + system size. + Memory consists of static, dynamic, and temporary memory. + Static memory is the TEXT (code), DATA (initialized data + in the code), and BSS (uninitialized data) sections. + Dynamic memory represents memory that is allocated at runtime: + stacks, hash tables, and so forth. + Temporary memory is recovered after the boot process. + This memory consists of memory used for decompressing + the kernel and for the <filename>__init__</filename> + functions. + </para> + + <para> + To help you see where you currently are with kernel and root + filesystem sizes, you can use two tools found in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> in + the <filename>scripts/tiny/</filename> directory: + <itemizedlist> + <listitem><para><filename>ksize.py</filename>: Reports + component sizes for the kernel build objects. + </para></listitem> + <listitem><para><filename>dirsize.py</filename>: Reports + component sizes for the root filesystem.</para></listitem> + </itemizedlist> + This next tool and command help you organize configuration + fragments and view file dependencies in a human-readable form: + <itemizedlist> + <listitem><para><filename>merge_config.sh</filename>: + Helps you manage configuration files and fragments + within the kernel. + With this tool, you can merge individual configuration + fragments together. + The tool allows you to make overrides and warns you + of any missing configuration options. + The tool is ideal for allowing you to iterate on + configurations, create minimal configurations, and + create configuration files for different machines + without having to duplicate your process.</para> + <para>The <filename>merge_config.sh</filename> script is + part of the Linux Yocto kernel Git repositories + (i.e. <filename>linux-yocto-3.14</filename>, + <filename>linux-yocto-3.10</filename>, + <filename>linux-yocto-3.8</filename>, and so forth) + in the + <filename>scripts/kconfig</filename> directory.</para> + <para>For more information on configuration fragments, + see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#creating-config-fragments'>Creating Configuration Fragments</ulink>" + section in the Yocto Project Linux Kernel Development + Manual. + </para></listitem> + <listitem><para><filename>bitbake -u taskexp -g <replaceable>bitbake_target</replaceable></filename>: + Using the BitBake command with these options brings up + a Dependency Explorer from which you can view file + dependencies. + Understanding these dependencies allows you to make + informed decisions when cutting out various pieces of the + kernel and root filesystem.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='trim-the-root-filesystem'> + <title>Trim the Root Filesystem</title> + + <para> + The root filesystem is made up of packages for booting, + libraries, and applications. + To change things, you can configure how the packaging happens, + which changes the way you build them. + You can also modify the filesystem itself or select a different + filesystem. + </para> + + <para> + First, find out what is hogging your root filesystem by running the + <filename>dirsize.py</filename> script from your root directory: + <literallayout class='monospaced'> + $ cd <replaceable>root-directory-of-image</replaceable> + $ dirsize.py 100000 > dirsize-100k.log + $ cat dirsize-100k.log + </literallayout> + You can apply a filter to the script to ignore files under + a certain size. + The previous example filters out any files below 100 Kbytes. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a + compressed root filesystem. + When you examine your log file, you can focus on areas of the + root filesystem that take up large amounts of memory. + </para> + + <para> + You need to be sure that what you eliminate does not cripple + the functionality you need. + One way to see how packages relate to each other is by using + the Dependency Explorer UI with the BitBake command: + <literallayout class='monospaced'> + $ cd <replaceable>image-directory</replaceable> + $ bitbake -u taskexp -g <replaceable>image</replaceable> + </literallayout> + Use the interface to select potential packages you wish to + eliminate and see their dependency relationships. + </para> + + <para> + When deciding how to reduce the size, get rid of packages that + result in minimal impact on the feature set. + For example, you might not need a VGA display. + Or, you might be able to get by with <filename>devtmpfs</filename> + and <filename>mdev</filename> instead of + <filename>udev</filename>. + </para> + + <para> + Use your <filename>local.conf</filename> file to make changes. + For example, to eliminate <filename>udev</filename> and + <filename>glib</filename>, set the following in the + local configuration file: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_dev_manager = "" + </literallayout> + </para> + + <para> + Finally, you should consider exactly the type of root + filesystem you need to meet your needs while also reducing + its size. + For example, consider <filename>cramfs</filename>, + <filename>squashfs</filename>, <filename>ubifs</filename>, + <filename>ext2</filename>, or an <filename>initramfs</filename> + using <filename>initramfs</filename>. + Be aware that <filename>ext3</filename> requires a 1 Mbyte + journal. + If you are okay with running read-only, you do not need this + journal. + </para> + + <note> + After each round of elimination, you need to rebuild your + system and then use the tools to see the effects of your + reductions. + </note> + </section> + + <section id='trim-the-kernel'> + <title>Trim the Kernel</title> + + <para> + The kernel is built by including policies for hardware-independent + aspects. + What subsystems do you enable? + For what architecture are you building? + Which drivers do you build by default? + <note>You can modify the kernel source if you want to help + with boot time. + </note> + </para> + + <para> + Run the <filename>ksize.py</filename> script from the top-level + Linux build directory to get an idea of what is making up + the kernel: + <literallayout class='monospaced'> + $ cd <replaceable>top-level-linux-build-directory</replaceable> + $ ksize.py > ksize.log + $ cat ksize.log + </literallayout> + When you examine the log, you will see how much space is + taken up with the built-in <filename>.o</filename> files for + drivers, networking, core kernel files, filesystem, sound, + and so forth. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a compressed + kernel image. + Look to reduce the areas that are large and taking up around + the "90% rule." + </para> + + <para> + To examine, or drill down, into any particular area, use the + <filename>-d</filename> option with the script: + <literallayout class='monospaced'> + $ ksize.py -d > ksize.log + </literallayout> + Using this option breaks out the individual file information + for each area of the kernel (e.g. drivers, networking, and + so forth). + </para> + + <para> + Use your log file to see what you can eliminate from the kernel + based on features you can let go. + For example, if you are not going to need sound, you do not + need any drivers that support sound. + </para> + + <para> + After figuring out what to eliminate, you need to reconfigure + the kernel to reflect those changes during the next build. + You could run <filename>menuconfig</filename> and make all your + changes at once. + However, that makes it difficult to see the effects of your + individual eliminations and also makes it difficult to replicate + the changes for perhaps another target device. + A better method is to start with no configurations using + <filename>allnoconfig</filename>, create configuration + fragments for individual changes, and then manage the + fragments into a single configuration file using + <filename>merge_config.sh</filename>. + The tool makes it easy for you to iterate using the + configuration change and build cycle. + </para> + + <para> + Each time you make configuration changes, you need to rebuild + the kernel and check to see what impact your changes had on + the overall size. + </para> + </section> + + <section id='remove-package-management-requirements'> + <title>Remove Package Management Requirements</title> + + <para> + Packaging requirements add size to the image. + One way to reduce the size of the image is to remove all the + packaging requirements from the image. + This reduction includes both removing the package manager + and its unique dependencies as well as removing the package + management data itself. + </para> + + <para> + To eliminate all the packaging requirements for an image, + be sure that "package-management" is not part of your + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + statement for the image. + When you remove this feature, you are removing the package + manager as well as its dependencies from the root filesystem. + </para> + </section> + + <section id='look-for-other-ways-to-minimize-size'> + <title>Look for Other Ways to Minimize Size</title> + + <para> + Depending on your particular circumstances, other areas that you + can trim likely exist. + The key to finding these areas is through tools and methods + described here combined with experimentation and iteration. + Here are a couple of areas to experiment with: + <itemizedlist> + <listitem><para><filename>glibc</filename>: + In general, follow this process: + <orderedlist> + <listitem><para>Remove <filename>glibc</filename> + features from + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + that you think you do not need.</para></listitem> + <listitem><para>Build your distribution. + </para></listitem> + <listitem><para>If the build fails due to missing + symbols in a package, determine if you can + reconfigure the package to not need those + features. + For example, change the configuration to not + support wide character support as is done for + <filename>ncurses</filename>. + Or, if support for those characters is needed, + determine what <filename>glibc</filename> + features provide the support and restore the + configuration. + </para></listitem> + <listitem><para>Rebuild and repeat the process. + </para></listitem> + </orderedlist></para></listitem> + <listitem><para><filename>busybox</filename>: + For BusyBox, use a process similar as described for + <filename>glibc</filename>. + A difference is you will need to boot the resulting + system to see if you are able to do everything you + expect from the running system. + You need to be sure to integrate configuration fragments + into Busybox because BusyBox handles its own core + features and then allows you to add configuration + fragments on top. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='iterate-on-the-process'> + <title>Iterate on the Process</title> + + <para> + If you have not reached your goals on system size, you need + to iterate on the process. + The process is the same. + Use the tools and see just what is taking up 90% of the root + filesystem and the kernel. + Decide what you can eliminate without limiting your device + beyond what you need. + </para> + + <para> + Depending on your system, a good place to look might be + Busybox, which provides a stripped down + version of Unix tools in a single, executable file. + You might be able to drop virtual terminal services or perhaps + ipv6. + </para> + </section> + </section> + + <section id='building-images-for-more-than-one-machine'> + <title>Building Images for More than One Machine</title> + + <para> + A common scenario developers face is creating images for several + different machines that use the same software environment. + In this situation, it is tempting to set the + tunings and optimization flags for each build specifically for + the targeted hardware (i.e. "maxing out" the tunings). + Doing so can considerably add to build times and package feed + maintenance collectively for the machines. + For example, selecting tunes that are extremely specific to a + CPU core used in a system might enable some micro optimizations + in GCC for that particular system but would otherwise not gain + you much of a performance difference across the other systems + as compared to using a more general tuning across all the builds + (e.g. setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEFAULTTUNE'><filename>DEFAULTTUNE</filename></ulink> + specifically for each machine's build). + Rather than "max out" each build's tunings, you can take steps that + cause the OpenEmbedded build system to reuse software across the + various machines where it makes sense. + </para> + + <para> + If build speed and package feed maintenance are considerations, + you should consider the points in this section that can help you + optimize your tunings to best consider build times and package + feed maintenance. + <itemizedlist> + <listitem><para> + <emphasis>Share the Build Directory:</emphasis> + If at all possible, share the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + across builds. + The Yocto Project supports switching between different + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + values in the same <filename>TMPDIR</filename>. + This practice is well supported and regularly used by + developers when building for multiple machines. + When you use the same <filename>TMPDIR</filename> for + multiple machine builds, the OpenEmbedded build system can + reuse the existing native and often cross-recipes for + multiple machines. + Thus, build time decreases. + <note> + If + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + settings change or fundamental configuration settings + such as the filesystem layout, you need to work with + a clean <filename>TMPDIR</filename>. + Sharing <filename>TMPDIR</filename> under these + circumstances might work but since it is not + guaranteed, you should use a clean + <filename>TMPDIR</filename>. + </note> + </para></listitem> + <listitem><para> + <emphasis>Enable the Appropriate Package Architecture:</emphasis> + By default, the OpenEmbedded build system enables three + levels of package architectures: "all", "tune" or "package", + and "machine". + Any given recipe usually selects one of these package + architectures (types) for its output. + Depending for what a given recipe creates packages, making + sure you enable the appropriate package architecture can + directly impact the build time.</para> + + <para>A recipe that just generates scripts can enable + "all" architecture because there are no binaries to build. + To specifically enable "all" architecture, be sure your + recipe inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> + class. + This class is useful for "all" architectures because it + configures many variables so packages can be used across + multiple architectures.</para> + + <para>If your recipe needs to generate packages that are + machine-specific or when one of the build or runtime + dependencies is already machine-architecture dependent, + which makes your recipe also machine-architecture dependent, + make sure your recipe enables the "machine" package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink> + variable: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${MACHINE_ARCH}" + </literallayout> + When you do not specifically enable a package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>, + The OpenEmbedded build system defaults to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TUNE_PKGARCH'><filename>TUNE_PKGARCH</filename></ulink> + setting: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${TUNE_PKGARCH}" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Choose a Generic Tuning File if Possible:</emphasis> + Some tunes are more generic and can run on multiple targets + (e.g. an <filename>armv5</filename> set of packages could + run on <filename>armv6</filename> and + <filename>armv7</filename> processors in most cases). + Similarly, <filename>i486</filename> binaries could work + on <filename>i586</filename> and higher processors. + You should realize, however, that advances on newer + processor versions would not be used.</para> + + <para>If you select the same tune for several different + machines, the OpenEmbedded build system reuses software + previously built, thus speeding up the overall build time. + Realize that even though a new sysroot for each machine is + generated, the software is not recompiled and only one + package feed exists. + </para></listitem> + <listitem><para> + <emphasis>Manage Granular Level Packaging:</emphasis> + Sometimes cases exist where injecting another level of + package architecture beyond the three higher levels noted + earlier can be useful. + For example, consider how NXP (formerly Freescale) allows + for the easy reuse of binary packages in their layer + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/meta-freescale/'><filename>meta-freescale</filename></ulink>. + In this example, the + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass'><filename>fsl-dynamic-packagearch</filename></ulink> + class shares GPU packages for i.MX53 boards because + all boards share the AMD GPU. + The i.MX6-based boards can do the same because all boards + share the Vivante GPU. + This class inspects the BitBake datastore to identify if + the package provides or depends on one of the + sub-architecture values. + If so, the class sets the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> + value based on the <filename>MACHINE_SUBARCH</filename> + value. + If the package does not provide or depend on one of the + sub-architecture values but it matches a value in the + machine-specific filter, it sets + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink>. + This behavior reduces the number of packages built and + saves build time by reusing binaries. + </para></listitem> + <listitem><para> + <emphasis>Use Tools to Debug Issues:</emphasis> + Sometimes you can run into situations where software is + being rebuilt when you think it should not be. + For example, the OpenEmbedded build system might not be + using shared state between machines when you think it + should be. + These types of situations are usually due to references + to machine-specific variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLES'><filename>SERIAL_CONSOLES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-XSERVER'><filename>XSERVER</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'><filename>MACHINE_FEATURES</filename></ulink>, + and so forth in code that is supposed to only be + tune-specific or when the recipe depends + (<ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RSUGGESTS'><filename>RSUGGESTS</filename></ulink>, + and so forth) on some other recipe that already has + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> + defined as "${MACHINE_ARCH}". + <note> + Patches to fix any issues identified are most welcome + as these issues occasionally do occur. + </note></para> + + <para>For such cases, you can use some tools to help you + sort out the situation: + <itemizedlist> + <listitem><para> + <emphasis><filename>sstate-diff-machines.sh</filename>:</emphasis> + You can find this tool in the + <filename>scripts</filename> directory of the + Source Repositories. + See the comments in the script for information on + how to use the tool. + </para></listitem> + <listitem><para> + <emphasis>BitBake's "-S printdiff" Option:</emphasis> + Using this option causes BitBake to try to + establish the closest signature match it can + (e.g. in the shared state cache) and then run + <filename>bitbake-diffsigs</filename> over the + matches to determine the stamps and delta where + these two stamp trees diverge. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id="building-software-from-an-external-source"> + <title>Building Software from an External Source</title> + + <para> + By default, the OpenEmbedded build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + when building source code. + The build process involves fetching the source files, unpacking + them, and then patching them if necessary before the build takes + place. + </para> + + <para> + Situations exist where you might want to build software from source + files that are external to and thus outside of the + OpenEmbedded build system. + For example, suppose you have a project that includes a new BSP with + a heavily customized kernel. + And, you want to minimize exposing the build system to the + development team so that they can focus on their project and + maintain everyone's workflow as much as possible. + In this case, you want a kernel source directory on the development + machine where the development occurs. + You want the recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable to point to the external directory and use it as is, not + copy it. + </para> + + <para> + To build from software that comes from an external source, all you + need to do is inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class and then set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink> + variable to point to your external source code. + Here are the statements to put in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT += "externalsrc" + EXTERNALSRC_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + + <para> + This next example shows how to accomplish the same thing by setting + <filename>EXTERNALSRC</filename> in the recipe itself or in the + recipe's append file: + <literallayout class='monospaced'> + EXTERNALSRC = "<replaceable>path</replaceable>" + EXTERNALSRC_BUILD = "<replaceable>path</replaceable>" + </literallayout> + <note> + In order for these settings to take effect, you must globally + or locally inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class. + </note> + </para> + + <para> + By default, <filename>externalsrc.bbclass</filename> builds + the source code in a directory separate from the external source + directory as specified by + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink>. + If you need to have the source built in the same directory in + which it resides, or some other nominated directory, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC_BUILD'><filename>EXTERNALSRC_BUILD</filename></ulink> + to point to that directory: + <literallayout class='monospaced'> + EXTERNALSRC_BUILD_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + </section> + </section> + + + + <section id='speeding-up-a-build'> + <title>Speeding Up a Build</title> + + <para> + Build time can be an issue. + By default, the build system uses simple controls to try and maximize + build efficiency. + In general, the default settings for all the following variables + result in the most efficient build times when dealing with single + socket systems (i.e. a single CPU). + If you have multiple CPUs, you might try increasing the default + values to gain more speed. + See the descriptions in the glossary for each variable for more + information: + <itemizedlist> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename>:</ulink> + The maximum number of threads BitBake simultaneously executes. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename>:</ulink> + The number of threads BitBake uses during parsing. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename>:</ulink> + Extra options passed to the <filename>make</filename> command + during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + task in order to specify parallel compilation on the + local build host. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename>:</ulink> + Extra options passed to the <filename>make</filename> command + during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task in order to specify parallel installation on the + local build host. + </para></listitem> + </itemizedlist> + As mentioned, these variables all scale to the number of processor + cores available on the build system. + For single socket systems, this auto-scaling ensures that the build + system fundamentally takes advantage of potential parallel operations + during the build based on the build machine's capabilities. + </para> + + <para> + Following are additional factors that can affect build speed: + <itemizedlist> + <listitem><para> + File system type: + The file system type that the build is being performed on can + also influence performance. + Using <filename>ext4</filename> is recommended as compared + to <filename>ext2</filename> and <filename>ext3</filename> + due to <filename>ext4</filename> improved features + such as extents. + </para></listitem> + <listitem><para> + Disabling the updating of access time using + <filename>noatime</filename>: + The <filename>noatime</filename> mount option prevents the + build system from updating file and directory access times. + </para></listitem> + <listitem><para> + Setting a longer commit: + Using the "commit=" mount option increases the interval + in seconds between disk cache writes. + Changing this interval from the five second default to + something longer increases the risk of data loss but decreases + the need to write to the disk, thus increasing the build + performance. + </para></listitem> + <listitem><para> + Choosing the packaging backend: + Of the available packaging backends, IPK is the fastest. + Additionally, selecting a singular packaging backend also + helps. + </para></listitem> + <listitem><para> + Using <filename>tmpfs</filename> for + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + as a temporary file system: + While this can help speed up the build, the benefits are + limited due to the compiler using + <filename>-pipe</filename>. + The build system goes to some lengths to avoid + <filename>sync()</filename> calls into the + file system on the principle that if there was a significant + failure, the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + contents could easily be rebuilt. + </para></listitem> + <listitem><para> + Inheriting the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + class: + Inheriting this class has shown to speed up builds due to + significantly lower amounts of data stored in the data + cache as well as on disk. + Inheriting this class also makes cleanup of + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + faster, at the expense of being easily able to dive into the + source code. + File system maintainers have recommended that the fastest way + to clean up large numbers of files is to reformat partitions + rather than delete files due to the linear nature of + partitions. + This, of course, assumes you structure the disk partitions and + file systems in a way that this is practical. + </para></listitem> + </itemizedlist> + Aside from the previous list, you should keep some trade offs in + mind that can help you speed up the build: + <itemizedlist> + <listitem><para> + Remove items from + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + that you might not need. + </para></listitem> + <listitem><para> + Exclude debug symbols and other debug information: + If you do not need these symbols and other debug information, + disabling the <filename>*-dbg</filename> package generation + can speed up the build. + You can disable this generation by setting the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INHIBIT_PACKAGE_DEBUG_SPLIT'><filename>INHIBIT_PACKAGE_DEBUG_SPLIT</filename></ulink> + variable to "1". + </para></listitem> + <listitem><para> + Disable static library generation for recipes derived from + <filename>autoconf</filename> or <filename>libtool</filename>: + Following is an example showing how to disable static + libraries and still provide an override to handle exceptions: + <literallayout class='monospaced'> + STATICLIBCONF = "--disable-static" + STATICLIBCONF_sqlite3-native = "" + EXTRA_OECONF += "${STATICLIBCONF}" + </literallayout> + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + Some recipes need static libraries in order to work + correctly (e.g. <filename>pseudo-native</filename> + needs <filename>sqlite3-native</filename>). + Overrides, as in the previous example, account for + these kinds of exceptions. + </para></listitem> + <listitem><para> + Some packages have packaging code that assumes the + presence of the static libraries. + If so, you might need to exclude them as well. + </para></listitem> + </itemizedlist> + </note> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id="platdev-working-with-libraries"> + <title>Working With Libraries</title> + + <para> + Libraries are an integral part of your system. + This section describes some common practices you might find + helpful when working with libraries to build your system: + <itemizedlist> + <listitem><para><link linkend='including-static-library-files'>How to include static library files</link> + </para></listitem> + <listitem><para><link linkend='combining-multiple-versions-library-files-into-one-image'>How to use the Multilib feature to combine multiple versions of library files into a single image</link> + </para></listitem> + <listitem><para><link linkend='installing-multiple-versions-of-the-same-library'>How to install multiple versions of the same library in parallel on the same system</link> + </para></listitem> + </itemizedlist> + </para> + + <section id='including-static-library-files'> + <title>Including Static Library Files</title> + + <para> + If you are building a library and the library offers static linking, you can control + which static library files (<filename>*.a</filename> files) get included in the + built library. + </para> + + <para> + The <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES_*</filename></ulink> + variables in the + <filename>meta/conf/bitbake.conf</filename> configuration file define how files installed + by the <filename>do_install</filename> task are packaged. + By default, the <filename>PACKAGES</filename> variable includes + <filename>${PN}-staticdev</filename>, which represents all static library files. + <note> + Some previously released versions of the Yocto Project + defined the static library files through + <filename>${PN}-dev</filename>. + </note> + Following is part of the BitBake configuration file, where + you can see how the static library files are defined: + <literallayout class='monospaced'> + PACKAGE_BEFORE_PN ?= "" + PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" + PACKAGES_DYNAMIC = "^${PN}-locale-.*" + FILES = "" + + FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \ + ${sysconfdir} ${sharedstatedir} ${localstatedir} \ + ${base_bindir}/* ${base_sbindir}/* \ + ${base_libdir}/*${SOLIBS} \ + ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \ + ${datadir}/${BPN} ${libdir}/${BPN}/* \ + ${datadir}/pixmaps ${datadir}/applications \ + ${datadir}/idl ${datadir}/omf ${datadir}/sounds \ + ${libdir}/bonobo/servers" + + FILES_${PN}-bin = "${bindir}/* ${sbindir}/*" + + FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \ + ${datadir}/gnome/help" + SECTION_${PN}-doc = "doc" + + FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}" + FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \ + ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \ + ${datadir}/aclocal ${base_libdir}/*.o \ + ${libdir}/${BPN}/*.la ${base_libdir}/*.la" + SECTION_${PN}-dev = "devel" + ALLOW_EMPTY_${PN}-dev = "1" + RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})" + + FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a" + SECTION_${PN}-staticdev = "devel" + RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})" + </literallayout> + </para> + </section> + + <section id="combining-multiple-versions-library-files-into-one-image"> + <title>Combining Multiple Versions of Library Files into One Image</title> + + <para> + The build system offers the ability to build libraries with different + target optimizations or architecture formats and combine these together + into one system image. + You can link different binaries in the image + against the different libraries as needed for specific use cases. + This feature is called "Multilib." + </para> + + <para> + An example would be where you have most of a system compiled in 32-bit + mode using 32-bit libraries, but you have something large, like a database + engine, that needs to be a 64-bit application and uses 64-bit libraries. + Multilib allows you to get the best of both 32-bit and 64-bit libraries. + </para> + + <para> + While the Multilib feature is most commonly used for 32 and 64-bit differences, + the approach the build system uses facilitates different target optimizations. + You could compile some binaries to use one set of libraries and other binaries + to use a different set of libraries. + The libraries could differ in architecture, compiler options, or other + optimizations. + </para> + + <para> + Several examples exist in the + <filename>meta-skeleton</filename> layer found in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>: + <itemizedlist> + <listitem><para><filename>conf/multilib-example.conf</filename> + configuration file</para></listitem> + <listitem><para><filename>conf/multilib-example2.conf</filename> + configuration file</para></listitem> + <listitem><para><filename>recipes-multilib/images/core-image-multilib-example.bb</filename> + recipe</para></listitem> + </itemizedlist> + </para> + + <section id='preparing-to-use-multilib'> + <title>Preparing to Use Multilib</title> + + <para> + User-specific requirements drive the Multilib feature. + Consequently, there is no one "out-of-the-box" configuration that likely + exists to meet your needs. + </para> + + <para> + In order to enable Multilib, you first need to ensure your recipe is + extended to support multiple libraries. + Many standard recipes are already extended and support multiple libraries. + You can check in the <filename>meta/conf/multilib.conf</filename> + configuration file in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> to see how this is + done using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></ulink> + variable. + Eventually, all recipes will be covered and this list will + not be needed. + </para> + + <para> + For the most part, the Multilib class extension works automatically to + extend the package name from <filename>${PN}</filename> to + <filename>${MLPREFIX}${PN}</filename>, where <filename>MLPREFIX</filename> + is the particular multilib (e.g. "lib32-" or "lib64-"). + Standard variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RPROVIDES'><filename>RPROVIDES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES_DYNAMIC'><filename>PACKAGES_DYNAMIC</filename></ulink> + are automatically extended by the system. + If you are extending any manual code in the recipe, you can use the + <filename>${MLPREFIX}</filename> variable to ensure those names are extended + correctly. + This automatic extension code resides in <filename>multilib.bbclass</filename>. + </para> + </section> + + <section id='using-multilib'> + <title>Using Multilib</title> + + <para> + After you have set up the recipes, you need to define the actual + combination of multiple libraries you want to build. + You accomplish this through your <filename>local.conf</filename> + configuration file in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + An example configuration would be as follows: + <literallayout class='monospaced'> + MACHINE = "qemux86-64" + require conf/multilib.conf + MULTILIBS = "multilib:lib32" + DEFAULTTUNE_virtclass-multilib-lib32 = "x86" + IMAGE_INSTALL_append = " lib32-glib-2.0" + </literallayout> + This example enables an + additional library named <filename>lib32</filename> alongside the + normal target packages. + When combining these "lib32" alternatives, the example uses "x86" for tuning. + For information on this particular tuning, see + <filename>meta/conf/machine/include/ia32/arch-ia32.inc</filename>. + </para> + + <para> + The example then includes <filename>lib32-glib-2.0</filename> + in all the images, which illustrates one method of including a + multiple library dependency. + You can use a normal image build to include this dependency, + for example: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + You can also build Multilib packages specifically with a command like this: + <literallayout class='monospaced'> + $ bitbake lib32-glib-2.0 + </literallayout> + </para> + </section> + + <section id='additional-implementation-details'> + <title>Additional Implementation Details</title> + + <para> + Generic implementation details as well as details that are + specific to package management systems exist. + Following are implementation details that exist regardless + of the package management system: + <itemizedlist> + <listitem><para>The typical convention used for the + class extension code as used by + Multilib assumes that all package names specified + in + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + that contain <filename>${PN}</filename> have + <filename>${PN}</filename> at the start of the name. + When that convention is not followed and + <filename>${PN}</filename> appears at + the middle or the end of a name, problems occur. + </para></listitem> + <listitem><para>The + <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_VENDOR'><filename>TARGET_VENDOR</filename></ulink> + value under Multilib will be extended to + "-<replaceable>vendor</replaceable>ml<replaceable>multilib</replaceable>" + (e.g. "-pokymllib32" for a "lib32" Multilib with + Poky). + The reason for this slightly unwieldy contraction + is that any "-" characters in the vendor + string presently break Autoconf's + <filename>config.sub</filename>, and + other separators are problematic for different + reasons. + </para></listitem> + </itemizedlist> + </para> + + <para> + For the RPM Package Management System, the following implementation details + exist: + <itemizedlist> + <listitem><para>A unique architecture is defined for the Multilib packages, + along with creating a unique deploy folder under + <filename>tmp/deploy/rpm</filename> in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + For example, consider <filename>lib32</filename> in a + <filename>qemux86-64</filename> image. + The possible architectures in the system are "all", "qemux86_64", + "lib32_qemux86_64", and "lib32_x86".</para></listitem> + <listitem><para>The <filename>${MLPREFIX}</filename> variable is stripped from + <filename>${PN}</filename> during RPM packaging. + The naming for a normal RPM package and a Multilib RPM package in a + <filename>qemux86-64</filename> system resolves to something similar to + <filename>bash-4.1-r2.x86_64.rpm</filename> and + <filename>bash-4.1.r2.lib32_x86.rpm</filename>, respectively. + </para></listitem> + <listitem><para>When installing a Multilib image, the RPM backend first + installs the base image and then installs the Multilib libraries. + </para></listitem> + <listitem><para>The build system relies on RPM to resolve the identical files in the + two (or more) Multilib packages.</para></listitem> + </itemizedlist> + </para> + + <para> + For the IPK Package Management System, the following implementation details exist: + <itemizedlist> + <listitem><para>The <filename>${MLPREFIX}</filename> is not stripped from + <filename>${PN}</filename> during IPK packaging. + The naming for a normal RPM package and a Multilib IPK package in a + <filename>qemux86-64</filename> system resolves to something like + <filename>bash_4.1-r2.x86_64.ipk</filename> and + <filename>lib32-bash_4.1-rw_x86.ipk</filename>, respectively. + </para></listitem> + <listitem><para>The IPK deploy folder is not modified with + <filename>${MLPREFIX}</filename> because packages with and without + the Multilib feature can exist in the same folder due to the + <filename>${PN}</filename> differences.</para></listitem> + <listitem><para>IPK defines a sanity check for Multilib installation + using certain rules for file comparison, overridden, etc. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='installing-multiple-versions-of-the-same-library'> + <title>Installing Multiple Versions of the Same Library</title> + + <para> + Situations can exist where you need to install and use + multiple versions of the same library on the same system + at the same time. + These situations almost always exist when a library API + changes and you have multiple pieces of software that + depend on the separate versions of the library. + To accommodate these situations, you can install multiple + versions of the same library in parallel on the same system. + </para> + + <para> + The process is straightforward as long as the libraries use + proper versioning. + With properly versioned libraries, all you need to do to + individually specify the libraries is create separate, + appropriately named recipes where the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> part of the + name includes a portion that differentiates each library version + (e.g.the major part of the version number). + Thus, instead of having a single recipe that loads one version + of a library (e.g. <filename>clutter</filename>), you provide + multiple recipes that result in different versions + of the libraries you want. + As an example, the following two recipes would allow the + two separate versions of the <filename>clutter</filename> + library to co-exist on the same system: + <literallayout class='monospaced'> + clutter-1.6_1.6.20.bb + clutter-1.8_1.8.4.bb + </literallayout> + Additionally, if you have other recipes that depend on a given + library, you need to use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable to create the dependency. + Continuing with the same example, if you want to have a recipe + depend on the 1.8 version of the <filename>clutter</filename> + library, use the following in your recipe: + <literallayout class='monospaced'> + DEPENDS = "clutter-1.8" + </literallayout> + </para> + </section> + </section> + + <section id='using-x32-psabi'> + <title>Using x32 psABI</title> + + <para> + x32 processor-specific Application Binary Interface + (<ulink url='https://software.intel.com/en-us/node/628948'>x32 psABI</ulink>) + is a native 32-bit processor-specific ABI for + <trademark class='registered'>Intel</trademark> 64 (x86-64) + architectures. + An ABI defines the calling conventions between functions in a + processing environment. + The interface determines what registers are used and what the + sizes are for various C data types. + </para> + + <para> + Some processing environments prefer using 32-bit applications even + when running on Intel 64-bit platforms. + Consider the i386 psABI, which is a very old 32-bit ABI for Intel + 64-bit platforms. + The i386 psABI does not provide efficient use and access of the + Intel 64-bit processor resources, leaving the system underutilized. + Now consider the x86_64 psABI. + This ABI is newer and uses 64-bits for data sizes and program + pointers. + The extra bits increase the footprint size of the programs, + libraries, and also increases the memory and file system size + requirements. + Executing under the x32 psABI enables user programs to utilize CPU + and system resources more efficiently while keeping the memory + footprint of the applications low. + Extra bits are used for registers but not for addressing mechanisms. + </para> + + <para> + The Yocto Project supports the final specifications of x32 psABI + as follows: + <itemizedlist> + <listitem><para> + You can create packages and images in x32 psABI format on + x86_64 architecture targets. + </para></listitem> + <listitem><para> + You can successfully build recipes with the x32 toolchain. + </para></listitem> + <listitem><para> + You can create and boot + <filename>core-image-minimal</filename> and + <filename>core-image-sato</filename> images. + </para></listitem> + <listitem><para> + RPM Package Manager (RPM) support exists for x32 binaries. + </para></listitem> + <listitem><para> + Support for large images exists. + </para></listitem> + </itemizedlist> + </para> + + <para> + To use the x32 psABI, you need to edit your + <filename>conf/local.conf</filename> configuration file as + follows: + <literallayout class='monospaced'> + MACHINE = "qemux86-64" + DEFAULTTUNE = "x86-64-x32" + baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE', True) \ + or 'INVALID'), True) or 'lib'}" + </literallayout> + Once you have set up your configuration file, use BitBake to + build an image that supports the x32 psABI. + Here is an example: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para> + </section> + + <section id='enabling-gobject-introspection-support'> + <title>Enabling GObject Introspection Support</title> + + <para> + <ulink url='https://wiki.gnome.org/Projects/GObjectIntrospection'>GObject introspection</ulink> + is the standard mechanism for accessing GObject-based software + from runtime environments. + GObject is a feature of the GLib library that provides an object + framework for the GNOME desktop and related software. + GObject Introspection adds information to GObject that allows + objects created within it to be represented across different + programming languages. + If you want to construct GStreamer pipelines using Python, or + control UPnP infrastructure using Javascript and GUPnP, + GObject introspection is the only way to do it. + </para> + + <para> + This section describes the Yocto Project support for generating + and packaging GObject introspection data. + GObject introspection data is a description of the + API provided by libraries built on top of GLib framework, + and, in particular, that framework's GObject mechanism. + GObject Introspection Repository (GIR) files go to + <filename>-dev</filename> packages, + <filename>typelib</filename> files go to main packages as they + are packaged together with libraries that are introspected. + </para> + + <para> + The data is generated when building such a library, by linking + the library with a small executable binary that asks the library + to describe itself, and then executing the binary and + processing its output. + </para> + + <para> + Generating this data in a cross-compilation environment + is difficult because the library is produced for the target + architecture, but its code needs to be executed on the build host. + This problem is solved with the OpenEmbedded build system by + running the code through QEMU, which allows precisely that. + Unfortunately, QEMU does not always work perfectly as mentioned + in the xxx section. + </para> + + <section id='enabling-the-generation-of-introspection-data'> + <title>Enabling the Generation of Introspection Data</title> + + <para> + Enabling the generation of introspection data (GIR files) + in your library package involves the following: + <orderedlist> + <listitem><para> + Inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-gobject-introspection'><filename>gobject-introspection</filename></ulink> + class. + </para></listitem> + <listitem><para> + Make sure introspection is not disabled anywhere in + the recipe or from anything the recipe includes. + Also, make sure that "gobject-introspection-data" is + not in + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES_BACKFILL_CONSIDERED'><filename>DISTRO_FEATURES_BACKFILL_CONSIDERED</filename></ulink> + and that "qemu-usermode" is not in + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES_BACKFILL_CONSIDERED'><filename>MACHINE_FEATURES_BACKFILL_CONSIDERED</filename></ulink>. + If either of these conditions exist, nothing will + happen. + </para></listitem> + <listitem><para> + Try to build the recipe. + If you encounter build errors that look like + something is unable to find + <filename>.so</filename> libraries, check where these + libraries are located in the source tree and add + the following to the recipe: + <literallayout class='monospaced'> + GIR_EXTRA_LIBS_PATH = "${B}/<replaceable>something</replaceable>/.libs" + </literallayout> + <note> + See recipes in the <filename>oe-core</filename> + repository that use that + <filename>GIR_EXTRA_LIBS_PATH</filename> variable + as an example. + </note> + </para></listitem> + <listitem><para> + Look for any other errors, which probably mean that + introspection support in a package is not entirely + standard, and thus breaks down in a cross-compilation + environment. + For such cases, custom-made fixes are needed. + A good place to ask and receive help in these cases + is the + <ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Yocto Project mailing lists</ulink>. + </para></listitem> + </orderedlist> + <note> + Using a library that no longer builds against the latest + Yocto Project release and prints introspection related + errors is a good candidate for the previous procedure. + </note> + </para> + </section> + + <section id='disabling-the-generation-of-introspection-data'> + <title>Disabling the Generation of Introspection Data</title> + + <para> + You might find that you do not want to generate + introspection data. + Or, perhaps QEMU does not work on your build host and + target architecture combination. + If so, you can use either of the following methods to + disable GIR file generations: + <itemizedlist> + <listitem><para> + Add the following to your distro configuration: + <literallayout class='monospaced'> + DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data" + </literallayout> + Adding this statement disables generating + introspection data using QEMU but will still enable + building introspection tools and libraries + (i.e. building them does not require the use of QEMU). + </para></listitem> + <listitem><para> + Add the following to your machine configuration: + <literallayout class='monospaced'> + MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode" + </literallayout> + Adding this statement disables the use of QEMU + when building packages for your machine. + Currently, this feature is used only by introspection + recipes and has the same effect as the previously + described option. + <note> + Future releases of the Yocto Project might have + other features affected by this option. + </note> + </para></listitem> + </itemizedlist> + If you disable introspection data, you can still + obtain it through other means such as copying the data + from a suitable sysroot, or by generating it on the + target hardware. + The OpenEmbedded build system does not currently + provide specific support for these techniques. + </para> + </section> + + <section id='testing-that-introspection-works-in-an-image'> + <title>Testing that Introspection Works in an Image</title> + + <para> + Use the following procedure to test if generating + introspection data is working in an image: + <orderedlist> + <listitem><para> + Make sure that "gobject-introspection-data" is not in + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES_BACKFILL_CONSIDERED'><filename>DISTRO_FEATURES_BACKFILL_CONSIDERED</filename></ulink> + and that "qemu-usermode" is not in + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES_BACKFILL_CONSIDERED'><filename>MACHINE_FEATURES_BACKFILL_CONSIDERED</filename></ulink>. + </para></listitem> + <listitem><para> + Build <filename>core-image-sato</filename>. + </para></listitem> + <listitem><para> + Launch a Terminal and then start Python in the + terminal. + </para></listitem> + <listitem><para> + Enter the following in the terminal: + <literallayout class='monospaced'> + >>> from gi.repository import GLib + >>> GLib.get_host_name() + </literallayout> + </para></listitem> + <listitem><para> + For something a little more advanced, enter the + following: + <literallayout class='monospaced'> + http://python-gtk-3-tutorial.readthedocs.org/en/latest/introduction.html + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='known-issues'> + <title>Known Issues</title> + + <para> + The following know issues exist for + GObject Introspection Support: + <itemizedlist> + <listitem><para> + <filename>qemu-ppc64</filename> immediately crashes. + Consequently, you cannot build introspection data on + that architecture. + </para></listitem> + <listitem><para> + x32 is not supported by QEMU. + Consequently, introspection data is disabled. + </para></listitem> + <listitem><para> + musl causes transient GLib binaries to crash on + assertion failures. + Consequently, generating introspection data is + disabled. + </para></listitem> + <listitem><para> + Because QEMU is not able to run the binaries correctly, + introspection is disabled for some specific packages + under specific architectures (e.g. + <filename>gcr</filename>, + <filename>libsecret</filename>, and + <filename>webkit</filename>). + </para></listitem> + <listitem><para> + QEMU usermode might not work properly when running + 64-bit binaries under 32-bit host machines. + In particular, "qemumips64" is known to not work under + i686. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='dev-optionally-using-an-external-toolchain'> + <title>Optionally Using an External Toolchain</title> + + <para> + You might want to use an external toolchain as part of your + development. + If this is the case, the fundamental steps you need to accomplish + are as follows: + <itemizedlist> + <listitem><para> + Understand where the installed toolchain resides. + For cases where you need to build the external toolchain, + you would need to take separate steps to build and install + the toolchain. + </para></listitem> + <listitem><para> + Make sure you add the layer that contains the toolchain to + your <filename>bblayers.conf</filename> file through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBLAYERS'><filename>BBLAYERS</filename></ulink> + variable. + </para></listitem> + <listitem><para> + Set the <filename>EXTERNAL_TOOLCHAIN</filename> + variable in your <filename>local.conf</filename> file + to the location in which you installed the toolchain. + </para></listitem> + </itemizedlist> + A good example of an external toolchain used with the Yocto Project + is <trademark class='registered'>Mentor Graphics</trademark> + Sourcery G++ Toolchain. + You can see information on how to use that particular layer in the + <filename>README</filename> file at + <ulink url='http://github.com/MentorEmbedded/meta-sourcery/'></ulink>. + You can find further information by reading about the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TCMODE'><filename>TCMODE</filename></ulink> + variable in the Yocto Project Reference Manual's variable glossary. + </para> + </section> + + <section id='creating-partitioned-images-using-wic'> + <title>Creating Partitioned Images Using Wic</title> + + <para> + Creating an image for a particular hardware target using the + OpenEmbedded build system does not necessarily mean you can boot + that image as is on your device. + Physical devices accept and boot images in various ways depending + on the specifics of the device. + Usually, information about the hardware can tell you what image + format the device requires. + Should your device require multiple partitions on an SD card, flash, + or an HDD, you can use the OpenEmbedded Image Creator, + Wic, to create the properly partitioned image. + </para> + + <para> + The <filename>wic</filename> command generates partitioned + images from existing OpenEmbedded build artifacts. + Image generation is driven by partitioning commands + contained in an Openembedded kickstart file + (<filename>.wks</filename>) specified either directly on + the command line or as one of a selection of canned + kickstart files as shown with the + <filename>wic list images</filename> command in the + "<link linkend='using-a-provided-kickstart-file'>Using an Existing Kickstart File</link>" + section. + When you apply the command to a given set of build + artifacts, the result is an image or set of images that + can be directly written onto media and used on a particular + system. + <note> + For a kickstart file reference, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-kickstart'>OpenEmbedded Kickstart (<filename>.wks</filename>) Reference</ulink>" + Chapter in the Yocto Project Reference Manual. + </note> + </para> + + <para> + The <filename>wic</filename> command and the infrastructure + it is based on is by definition incomplete. + The purpose of the command is to allow the generation of + customized images, and as such, was designed to be + completely extensible through a plug-in interface. + See the + "<link linkend='wic-using-the-wic-plug-ins-interface'>Using the Wic Plug-Ins Interface</link>" + section for information on these plug-ins. + </para> + + <para> + This section provides some background information on Wic, + describes what you need to have in + place to run the tool, provides instruction on how to use + the Wic utility, provides information on using the Wic plug-ins + interface, and provides several examples that show how to use + Wic. + </para> + + <section id='wic-background'> + <title>Background</title> + + <para> + This section provides some background on the Wic utility. + While none of this information is required to use + Wic, you might find it interesting. + <itemizedlist> + <listitem><para> + The name "Wic" is derived from OpenEmbedded + Image Creator (oeic). + The "oe" diphthong in "oeic" was promoted to the + letter "w", because "oeic" is both difficult to + remember and to pronounce. + </para></listitem> + <listitem><para> + Wic is loosely based on the + Meego Image Creator (<filename>mic</filename>) + framework. + The Wic implementation has been + heavily modified to make direct use of OpenEmbedded + build artifacts instead of package installation and + configuration, which are already incorporated within + the OpenEmbedded artifacts. + </para></listitem> + <listitem><para> + Wic is a completely independent + standalone utility that initially provides + easier-to-use and more flexible replacements for an + existing functionality in OE-Core's + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image-live'><filename>image-live</filename></ulink> + class and <filename>mkefidisk.sh</filename> script. + The difference between + Wic and those examples is + that with Wic the + functionality of those scripts is implemented + by a general-purpose partitioning language, which is + based on Redhat kickstart syntax.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='wic-requirements'> + <title>Requirements</title> + + <para> + In order to use the Wic utility with the OpenEmbedded Build + system, your system needs to meet the following + requirements: + <itemizedlist> + <listitem><para> + The Linux distribution on your development host must + support the Yocto Project. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#detailed-supported-distros'>Supported Linux Distributions</ulink>" + section in the Yocto Project Reference Manual for + the list of distributions that support the + Yocto Project. + </para></listitem> + <listitem><para> + The standard system utilities, such as + <filename>cp</filename>, must be installed on your + development host system. + </para></listitem> + <listitem><para> + You must have sourced the build environment + setup script (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>) + found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + </para></listitem> + <listitem><para> + You need to have the build artifacts already + available, which typically means that you must + have already created an image using the + Openembedded build system (e.g. + <filename>core-image-minimal</filename>). + While it might seem redundant to generate an image + in order to create an image using + Wic, the current version of + Wic requires the artifacts + in the form generated by the OpenEmbedded build + system. + </para></listitem> + <listitem><para> + You must build several native tools, which are + built to run on the build system: + <literallayout class='monospaced'> + $ bitbake parted-native dosfstools-native mtools-native + </literallayout> + </para></listitem> + <listitem><para> + Include "wic" as part of the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> + variable. + </para></listitem> + <listitem><para> + Include the name of the + <ulink url='&YOCTO_DOCS_REF_URL;#openembedded-kickstart-wks-reference'>wic kickstart file</ulink> + as part of the + <ulink url='&YOCTO_DOCS_REF_URL;#var-WKS_FILE'><filename>WKS_FILE</filename></ulink> + variable + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='wic-getting-help'> + <title>Getting Help</title> + + <para> + You can get general help for the <filename>wic</filename> + command by entering the <filename>wic</filename> command + by itself or by entering the command with a help argument + as follows: + <literallayout class='monospaced'> + $ wic -h + $ wic --help + $ wic help + </literallayout> + </para> + + <para> + Currently, Wic supports seven commands: + <filename>cp</filename>, <filename>create</filename>, + <filename>help</filename>, <filename>list</filename>, + <filename>ls</filename>, <filename>rm</filename>, and + <filename>write</filename>. + You can get help for all these commands except "help" by + using the following form: + <literallayout class='monospaced'> + $ wic help <replaceable>command</replaceable> + </literallayout> + For example, the following command returns help for the + <filename>write</filename> command: + <literallayout class='monospaced'> + $ wic help write + </literallayout> + </para> + + <para> + Wic supports help for three topics: + <filename>overview</filename>, + <filename>plugins</filename>, and + <filename>kickstart</filename>. + You can get help for any topic using the following form: + <literallayout class='monospaced'> + $ wic help <replaceable>topic</replaceable> + </literallayout> + For example, the following returns overview help for Wic: + <literallayout class='monospaced'> + $ wic help overview + </literallayout> + </para> + + <para> + One additional level of help exists for Wic. + You can get help on individual images through the + <filename>list</filename> command. + You can use the <filename>list</filename> command to return the + available Wic images as follows: + <literallayout class='monospaced'> + $ wic list images + mpc8315e-rdb Create SD card image for MPC8315E-RDB + genericx86 Create an EFI disk image for genericx86* + beaglebone-yocto Create SD card image for Beaglebone + edgerouter Create SD card image for Edgerouter + qemux86-directdisk Create a qemu machine 'pcbios' direct disk image + directdisk-gpt Create a 'pcbios' direct disk image + mkefidisk Create an EFI disk image + directdisk Create a 'pcbios' direct disk image + systemd-bootdisk Create an EFI disk image with systemd-boot + mkhybridiso Create a hybrid ISO image + sdimage-bootpart Create SD card image with a boot partition + directdisk-multi-rootfs Create multi rootfs image using rootfs plugin + directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config + </literallayout> + Once you know the list of available Wic images, you can use + <filename>help</filename> with the command to get help on a + particular image. + For example, the following command returns help on the + "beaglebone-yocto" image: + <literallayout class='monospaced'> + $ wic list beaglebone-yocto help + + + Creates a partitioned SD card image for Beaglebone. + Boot files are located in the first vfat partition. + </literallayout> + </para> + </section> + + <section id='operational-modes'> + <title>Operational Modes</title> + + <para> + You can use Wic in two different + modes, depending on how much control you need for + specifying the Openembedded build artifacts that are + used for creating the image: Raw and Cooked: + <itemizedlist> + <listitem><para> + <emphasis>Raw Mode:</emphasis> + You explicitly specify build artifacts through + Wic command-line arguments. + </para></listitem> + <listitem><para> + <emphasis>Cooked Mode:</emphasis> + The current + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + setting and image name are used to automatically + locate and provide the build artifacts. + You just supply a kickstart file and the name + of the image from which to use artifacts. + </para></listitem> + </itemizedlist> + </para> + + <para> + Regardless of the mode you use, you need to have the build + artifacts ready and available. + </para> + + <section id='raw-mode'> + <title>Raw Mode</title> + + <para> + Running Wic in raw mode allows you to specify all the + partitions through the <filename>wic</filename> + command line. + The primary use for raw mode is if you have built + your kernel outside of the Yocto Project + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + In other words, you can point to arbitrary kernel, + root filesystem locations, and so forth. + Contrast this behavior with cooked mode where Wic + looks in the Build Directory (e.g. + <filename>tmp/deploy/images/</filename><replaceable>machine</replaceable>). + </para> + + <para> + The general form of the + <filename>wic</filename> command in raw mode is: + <literallayout class='monospaced'> + $ wic create <replaceable>wks_file</replaceable> <replaceable>options</replaceable> ... + + Where: + + <replaceable>wks_file</replaceable>: + An OpenEmbedded kickstart file. You can provide + your own custom file or use a file from a set of + existing files as described by further options. + + optional arguments: + -h, --help show this help message and exit + -o <replaceable>OUTDIR</replaceable>, --outdir <replaceable>OUTDIR</replaceable> + name of directory to create image in + -e <replaceable>IMAGE_NAME</replaceable>, --image-name <replaceable>IMAGE_NAME</replaceable> + name of the image to use the artifacts from e.g. core- + image-sato + -r <replaceable>ROOTFS_DIR</replaceable>, --rootfs-dir <replaceable>ROOTFS_DIR</replaceable> + path to the /rootfs dir to use as the .wks rootfs + source + -b <replaceable>BOOTIMG_DIR</replaceable>, --bootimg-dir <replaceable>BOOTIMG_DIR</replaceable> + path to the dir containing the boot artifacts (e.g. + /EFI or /syslinux dirs) to use as the .wks bootimg + source + -k <replaceable>KERNEL_DIR</replaceable>, --kernel-dir <replaceable>KERNEL_DIR</replaceable> + path to the dir containing the kernel to use in the + .wks bootimg + -n <replaceable>NATIVE_SYSROOT</replaceable>, --native-sysroot <replaceable>NATIVE_SYSROOT</replaceable> + path to the native sysroot containing the tools to use + to build the image + -s, --skip-build-check + skip the build check + -f, --build-rootfs build rootfs + -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz} + compress image with specified compressor + -m, --bmap generate .bmap + --no-fstab-update Do not change fstab file. + -v <replaceable>VARS_DIR</replaceable>, --vars <replaceable>VARS_DIR</replaceable> + directory with <image>.env files that store bitbake + variables + -D, --debug output debug information + </literallayout> + <note> + You do not need root privileges to run + Wic. + In fact, you should not run as root when using the + utility. + </note> + </para> + </section> + + <section id='cooked-mode'> + <title>Cooked Mode</title> + + <para> + Running Wic in cooked mode leverages off artifacts in + the Build Directory. + In other words, you do not have to specify kernel or + root filesystem locations as part of the command. + All you need to provide is a kickstart file and the + name of the image from which to use artifacts by using + the "-e" option. + Wic looks in the Build Directory (e.g. + <filename>tmp/deploy/images/</filename><replaceable>machine</replaceable>) + for artifacts. + </para> + + <para> + The general form of the <filename>wic</filename> + command using Cooked Mode is as follows: + <literallayout class='monospaced'> + $ wic create <replaceable>wks_file</replaceable> -e <replaceable>IMAGE_NAME</replaceable> + + Where: + + <replaceable>wks_file</replaceable>: + An OpenEmbedded kickstart file. You can provide + your own custom file or use a file from a set of + existing files provided with the Yocto Project + release. + + required argument: + -e <replaceable>IMAGE_NAME</replaceable>, --image-name <replaceable>IMAGE_NAME</replaceable> + name of the image to use the artifacts from e.g. core- + image-sato + </literallayout> + </para> + </section> + </section> + + <section id='using-a-provided-kickstart-file'> + <title>Using an Existing Kickstart File</title> + + <para> + If you do not want to create your own kickstart file, you + can use an existing file provided by the Wic installation. + As shipped, kickstart files can be found in the + Yocto Project + <ulink url='&YOCTO_DOCS_OM_URL;#source-repositories'>Source Repositories</ulink> + in the following two locations: + <literallayout class='monospaced'> + poky/meta-yocto-bsp/wic + poky/scripts/lib/wic/canned-wks + </literallayout> + Use the following command to list the available kickstart + files: + <literallayout class='monospaced'> + $ wic list images + mpc8315e-rdb Create SD card image for MPC8315E-RDB + genericx86 Create an EFI disk image for genericx86* + beaglebone-yocto Create SD card image for Beaglebone + edgerouter Create SD card image for Edgerouter + qemux86-directdisk Create a qemu machine 'pcbios' direct disk image + directdisk-gpt Create a 'pcbios' direct disk image + mkefidisk Create an EFI disk image + directdisk Create a 'pcbios' direct disk image + systemd-bootdisk Create an EFI disk image with systemd-boot + mkhybridiso Create a hybrid ISO image + sdimage-bootpart Create SD card image with a boot partition + directdisk-multi-rootfs Create multi rootfs image using rootfs plugin + directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config + </literallayout> + When you use an existing file, you do not have to use the + <filename>.wks</filename> extension. + Here is an example in Raw Mode that uses the + <filename>directdisk</filename> file: + <literallayout class='monospaced'> + $ wic create directdisk -r <replaceable>rootfs_dir</replaceable> -b <replaceable>bootimg_dir</replaceable> \ + -k <replaceable>kernel_dir</replaceable> -n <replaceable>native_sysroot</replaceable> + </literallayout> + </para> + + <para> + Here are the actual partition language commands + used in the <filename>genericx86.wks</filename> file to + generate an image: + <literallayout class='monospaced'> + # short-description: Create an EFI disk image for genericx86* + # long-description: Creates a partitioned EFI disk image for genericx86* machines + part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024 + part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid + part swap --ondisk sda --size 44 --label swap1 --fstype=swap + + bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0" + </literallayout> + </para> + </section> + + <section id='wic-using-the-wic-plug-ins-interface'> + <title>Using the Wic Plug-Ins Interface</title> + + <para> + You can extend and specialize Wic functionality by using + Wic plug-ins. + This section explains the Wic plug-in interface. + <note> + Wic plug-ins consist of "source" and "imager" plug-ins. + Imager plug-ins are beyond the scope of this section. + </note> + </para> + + <para> + Source plug-ins provide a mechanism to customize partition + content during the Wic image generation process. + You can use source plug-ins to map values that you specify + using <filename>--source</filename> commands in kickstart + files (i.e. <filename>*.wks</filename>) to a plug-in + implementation used to populate a given partition. + <note> + If you use plug-ins that have build-time dependencies + (e.g. native tools, bootloaders, and so forth) + when building a Wic image, you need to specify those + dependencies using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-WKS_FILE_DEPENDS'><filename>WKS_FILE_DEPENDS</filename></ulink> + variable. + </note> + </para> + + <para> + Source plug-ins are subclasses defined in plug-in files. + As shipped, the Yocto Project provides several plug-in + files. + You can see the source plug-in files that ship with the + Yocto Project + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/source'>here</ulink>. + Each of these plug-in files contains source plug-ins that + are designed to populate a specific Wic image partition. + </para> + + <para> + Source plug-ins are subclasses of the + <filename>SourcePlugin</filename> class, which is + defined in the + <filename>poky/scripts/lib/wic/pluginbase.py</filename> + file. + For example, the <filename>BootimgEFIPlugin</filename> + source plug-in found in the + <filename>bootimg-efi.py</filename> file is a subclass of + the <filename>SourcePlugin</filename> class, which is found + in the <filename>pluginbase.py</filename> file. + </para> + + <para> + You can also implement source plug-ins in a layer outside + of the Source Repositories (external layer). + To do so, be sure that your plug-in files are located in + a directory whose path is + <filename>scripts/lib/wic/plugins/source/</filename> + within your external layer. + When the plug-in files are located there, the source + plug-ins they contain are made available to Wic. + </para> + + <para> + When the Wic implementation needs to invoke a + partition-specific implementation, it looks for the plug-in + with the same name as the <filename>--source</filename> + parameter used in the kickstart file given to that + partition. + For example, if the partition is set up using the following + command in a kickstart file: + <literallayout class='monospaced'> + part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 + </literallayout> + The methods defined as class members of the matching + source plug-in (i.e. <filename>bootimg-pcbios</filename>) + in the <filename>bootimg-pcbios.py</filename> plug-in file + are used. + </para> + + <para> + To be more concrete, here is the corresponding plug-in + definition from the <filename>bootimg-pcbios.py</filename> + file for the previous command along with an example + method called by the Wic implementation when it needs to + prepare a partition using an implementation-specific + function: + <literallayout class='monospaced'> + . + . + . + class BootimgPcbiosPlugin(SourcePlugin): + """ + Create MBR boot partition and install syslinux on it. + """ + + name = 'bootimg-pcbios' + . + . + . + @classmethod + def do_prepare_partition(cls, part, source_params, creator, cr_workdir, + oe_builddir, bootimg_dir, kernel_dir, + rootfs_dir, native_sysroot): + """ + Called to do the actual content population for a partition i.e. it + 'prepares' the partition to be incorporated into the image. + In this case, prepare content for legacy bios boot partition. + """ + . + . + . + </literallayout> + If a subclass (plug-in) itself does not implement a + particular function, Wic locates and uses the default + version in the superclass. + It is for this reason that all source plug-ins are derived + from the <filename>SourcePlugin</filename> class. + </para> + + <para> + The <filename>SourcePlugin</filename> class defined in + the <filename>pluginbase.py</filename> file defines + a set of methods that source plug-ins can implement or + override. + Any plug-ins (subclass of + <filename>SourcePlugin</filename>) that do not implement + a particular method inherit the implementation of the + method from the <filename>SourcePlugin</filename> class. + For more information, see the + <filename>SourcePlugin</filename> class in the + <filename>pluginbase.py</filename> file for details: + </para> + + <para> + The following list describes the methods implemented in the + <filename>SourcePlugin</filename> class: + <itemizedlist> + <listitem><para> + <emphasis><filename>do_prepare_partition()</filename>:</emphasis> + Called to populate a partition with actual content. + In other words, the method prepares the final + partition image that is incorporated into the + disk image. + </para></listitem> + <listitem><para> + <emphasis><filename>do_configure_partition()</filename>:</emphasis> + Called before + <filename>do_prepare_partition()</filename> to + create custom configuration files for a partition + (e.g. syslinux or grub configuration files). + </para></listitem> + <listitem><para> + <emphasis><filename>do_install_disk()</filename>:</emphasis> + Called after all partitions have been prepared and + assembled into a disk image. + This method provides a hook to allow finalization + of a disk image (e.g. writing an MBR). + </para></listitem> + <listitem><para> + <emphasis><filename>do_stage_partition()</filename>:</emphasis> + Special content-staging hook called before + <filename>do_prepare_partition()</filename>. + This method is normally empty.</para> + + <para>Typically, a partition just uses the passed-in + parameters (e.g. the unmodified value of + <filename>bootimg_dir</filename>). + However, in some cases, things might need to be + more tailored. + As an example, certain files might additionally + need to be taken from + <filename>bootimg_dir + /boot</filename>. + This hook allows those files to be staged in a + customized fashion. + <note> + <filename>get_bitbake_var()</filename> + allows you to access non-standard variables + that you might want to use for this + behavior. + </note> + </para></listitem> + </itemizedlist> + </para> + + <para> + You can extend the source plug-in mechanism. + To add more hooks, create more source plug-in methods + within <filename>SourcePlugin</filename> and the + corresponding derived subclasses. + The code that calls the plug-in methods uses the + <filename>plugin.get_source_plugin_methods()</filename> + function to find the method or methods needed by the call. + Retrieval of those methods is accomplished by filling up + a dict with keys that contain the method names of interest. + On success, these will be filled in with the actual + methods. + See the Wic implementation for examples and details. + </para> + </section> + + <section id='wic-usage-examples'> + <title>Examples</title> + + <para> + This section provides several examples that show how to use + the Wic utility. + All the examples assume the list of requirements in the + "<link linkend='wic-requirements'>Requirements</link>" + section have been met. + The examples assume the previously generated image is + <filename>core-image-minimal</filename>. + </para> + + <section id='generate-an-image-using-a-provided-kickstart-file'> + <title>Generate an Image using an Existing Kickstart File</title> + + <para> + This example runs in Cooked Mode and uses the + <filename>mkefidisk</filename> kickstart file: + <literallayout class='monospaced'> + $ wic create mkefidisk -e core-image-minimal + INFO: Building wic-tools... + . + . + . + INFO: The new image(s) can be found here: + ./mkefidisk-201804191017-sda.direct + + The following build artifacts were used to create the image(s): + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native + + INFO: The image(s) were created using OE kickstart file: + /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks + </literallayout> + The previous example shows the easiest way to create + an image by running in cooked mode and supplying + a kickstart file and the "-e" option to point to the + existing build artifacts. + Your <filename>local.conf</filename> file needs to have + the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + variable set to the machine you are using, which is + "qemux86" in this example. + </para> + + <para> + Once the image builds, the output provides image + location, artifact use, and kickstart file information. + <note> + You should always verify the details provided in the + output to make sure that the image was indeed + created exactly as expected. + </note> + </para> + + <para> + Continuing with the example, you can now write the + image from the Build Directory onto a USB stick, or + whatever media for which you built your image, and boot + from the media. + You can write the image by using + <filename>bmaptool</filename> or + <filename>dd</filename>: + <literallayout class='monospaced'> + $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sd<replaceable>X</replaceable> + </literallayout> + or + <literallayout class='monospaced'> + $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sd<replaceable>X</replaceable> + </literallayout> + <note> + For more information on how to use the + <filename>bmaptool</filename> to flash a device + with an image, see the + "<link linkend='flashing-images-using-bmaptool'>Flashing Images Using <filename>bmaptool</filename></link>" + section. + </note> + </para> + </section> + + <section id='using-a-modified-kickstart-file'> + <title>Using a Modified Kickstart File</title> + + <para> + Because partitioned image creation is driven by the + kickstart file, it is easy to affect image creation by + changing the parameters in the file. + This next example demonstrates that through modification + of the <filename>directdisk-gpt</filename> kickstart + file. + </para> + + <para> + As mentioned earlier, you can use the command + <filename>wic list images</filename> to show the list + of existing kickstart files. + The directory in which the + <filename>directdisk-gpt.wks</filename> file resides is + <filename>scripts/lib/image/canned-wks/</filename>, + which is located in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. <filename>poky</filename>). + Because available files reside in this directory, + you can create and add your own custom files to the + directory. + Subsequent use of the + <filename>wic list images</filename> command would then + include your kickstart files. + </para> + + <para> + In this example, the existing + <filename>directdisk-gpt</filename> file already does + most of what is needed. + However, for the hardware in this example, the image + will need to boot from <filename>sdb</filename> instead + of <filename>sda</filename>, which is what the + <filename>directdisk-gpt</filename> kickstart file + uses. + </para> + + <para> + The example begins by making a copy of the + <filename>directdisk-gpt.wks</filename> file in the + <filename>scripts/lib/image/canned-wks</filename> + directory and then by changing the lines that specify + the target disk from which to boot. + <literallayout class='monospaced'> + $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \ + /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks + </literallayout> + Next, the example modifies the + <filename>directdisksdb-gpt.wks</filename> file and + changes all instances of + "<filename>--ondisk sda</filename>" to + "<filename>--ondisk sdb</filename>". + The example changes the following two lines and leaves + the remaining lines untouched: + <literallayout class='monospaced'> + part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024 + part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid + </literallayout> + Once the lines are changed, the example generates the + <filename>directdisksdb-gpt</filename> image. + The command points the process at the + <filename>core-image-minimal</filename> artifacts for + the Next Unit of Computing (nuc) + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + the <filename>local.conf</filename>. + <literallayout class='monospaced'> + $ wic create directdisksdb-gpt -e core-image-minimal + INFO: Building wic-tools... + . + . + . + Initialising tasks: 100% |#######################################| Time: 0:00:01 + NOTE: Executing SetScene Tasks + NOTE: Executing RunQueue Tasks + NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded. + INFO: Creating image(s)... + + INFO: The new image(s) can be found here: + ./directdisksdb-gpt-201710090938-sdb.direct + + The following build artifacts were used to create the image(s): + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native + + INFO: The image(s) were created using OE kickstart file: + /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks + </literallayout> + Continuing with the example, you can now directly + <filename>dd</filename> the image to a USB stick, or + whatever media for which you built your image, + and boot the resulting media: + <literallayout class='monospaced'> + $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb + 140966+0 records in + 140966+0 records out + 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s + $ sudo eject /dev/sdb + </literallayout> + </para> + </section> + + <section id='using-a-modified-kickstart-file-and-running-in-raw-mode'> + <title>Using a Modified Kickstart File and Running in Raw Mode</title> + + <para> + This next example manually specifies each build artifact + (runs in Raw Mode) and uses a modified kickstart file. + The example also uses the <filename>-o</filename> option + to cause Wic to create the output + somewhere other than the default output directory, + which is the current directory: + <literallayout class='monospaced'> + $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \ + --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \ + --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \ + --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \ + --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native + + INFO: Creating image(s)... + + INFO: The new image(s) can be found here: + /home/stephano/testwic/test-201710091445-sdb.direct + + The following build artifacts were used to create the image(s): + ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share + KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86 + NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native + + INFO: The image(s) were created using OE kickstart file: + /home/stephano/my_yocto/test.wks + </literallayout> + For this example, + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + did not have to be specified in the + <filename>local.conf</filename> file since the + artifact is manually specified. + </para> + </section> + + <section id='using-wic-to-manipulate-an-image'> + <title>Using Wic to Manipulate an Image</title> + + <para> + Wic image manipulation allows you to shorten turnaround + time during image development. + For example, you can use Wic to delete the kernel partition + of a Wic image and then insert a newly built kernel. + This saves you time from having to rebuild the entire image + each time you modify the kernel. + <note> + In order to use Wic to manipulate a Wic image as in + this example, your development machine must have the + <filename>mtools</filename> package installed. + </note> + </para> + + <para> + The following example examines the contents of the Wic + image, deletes the existing kernel, and then inserts a + new kernel: + <orderedlist> + <listitem><para> + <emphasis>List the Partitions:</emphasis> + Use the <filename>wic ls</filename> command to list + all the partitions in the Wic image: + <literallayout class='monospaced'> + $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic + Num Start End Size Fstype + 1 1048576 25041919 23993344 fat16 + 2 25165824 72157183 46991360 ext4 + </literallayout> + The previous output shows two partitions in the + <filename>core-image-minimal-qemux86.wic</filename> + image. + </para></listitem> + <listitem><para> + <emphasis>Examine a Particular Partition:</emphasis> + Use the <filename>wic ls</filename> command again + but in a different form to examine a particular + partition. + <note> + You can get command usage on any Wic command + using the following form: + <literallayout class='monospaced'> + $ wic help <replaceable>command</replaceable> + </literallayout> + For example, the following command shows you + the various ways to use the + <filename>wic ls</filename> command: + <literallayout class='monospaced'> + $ wic help ls + </literallayout> + </note> + The following command shows what is in Partition + one: + <literallayout class='monospaced'> + $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1 + Volume in drive : is boot + Volume Serial Number is E894-1809 + Directory for ::/ + + libcom32 c32 186500 2017-10-09 16:06 + libutil c32 24148 2017-10-09 16:06 + syslinux cfg 220 2017-10-09 16:06 + vesamenu c32 27104 2017-10-09 16:06 + vmlinuz 6904608 2017-10-09 16:06 + 5 files 7 142 580 bytes + 16 582 656 bytes free + </literallayout> + The previous output shows five files, with the + <filename>vmlinuz</filename> being the kernel. + <note> + If you see the following error, you need to + update or create a + <filename>~/.mtoolsrc</filename> file and + be sure to have the line “mtools_skip_check=1“ + in the file. + Then, run the Wic command again: + <literallayout class='monospaced'> + ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0 + output: Total number of sectors (47824) not a multiple of sectors per track (32)! + Add mtools_skip_check=1 to your .mtoolsrc file to skip this test + </literallayout> + </note> + </para></listitem> + <listitem><para> + <emphasis>Remove the Old Kernel:</emphasis> + Use the <filename>wic rm</filename> command to + remove the <filename>vmlinuz</filename> file + (kernel): + <literallayout class='monospaced'> + $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Add In the New Kernel:</emphasis> + Use the <filename>wic cp</filename> command to + add the updated kernel to the Wic image. + Depending on how you built your kernel, it could + be in different places. + If you used <filename>devtool</filename> and + an SDK to build your kernel, it resides in the + <filename>tmp/work</filename> directory of the + extensible SDK. + If you used <filename>make</filename> to build the + kernel, the kernel will be in the + <filename>workspace/sources</filename> area. + </para> + + <para>The following example assumes + <filename>devtool</filename> was used to build + the kernel: + <literallayout class='monospaced'> + cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \ + ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz + </literallayout> + Once the new kernel is added back into the image, + you can use the <filename>dd</filename> + command or + <link linkend='flashing-images-using-bmaptool'><filename>bmaptool</filename></link> + to flash your wic image onto an SD card + or USB stick and test your target. + <note> + Using <filename>bmaptool</filename> is + generally 10 to 20 times faster than using + <filename>dd</filename>. + </note> + </para></listitem> + </orderedlist> + </para> + </section> + </section> + </section> + + <section id='flashing-images-using-bmaptool'> + <title>Flashing Images Using <filename>bmaptool</filename></title> + + <para> + A fast and easy way to flash an image to a bootable device + is to use Bmaptool, which is integrated into the OpenEmbedded + build system. + Bmaptool is a generic tool that creates a file's block map (bmap) + and then uses that map to copy the file. + As compared to traditional tools such as dd or cp, Bmaptool + can copy (or flash) large files like raw system image files + much faster. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + If you are using Ubuntu or Debian distributions, you + can install the <filename>bmap-tools</filename> package + using the following command and then use the tool + without specifying <filename>PATH</filename> even from + the root account: + <literallayout class='monospaced'> + $ sudo apt-get install bmap-tools + </literallayout> + </para></listitem> + <listitem><para> + If you are unable to install the + <filename>bmap-tools</filename> package, you will + need to build Bmaptool before using it. + Use the following command: + <literallayout class='monospaced'> + $ bitbake bmap-tools-native + </literallayout> + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + Following, is an example that shows how to flash a Wic image. + Realize that while this example uses a Wic image, you can use + Bmaptool to flash any type of image. + Use these steps to flash an image using Bmaptool: + <orderedlist> + <listitem><para> + <emphasis>Update your <filename>local.conf</filename> File:</emphasis> + You need to have the following set in your + <filename>local.conf</filename> file before building + your image: + <literallayout class='monospaced'> + IMAGE_FSTYPES += "wic wic.bmap" + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Get Your Image:</emphasis> + Either have your image ready (pre-built with the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink> + setting previously mentioned) or take the step to build + the image: + <literallayout class='monospaced'> + $ bitbake <replaceable>image</replaceable> + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Flash the Device:</emphasis> + Flash the device with the image by using Bmaptool + depending on your particular setup. + The following commands assume the image resides in the + Build Directory's <filename>deploy/images/</filename> + area: + <itemizedlist> + <listitem><para> + If you have write access to the media, use this + command form: + <literallayout class='monospaced'> + $ oe-run-native bmap-tools-native bmaptool copy <replaceable>build-directory</replaceable>/tmp/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.wic /dev/sd<replaceable>X</replaceable> + </literallayout> + </para></listitem> + <listitem><para> + If you do not have write access to the media, set + your permissions first and then use the same + command form: + <literallayout class='monospaced'> + $ sudo chmod 666 /dev/sd<replaceable>X</replaceable> + $ oe-run-native bmap-tools-native bmaptool copy <replaceable>build-directory</replaceable>/tmp/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.wic /dev/sd<replaceable>X</replaceable> + </literallayout> + </para></listitem> + </itemizedlist> + </para></listitem> + </orderedlist> + </para> + + <para> + For help on the <filename>bmaptool</filename> command, use the + following command: + <literallayout class='monospaced'> + $ bmaptool --help + </literallayout> + </para> + </section> + + <section id='making-images-more-secure'> + <title>Making Images More Secure</title> + + <para> + Security is of increasing concern for embedded devices. + Consider the issues and problems discussed in just this + sampling of work found across the Internet: + <itemizedlist> + <listitem><para><emphasis> + "<ulink url='https://www.schneier.com/blog/archives/2014/01/security_risks_9.html'>Security Risks of Embedded Systems</ulink>"</emphasis> + by Bruce Schneier + </para></listitem> + <listitem><para><emphasis> + "<ulink url='http://census2012.sourceforge.net/paper.html'>Internet Census 2012</ulink>"</emphasis> + by Carna Botnet</para></listitem> + <listitem><para><emphasis> + "<ulink url='http://elinux.org/images/6/6f/Security-issues.pdf'>Security Issues for Embedded Devices</ulink>"</emphasis> + by Jake Edge + </para></listitem> + </itemizedlist> + </para> + + <para> + When securing your image is of concern, there are steps, tools, + and variables that you can consider to help you reach the + security goals you need for your particular device. + Not all situations are identical when it comes to making an + image secure. + Consequently, this section provides some guidance and suggestions + for consideration when you want to make your image more secure. + <note> + Because the security requirements and risks are + different for every type of device, this section cannot + provide a complete reference on securing your custom OS. + It is strongly recommended that you also consult other sources + of information on embedded Linux system hardening and on + security. + </note> + </para> + + <section id='general-considerations'> + <title>General Considerations</title> + + <para> + General considerations exist that help you create more + secure images. + You should consider the following suggestions to help + make your device more secure: + <itemizedlist> + <listitem><para> + Scan additional code you are adding to the system + (e.g. application code) by using static analysis + tools. + Look for buffer overflows and other potential + security problems. + </para></listitem> + <listitem><para> + Pay particular attention to the security for + any web-based administration interface. + </para> + <para>Web interfaces typically need to perform + administrative functions and tend to need to run with + elevated privileges. + Thus, the consequences resulting from the interface's + security becoming compromised can be serious. + Look for common web vulnerabilities such as + cross-site-scripting (XSS), unvalidated inputs, + and so forth.</para> + <para>As with system passwords, the default credentials + for accessing a web-based interface should not be the + same across all devices. + This is particularly true if the interface is enabled + by default as it can be assumed that many end-users + will not change the credentials. + </para></listitem> + <listitem><para> + Ensure you can update the software on the device to + mitigate vulnerabilities discovered in the future. + This consideration especially applies when your + device is network-enabled. + </para></listitem> + <listitem><para> + Ensure you remove or disable debugging functionality + before producing the final image. + For information on how to do this, see the + "<link linkend='considerations-specific-to-the-openembedded-build-system'>Considerations Specific to the OpenEmbedded Build System</link>" + section. + </para></listitem> + <listitem><para> + Ensure you have no network services listening that + are not needed. + </para></listitem> + <listitem><para> + Remove any software from the image that is not needed. + </para></listitem> + <listitem><para> + Enable hardware support for secure boot functionality + when your device supports this functionality. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='security-flags'> + <title>Security Flags</title> + + <para> + The Yocto Project has security flags that you can enable that + help make your build output more secure. + The security flags are in the + <filename>meta/conf/distro/include/security_flags.inc</filename> + file in your + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. <filename>poky</filename>). + <note> + Depending on the recipe, certain security flags are enabled + and disabled by default. + </note> + </para> + + <para> +<!-- + The GCC/LD flags in <filename>security_flags.inc</filename> + enable more secure code generation. + By including the <filename>security_flags.inc</filename> + file, you enable flags to the compiler and linker that cause + them to generate more secure code. + <note> + The GCC/LD flags are enabled by default in the + <filename>poky-lsb</filename> distribution. + </note> +--> + Use the following line in your + <filename>local.conf</filename> file or in your custom + distribution configuration file to enable the security + compiler and linker flags for your build: + <literallayout class='monospaced'> + require conf/distro/include/security_flags.inc + </literallayout> + </para> + </section> + + <section id='considerations-specific-to-the-openembedded-build-system'> + <title>Considerations Specific to the OpenEmbedded Build System</title> + + <para> + You can take some steps that are specific to the + OpenEmbedded build system to make your images more secure: + <itemizedlist> + <listitem><para> + Ensure "debug-tweaks" is not one of your selected + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>. + When creating a new project, the default is to provide you + with an initial <filename>local.conf</filename> file that + enables this feature using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> variable with the line: + <literallayout class='monospaced'> + EXTRA_IMAGE_FEATURES = "debug-tweaks" + </literallayout> + To disable that feature, simply comment out that line in your + <filename>local.conf</filename> file, or + make sure <filename>IMAGE_FEATURES</filename> does not contain + "debug-tweaks" before producing your final image. + Among other things, leaving this in place sets the + root password as blank, which makes logging in for + debugging or inspection easy during + development but also means anyone can easily log in + during production. + </para></listitem> + <listitem><para> + It is possible to set a root password for the image + and also to set passwords for any extra users you might + add (e.g. administrative or service type users). + When you set up passwords for multiple images or + users, you should not duplicate passwords. + </para> + <para> + To set up passwords, use the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-extrausers'><filename>extrausers</filename></ulink> + class, which is the preferred method. + For an example on how to set up both root and user + passwords, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-extrausers'><filename>extrausers.bbclass</filename></ulink>" + section. + <note> + When adding extra user accounts or setting a + root password, be cautious about setting the + same password on every device. + If you do this, and the password you have set + is exposed, then every device is now potentially + compromised. + If you need this access but want to ensure + security, consider setting a different, + random password for each device. + Typically, you do this as a separate step after + you deploy the image onto the device. + </note> + </para></listitem> + <listitem><para> + Consider enabling a Mandatory Access Control (MAC) + framework such as SMACK or SELinux and tuning it + appropriately for your device's usage. + You can find more information in the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/meta-selinux/'><filename>meta-selinux</filename></ulink> + layer. + </para></listitem> + </itemizedlist> + </para> + + <para> + </para> + </section> + + <section id='tools-for-hardening-your-image'> + <title>Tools for Hardening Your Image</title> + + <para> + The Yocto Project provides tools for making your image + more secure. + You can find these tools in the + <filename>meta-security</filename> layer of the + <ulink url='&YOCTO_GIT_URL;'>Yocto Project Source Repositories</ulink>. + </para> + </section> + </section> + + <section id='creating-your-own-distribution'> + <title>Creating Your Own Distribution</title> + + <para> + When you build an image using the Yocto Project and + do not alter any distribution + <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>, + you are creating a Poky distribution. + If you wish to gain more control over package alternative + selections, compile-time options, and other low-level + configurations, you can create your own distribution. + </para> + + <para> + To create your own distribution, the basic steps consist of + creating your own distribution layer, creating your own + distribution configuration file, and then adding any needed + code and Metadata to the layer. + The following steps provide some more detail: + <itemizedlist> + <listitem><para><emphasis>Create a layer for your new distro:</emphasis> + Create your distribution layer so that you can keep your + Metadata and code for the distribution separate. + It is strongly recommended that you create and use your own + layer for configuration and code. + Using your own layer as compared to just placing + configurations in a <filename>local.conf</filename> + configuration file makes it easier to reproduce the same + build configuration when using multiple build machines. + See the + "<link linkend='creating-a-general-layer-using-the-bitbake-layers-script'>Creating a General Layer Using the <filename>bitbake-layers</filename> Script</link>" + section for information on how to quickly set up a layer. + </para></listitem> + <listitem><para><emphasis>Create the distribution configuration file:</emphasis> + The distribution configuration file needs to be created in + the <filename>conf/distro</filename> directory of your + layer. + You need to name it using your distribution name + (e.g. <filename>mydistro.conf</filename>). + <note> + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable in your + <filename>local.conf</filename> file determines the + name of your distribution. + </note></para> + <para>You can split out parts of your configuration file + into include files and then "require" them from within + your distribution configuration file. + Be sure to place the include files in the + <filename>conf/distro/include</filename> directory of + your layer. + A common example usage of include files would be to + separate out the selection of desired version and revisions + for individual recipes. +</para> + <para>Your configuration file needs to set the following + required variables: + <literallayout class='monospaced'> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_NAME'><filename>DISTRO_NAME</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_VERSION'><filename>DISTRO_VERSION</filename></ulink> + </literallayout> + These following variables are optional and you typically + set them from the distribution configuration file: + <literallayout class='monospaced'> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_EXTRA_RDEPENDS'><filename>DISTRO_EXTRA_RDEPENDS</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_EXTRA_RRECOMMENDS'><filename>DISTRO_EXTRA_RRECOMMENDS</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-TCLIBC'><filename>TCLIBC</filename></ulink> + </literallayout> + <tip> + If you want to base your distribution configuration file + on the very basic configuration from OE-Core, you + can use + <filename>conf/distro/defaultsetup.conf</filename> as + a reference and just include variables that differ + as compared to <filename>defaultsetup.conf</filename>. + Alternatively, you can create a distribution + configuration file from scratch using the + <filename>defaultsetup.conf</filename> file + or configuration files from other distributions + such as Poky or Angstrom as references. + </tip></para></listitem> + <listitem><para><emphasis>Provide miscellaneous variables:</emphasis> + Be sure to define any other variables for which you want to + create a default or enforce as part of the distribution + configuration. + You can include nearly any variable from the + <filename>local.conf</filename> file. + The variables you use are not limited to the list in the + previous bulleted item.</para></listitem> + <listitem><para><emphasis>Point to Your distribution configuration file:</emphasis> + In your <filename>local.conf</filename> file in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>, + set your + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable to point to your distribution's configuration file. + For example, if your distribution's configuration file is + named <filename>mydistro.conf</filename>, then you point + to it as follows: + <literallayout class='monospaced'> + DISTRO = "mydistro" + </literallayout></para></listitem> + <listitem><para><emphasis>Add more to the layer if necessary:</emphasis> + Use your layer to hold other information needed for the + distribution: + <itemizedlist> + <listitem><para>Add recipes for installing + distro-specific configuration files that are not + already installed by another recipe. + If you have distro-specific configuration files + that are included by an existing recipe, you should + add an append file (<filename>.bbappend</filename>) + for those. + For general information and recommendations + on how to add recipes to your layer, see the + "<link linkend='creating-your-own-layer'>Creating Your Own Layer</link>" + and + "<link linkend='best-practices-to-follow-when-creating-layers'>Following Best Practices When Creating Layers</link>" + sections.</para></listitem> + <listitem><para>Add any image recipes that are specific + to your distribution.</para></listitem> + <listitem><para>Add a <filename>psplash</filename> + append file for a branded splash screen. + For information on append files, see the + "<link linkend='using-bbappend-files'>Using .bbappend Files in Your Layer</link>" + section.</para></listitem> + <listitem><para>Add any other append files to make + custom changes that are specific to individual + recipes.</para></listitem> + </itemizedlist></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='creating-a-custom-template-configuration-directory'> + <title>Creating a Custom Template Configuration Directory</title> + + <para> + If you are producing your own customized version + of the build system for use by other users, you might + want to customize the message shown by the setup script or + you might want to change the template configuration files (i.e. + <filename>local.conf</filename> and + <filename>bblayers.conf</filename>) that are created in + a new build directory. + </para> + + <para> + The OpenEmbedded build system uses the environment variable + <filename>TEMPLATECONF</filename> to locate the directory + from which it gathers configuration information that ultimately + ends up in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + <filename>conf</filename> directory. + By default, <filename>TEMPLATECONF</filename> is set as + follows in the <filename>poky</filename> repository: + <literallayout class='monospaced'> + TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf} + </literallayout> + This is the directory used by the build system to find templates + from which to build some key configuration files. + If you look at this directory, you will see the + <filename>bblayers.conf.sample</filename>, + <filename>local.conf.sample</filename>, and + <filename>conf-notes.txt</filename> files. + The build system uses these files to form the respective + <filename>bblayers.conf</filename> file, + <filename>local.conf</filename> file, and display the list of + BitBake targets when running the setup script. + </para> + + <para> + To override these default configuration files with + configurations you want used within every new + Build Directory, simply set the + <filename>TEMPLATECONF</filename> variable to your directory. + The <filename>TEMPLATECONF</filename> variable is set in the + <filename>.templateconf</filename> file, which is in the + top-level + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + folder (e.g. <filename>poky</filename>). + Edit the <filename>.templateconf</filename> so that it can locate + your directory. + </para> + + <para> + Best practices dictate that you should keep your + template configuration directory in your custom distribution layer. + For example, suppose you have a layer named + <filename>meta-mylayer</filename> located in your home directory + and you want your template configuration directory named + <filename>myconf</filename>. + Changing the <filename>.templateconf</filename> as follows + causes the OpenEmbedded build system to look in your directory + and base its configuration files on the + <filename>*.sample</filename> configuration files it finds. + The final configuration files (i.e. + <filename>local.conf</filename> and + <filename>bblayers.conf</filename> ultimately still end up in + your Build Directory, but they are based on your + <filename>*.sample</filename> files. + <literallayout class='monospaced'> + TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf} + </literallayout> + </para> + + <para> + Aside from the <filename>*.sample</filename> configuration files, + the <filename>conf-notes.txt</filename> also resides in the + default <filename>meta-poky/conf</filename> directory. + The script that sets up the build environment + (i.e. + <ulink url="&YOCTO_DOCS_REF_URL;#structure-core-script"><filename>&OE_INIT_FILE;</filename></ulink>) + uses this file to display BitBake targets as part of the script + output. + Customizing this <filename>conf-notes.txt</filename> file is a + good way to make sure your list of custom targets appears + as part of the script's output. + </para> + + <para> + Here is the default list of targets displayed as a result of + running either of the setup scripts: + <literallayout class='monospaced'> + You can now run 'bitbake <target>' + + Common targets are: + core-image-minimal + core-image-sato + meta-toolchain + meta-ide-support + </literallayout> + </para> + + <para> + Changing the listed common targets is as easy as editing your + version of <filename>conf-notes.txt</filename> in your + custom template configuration directory and making sure you + have <filename>TEMPLATECONF</filename> set to your directory. + </para> + </section> + + <section id='dev-saving-memory-during-a-build'> + <title>Conserving Disk Space During Builds</title> + + <para> + To help conserve disk space during builds, you can add the + following statement to your project's + <filename>local.conf</filename> configuration file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "rm_work" + </literallayout> + Adding this statement deletes the work directory used for building + a recipe once the recipe is built. + For more information on "rm_work", see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + class in the Yocto Project Reference Manual. + </para> + </section> + + <section id='working-with-packages'> + <title>Working with Packages</title> + + <para> + This section describes a few tasks that involve packages: + <itemizedlist> + <listitem><para> + <link linkend='excluding-packages-from-an-image'>Excluding packages from an image</link> + </para></listitem> + <listitem><para> + <link linkend='incrementing-a-binary-package-version'>Incrementing a binary package version</link> + </para></listitem> + <listitem><para> + <link linkend='handling-optional-module-packaging'>Handling optional module packaging</link> + </para></listitem> + <listitem><para> + <link linkend='using-runtime-package-management'>Using Runtime Package Management</link> + </para></listitem> + <listitem><para> + <link linkend='testing-packages-with-ptest'>Setting up and running package test (ptest)</link> + </para></listitem> + </itemizedlist> + </para> + + <section id='excluding-packages-from-an-image'> + <title>Excluding Packages from an Image</title> + + <para> + You might find it necessary to prevent specific packages + from being installed into an image. + If so, you can use several variables to direct the build + system to essentially ignore installing recommended packages + or to not install a package at all. + </para> + + <para> + The following list introduces variables you can use to + prevent packages from being installed into your image. + Each of these variables only works with IPK and RPM + package types. + Support for Debian packages does not exist. + Also, you can use these variables from your + <filename>local.conf</filename> file or attach them to a + specific image recipe by using a recipe name override. + For more detail on the variables, see the descriptions in the + Yocto Project Reference Manual's glossary chapter. + <itemizedlist> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-BAD_RECOMMENDATIONS'><filename>BAD_RECOMMENDATIONS</filename></ulink>: + Use this variable to specify "recommended-only" + packages that you do not want installed. + </para></listitem> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-NO_RECOMMENDATIONS'><filename>NO_RECOMMENDATIONS</filename></ulink>: + Use this variable to prevent all "recommended-only" + packages from being installed. + </para></listitem> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>: + Use this variable to prevent specific packages from + being installed regardless of whether they are + "recommended-only" or not. + You need to realize that the build process could + fail with an error when you + prevent the installation of a package whose presence + is required by an installed package. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='incrementing-a-binary-package-version'> + <title>Incrementing a Package Version</title> + + <para> + This section provides some background on how binary package + versioning is accomplished and presents some of the services, + variables, and terminology involved. + </para> + + <para> + In order to understand binary package versioning, you need + to consider the following: + <itemizedlist> + <listitem><para> + Binary Package: The binary package that is eventually + built and installed into an image. + </para></listitem> + <listitem><para> + Binary Package Version: The binary package version + is composed of two components - a version and a + revision. + <note> + Technically, a third component, the "epoch" (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>) + is involved but this discussion for the most part + ignores <filename>PE</filename>. + </note> + The version and revision are taken from the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + variables, respectively. + </para></listitem> + <listitem><para> + <filename>PV</filename>: The recipe version. + <filename>PV</filename> represents the version of the + software being packaged. + Do not confuse <filename>PV</filename> with the + binary package version. + </para></listitem> + <listitem><para> + <filename>PR</filename>: The recipe revision. + </para></listitem> + <listitem><para> + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCPV'><filename>SRCPV</filename></ulink>: + The OpenEmbedded build system uses this string + to help define the value of <filename>PV</filename> + when the source code revision needs to be included + in it. + </para></listitem> + <listitem><para> + <ulink url='https://wiki.yoctoproject.org/wiki/PR_Service'>PR Service</ulink>: + A network-based service that helps automate keeping + package feeds compatible with existing package + manager applications such as RPM, APT, and OPKG. + </para></listitem> + </itemizedlist> + </para> + + <para> + Whenever the binary package content changes, the binary package + version must change. + Changing the binary package version is accomplished by changing + or "bumping" the <filename>PR</filename> and/or + <filename>PV</filename> values. + Increasing these values occurs one of two ways: + <itemizedlist> + <listitem><para>Automatically using a Package Revision + Service (PR Service). + </para></listitem> + <listitem><para>Manually incrementing the + <filename>PR</filename> and/or + <filename>PV</filename> variables. + </para></listitem> + </itemizedlist> + </para> + + <para> + Given a primary challenge of any build system and its users + is how to maintain a package feed that is compatible with + existing package manager applications such as RPM, APT, and + OPKG, using an automated system is much preferred over a + manual system. + In either system, the main requirement is that binary package + version numbering increases in a linear fashion and that a + number of version components exist that support that linear + progression. + For information on how to ensure package revisioning remains + linear, see the + "<link linkend='automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</link>" + section. + </para> + + <para> + The following three sections provide related information on the + PR Service, the manual method for "bumping" + <filename>PR</filename> and/or <filename>PV</filename>, and + on how to ensure binary package revisioning remains linear. + </para> + + <section id='working-with-a-pr-service'> + <title>Working With a PR Service</title> + + <para> + As mentioned, attempting to maintain revision numbers in the + <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> + is error prone, inaccurate, and causes problems for people + submitting recipes. + Conversely, the PR Service automatically generates + increasing numbers, particularly the revision field, + which removes the human element. + <note> + For additional information on using a PR Service, you + can see the + <ulink url='&YOCTO_WIKI_URL;/wiki/PR_Service'>PR Service</ulink> + wiki page. + </note> + </para> + + <para> + The Yocto Project uses variables in order of + decreasing priority to facilitate revision numbering (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + for epoch, version, and revision, respectively). + The values are highly dependent on the policies and + procedures of a given distribution and package feed. + </para> + + <para> + Because the OpenEmbedded build system uses + "<ulink url='&YOCTO_DOCS_OM_URL;#overview-checksums'>signatures</ulink>", + which are unique to a given build, the build system + knows when to rebuild packages. + All the inputs into a given task are represented by a + signature, which can trigger a rebuild when different. + Thus, the build system itself does not rely on the + <filename>PR</filename>, <filename>PV</filename>, and + <filename>PE</filename> numbers to trigger a rebuild. + The signatures, however, can be used to generate + these values. + </para> + + <para> + The PR Service works with both + <filename>OEBasic</filename> and + <filename>OEBasicHash</filename> generators. + The value of <filename>PR</filename> bumps when the + checksum changes and the different generator mechanisms + change signatures under different circumstances. + </para> + + <para> + As implemented, the build system includes values from + the PR Service into the <filename>PR</filename> field as + an addition using the form "<filename>.x</filename>" so + <filename>r0</filename> becomes <filename>r0.1</filename>, + <filename>r0.2</filename> and so forth. + This scheme allows existing <filename>PR</filename> values + to be used for whatever reasons, which include manual + <filename>PR</filename> bumps, should it be necessary. + </para> + + <para> + By default, the PR Service is not enabled or running. + Thus, the packages generated are just "self consistent". + The build system adds and removes packages and + there are no guarantees about upgrade paths but images + will be consistent and correct with the latest changes. + </para> + + <para> + The simplest form for a PR Service is for it to exist + for a single host development system that builds the + package feed (building system). + For this scenario, you can enable a local PR Service by + setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-PRSERV_HOST'><filename>PRSERV_HOST</filename></ulink> + in your <filename>local.conf</filename> file in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + PRSERV_HOST = "localhost:0" + </literallayout> + Once the service is started, packages will automatically + get increasing <filename>PR</filename> values and + BitBake takes care of starting and stopping the server. + </para> + + <para> + If you have a more complex setup where multiple host + development systems work against a common, shared package + feed, you have a single PR Service running and it is + connected to each building system. + For this scenario, you need to start the PR Service using + the <filename>bitbake-prserv</filename> command: + <literallayout class='monospaced'> + bitbake-prserv --host <replaceable>ip</replaceable> --port <replaceable>port</replaceable> --start + </literallayout> + In addition to hand-starting the service, you need to + update the <filename>local.conf</filename> file of each + building system as described earlier so each system + points to the server and port. + </para> + + <para> + It is also recommended you use build history, which adds + some sanity checks to binary package versions, in + conjunction with the server that is running the PR Service. + To enable build history, add the following to each building + system's <filename>local.conf</filename> file: + <literallayout class='monospaced'> + # It is recommended to activate "buildhistory" for testing the PR service + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + For information on build history, see the + "<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>" + section. + </para> + + <note> + <para> + The OpenEmbedded build system does not maintain + <filename>PR</filename> information as part of the + shared state (sstate) packages. + If you maintain an sstate feed, its expected that either + all your building systems that contribute to the sstate + feed use a shared PR Service, or you do not run a PR + Service on any of your building systems. + Having some systems use a PR Service while others do + not leads to obvious problems. + </para> + + <para> + For more information on shared state, see the + "<ulink url='&YOCTO_DOCS_OM_URL;#shared-state-cache'>Shared State Cache</ulink>" + section in the Yocto Project Overview and Concepts + Manual. + </para> + </note> + </section> + + <section id='manually-bumping-pr'> + <title>Manually Bumping PR</title> + + <para> + The alternative to setting up a PR Service is to manually + "bump" the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + variable. + </para> + + <para> + If a committed change results in changing the package + output, then the value of the PR variable needs to be + increased (or "bumped") as part of that commit. + For new recipes you should add the <filename>PR</filename> + variable and set its initial value equal to "r0", which is + the default. + Even though the default value is "r0", the practice of + adding it to a new recipe makes it harder to forget to bump + the variable when you make changes to the recipe in future. + </para> + + <para> + If you are sharing a common <filename>.inc</filename> file + with multiple recipes, you can also use the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-INC_PR'>INC_PR</ulink></filename> + variable to ensure that the recipes sharing the + <filename>.inc</filename> file are rebuilt when the + <filename>.inc</filename> file itself is changed. + The <filename>.inc</filename> file must set + <filename>INC_PR</filename> (initially to "r0"), and all + recipes referring to it should set <filename>PR</filename> + to "${INC_PR}.0" initially, incrementing the last number + when the recipe is changed. + If the <filename>.inc</filename> file is changed then its + <filename>INC_PR</filename> should be incremented. + </para> + + <para> + When upgrading the version of a binary package, assuming the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'>PV</ulink></filename> + changes, the <filename>PR</filename> variable should be + reset to "r0" (or "${INC_PR}.0" if you are using + <filename>INC_PR</filename>). + </para> + + <para> + Usually, version increases occur only to binary packages. + However, if for some reason <filename>PV</filename> changes + but does not increase, you can increase the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PE'>PE</ulink></filename> + variable (Package Epoch). + The <filename>PE</filename> variable defaults to "0". + </para> + + <para> + Binary package version numbering strives to follow the + <ulink url='http://www.debian.org/doc/debian-policy/ch-controlfields.html'> + Debian Version Field Policy Guidelines</ulink>. + These guidelines define how versions are compared and what + "increasing" a version means. + </para> + </section> + + <section id='automatically-incrementing-a-binary-package-revision-number'> + <title>Automatically Incrementing a Package Version Number</title> + + <para> + When fetching a repository, BitBake uses the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + variable to determine the specific source code revision + from which to build. + You set the <filename>SRCREV</filename> variable to + <ulink url='&YOCTO_DOCS_REF_URL;#var-AUTOREV'><filename>AUTOREV</filename></ulink> + to cause the OpenEmbedded build system to automatically use the + latest revision of the software: + <literallayout class='monospaced'> + SRCREV = "${AUTOREV}" + </literallayout> + </para> + + <para> + Furthermore, you need to reference <filename>SRCPV</filename> + in <filename>PV</filename> in order to automatically update + the version whenever the revision of the source code + changes. + Here is an example: + <literallayout class='monospaced'> + PV = "1.0+git${SRCPV}" + </literallayout> + The OpenEmbedded build system substitutes + <filename>SRCPV</filename> with the following: + <literallayout class='monospaced'> + AUTOINC+<replaceable>source_code_revision</replaceable> + </literallayout> + The build system replaces the <filename>AUTOINC</filename> with + a number. + The number used depends on the state of the PR Service: + <itemizedlist> + <listitem><para> + If PR Service is enabled, the build system increments + the number, which is similar to the behavior of + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>. + This behavior results in linearly increasing package + versions, which is desirable. + Here is an example: + <literallayout class='monospaced'> + hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk + hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk + </literallayout> + </para></listitem> + <listitem><para> + If PR Service is not enabled, the build system + replaces the <filename>AUTOINC</filename> + placeholder with zero (i.e. "0"). + This results in changing the package version since + the source revision is included. + However, package versions are not increased linearly. + Here is an example: + <literallayout class='monospaced'> + hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk + hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + In summary, the OpenEmbedded build system does not track the + history of binary package versions for this purpose. + <filename>AUTOINC</filename>, in this case, is comparable to + <filename>PR</filename>. + If PR server is not enabled, <filename>AUTOINC</filename> + in the package version is simply replaced by "0". + If PR server is enabled, the build system keeps track of the + package versions and bumps the number when the package + revision changes. + </para> + </section> + </section> + + <section id='handling-optional-module-packaging'> + <title>Handling Optional Module Packaging</title> + + <para> + Many pieces of software split functionality into optional + modules (or plug-ins) and the plug-ins that are built + might depend on configuration options. + To avoid having to duplicate the logic that determines what + modules are available in your recipe or to avoid having + to package each module by hand, the OpenEmbedded build system + provides functionality to handle module packaging dynamically. + </para> + + <para> + To handle optional module packaging, you need to do two things: + <itemizedlist> + <listitem><para>Ensure the module packaging is actually + done.</para></listitem> + <listitem><para>Ensure that any dependencies on optional + modules from other recipes are satisfied by your recipe. + </para></listitem> + </itemizedlist> + </para> + + <section id='making-sure-the-packaging-is-done'> + <title>Making Sure the Packaging is Done</title> + + <para> + To ensure the module packaging actually gets done, you use + the <filename>do_split_packages</filename> function within + the <filename>populate_packages</filename> Python function + in your recipe. + The <filename>do_split_packages</filename> function + searches for a pattern of files or directories under a + specified path and creates a package for each one it finds + by appending to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + variable and setting the appropriate values for + <filename>FILES_packagename</filename>, + <filename>RDEPENDS_packagename</filename>, + <filename>DESCRIPTION_packagename</filename>, and so forth. + Here is an example from the <filename>lighttpd</filename> + recipe: + <literallayout class='monospaced'> + python populate_packages_prepend () { + lighttpd_libdir = d.expand('${libdir}') + do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', + 'lighttpd-module-%s', 'Lighttpd module for %s', + extra_depends='') + } + </literallayout> + The previous example specifies a number of things in the + call to <filename>do_split_packages</filename>. + <itemizedlist> + <listitem><para>A directory within the files installed + by your recipe through <filename>do_install</filename> + in which to search.</para></listitem> + <listitem><para>A regular expression used to match module + files in that directory. + In the example, note the parentheses () that mark + the part of the expression from which the module + name should be derived.</para></listitem> + <listitem><para>A pattern to use for the package names. + </para></listitem> + <listitem><para>A description for each package. + </para></listitem> + <listitem><para>An empty string for + <filename>extra_depends</filename>, which disables + the default dependency on the main + <filename>lighttpd</filename> package. + Thus, if a file in <filename>${libdir}</filename> + called <filename>mod_alias.so</filename> is found, + a package called <filename>lighttpd-module-alias</filename> + is created for it and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DESCRIPTION'><filename>DESCRIPTION</filename></ulink> + is set to "Lighttpd module for alias".</para></listitem> + </itemizedlist> + </para> + + <para> + Often, packaging modules is as simple as the previous + example. + However, more advanced options exist that you can use + within <filename>do_split_packages</filename> to modify its + behavior. + And, if you need to, you can add more logic by specifying + a hook function that is called for each package. + It is also perfectly acceptable to call + <filename>do_split_packages</filename> multiple times if + you have more than one set of modules to package. + </para> + + <para> + For more examples that show how to use + <filename>do_split_packages</filename>, see the + <filename>connman.inc</filename> file in the + <filename>meta/recipes-connectivity/connman/</filename> + directory of the <filename>poky</filename> + <ulink url='&YOCTO_DOCS_OM_URL;#yocto-project-repositories'>source repository</ulink>. + You can also find examples in + <filename>meta/classes/kernel.bbclass</filename>. + </para> + + <para> + Following is a reference that shows + <filename>do_split_packages</filename> mandatory and + optional arguments: + <literallayout class='monospaced'> + Mandatory arguments + + root + The path in which to search + file_regex + Regular expression to match searched files. + Use parentheses () to mark the part of this + expression that should be used to derive the + module name (to be substituted where %s is + used in other function arguments as noted below) + output_pattern + Pattern to use for the package names. Must + include %s. + description + Description to set for each package. Must + include %s. + + Optional arguments + + postinst + Postinstall script to use for all packages + (as a string) + recursive + True to perform a recursive search - default + False + hook + A hook function to be called for every match. + The function will be called with the following + arguments (in the order listed): + + f + Full path to the file/directory match + pkg + The package name + file_regex + As above + output_pattern + As above + modulename + The module name derived using file_regex + + extra_depends + Extra runtime dependencies (RDEPENDS) to be + set for all packages. The default value of None + causes a dependency on the main package + (${PN}) - if you do not want this, pass empty + string '' for this parameter. + aux_files_pattern + Extra item(s) to be added to FILES for each + package. Can be a single string item or a list + of strings for multiple items. Must include %s. + postrm + postrm script to use for all packages (as a + string) + allow_dirs + True to allow directories to be matched - + default False + prepend + If True, prepend created packages to PACKAGES + instead of the default False which appends them + match_path + match file_regex on the whole relative path to + the root rather than just the file name + aux_files_pattern_verbatim + Extra item(s) to be added to FILES for each + package, using the actual derived module name + rather than converting it to something legal + for a package name. Can be a single string item + or a list of strings for multiple items. Must + include %s. + allow_links + True to allow symlinks to be matched - default + False + summary + Summary to set for each package. Must include %s; + defaults to description if not set. + </literallayout> + </para> + </section> + + <section id='satisfying-dependencies'> + <title>Satisfying Dependencies</title> + + <para> + The second part for handling optional module packaging + is to ensure that any dependencies on optional modules + from other recipes are satisfied by your recipe. + You can be sure these dependencies are satisfied by + using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES_DYNAMIC'><filename>PACKAGES_DYNAMIC</filename></ulink> variable. + Here is an example that continues with the + <filename>lighttpd</filename> recipe shown earlier: + <literallayout class='monospaced'> + PACKAGES_DYNAMIC = "lighttpd-module-.*" + </literallayout> + The name specified in the regular expression can of + course be anything. + In this example, it is <filename>lighttpd-module-</filename> + and is specified as the prefix to ensure that any + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink> + on a package name starting with the prefix are satisfied + during build time. + If you are using <filename>do_split_packages</filename> + as described in the previous section, the value you put in + <filename>PACKAGES_DYNAMIC</filename> should correspond to + the name pattern specified in the call to + <filename>do_split_packages</filename>. + </para> + </section> + </section> + + <section id='using-runtime-package-management'> + <title>Using Runtime Package Management</title> + + <para> + During a build, BitBake always transforms a recipe into one or + more packages. + For example, BitBake takes the <filename>bash</filename> recipe + and produces a number of packages (e.g. + <filename>bash</filename>, <filename>bash-bashbug</filename>, + <filename>bash-completion</filename>, + <filename>bash-completion-dbg</filename>, + <filename>bash-completion-dev</filename>, + <filename>bash-completion-extra</filename>, + <filename>bash-dbg</filename>, and so forth). + Not all generated packages are included in an image. + </para> + + <para> + In several situations, you might need to update, add, remove, + or query the packages on a target device at runtime + (i.e. without having to generate a new image). + Examples of such situations include: + <itemizedlist> + <listitem><para> + You want to provide in-the-field updates to deployed + devices (e.g. security updates). + </para></listitem> + <listitem><para> + You want to have a fast turn-around development cycle + for one or more applications that run on your device. + </para></listitem> + <listitem><para> + You want to temporarily install the "debug" packages + of various applications on your device so that + debugging can be greatly improved by allowing + access to symbols and source debugging. + </para></listitem> + <listitem><para> + You want to deploy a more minimal package selection of + your device but allow in-the-field updates to add a + larger selection for customization. + </para></listitem> + </itemizedlist> + </para> + + <para> + In all these situations, you have something similar to a more + traditional Linux distribution in that in-field devices + are able to receive pre-compiled packages from a server for + installation or update. + Being able to install these packages on a running, + in-field device is what is termed "runtime package + management". + </para> + + <para> + In order to use runtime package management, you + need a host or server machine that serves up the pre-compiled + packages plus the required metadata. + You also need package manipulation tools on the target. + The build machine is a likely candidate to act as the server. + However, that machine does not necessarily have to be the + package server. + The build machine could push its artifacts to another machine + that acts as the server (e.g. Internet-facing). + In fact, doing so is advantageous for a production + environment as getting the packages away from the + development system's build directory prevents accidental + overwrites. + </para> + + <para> + A simple build that targets just one device produces + more than one package database. + In other words, the packages produced by a build are separated + out into a couple of different package groupings based on + criteria such as the target's CPU architecture, the target + board, or the C library used on the target. + For example, a build targeting the <filename>qemux86</filename> + device produces the following three package databases: + <filename>noarch</filename>, <filename>i586</filename>, and + <filename>qemux86</filename>. + If you wanted your <filename>qemux86</filename> device to be + aware of all the packages that were available to it, + you would need to point it to each of these databases + individually. + In a similar way, a traditional Linux distribution usually is + configured to be aware of a number of software repositories + from which it retrieves packages. + </para> + + <para> + Using runtime package management is completely optional and + not required for a successful build or deployment in any + way. + But if you want to make use of runtime package management, + you need to do a couple things above and beyond the basics. + The remainder of this section describes what you need to do. + </para> + + <section id='runtime-package-management-build'> + <title>Build Considerations</title> + + <para> + This section describes build considerations of which you + need to be aware in order to provide support for runtime + package management. + </para> + + <para> + When BitBake generates packages, it needs to know + what format or formats to use. + In your configuration, you use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> + variable to specify the format: + <orderedlist> + <listitem><para> + Open the <filename>local.conf</filename> file + inside your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + (e.g. <filename>~/poky/build/conf/local.conf</filename>). + </para></listitem> + <listitem><para> + Select the desired package format as follows: + <literallayout class='monospaced'> + PACKAGE_CLASSES ?= “package_<replaceable>packageformat</replaceable>” + </literallayout> + where <replaceable>packageformat</replaceable> + can be "ipk", "rpm", "deb", or "tar" which are the + supported package formats. + <note> + Because the Yocto Project supports four + different package formats, you can set the + variable with more than one argument. + However, the OpenEmbedded build system only + uses the first argument when creating an image + or Software Development Kit (SDK). + </note> + </para></listitem> + </orderedlist> + </para> + + <para> + If you would like your image to start off with a basic + package database containing the packages in your current + build as well as to have the relevant tools available on the + target for runtime package management, you can include + "package-management" in the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + variable. + Including "package-management" in this configuration + variable ensures that when the image is assembled for your + target, the image includes the currently-known package + databases as well as the target-specific tools required + for runtime package management to be performed on the + target. + However, this is not strictly necessary. + You could start your image off without any databases + but only include the required on-target package + tool(s). + As an example, you could include "opkg" in your + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink> + variable if you are using the IPK package format. + You can then initialize your target's package database(s) + later once your image is up and running. + </para> + + <para> + Whenever you perform any sort of build step that can + potentially generate a package or modify existing + package, it is always a good idea to re-generate the + package index after the build by using the following + command: + <literallayout class='monospaced'> + $ bitbake package-index + </literallayout> + It might be tempting to build the package and the + package index at the same time with a command such as + the following: + <literallayout class='monospaced'> + $ bitbake <replaceable>some-package</replaceable> package-index + </literallayout> + Do not do this as BitBake does not schedule the package + index for after the completion of the package you are + building. + Consequently, you cannot be sure of the package index + including information for the package you just built. + Thus, be sure to run the package update step separately + after building any packages. + </para> + + <para> + You can use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> + variables to pre-configure target images to use a package + feed. + If you do not define these variables, then manual steps + as described in the subsequent sections are necessary to + configure the target. + You should set these variables before building the image + in order to produce a correctly configured image. + </para> + + <para> + When your build is complete, your packages reside in the + <filename>${TMPDIR}/deploy/<replaceable>packageformat</replaceable></filename> + directory. + For example, if + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink><filename>}</filename> + is <filename>tmp</filename> and your selected package type + is RPM, then your RPM packages are available in + <filename>tmp/deploy/rpm</filename>. + </para> + </section> + + <section id='runtime-package-management-server'> + <title>Host or Server Machine Setup</title> + + <para> + Although other protocols are possible, a server using HTTP + typically serves packages. + If you want to use HTTP, then set up and configure a + web server such as Apache 2, lighttpd, or + SimpleHTTPServer on the machine serving the packages. + </para> + + <para> + To keep things simple, this section describes how to set + up a SimpleHTTPServer web server to share package feeds + from the developer's machine. + Although this server might not be the best for a production + environment, the setup is simple and straight forward. + Should you want to use a different server more suited for + production (e.g. Apache 2, Lighttpd, or Nginx), take the + appropriate steps to do so. + </para> + + <para> + From within the build directory where you have built an + image based on your packaging choice (i.e. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> + setting), simply start the server. + The following example assumes a build directory of + <filename>~/poky/build/tmp/deploy/rpm</filename> and a + <filename>PACKAGE_CLASSES</filename> setting of + "package_rpm": + <literallayout class='monospaced'> + $ cd ~/poky/build/tmp/deploy/rpm + $ python -m SimpleHTTPServer + </literallayout> + </para> + </section> + + <section id='runtime-package-management-target'> + <title>Target Setup</title> + + <para> + Setting up the target differs depending on the + package management system. + This section provides information for RPM, IPK, and DEB. + </para> + + <section id='runtime-package-management-target-rpm'> + <title>Using RPM</title> + + <para> + The + <ulink url='https://en.wikipedia.org/wiki/DNF_(software)'>Dandified Packaging Tool</ulink> + (DNF) performs runtime package management of RPM + packages. + In order to use DNF for runtime package management, + you must perform an initial setup on the target + machine for cases where the + <filename>PACKAGE_FEED_*</filename> variables were not + set as part of the image that is running on the + target. + This means if you built your image and did not not use + these variables as part of the build and your image is + now running on the target, you need to perform the + steps in this section if you want to use runtime + package management. + <note> + For information on the + <filename>PACKAGE_FEED_*</filename> variables, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> + in the Yocto Project Reference Manual variables + glossary. + </note> + </para> + + <para> + On the target, you must inform DNF that package + databases are available. + You do this by creating a file named + <filename>/etc/yum.repos.d/oe-packages.repo</filename> + and defining the <filename>oe-packages</filename>. + </para> + + <para> + As an example, assume the target is able to use the + following package databases: + <filename>all</filename>, <filename>i586</filename>, + and <filename>qemux86</filename> from a server named + <filename>my.server</filename>. + The specifics for setting up the web server are up to + you. + The critical requirement is that the URIs in the + target repository configuration point to the + correct remote location for the feeds. + <note><title>Tip</title> + For development purposes, you can point the web + server to the build system's + <filename>deploy</filename> directory. + However, for production use, it is better to copy + the package directories to a location outside of + the build area and use that location. + Doing so avoids situations where the build system + overwrites or changes the + <filename>deploy</filename> directory. + </note> + </para> + + <para> + When telling DNF where to look for the package + databases, you must declare individual locations + per architecture or a single location used for all + architectures. + You cannot do both: + <itemizedlist> + <listitem><para> + <emphasis>Create an Explicit List of Architectures:</emphasis> + Define individual base URLs to identify where + each package database is located: + <literallayout class='monospaced'> + [oe-packages] + baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all + </literallayout> + This example informs DNF about individual + package databases for all three architectures. + </para></listitem> + <listitem><para> + <emphasis>Create a Single (Full) Package Index:</emphasis> + Define a single base URL that identifies where + a full package database is located: + <literallayout class='monospaced'> + [oe-packages] + baseurl=http://my.server/rpm + </literallayout> + This example informs DNF about a single package + database that contains all the package index + information for all supported architectures. + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you have informed DNF where to find the package + databases, you need to fetch them: + <literallayout class='monospaced'> + # dnf makecache + </literallayout> + DNF is now able to find, install, and upgrade packages + from the specified repository or repositories. + <note> + See the + <ulink url='http://dnf.readthedocs.io/en/latest/'>DNF documentation</ulink> + for additional information. + </note> + </para> + </section> + + <section id='runtime-package-management-target-ipk'> + <title>Using IPK</title> + + <para> + The <filename>opkg</filename> application performs + runtime package management of IPK packages. + You must perform an initial setup for + <filename>opkg</filename> on the target machine + if the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> + variables have not been set or the target image was + built before the variables were set. + </para> + + <para> + The <filename>opkg</filename> application uses + configuration files to find available package + databases. + Thus, you need to create a configuration file inside + the <filename>/etc/opkg/</filename> direction, which + informs <filename>opkg</filename> of any repository + you want to use. + </para> + + <para> + As an example, suppose you are serving packages from a + <filename>ipk/</filename> directory containing the + <filename>i586</filename>, + <filename>all</filename>, and + <filename>qemux86</filename> databases through an + HTTP server named <filename>my.server</filename>. + On the target, create a configuration file + (e.g. <filename>my_repo.conf</filename>) inside the + <filename>/etc/opkg/</filename> directory containing + the following: + <literallayout class='monospaced'> + src/gz all http://my.server/ipk/all + src/gz i586 http://my.server/ipk/i586 + src/gz qemux86 http://my.server/ipk/qemux86 + </literallayout> + Next, instruct <filename>opkg</filename> to fetch + the repository information: + <literallayout class='monospaced'> + # opkg update + </literallayout> + The <filename>opkg</filename> application is now able + to find, install, and upgrade packages from the + specified repository. + </para> + </section> + + <section id='runtime-package-management-target-deb'> + <title>Using DEB</title> + + <para> + The <filename>apt</filename> application performs + runtime package management of DEB packages. + This application uses a source list file to find + available package databases. + You must perform an initial setup for + <filename>apt</filename> on the target machine + if the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_ARCHS'><filename>PACKAGE_FEED_ARCHS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_BASE_PATHS'><filename>PACKAGE_FEED_BASE_PATHS</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_FEED_URIS'><filename>PACKAGE_FEED_URIS</filename></ulink> + variables have not been set or the target image was + built before the variables were set. + </para> + + <para> + To inform <filename>apt</filename> of the repository + you want to use, you might create a list file (e.g. + <filename>my_repo.list</filename>) inside the + <filename>/etc/apt/sources.list.d/</filename> + directory. + As an example, suppose you are serving packages from a + <filename>deb/</filename> directory containing the + <filename>i586</filename>, + <filename>all</filename>, and + <filename>qemux86</filename> databases through an + HTTP server named <filename>my.server</filename>. + The list file should contain: + <literallayout class='monospaced'> + deb http://my.server/deb/all ./ + deb http://my.server/deb/i586 ./ + deb http://my.server/deb/qemux86 ./ + </literallayout> + Next, instruct the <filename>apt</filename> + application to fetch the repository information: + <literallayout class='monospaced'> + # apt-get update + </literallayout> + After this step, <filename>apt</filename> is able + to find, install, and upgrade packages from the + specified repository. + </para> + </section> + </section> + </section> + + <section id='generating-and-using-signed-packages'> + <title>Generating and Using Signed Packages</title> + <para> + In order to add security to RPM packages used during a build, + you can take steps to securely sign them. + Once a signature is verified, the OpenEmbedded build system + can use the package in the build. + If security fails for a signed package, the build system + aborts the build. + </para> + + <para> + This section describes how to sign RPM packages during a build + and how to use signed package feeds (repositories) when + doing a build. + </para> + + <section id='signing-rpm-packages'> + <title>Signing RPM Packages</title> + + <para> + To enable signing RPM packages, you must set up the + following configurations in either your + <filename>local.config</filename> or + <filename>distro.config</filename> file: + <literallayout class='monospaced'> + # Inherit sign_rpm.bbclass to enable signing functionality + INHERIT += " sign_rpm" + # Define the GPG key that will be used for signing. + RPM_GPG_NAME = "<replaceable>key_name</replaceable>" + # Provide passphrase for the key + RPM_GPG_PASSPHRASE = "<replaceable>passphrase</replaceable>" + </literallayout> + <note> + Be sure to supply appropriate values for both + <replaceable>key_name</replaceable> and + <replaceable>passphrase</replaceable> + </note> + Aside from the + <filename>RPM_GPG_NAME</filename> and + <filename>RPM_GPG_PASSPHRASE</filename> variables in the + previous example, two optional variables related to signing + exist: + <itemizedlist> + <listitem><para> + <emphasis><filename>GPG_BIN</filename>:</emphasis> + Specifies a <filename>gpg</filename> binary/wrapper + that is executed when the package is signed. + </para></listitem> + <listitem><para> + <emphasis><filename>GPG_PATH</filename>:</emphasis> + Specifies the <filename>gpg</filename> home + directory used when the package is signed. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='processing-package-feeds'> + <title>Processing Package Feeds</title> + + <para> + In addition to being able to sign RPM packages, you can + also enable signed package feeds for IPK and RPM packages. + </para> + + <para> + The steps you need to take to enable signed package feed + use are similar to the steps used to sign RPM packages. + You must define the following in your + <filename>local.config</filename> or + <filename>distro.config</filename> file: + <literallayout class='monospaced'> + INHERIT += "sign_package_feed" + PACKAGE_FEED_GPG_NAME = "<replaceable>key_name</replaceable>" + PACKAGE_FEED_GPG_PASSPHRASE_FILE = "<replaceable>path_to_file_containing_passphrase</replaceable>" + </literallayout> + For signed package feeds, the passphrase must exist in a + separate file, which is pointed to by the + <filename>PACKAGE_FEED_GPG_PASSPHRASE_FILE</filename> + variable. + Regarding security, keeping a plain text passphrase out of + the configuration is more secure. + </para> + + <para> + Aside from the + <filename>PACKAGE_FEED_GPG_NAME</filename> and + <filename>PACKAGE_FEED_GPG_PASSPHRASE_FILE</filename> + variables, three optional variables related to signed + package feeds exist: + <itemizedlist> + <listitem><para> + <emphasis><filename>GPG_BIN</filename>:</emphasis> + Specifies a <filename>gpg</filename> binary/wrapper + that is executed when the package is signed. + </para></listitem> + <listitem><para> + <emphasis><filename>GPG_PATH</filename>:</emphasis> + Specifies the <filename>gpg</filename> home + directory used when the package is signed. + </para></listitem> + <listitem><para> + <emphasis><filename>PACKAGE_FEED_GPG_SIGNATURE_TYPE</filename>:</emphasis> + Specifies the type of <filename>gpg</filename> + signature. + This variable applies only to RPM and IPK package + feeds. + Allowable values for the + <filename>PACKAGE_FEED_GPG_SIGNATURE_TYPE</filename> + are "ASC", which is the default and specifies ascii + armored, and "BIN", which specifies binary. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='testing-packages-with-ptest'> + <title>Testing Packages With ptest</title> + + <para> + A Package Test (ptest) runs tests against packages built + by the OpenEmbedded build system on the target machine. + A ptest contains at least two items: the actual test, and + a shell script (<filename>run-ptest</filename>) that starts + the test. + The shell script that starts the test must not contain + the actual test - the script only starts the test. + On the other hand, the test can be anything from a simple + shell script that runs a binary and checks the output to + an elaborate system of test binaries and data files. + </para> + + <para> + The test generates output in the format used by + Automake: + <literallayout class='monospaced'> + <replaceable>result</replaceable>: <replaceable>testname</replaceable> + </literallayout> + where the result can be <filename>PASS</filename>, + <filename>FAIL</filename>, or <filename>SKIP</filename>, + and the testname can be any identifying string. + </para> + + <para> + For a list of Yocto Project recipes that are already + enabled with ptest, see the + <ulink url='https://wiki.yoctoproject.org/wiki/Ptest'>Ptest</ulink> + wiki page. + <note> + A recipe is "ptest-enabled" if it inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-ptest'><filename>ptest</filename></ulink> + class. + </note> + </para> + + <section id='adding-ptest-to-your-build'> + <title>Adding ptest to Your Build</title> + + <para> + To add package testing to your build, add the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> + variables to your <filename>local.conf</filename> file, + which is found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " ptest" + EXTRA_IMAGE_FEATURES += "ptest-pkgs" + </literallayout> + Once your build is complete, the ptest files are installed + into the + <filename>/usr/lib/<replaceable>package</replaceable>/ptest</filename> + directory within the image, where + <filename><replaceable>package</replaceable></filename> + is the name of the package. + </para> + </section> + + <section id='running-ptest'> + <title>Running ptest</title> + + <para> + The <filename>ptest-runner</filename> package installs a + shell script that loops through all installed ptest test + suites and runs them in sequence. + Consequently, you might want to add this package to + your image. + </para> + </section> + + <section id='getting-your-package-ready'> + <title>Getting Your Package Ready</title> + + <para> + In order to enable a recipe to run installed ptests + on target hardware, + you need to prepare the recipes that build the packages + you want to test. + Here is what you have to do for each recipe: + <itemizedlist> + <listitem><para><emphasis>Be sure the recipe + inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-ptest'><filename>ptest</filename></ulink> + class:</emphasis> + Include the following line in each recipe: + <literallayout class='monospaced'> + inherit ptest + </literallayout> + </para></listitem> + <listitem><para><emphasis>Create <filename>run-ptest</filename>:</emphasis> + This script starts your test. + Locate the script where you will refer to it + using + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>. + Here is an example that starts a test for + <filename>dbus</filename>: + <literallayout class='monospaced'> + #!/bin/sh + cd test + make -k runtest-TESTS + </literallayout> + </para></listitem> + <listitem><para><emphasis>Ensure dependencies are + met:</emphasis> + If the test adds build or runtime dependencies + that normally do not exist for the package + (such as requiring "make" to run the test suite), + use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + variables in your recipe in order for the package + to meet the dependencies. + Here is an example where the package has a runtime + dependency on "make": + <literallayout class='monospaced'> + RDEPENDS_${PN}-ptest += "make" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Add a function to build the + test suite:</emphasis> + Not many packages support cross-compilation of + their test suites. + Consequently, you usually need to add a + cross-compilation function to the package. + </para> + + <para>Many packages based on Automake compile and + run the test suite by using a single command + such as <filename>make check</filename>. + However, the host <filename>make check</filename> + builds and runs on the same computer, while + cross-compiling requires that the package is built + on the host but executed for the target + architecture (though often, as in the case for + ptest, the execution occurs on the host). + The built version of Automake that ships with the + Yocto Project includes a patch that separates + building and execution. + Consequently, packages that use the unaltered, + patched version of <filename>make check</filename> + automatically cross-compiles.</para> + <para>Regardless, you still must add a + <filename>do_compile_ptest</filename> function to + build the test suite. + Add a function similar to the following to your + recipe: + <literallayout class='monospaced'> + do_compile_ptest() { + oe_runmake buildtest-TESTS + } + </literallayout> + </para></listitem> + <listitem><para><emphasis>Ensure special configurations + are set:</emphasis> + If the package requires special configurations + prior to compiling the test code, you must + insert a <filename>do_configure_ptest</filename> + function into the recipe. + </para></listitem> + <listitem><para><emphasis>Install the test + suite:</emphasis> + The <filename>ptest</filename> class + automatically copies the file + <filename>run-ptest</filename> to the target and + then runs make <filename>install-ptest</filename> + to run the tests. + If this is not enough, you need to create a + <filename>do_install_ptest</filename> function and + make sure it gets called after the + "make install-ptest" completes. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + </section> + + <section id='efficiently-fetching-source-files-during-a-build'> + <title>Efficiently Fetching Source Files During a Build</title> + + <para> + The OpenEmbedded build system works with source files located + through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable. + When you build something using BitBake, a big part of the operation + is locating and downloading all the source tarballs. + For images, downloading all the source for various packages can + take a significant amount of time. + </para> + + <para> + This section shows you how you can use mirrors to speed up + fetching source files and how you can pre-fetch files all of which + leads to more efficient use of resources and time. + </para> + + <section id='setting-up-effective-mirrors'> + <title>Setting up Effective Mirrors</title> + + <para> + A good deal that goes into a Yocto Project + build is simply downloading all of the source tarballs. + Maybe you have been working with another build system + (OpenEmbedded or Angstrom) for which you have built up a + sizable directory of source tarballs. + Or, perhaps someone else has such a directory for which you + have read access. + If so, you can save time by adding statements to your + configuration file so that the build process checks local + directories first for existing tarballs before checking the + Internet. + </para> + + <para> + Here is an efficient way to set it up in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/" + INHERIT += "own-mirrors" + BB_GENERATE_MIRROR_TARBALLS = "1" + # BB_NO_NETWORK = "1" + </literallayout> + </para> + + <para> + In the previous example, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> + variable causes the OpenEmbedded build system to generate + tarballs of the Git repositories and store them in the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> + directory. + Due to performance reasons, generating and storing these + tarballs is not the build system's default behavior. + </para> + + <para> + You can also use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink> + variable. + For an example, see the variable's glossary entry in the + Yocto Project Reference Manual. + </para> + </section> + + <section id='getting-source-files-and-suppressing-the-build'> + <title>Getting Source Files and Suppressing the Build</title> + + <para> + Another technique you can use to ready yourself for a + successive string of build operations, is to pre-fetch + all the source files without actually starting a build. + This technique lets you work through any download issues + and ultimately gathers all the source files into your + download directory + <ulink url='&YOCTO_DOCS_REF_URL;#structure-build-downloads'><filename>build/downloads</filename></ulink>, + which is located with + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>. + </para> + + <para> + Use the following BitBake command form to fetch all the + necessary sources without starting the build: + <literallayout class='monospaced'> + $ bitbake -c <replaceable>target</replaceable> runall="fetch" + </literallayout> + This variation of the BitBake command guarantees that you + have all the sources for that BitBake target should you + disconnect from the Internet and want to do the build + later offline. + </para> + </section> + </section> + + <section id="selecting-an-initialization-manager"> + <title>Selecting an Initialization Manager</title> + + <para> + By default, the Yocto Project uses SysVinit as the initialization + manager. + However, support also exists for systemd, + which is a full replacement for init with + parallel starting of services, reduced shell overhead and other + features that are used by many distributions. + </para> + + <para> + If you want to use SysVinit, you do + not have to do anything. + But, if you want to use systemd, you must + take some steps as described in the following sections. + </para> + + <section id='using-systemd-exclusively'> + <title>Using systemd Exclusively</title> + + <para> + Set the these variables in your distribution configuration + file as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " systemd" + VIRTUAL-RUNTIME_init_manager = "systemd" + </literallayout> + You can also prevent the SysVinit + distribution feature from + being automatically enabled as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit" + </literallayout> + Doing so removes any redundant SysVinit scripts. + </para> + + <para> + To remove initscripts from your image altogether, + set this variable also: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_initscripts = "" + </literallayout> + </para> + + <para> + For information on the backfill variable, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES_BACKFILL_CONSIDERED'><filename>DISTRO_FEATURES_BACKFILL_CONSIDERED</filename></ulink>. + </para> + </section> + + <section id='using-systemd-for-the-main-image-and-using-sysvinit-for-the-rescue-image'> + <title>Using systemd for the Main Image and Using SysVinit for the Rescue Image</title> + + <para> + Set these variables in your distribution configuration + file as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " systemd" + VIRTUAL-RUNTIME_init_manager = "systemd" + </literallayout> + Doing so causes your main image to use the + <filename>packagegroup-core-boot.bb</filename> recipe and + systemd. + The rescue/minimal image cannot use this package group. + However, it can install SysVinit + and the appropriate packages will have support for both + systemd and SysVinit. + </para> + </section> + </section> + + <section id="selecting-dev-manager"> + <title>Selecting a Device Manager</title> + + <para> + The Yocto Project provides multiple ways to manage the device + manager (<filename>/dev</filename>): + <itemizedlist> + <listitem><para><emphasis>Persistent and Pre-Populated<filename>/dev</filename>:</emphasis> + For this case, the <filename>/dev</filename> directory + is persistent and the required device nodes are created + during the build. + </para></listitem> + <listitem><para><emphasis>Use <filename>devtmpfs</filename> with a Device Manager:</emphasis> + For this case, the <filename>/dev</filename> directory + is provided by the kernel as an in-memory file system and + is automatically populated by the kernel at runtime. + Additional configuration of device nodes is done in user + space by a device manager like + <filename>udev</filename> or + <filename>busybox-mdev</filename>. + </para></listitem> + </itemizedlist> + </para> + + <section id="static-dev-management"> + <title>Using Persistent and Pre-Populated<filename>/dev</filename></title> + + <para> + To use the static method for device population, you need to + set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-USE_DEVFS'><filename>USE_DEVFS</filename></ulink> + variable to "0" as follows: + <literallayout class='monospaced'> + USE_DEVFS = "0" + </literallayout> + </para> + + <para> + The content of the resulting <filename>/dev</filename> + directory is defined in a Device Table file. + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_DEVICE_TABLES'><filename>IMAGE_DEVICE_TABLES</filename></ulink> + variable defines the Device Table to use and should be set + in the machine or distro configuration file. + Alternatively, you can set this variable in your + <filename>local.conf</filename> configuration file. + </para> + + <para> + If you do not define the + <filename>IMAGE_DEVICE_TABLES</filename> variable, the default + <filename>device_table-minimal.txt</filename> is used: + <literallayout class='monospaced'> + IMAGE_DEVICE_TABLES = "device_table-mymachine.txt" + </literallayout> + </para> + + <para> + The population is handled by the <filename>makedevs</filename> + utility during image creation: + </para> + </section> + + <section id="devtmpfs-dev-management"> + <title>Using <filename>devtmpfs</filename> and a Device Manager</title> + + <para> + To use the dynamic method for device population, you need to + use (or be sure to set) the + <ulink url='&YOCTO_DOCS_REF_URL;#var-USE_DEVFS'><filename>USE_DEVFS</filename></ulink> + variable to "1", which is the default: + <literallayout class='monospaced'> + USE_DEVFS = "1" + </literallayout> + With this setting, the resulting <filename>/dev</filename> + directory is populated by the kernel using + <filename>devtmpfs</filename>. + Make sure the corresponding kernel configuration variable + <filename>CONFIG_DEVTMPFS</filename> is set when building + you build a Linux kernel. + </para> + + <para> + All devices created by <filename>devtmpfs</filename> will be + owned by <filename>root</filename> and have permissions + <filename>0600</filename>. + </para> + + <para> + To have more control over the device nodes, you can use a + device manager like <filename>udev</filename> or + <filename>busybox-mdev</filename>. + You choose the device manager by defining the + <filename>VIRTUAL-RUNTIME_dev_manager</filename> variable + in your machine or distro configuration file. + Alternatively, you can set this variable in your + <filename>local.conf</filename> configuration file: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_dev_manager = "udev" + + # Some alternative values + # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev" + # VIRTUAL-RUNTIME_dev_manager = "systemd" + </literallayout> + </para> + </section> + </section> + + <section id="platdev-appdev-srcrev"> + <title>Using an External SCM</title> + + <para> + If you're working on a recipe that pulls from an external Source + Code Manager (SCM), it is possible to have the OpenEmbedded build + system notice new recipe changes added to the SCM and then build + the resulting packages that depend on the new recipes by using + the latest versions. + This only works for SCMs from which it is possible to get a + sensible revision number for changes. + Currently, you can do this with Apache Subversion (SVN), Git, and + Bazaar (BZR) repositories. + </para> + + <para> + To enable this behavior, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + of the recipe needs to reference + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCPV'><filename>SRCPV</filename></ulink>. + Here is an example: + <literallayout class='monospaced'> + PV = "1.2.3+git${SRCPV}" + </literallayout> + Then, you can add the following to your + <filename>local.conf</filename>: + <literallayout class='monospaced'> + SRCREV_pn-<replaceable>PN</replaceable> = "${AUTOREV}" + </literallayout> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> + is the name of the recipe for which you want to enable automatic source + revision updating. + </para> + + <para> + If you do not want to update your local configuration file, you can + add the following directly to the recipe to finish enabling + the feature: + <literallayout class='monospaced'> + SRCREV = "${AUTOREV}" + </literallayout> + </para> + + <para> + The Yocto Project provides a distribution named + <filename>poky-bleeding</filename>, whose configuration + file contains the line: + <literallayout class='monospaced'> + require conf/distro/include/poky-floating-revisions.inc + </literallayout> + This line pulls in the listed include file that contains + numerous lines of exactly that form: + <literallayout class='monospaced'> + #SRCREV_pn-opkg-native ?= "${AUTOREV}" + #SRCREV_pn-opkg-sdk ?= "${AUTOREV}" + #SRCREV_pn-opkg ?= "${AUTOREV}" + #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}" + #SRCREV_pn-opkg-utils ?= "${AUTOREV}" + SRCREV_pn-gconf-dbus ?= "${AUTOREV}" + SRCREV_pn-matchbox-common ?= "${AUTOREV}" + SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}" + SRCREV_pn-matchbox-desktop ?= "${AUTOREV}" + SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}" + SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}" + SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}" + SRCREV_pn-matchbox-terminal ?= "${AUTOREV}" + SRCREV_pn-matchbox-wm ?= "${AUTOREV}" + SRCREV_pn-settings-daemon ?= "${AUTOREV}" + SRCREV_pn-screenshot ?= "${AUTOREV}" + . + . + . + </literallayout> + These lines allow you to experiment with building a + distribution that tracks the latest development source + for numerous packages. + <note><title>Caution</title> + The <filename>poky-bleeding</filename> distribution + is not tested on a regular basis. + Keep this in mind if you use it. + </note> + </para> + </section> + + <section id='creating-a-read-only-root-filesystem'> + <title>Creating a Read-Only Root Filesystem</title> + + <para> + Suppose, for security reasons, you need to disable + your target device's root filesystem's write permissions + (i.e. you need a read-only root filesystem). + Or, perhaps you are running the device's operating system + from a read-only storage device. + For either case, you can customize your image for + that behavior. + </para> + + <note> + Supporting a read-only root filesystem requires that the system and + applications do not try to write to the root filesystem. + You must configure all parts of the target system to write + elsewhere, or to gracefully fail in the event of attempting to + write to the root filesystem. + </note> + + <section id='creating-the-root-filesystem'> + <title>Creating the Root Filesystem</title> + + <para> + To create the read-only root filesystem, simply add the + "read-only-rootfs" feature to your image. + Using either of the following statements in your + image recipe or from within the + <filename>local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + causes the build system to create a read-only root filesystem: + <literallayout class='monospaced'> + IMAGE_FEATURES = "read-only-rootfs" + </literallayout> + or + <literallayout class='monospaced'> + EXTRA_IMAGE_FEATURES += "read-only-rootfs" + </literallayout> + </para> + + <para> + For more information on how to use these variables, see the + "<link linkend='usingpoky-extend-customimage-imagefeatures'>Customizing Images Using Custom <filename>IMAGE_FEATURES</filename> and <filename>EXTRA_IMAGE_FEATURES</filename></link>" + section. + For information on the variables, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink>. + </para> + </section> + + <section id='post-installation-scripts'> + <title>Post-Installation Scripts</title> + + <para> + It is very important that you make sure all + post-Installation (<filename>pkg_postinst</filename>) scripts + for packages that are installed into the image can be run + at the time when the root filesystem is created during the + build on the host system. + These scripts cannot attempt to run during first-boot on the + target device. + With the "read-only-rootfs" feature enabled, + the build system checks during root filesystem creation to make + sure all post-installation scripts succeed. + If any of these scripts still need to be run after the root + filesystem is created, the build immediately fails. + These build-time checks ensure that the build fails + rather than the target device fails later during its + initial boot operation. + </para> + + <para> + Most of the common post-installation scripts generated by the + build system for the out-of-the-box Yocto Project are engineered + so that they can run during root filesystem creation + (e.g. post-installation scripts for caching fonts). + However, if you create and add custom scripts, you need + to be sure they can be run during this file system creation. + </para> + + <para> + Here are some common problems that prevent + post-installation scripts from running during root filesystem + creation: + <itemizedlist> + <listitem><para> + <emphasis>Not using $D in front of absolute + paths:</emphasis> + The build system defines + <filename>$</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> + when the root filesystem is created. + Furthermore, <filename>$D</filename> is blank when the + script is run on the target device. + This implies two purposes for <filename>$D</filename>: + ensuring paths are valid in both the host and target + environments, and checking to determine which + environment is being used as a method for taking + appropriate actions. + </para></listitem> + <listitem><para> + <emphasis>Attempting to run processes that are + specific to or dependent on the target + architecture:</emphasis> + You can work around these attempts by using native + tools, which run on the host system, + to accomplish the same tasks, or + by alternatively running the processes under QEMU, + which has the <filename>qemu_run_binary</filename> + function. + For more information, see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-qemu'><filename>qemu</filename></ulink> + class.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='areas-with-write-access'> + <title>Areas With Write Access</title> + + <para> + With the "read-only-rootfs" feature enabled, + any attempt by the target to write to the root filesystem at + runtime fails. + Consequently, you must make sure that you configure processes + and applications that attempt these types of writes do so + to directories with write access (e.g. + <filename>/tmp</filename> or <filename>/var/run</filename>). + </para> + </section> + </section> + + + + + <section id='maintaining-build-output-quality'> + <title>Maintaining Build Output Quality</title> + + <para> + Many factors can influence the quality of a build. + For example, if you upgrade a recipe to use a new version of an + upstream software package or you experiment with some new + configuration options, subtle changes can occur that you might + not detect until later. + Consider the case where your recipe is using a newer version of + an upstream package. + In this case, a new version of a piece of software might + introduce an optional dependency on another library, which is + auto-detected. + If that library has already been built when the software is + building, the software will link to the built library and that + library will be pulled into your image along with the new + software even if you did not want the library. + </para> + + <para> + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-buildhistory'><filename>buildhistory</filename></ulink> + class exists to help you maintain the quality of your build + output. + You can use the class to highlight unexpected and possibly + unwanted changes in the build output. + When you enable build history, it records information about the + contents of each package and image and then commits that + information to a local Git repository where you can examine + the information. + </para> + + <para> + The remainder of this section describes the following: + <itemizedlist> + <listitem><para> + How you can enable and disable build history + </para></listitem> + <listitem><para> + How to understand what the build history contains + </para></listitem> + <listitem><para> + How to limit the information used for build history + </para></listitem> + <listitem><para> + How to examine the build history from both a + command-line and web interface + </para></listitem> + </itemizedlist> + </para> + + <section id='enabling-and-disabling-build-history'> + <title>Enabling and Disabling Build History</title> + + <para> + Build history is disabled by default. + To enable it, add the following <filename>INHERIT</filename> + statement and set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink> + variable to "1" at the end of your + <filename>conf/local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + Enabling build history as previously described causes the + OpenEmbedded build system to collect build output information + and commit it as a single commit to a local + <ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink> + repository. + <note> + Enabling build history increases your build times slightly, + particularly for images, and increases the amount of disk + space used during the build. + </note> + </para> + + <para> + You can disable build history by removing the previous + statements from your <filename>conf/local.conf</filename> + file. + </para> + </section> + + <section id='understanding-what-the-build-history-contains'> + <title>Understanding What the Build History Contains</title> + + <para> + Build history information is kept in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink><filename>}/buildhistory</filename> + in the Build Directory as defined by the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></ulink> + variable. + The following is an example abbreviated listing: + <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" /> + </para> + + <para> + At the top level, a <filename>metadata-revs</filename> + file exists that lists the revisions of the repositories for + the enabled layers when the build was produced. + The rest of the data splits into separate + <filename>packages</filename>, <filename>images</filename> + and <filename>sdk</filename> directories, the contents of + which are described as follows. + </para> + + <section id='build-history-package-information'> + <title>Build History Package Information</title> + + <para> + The history for each package contains a text file that has + name-value pairs with information about the package. + For example, + <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename> + contains the following: + <literallayout class='monospaced'> + PV = 1.22.1 + PR = r32 + RPROVIDES = + RDEPENDS = glibc (>= 2.20) update-alternatives-opkg + RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d + PKGSIZE = 540168 + FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ + /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ + /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ + /usr/share/pixmaps /usr/share/applications /usr/share/idl \ + /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers + FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ + /etc/busybox.links.nosuid /etc/busybox.links.suid + </literallayout> + Most of these name-value pairs correspond to variables + used to produce the package. + The exceptions are <filename>FILELIST</filename>, which + is the actual list of files in the package, and + <filename>PKGSIZE</filename>, which is the total size of + files in the package in bytes. + </para> + + <para> + A file also exists that corresponds to the recipe from + which the package came (e.g. + <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>): + <literallayout class='monospaced'> + PV = 1.22.1 + PR = r32 + DEPENDS = initscripts kern-tools-native update-rc.d-native \ + virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ + virtual/libc virtual/update-alternatives + PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ + busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ + busybox-staticdev busybox-dev busybox-doc busybox-locale busybox + </literallayout> + </para> + + <para> + Finally, for those recipes fetched from a version control + system (e.g., Git), a file exists that lists source + revisions that are specified in the recipe and lists + the actual revisions used during the build. + Listed and actual revisions might differ when + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + is set to + ${<ulink url='&YOCTO_DOCS_REF_URL;#var-AUTOREV'><filename>AUTOREV</filename></ulink>}. + Here is an example assuming + <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>): + <literallayout class='monospaced'> + # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + </literallayout> + You can use the + <filename>buildhistory-collect-srcrevs</filename> + command with the <filename>-a</filename> option to + collect the stored <filename>SRCREV</filename> values + from build history and report them in a format suitable for + use in global configuration (e.g., + <filename>local.conf</filename> or a distro include file) + to override floating <filename>AUTOREV</filename> values + to a fixed set of revisions. + Here is some example output from this command: + <literallayout class='monospaced'> + $ buildhistory-collect-srcrevs -a + # i586-poky-linux + SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # x86_64-linux + SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa" + SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf" + SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11" + SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3" + SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca" + SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff" + SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # qemux86-poky-linux + SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f" + # all-poky-linux + SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11" + </literallayout> + <note> + Here are some notes on using the + <filename>buildhistory-collect-srcrevs</filename> + command: + <itemizedlist> + <listitem><para> + By default, only values where the + <filename>SRCREV</filename> was not hardcoded + (usually when <filename>AUTOREV</filename> + is used) are reported. + Use the <filename>-a</filename> option to + see all <filename>SRCREV</filename> values. + </para></listitem> + <listitem><para> + The output statements might not have any effect + if overrides are applied elsewhere in the + build system configuration. + Use the <filename>-f</filename> option to add + the <filename>forcevariable</filename> override + to each output line if you need to work around + this restriction. + </para></listitem> + <listitem><para> + The script does apply special handling when + building for multiple machines. + However, the script does place a comment before + each set of values that specifies which + triplet to which they belong as previously + shown (e.g., + <filename>i586-poky-linux</filename>). + </para></listitem> + </itemizedlist> + </note> + </para> + </section> + + <section id='build-history-image-information'> + <title>Build History Image Information</title> + + <para> + The files produced for each image are as follows: + <itemizedlist> + <listitem><para> + <filename>image-files:</filename> + A directory containing selected files from the root + filesystem. + The files are defined by + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></ulink>. + </para></listitem> + <listitem><para> + <filename>build-id.txt:</filename> + Human-readable information about the build + configuration and metadata source revisions. + This file contains the full build header as printed + by BitBake. + </para></listitem> + <listitem><para> + <filename>*.dot:</filename> + Dependency graphs for the image that are + compatible with <filename>graphviz</filename>. + </para></listitem> + <listitem><para> + <filename>files-in-image.txt:</filename> + A list of files in the image with permissions, + owner, group, size, and symlink information. + </para></listitem> + <listitem><para> + <filename>image-info.txt:</filename> + A text file containing name-value pairs with + information about the image. + See the following listing example for more + information. + </para></listitem> + <listitem><para> + <filename>installed-package-names.txt:</filename> + A list of installed packages by name only. + </para></listitem> + <listitem><para> + <filename>installed-package-sizes.txt:</filename> + A list of installed packages ordered by size. + </para></listitem> + <listitem><para> + <filename>installed-packages.txt:</filename> + A list of installed packages with full package + filenames. + </para></listitem> + </itemizedlist> + <note> + Installed package information is able to be gathered + and produced even if package management is disabled + for the final image. + </note> + </para> + + <para> + Here is an example of <filename>image-info.txt</filename>: + <literallayout class='monospaced'> + DISTRO = poky + DISTRO_VERSION = 1.7 + USER_CLASSES = buildstats image-mklibs image-prelink + IMAGE_CLASSES = image_types + IMAGE_FEATURES = debug-tweaks + IMAGE_LINGUAS = + IMAGE_INSTALL = packagegroup-core-boot run-postinsts + BAD_RECOMMENDATIONS = + NO_RECOMMENDATIONS = + PACKAGE_EXCLUDE = + ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \ + write_image_manifest ; buildhistory_list_installed_image ; \ + buildhistory_get_image_installed ; ssh_allow_empty_password; \ + postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ; + IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; + IMAGESIZE = 6900 + </literallayout> + Other than <filename>IMAGESIZE</filename>, which is the + total size of the files in the image in Kbytes, the + name-value pairs are variables that may have influenced the + content of the image. + This information is often useful when you are trying to + determine why a change in the package or file + listings has occurred. + </para> + </section> + + <section id='using-build-history-to-gather-image-information-only'> + <title>Using Build History to Gather Image Information Only</title> + + <para> + As you can see, build history produces image information, + including dependency graphs, so you can see why something + was pulled into the image. + If you are just interested in this information and not + interested in collecting specific package or SDK + information, you can enable writing only image information + without any history by adding the following to your + <filename>conf/local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "0" + BUILDHISTORY_FEATURES = "image" + </literallayout> + Here, you set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></ulink> + variable to use the image feature only. + </para> + </section> + + <section id='build-history-sdk-information'> + <title>Build History SDK Information</title> + + <para> + Build history collects similar information on the contents + of SDKs + (e.g. <filename>bitbake -c populate_sdk imagename</filename>) + as compared to information it collects for images. + Furthermore, this information differs depending on whether + an extensible or standard SDK is being produced. + </para> + + <para> + The following list shows the files produced for SDKs: + <itemizedlist> + <listitem><para> + <filename>files-in-sdk.txt:</filename> + A list of files in the SDK with permissions, + owner, group, size, and symlink information. + This list includes both the host and target parts + of the SDK. + </para></listitem> + <listitem><para> + <filename>sdk-info.txt:</filename> + A text file containing name-value pairs with + information about the SDK. + See the following listing example for more + information. + </para></listitem> + <listitem><para> + <filename>sstate-task-sizes.txt:</filename> + A text file containing name-value pairs with + information about task group sizes + (e.g. <filename>do_populate_sysroot</filename> + tasks have a total size). + The <filename>sstate-task-sizes.txt</filename> file + exists only when an extensible SDK is created. + </para></listitem> + <listitem><para> + <filename>sstate-package-sizes.txt:</filename> + A text file containing name-value pairs with + information for the shared-state packages and + sizes in the SDK. + The <filename>sstate-package-sizes.txt</filename> + file exists only when an extensible SDK is created. + </para></listitem> + <listitem><para> + <filename>sdk-files:</filename> + A folder that contains copies of the files + mentioned in + <filename>BUILDHISTORY_SDK_FILES</filename> if the + files are present in the output. + Additionally, the default value of + <filename>BUILDHISTORY_SDK_FILES</filename> is + specific to the extensible SDK although you can + set it differently if you would like to pull in + specific files from the standard SDK.</para> + + <para>The default files are + <filename>conf/local.conf</filename>, + <filename>conf/bblayers.conf</filename>, + <filename>conf/auto.conf</filename>, + <filename>conf/locked-sigs.inc</filename>, and + <filename>conf/devtool.conf</filename>. + Thus, for an extensible SDK, these files get + copied into the <filename>sdk-files</filename> + directory. + </para></listitem> + <listitem><para> + The following information appears under + each of the <filename>host</filename> + and <filename>target</filename> directories + for the portions of the SDK that run on the host + and on the target, respectively: + <note> + The following files for the most part are empty + when producing an extensible SDK because this + type of SDK is not constructed from packages + as is the standard SDK. + </note> + <itemizedlist> + <listitem><para> + <filename>depends.dot:</filename> + Dependency graph for the SDK that is + compatible with + <filename>graphviz</filename>. + </para></listitem> + <listitem><para> + <filename>installed-package-names.txt:</filename> + A list of installed packages by name only. + </para></listitem> + <listitem><para> + <filename>installed-package-sizes.txt:</filename> + A list of installed packages ordered by size. + </para></listitem> + <listitem><para> + <filename>installed-packages.txt:</filename> + A list of installed packages with full + package filenames. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + + <para> + Here is an example of <filename>sdk-info.txt</filename>: + <literallayout class='monospaced'> + DISTRO = poky + DISTRO_VERSION = 1.3+snapshot-20130327 + SDK_NAME = poky-glibc-i686-arm + SDK_VERSION = 1.3+snapshot + SDKMACHINE = + SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs + BAD_RECOMMENDATIONS = + SDKSIZE = 352712 + </literallayout> + Other than <filename>SDKSIZE</filename>, which is the + total size of the files in the SDK in Kbytes, the + name-value pairs are variables that might have influenced + the content of the SDK. + This information is often useful when you are trying to + determine why a change in the package or file listings + has occurred. + </para> + </section> + + <section id='examining-build-history-information'> + <title>Examining Build History Information</title> + + <para> + You can examine build history output from the command + line or from a web interface. + </para> + + <para> + To see any changes that have occurred (assuming you have + <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink><filename> = "1"</filename>), + you can simply use any Git command that allows you to + view the history of a repository. + Here is one method: + <literallayout class='monospaced'> + $ git log -p + </literallayout> + You need to realize, however, that this method does show + changes that are not significant (e.g. a package's size + changing by a few bytes). + </para> + + <para> + A command-line tool called + <filename>buildhistory-diff</filename> does exist, though, + that queries the Git repository and prints just the + differences that might be significant in human-readable + form. + Here is an example: + <literallayout class='monospaced'> + $ ~/poky/poky/scripts/buildhistory-diff . HEAD^ + Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): + /etc/anotherpkg.conf was added + /sbin/anotherpkg was added + * (installed-package-names.txt): + * anotherpkg was added + Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): + anotherpkg was added + packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + </literallayout> + <note> + The <filename>buildhistory-diff</filename> tool + requires the <filename>GitPython</filename> package. + Be sure to install it using Pip3 as follows: + <literallayout class='monospaced'> + $ pip3 install GitPython --user + </literallayout> + Alternatively, you can install + <filename>python3-git</filename> using the appropriate + distribution package manager (e.g. + <filename>apt-get</filename>, <filename>dnf</filename>, + or <filename>zipper</filename>). + </note> + </para> + + <para> + To see changes to the build history using a web interface, + follow the instruction in the <filename>README</filename> + file here. + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>. + </para> + + <para> + Here is a sample screenshot of the interface: + <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" /> + </para> + </section> + </section> + </section> + + <section id="performing-automated-runtime-testing"> + <title>Performing Automated Runtime Testing</title> + + <para> + The OpenEmbedded build system makes available a series of automated + tests for images to verify runtime functionality. + You can run these tests on either QEMU or actual target hardware. + Tests are written in Python making use of the + <filename>unittest</filename> module, and the majority of them + run commands on the target system over SSH. + This section describes how you set up the environment to use these + tests, run available tests, and write and add your own tests. + </para> + + <para> + For information on the test and QA infrastructure available + within the Yocto Project, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance'>Testing and Quality Assurance</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <section id='enabling-tests'> + <title>Enabling Tests</title> + + <para> + Depending on whether you are planning to run tests using + QEMU or on the hardware, you have to take + different steps to enable the tests. + See the following subsections for information on how to + enable both types of tests. + </para> + + <section id='qemu-image-enabling-tests'> + <title>Enabling Runtime Tests on QEMU</title> + + <para> + In order to run tests, you need to do the following: + <itemizedlist> + <listitem><para><emphasis>Set up to avoid interaction + with <filename>sudo</filename> for networking:</emphasis> + To accomplish this, you must do one of the + following: + <itemizedlist> + <listitem><para>Add + <filename>NOPASSWD</filename> for your user + in <filename>/etc/sudoers</filename> either for + all commands or just for + <filename>runqemu-ifup</filename>. + You must provide the full path as that can + change if you are using multiple clones of the + source repository. + <note> + On some distributions, you also need to + comment out "Defaults requiretty" in + <filename>/etc/sudoers</filename>. + </note></para></listitem> + <listitem><para>Manually configure a tap interface + for your system.</para></listitem> + <listitem><para>Run as root the script in + <filename>scripts/runqemu-gen-tapdevs</filename>, + which should generate a list of tap devices. + This is the option typically chosen for + Autobuilder-type environments. + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + Be sure to use an absolute path + when calling this script + with sudo. + </para></listitem> + <listitem><para> + The package recipe + <filename>qemu-helper-native</filename> + is required to run this script. + Build the package using the + following command: + <literallayout class='monospaced'> + $ bitbake qemu-helper-native + </literallayout> + </para></listitem> + </itemizedlist> + </note> + </para></listitem> + </itemizedlist></para></listitem> + <listitem><para><emphasis>Set the + <filename>DISPLAY</filename> variable:</emphasis> + You need to set this variable so that you have an X + server available (e.g. start + <filename>vncserver</filename> for a headless machine). + </para></listitem> + <listitem><para><emphasis>Be sure your host's firewall + accepts incoming connections from + 192.168.7.0/24:</emphasis> + Some of the tests (in particular DNF tests) start + an HTTP server on a random high number port, + which is used to serve files to the target. + The DNF module serves + <filename>${WORKDIR}/oe-rootfs-repo</filename> + so it can run DNF channel commands. + That means your host's firewall + must accept incoming connections from 192.168.7.0/24, + which is the default IP range used for tap devices + by <filename>runqemu</filename>.</para></listitem> + <listitem><para><emphasis>Be sure your host has the + correct packages installed:</emphasis> + Depending your host's distribution, you need + to have the following packages installed: + <itemizedlist> + <listitem><para>Ubuntu and Debian: + <filename>sysstat</filename> and + <filename>iproute2</filename> + </para></listitem> + <listitem><para>OpenSUSE: + <filename>sysstat</filename> and + <filename>iproute2</filename> + </para></listitem> + <listitem><para>Fedora: + <filename>sysstat</filename> and + <filename>iproute</filename> + </para></listitem> + <listitem><para>CentOS: + <filename>sysstat</filename> and + <filename>iproute</filename> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you start running the tests, the following happens: + <orderedlist> + <listitem><para>A copy of the root filesystem is written + to <filename>${WORKDIR}/testimage</filename>. + </para></listitem> + <listitem><para>The image is booted under QEMU using the + standard <filename>runqemu</filename> script. + </para></listitem> + <listitem><para>A default timeout of 500 seconds occurs + to allow for the boot process to reach the login prompt. + You can change the timeout period by setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_QEMUBOOT_TIMEOUT'><filename>TEST_QEMUBOOT_TIMEOUT</filename></ulink> + in the <filename>local.conf</filename> file. + </para></listitem> + <listitem><para>Once the boot process is reached and the + login prompt appears, the tests run. + The full boot log is written to + <filename>${WORKDIR}/testimage/qemu_boot_log</filename>. + </para></listitem> + <listitem><para>Each test module loads in the order found + in <filename>TEST_SUITES</filename>. + You can find the full output of the commands run over + SSH in + <filename>${WORKDIR}/testimgage/ssh_target_log</filename>. + </para></listitem> + <listitem><para>If no failures occur, the task running the + tests ends successfully. + You can find the output from the + <filename>unittest</filename> in the task log at + <filename>${WORKDIR}/temp/log.do_testimage</filename>. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='hardware-image-enabling-tests'> + <title>Enabling Runtime Tests on Hardware</title> + + <para> + The OpenEmbedded build system can run tests on real + hardware, and for certain devices it can also deploy + the image to be tested onto the device beforehand. + </para> + + <para> + For automated deployment, a "master image" is installed + onto the hardware once as part of setup. + Then, each time tests are to be run, the following + occurs: + <orderedlist> + <listitem><para>The master image is booted into and + used to write the image to be tested to + a second partition. + </para></listitem> + <listitem><para>The device is then rebooted using an + external script that you need to provide. + </para></listitem> + <listitem><para>The device boots into the image to be + tested. + </para></listitem> + </orderedlist> + </para> + + <para> + When running tests (independent of whether the image + has been deployed automatically or not), the device is + expected to be connected to a network on a + pre-determined IP address. + You can either use static IP addresses written into + the image, or set the image to use DHCP and have your + DHCP server on the test network assign a known IP address + based on the MAC address of the device. + </para> + + <para> + In order to run tests on hardware, you need to set + <filename>TEST_TARGET</filename> to an appropriate value. + For QEMU, you do not have to change anything, the default + value is "QemuTarget". + For running tests on hardware, the following options exist: + <itemizedlist> + <listitem><para><emphasis>"SimpleRemoteTarget":</emphasis> + Choose "SimpleRemoteTarget" if you are going to + run tests on a target system that is already + running the image to be tested and is available + on the network. + You can use "SimpleRemoteTarget" in conjunction + with either real hardware or an image running + within a separately started QEMU or any + other virtual machine manager. + </para></listitem> + <listitem><para><emphasis>"Systemd-bootTarget":</emphasis> + Choose "Systemd-bootTarget" if your hardware is + an EFI-based machine with + <filename>systemd-boot</filename> as bootloader and + <filename>core-image-testmaster</filename> + (or something similar) is installed. + Also, your hardware under test must be in a + DHCP-enabled network that gives it the same IP + address for each reboot.</para> + <para>If you choose "Systemd-bootTarget", there are + additional requirements and considerations. + See the + "<link linkend='selecting-systemd-boottarget'>Selecting Systemd-bootTarget</link>" + section, which follows, for more information. + </para></listitem> + <listitem><para><emphasis>"BeagleBoneTarget":</emphasis> + Choose "BeagleBoneTarget" if you are deploying + images and running tests on the BeagleBone + "Black" or original "White" hardware. + For information on how to use these tests, see the + comments at the top of the BeagleBoneTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"EdgeRouterTarget":</emphasis> + Choose "EdgeRouterTarget" is you are deploying + images and running tests on the Ubiquiti Networks + EdgeRouter Lite. + For information on how to use these tests, see the + comments at the top of the EdgeRouterTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"GrubTarget":</emphasis> + Choose the "supports deploying images and running + tests on any generic PC that boots using GRUB. + For information on how to use these tests, see the + comments at the top of the GrubTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"<replaceable>your-target</replaceable>":</emphasis> + Create your own custom target if you want to run + tests when you are deploying images and running + tests on a custom machine within your BSP layer. + To do this, you need to add a Python unit that + defines the target class under + <filename>lib/oeqa/controllers/</filename> within + your layer. + You must also provide an empty + <filename>__init__.py</filename>. + For examples, see files in + <filename>meta-yocto-bsp/lib/oeqa/controllers/</filename>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='selecting-systemd-boottarget'> + <title>Selecting Systemd-bootTarget</title> + + <para> + If you did not set <filename>TEST_TARGET</filename> to + "Systemd-bootTarget", then you do not need any information + in this section. + You can skip down to the + "<link linkend='qemu-image-running-tests'>Running Tests</link>" + section. + </para> + + <para> + If you did set <filename>TEST_TARGET</filename> to + "Systemd-bootTarget", you also need to perform a one-time + setup of your master image by doing the following: + <orderedlist> + <listitem><para><emphasis>Set <filename>EFI_PROVIDER</filename>:</emphasis> + Be sure that <filename>EFI_PROVIDER</filename> + is as follows: + <literallayout class='monospaced'> + EFI_PROVIDER = "systemd-boot" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Build the master image:</emphasis> + Build the <filename>core-image-testmaster</filename> + image. + The <filename>core-image-testmaster</filename> + recipe is provided as an example for a + "master" image and you can customize the image + recipe as you would any other recipe. + </para> + <para>Here are the image recipe requirements: + <itemizedlist> + <listitem><para>Inherits + <filename>core-image</filename> + so that kernel modules are installed. + </para></listitem> + <listitem><para>Installs normal linux utilities + not busybox ones (e.g. + <filename>bash</filename>, + <filename>coreutils</filename>, + <filename>tar</filename>, + <filename>gzip</filename>, and + <filename>kmod</filename>). + </para></listitem> + <listitem><para>Uses a custom + Initial RAM Disk (initramfs) image with a + custom installer. + A normal image that you can install usually + creates a single rootfs partition. + This image uses another installer that + creates a specific partition layout. + Not all Board Support Packages (BSPs) + can use an installer. + For such cases, you need to manually create + the following partition layout on the + target: + <itemizedlist> + <listitem><para>First partition mounted + under <filename>/boot</filename>, + labeled "boot". + </para></listitem> + <listitem><para>The main rootfs + partition where this image gets + installed, which is mounted under + <filename>/</filename>. + </para></listitem> + <listitem><para>Another partition + labeled "testrootfs" where test + images get deployed. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para><emphasis>Install image:</emphasis> + Install the image that you just built on the target + system. + </para></listitem> + </orderedlist> + </para> + + <para> + The final thing you need to do when setting + <filename>TEST_TARGET</filename> to "Systemd-bootTarget" is + to set up the test image: + <orderedlist> + <listitem><para><emphasis>Set up your <filename>local.conf</filename> file:</emphasis> + Make sure you have the following statements in + your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + IMAGE_FSTYPES += "tar.gz" + INHERIT += "testimage" + TEST_TARGET = "Systemd-bootTarget" + TEST_TARGET_IP = "192.168.2.3" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Build your test image:</emphasis> + Use BitBake to build the image: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='power-control'> + <title>Power Control</title> + + <para> + For most hardware targets other than SimpleRemoteTarget, + you can control power: + <itemizedlist> + <listitem><para> + You can use + <filename>TEST_POWERCONTROL_CMD</filename> + together with + <filename>TEST_POWERCONTROL_EXTRA_ARGS</filename> + as a command that runs on the host and does power + cycling. + The test code passes one argument to that command: + off, on or cycle (off then on). + Here is an example that could appear in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" + </literallayout> + In this example, the expect script does the + following: + <literallayout class='monospaced'> + ssh test@10.11.12.1 "pyctl nuc1 <replaceable>arg</replaceable>" + </literallayout> + It then runs a Python script that controls power + for a label called <filename>nuc1</filename>. + <note> + You need to customize + <filename>TEST_POWERCONTROL_CMD</filename> + and + <filename>TEST_POWERCONTROL_EXTRA_ARGS</filename> + for your own setup. + The one requirement is that it accepts + "on", "off", and "cycle" as the last argument. + </note> + </para></listitem> + <listitem><para> + When no command is defined, it connects to the + device over SSH and uses the classic reboot command + to reboot the device. + Classic reboot is fine as long as the machine + actually reboots (i.e. the SSH test has not + failed). + It is useful for scenarios where you have a simple + setup, typically with a single board, and where + some manual interaction is okay from time to time. + </para></listitem> + </itemizedlist> + If you have no hardware to automatically perform power + control but still wish to experiment with automated + hardware testing, you can use the dialog-power-control + script that shows a dialog prompting you to perform the + required power action. + This script requires either KDialog or Zenity to be + installed. + To use this script, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_POWERCONTROL_CMD'><filename>TEST_POWERCONTROL_CMD</filename></ulink> + variable as follows: + <literallayout class='monospaced'> + TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" + </literallayout> + </para> + </section> + + <section id='serial-console-connection'> + <title>Serial Console Connection</title> + + <para> + For test target classes requiring a serial console + to interact with the bootloader (e.g. BeagleBoneTarget, + EdgeRouterTarget, and GrubTarget), you need to + specify a command to use to connect to the serial console + of the target machine by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SERIALCONTROL_CMD'><filename>TEST_SERIALCONTROL_CMD</filename></ulink> + variable and optionally the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SERIALCONTROL_EXTRA_ARGS'><filename>TEST_SERIALCONTROL_EXTRA_ARGS</filename></ulink> + variable. + </para> + + <para> + These cases could be a serial terminal program if the + machine is connected to a local serial port, or a + <filename>telnet</filename> or + <filename>ssh</filename> command connecting to a remote + console server. + Regardless of the case, the command simply needs to + connect to the serial console and forward that connection + to standard input and output as any normal terminal + program does. + For example, to use the picocom terminal program on + serial device <filename>/dev/ttyUSB0</filename> + at 115200bps, you would set the variable as follows: + <literallayout class='monospaced'> + TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" + </literallayout> + For local devices where the serial port device disappears + when the device reboots, an additional "serdevtry" wrapper + script is provided. + To use this wrapper, simply prefix the terminal command + with + <filename>${COREBASE}/scripts/contrib/serdevtry</filename>: + <literallayout class='monospaced'> + TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b +115200 /dev/ttyUSB0" + </literallayout> + </para> + </section> + </section> + + <section id="qemu-image-running-tests"> + <title>Running Tests</title> + + <para> + You can start the tests automatically or manually: + <itemizedlist> + <listitem><para><emphasis>Automatically running tests:</emphasis> + To run the tests automatically after the + OpenEmbedded build system successfully creates an image, + first set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink> + variable to "1" in your <filename>local.conf</filename> + file in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + TEST_IMAGE = "1" + </literallayout> + Next, build your image. + If the image successfully builds, the tests will be + run: + <literallayout class='monospaced'> + bitbake core-image-sato + </literallayout></para></listitem> + <listitem><para><emphasis>Manually running tests:</emphasis> + To manually run the tests, first globally inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> + class by editing your <filename>local.conf</filename> + file: + <literallayout class='monospaced'> + INHERIT += "testimage" + </literallayout> + Next, use BitBake to run the tests: + <literallayout class='monospaced'> + bitbake -c testimage <replaceable>image</replaceable> + </literallayout></para></listitem> + </itemizedlist> + </para> + + <para> + All test files reside in + <filename>meta/lib/oeqa/runtime</filename> in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + A test name maps directly to a Python module. + Each test module may contain a number of individual tests. + Tests are usually grouped together by the area + tested (e.g tests for systemd reside in + <filename>meta/lib/oeqa/runtime/systemd.py</filename>). + </para> + + <para> + You can add tests to any layer provided you place them in the + proper area and you extend + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> + in the <filename>local.conf</filename> file as normal. + Be sure that tests reside in + <filename><replaceable>layer</replaceable>/lib/oeqa/runtime</filename>. + <note> + Be sure that module names do not collide with module names + used in the default set of test modules in + <filename>meta/lib/oeqa/runtime</filename>. + </note> + </para> + + <para> + You can change the set of tests run by appending or overriding + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SUITES'><filename>TEST_SUITES</filename></ulink> + variable in <filename>local.conf</filename>. + Each name in <filename>TEST_SUITES</filename> represents a + required test for the image. + Test modules named within <filename>TEST_SUITES</filename> + cannot be skipped even if a test is not suitable for an image + (e.g. running the RPM tests on an image without + <filename>rpm</filename>). + Appending "auto" to <filename>TEST_SUITES</filename> causes the + build system to try to run all tests that are suitable for the + image (i.e. each test module may elect to skip itself). + </para> + + <para> + The order you list tests in <filename>TEST_SUITES</filename> + is important and influences test dependencies. + Consequently, tests that depend on other tests should be added + after the test on which they depend. + For example, since the <filename>ssh</filename> test + depends on the + <filename>ping</filename> test, "ssh" needs to come after + "ping" in the list. + The test class provides no re-ordering or dependency handling. + <note> + Each module can have multiple classes with multiple test + methods. + And, Python <filename>unittest</filename> rules apply. + </note> + </para> + + <para> + Here are some things to keep in mind when running tests: + <itemizedlist> + <listitem><para>The default tests for the image are defined + as: + <literallayout class='monospaced'> + DEFAULT_TEST_SUITES_pn-<replaceable>image</replaceable> = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" + </literallayout></para></listitem> + <listitem><para>Add your own test to the list of the + by using the following: + <literallayout class='monospaced'> + TEST_SUITES_append = " mytest" + </literallayout></para></listitem> + <listitem><para>Run a specific list of tests as follows: + <literallayout class='monospaced'> + TEST_SUITES = "test1 test2 test3" + </literallayout> + Remember, order is important. + Be sure to place a test that is dependent on another test + later in the order.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id="exporting-tests"> + <title>Exporting Tests</title> + + <para> + You can export tests so that they can run independently of + the build system. + Exporting tests is required if you want to be able to hand + the test execution off to a scheduler. + You can only export tests that are defined in + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SUITES'><filename>TEST_SUITES</filename></ulink>. + </para> + + <para> + If your image is already built, make sure the following are set + in your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT +="testexport" + TEST_TARGET_IP = "<replaceable>IP-address-for-the-test-target</replaceable>" + TEST_SERVER_IP = "<replaceable>IP-address-for-the-test-server</replaceable>" + </literallayout> + You can then export the tests with the following BitBake + command form: + <literallayout class='monospaced'> + $ bitbake <replaceable>image</replaceable> -c testexport + </literallayout> + Exporting the tests places them in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + in + <filename>tmp/testexport/</filename><replaceable>image</replaceable>, + which is controlled by the + <filename>TEST_EXPORT_DIR</filename> variable. + </para> + + <para> + You can now run the tests outside of the build environment: + <literallayout class='monospaced'> + $ cd tmp/testexport/<replaceable>image</replaceable> + $ ./runexported.py testdata.json + </literallayout> + </para> + + <para> + Here is a complete example that shows IP addresses and uses + the <filename>core-image-sato</filename> image: + <literallayout class='monospaced'> + INHERIT +="testexport" + TEST_TARGET_IP = "192.168.7.2" + TEST_SERVER_IP = "192.168.7.1" + </literallayout> + Use BitBake to export the tests: + <literallayout class='monospaced'> + $ bitbake core-image-sato -c testexport + </literallayout> + Run the tests outside of the build environment using the + following: + <literallayout class='monospaced'> + $ cd tmp/testexport/core-image-sato + $ ./runexported.py testdata.json + </literallayout> + </para> + </section> + + <section id="qemu-image-writing-new-tests"> + <title>Writing New Tests</title> + + <para> + As mentioned previously, all new test files need to be in the + proper place for the build system to find them. + New tests for additional functionality outside of the core + should be added to the layer that adds the functionality, in + <filename><replaceable>layer</replaceable>/lib/oeqa/runtime</filename> + (as long as + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> + is extended in the layer's + <filename>layer.conf</filename> file as normal). + Just remember the following: + <itemizedlist> + <listitem><para>Filenames need to map directly to test + (module) names. + </para></listitem> + <listitem><para>Do not use module names that + collide with existing core tests. + </para></listitem> + <listitem><para>Minimally, an empty + <filename>__init__.py</filename> file must exist + in the runtime directory. + </para></listitem> + </itemizedlist> + </para> + + <para> + To create a new test, start by copying an existing module + (e.g. <filename>syslog.py</filename> or + <filename>gcc.py</filename> are good ones to use). + Test modules can use code from + <filename>meta/lib/oeqa/utils</filename>, which are helper + classes. + </para> + + <note> + Structure shell commands such that you rely on them and they + return a single code for success. + Be aware that sometimes you will need to parse the output. + See the <filename>df.py</filename> and + <filename>date.py</filename> modules for examples. + </note> + + <para> + You will notice that all test classes inherit + <filename>oeRuntimeTest</filename>, which is found in + <filename>meta/lib/oetest.py</filename>. + This base class offers some helper attributes, which are + described in the following sections: + </para> + + <section id='qemu-image-writing-tests-class-methods'> + <title>Class Methods</title> + + <para> + Class methods are as follows: + <itemizedlist> + <listitem><para><emphasis><filename>hasPackage(pkg)</filename>:</emphasis> + Returns "True" if <filename>pkg</filename> is in the + installed package list of the image, which is based + on the manifest file that is generated during the + <filename>do_rootfs</filename> task. + </para></listitem> + <listitem><para><emphasis><filename>hasFeature(feature)</filename>:</emphasis> + Returns "True" if the feature is in + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='qemu-image-writing-tests-class-attributes'> + <title>Class Attributes</title> + + <para> + Class attributes are as follows: + <itemizedlist> + <listitem><para><emphasis><filename>pscmd</filename>:</emphasis> + Equals "ps -ef" if <filename>procps</filename> is + installed in the image. + Otherwise, <filename>pscmd</filename> equals + "ps" (busybox). + </para></listitem> + <listitem><para><emphasis><filename>tc</filename>:</emphasis> + The called test context, which gives access to the + following attributes: + <itemizedlist> + <listitem><para><emphasis><filename>d</filename>:</emphasis> + The BitBake datastore, which allows you to + use stuff such as + <filename>oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")</filename>. + </para></listitem> + <listitem><para><emphasis><filename>testslist</filename> and <filename>testsrequired</filename>:</emphasis> + Used internally. + The tests do not need these. + </para></listitem> + <listitem><para><emphasis><filename>filesdir</filename>:</emphasis> + The absolute path to + <filename>meta/lib/oeqa/runtime/files</filename>, + which contains helper files for tests meant + for copying on the target such as small + files written in C for compilation. + </para></listitem> + <listitem><para><emphasis><filename>target</filename>:</emphasis> + The target controller object used to deploy + and start an image on a particular target + (e.g. QemuTarget, SimpleRemote, and + Systemd-bootTarget). + Tests usually use the following: + <itemizedlist> + <listitem><para><emphasis><filename>ip</filename>:</emphasis> + The target's IP address. + </para></listitem> + <listitem><para><emphasis><filename>server_ip</filename>:</emphasis> + The host's IP address, which is + usually used by the DNF test + suite. + </para></listitem> + <listitem><para><emphasis><filename>run(cmd, timeout=None)</filename>:</emphasis> + The single, most used method. + This command is a wrapper for: + <filename>ssh root@host "cmd"</filename>. + The command returns a tuple: + (status, output), which are what + their names imply - the return code + of "cmd" and whatever output + it produces. + The optional timeout argument + represents the number of seconds the + test should wait for "cmd" to + return. + If the argument is "None", the + test uses the default instance's + timeout period, which is 300 + seconds. + If the argument is "0", the test + runs until the command returns. + </para></listitem> + <listitem><para><emphasis><filename>copy_to(localpath, remotepath)</filename>:</emphasis> + <filename>scp localpath root@ip:remotepath</filename>. + </para></listitem> + <listitem><para><emphasis><filename>copy_from(remotepath, localpath)</filename>:</emphasis> + <filename>scp root@host:remotepath localpath</filename>. + </para></listitem> + </itemizedlist></para></listitem> + </itemizedlist></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='qemu-image-writing-tests-instance-attributes'> + <title>Instance Attributes</title> + + <para> + A single instance attribute exists, which is + <filename>target</filename>. + The <filename>target</filename> instance attribute is + identical to the class attribute of the same name, which + is described in the previous section. + This attribute exists as both an instance and class + attribute so tests can use + <filename>self.target.run(cmd)</filename> in instance + methods instead of + <filename>oeRuntimeTest.tc.target.run(cmd)</filename>. + </para> + </section> + </section> + + <section id='installing-packages-in-the-dut-without-the-package-manager'> + <title>Installing Packages in the DUT Without the Package Manager</title> + + <para> + When a test requires a package built by BitBake, it is possible + to install that package. + Installing the package does not require a package manager be + installed in the device under test (DUT). + It does, however, require an SSH connection and the target must + be using the <filename>sshcontrol</filename> class. + <note> + This method uses <filename>scp</filename> to copy files + from the host to the target, which causes permissions and + special attributes to be lost. + </note> + </para> + + <para> + A JSON file is used to define the packages needed by a test. + This file must be in the same path as the file used to define + the tests. + Furthermore, the filename must map directly to the test + module name with a <filename>.json</filename> extension. + </para> + + <para> + The JSON file must include an object with the test name as + keys of an object or an array. + This object (or array of objects) uses the following data: + <itemizedlist> + <listitem><para>"pkg" - A mandatory string that is the + name of the package to be installed. + </para></listitem> + <listitem><para>"rm" - An optional boolean, which defaults + to "false", that specifies to remove the package after + the test. + </para></listitem> + <listitem><para>"extract" - An optional boolean, which + defaults to "false", that specifies if the package must + be extracted from the package format. + When set to "true", the package is not automatically + installed into the DUT. + </para></listitem> + </itemizedlist> + </para> + + <para> + Following is an example JSON file that handles test "foo" + installing package "bar" and test "foobar" installing + packages "foo" and "bar". + Once the test is complete, the packages are removed from the + DUT. + <literallayout class='monospaced'> + { + "foo": { + "pkg": "bar" + }, + "foobar": [ + { + "pkg": "foo", + "rm": true + }, + { + "pkg": "bar", + "rm": true + } + ] + } + </literallayout> + </para> + </section> + </section> + + <section id='usingpoky-debugging-tools-and-techniques'> + <title>Debugging Tools and Techniques</title> + + <para> + The exact method for debugging build failures depends on the nature + of the problem and on the system's area from which the bug + originates. + Standard debugging practices such as comparison against the last + known working version with examination of the changes and the + re-application of steps to identify the one causing the problem are + valid for the Yocto Project just as they are for any other system. + Even though it is impossible to detail every possible potential + failure, this section provides some general tips to aid in + debugging given a variety of situations. + <note><title>Tip</title> + A useful feature for debugging is the error reporting tool. + Configuring the Yocto Project to use this tool causes the + OpenEmbedded build system to produce error reporting commands as + part of the console output. + You can enter the commands after the build completes to log + error information into a common database, that can help you + figure out what might be going wrong. + For information on how to enable and use this feature, see the + "<link linkend='using-the-error-reporting-tool'>Using the Error Reporting Tool</link>" + section. + </note> + </para> + + <para> + The following list shows the debugging topics in the remainder of + this section: + <itemizedlist> + <listitem><para> + "<link linkend='dev-debugging-viewing-logs-from-failed-tasks'>Viewing Logs from Failed Tasks</link>" + describes how to find and view logs from tasks that + failed during the build process. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-viewing-variable-values'>Viewing Variable Values</link>" + describes how to use the BitBake <filename>-e</filename> + option to examine variable values after a recipe has been + parsed. + </para></listitem> + <listitem><para> + "<link linkend='viewing-package-information-with-oe-pkgdata-util'>Viewing Package Information with <filename>oe-pkgdata-util</filename></link>" + describes how to use the + <filename>oe-pkgdata-util</filename> utility to query + <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> + and display package-related information for built + packages. + </para></listitem> + <listitem><para> + "<link linkend='dev-viewing-dependencies-between-recipes-and-tasks'>Viewing Dependencies Between Recipes and Tasks</link>" + describes how to use the BitBake <filename>-g</filename> + option to display recipe dependency information used + during the build. + </para></listitem> + <listitem><para> + "<link linkend='dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>" + describes how to use the + <filename>bitbake-dumpsig</filename> command in + conjunction with key subdirectories in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink> + to determine variable dependencies. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-taskrunning'>Running Specific Tasks</link>" + describes how to use several BitBake options (e.g. + <filename>-c</filename>, <filename>-C</filename>, and + <filename>-f</filename>) to run specific tasks in the + build chain. + It can be useful to run tasks "out-of-order" when trying + isolate build issues. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-bitbake'>General BitBake Problems</link>" + describes how to use BitBake's <filename>-D</filename> + debug output option to reveal more about what BitBake is + doing during the build. + </para></listitem> + <listitem><para> + "<link linkend='dev-debugging-buildfile'>Building with No Dependencies</link>" + describes how to use the BitBake <filename>-b</filename> + option to build a recipe while ignoring dependencies. + </para></listitem> + <listitem><para> + "<link linkend='recipe-logging-mechanisms'>Recipe Logging Mechanisms</link>" + describes how to use the many recipe logging functions + to produce debugging output and report errors and warnings. + </para></listitem> + <listitem><para> + "<link linkend='debugging-parallel-make-races'>Debugging Parallel Make Races</link>" + describes how to debug situations where the build consists + of several parts that are run simultaneously and when the + output or result of one part is not ready for use with a + different part of the build that depends on that output. + </para></listitem> + <listitem><para> + "<link linkend='platdev-gdb-remotedebug'>Debugging With the GNU Project Debugger (GDB) Remotely</link>" + describes how to use GDB to allow you to examine running + programs, which can help you fix problems. + </para></listitem> + <listitem><para> + "<link linkend='debugging-with-the-gnu-project-debugger-gdb-on-the-target'>Debugging with the GNU Project Debugger (GDB) on the Target</link>" + describes how to use GDB directly on target hardware for + debugging. + </para></listitem> + <listitem><para> + "<link linkend='dev-other-debugging-others'>Other Debugging Tips</link>" + describes miscellaneous debugging tips that can be useful. + </para></listitem> + </itemizedlist> + </para> + + <para> + For debugging information within the popular + <trademark class='trade'>Eclipse</trademark> IDE, see the + "<ulink url='&YOCTO_DOCS_SDK_URL;#adt-eclipse'>Working within Eclipse</ulink>" + section in the Yocto Project Application Development and the + Extensible Software Development Kit (eSDK) manual. + </para> + + <section id='dev-debugging-viewing-logs-from-failed-tasks'> + <title>Viewing Logs from Failed Tasks</title> + + <para> + You can find the log for a task in the file + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/temp/log.do_</filename><replaceable>taskname</replaceable>. + For example, the log for the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + task of the QEMU minimal image for the x86 machine + (<filename>qemux86</filename>) might be in + <filename>tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile</filename>. + To see the commands + <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> + ran to generate a log, look at the corresponding + <filename>run.do_</filename><replaceable>taskname</replaceable> + file in the same directory. + </para> + + <para> + <filename>log.do_</filename><replaceable>taskname</replaceable> + and + <filename>run.do_</filename><replaceable>taskname</replaceable> + are actually symbolic links to + <filename>log.do_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable> + and + <filename>log.run_</filename><replaceable>taskname</replaceable><filename>.</filename><replaceable>pid</replaceable>, + where <replaceable>pid</replaceable> is the PID the task had + when it ran. + The symlinks always point to the files corresponding to the most + recent run. + </para> + </section> + + <section id='dev-debugging-viewing-variable-values'> + <title>Viewing Variable Values</title> + + <para> + BitBake's <filename>-e</filename> option is used to display + variable values after parsing. + The following command displays the variable values after the + configuration files (i.e. <filename>local.conf</filename>, + <filename>bblayers.conf</filename>, + <filename>bitbake.conf</filename> and so forth) have been + parsed: + <literallayout class='monospaced'> + $ bitbake -e + </literallayout> + The following command displays variable values after a specific + recipe has been parsed. + The variables include those from the configuration as well: + <literallayout class='monospaced'> + $ bitbake -e recipename + </literallayout> + <note><para> + Each recipe has its own private set of variables + (datastore). + Internally, after parsing the configuration, a copy of the + resulting datastore is made prior to parsing each recipe. + This copying implies that variables set in one recipe will + not be visible to other recipes.</para> + + <para>Likewise, each task within a recipe gets a private + datastore based on the recipe datastore, which means that + variables set within one task will not be visible to + other tasks.</para> + </note> + </para> + + <para> + In the output of <filename>bitbake -e</filename>, each + variable is preceded by a description of how the variable + got its value, including temporary values that were later + overriden. + This description also includes variable flags (varflags) set on + the variable. + The output can be very helpful during debugging. + </para> + + <para> + Variables that are exported to the environment are preceded by + <filename>export</filename> in the output of + <filename>bitbake -e</filename>. + See the following example: + <literallayout class='monospaced'> + export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" + </literallayout> + </para> + + <para> + In addition to variable values, the output of the + <filename>bitbake -e</filename> and + <filename>bitbake -e</filename> <replaceable>recipe</replaceable> + commands includes the following information: + <itemizedlist> + <listitem><para> + The output starts with a tree listing all configuration + files and classes included globally, recursively listing + the files they include or inherit in turn. + Much of the behavior of the OpenEmbedded build system + (including the behavior of the + <ulink url='&YOCTO_DOCS_REF_URL;#normal-recipe-build-tasks'>normal recipe build tasks</ulink>) + is implemented in the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-base'><filename>base</filename></ulink> + class and the classes it inherits, rather than being + built into BitBake itself. + </para></listitem> + <listitem><para> + After the variable values, all functions appear in the + output. + For shell functions, variables referenced within the + function body are expanded. + If a function has been modified using overrides or + using override-style operators like + <filename>_append</filename> and + <filename>_prepend</filename>, then the final assembled + function body appears in the output. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='viewing-package-information-with-oe-pkgdata-util'> + <title>Viewing Package Information with <filename>oe-pkgdata-util</filename></title> + + <para> + You can use the <filename>oe-pkgdata-util</filename> + command-line utility to query + <ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> + and display various package-related information. + When you use the utility, you must use it to view information + on packages that have already been built. + </para> + + <para> + Following are a few of the available + <filename>oe-pkgdata-util</filename> subcommands. + <note> + You can use the standard * and ? globbing wildcards as part + of package names and paths. + </note> + <itemizedlist> + <listitem><para> + <filename>oe-pkgdata-util list-pkgs [</filename><replaceable>pattern</replaceable><filename>]</filename>: + Lists all packages that have been built, optionally + limiting the match to packages that match + <replaceable>pattern</replaceable>. + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util list-pkg-files </filename><replaceable>package</replaceable><filename> ...</filename>: + Lists the files and directories contained in the given + packages. + <note> + <para> + A different way to view the contents of a package is + to look at the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/packages-split</filename> + directory of the recipe that generates the + package. + This directory is created by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> + task and has one subdirectory for each package the + recipe generates, which contains the files stored in + that package.</para> + <para> + If you want to inspect the + <filename>${WORKDIR}/packages-split</filename> + directory, make sure that + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-rm-work'><filename>rm_work</filename></ulink> + is not enabled when you build the recipe. + </para> + </note> + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util find-path </filename><replaceable>path</replaceable><filename> ...</filename>: + Lists the names of the packages that contain the given + paths. + For example, the following tells us that + <filename>/usr/share/man/man1/make.1</filename> + is contained in the <filename>make-doc</filename> + package: + <literallayout class='monospaced'> + $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 + make-doc: /usr/share/man/man1/make.1 + </literallayout> + </para></listitem> + <listitem><para> + <filename>oe-pkgdata-util lookup-recipe </filename><replaceable>package</replaceable><filename> ...</filename>: + Lists the name of the recipes that + produce the given packages. + </para></listitem> + </itemizedlist> + </para> + + <para> + For more information on the <filename>oe-pkgdata-util</filename> + command, use the help facility: + <literallayout class='monospaced'> + $ oe-pkgdata-util ‐‐help + $ oe-pkgdata-util <replaceable>subcommand</replaceable> --help + </literallayout> + </para> + </section> + + <section id='dev-viewing-dependencies-between-recipes-and-tasks'> + <title>Viewing Dependencies Between Recipes and Tasks</title> + + <para> + Sometimes it can be hard to see why BitBake wants to build other + recipes before the one you have specified. + Dependency information can help you understand why a recipe is + built. + </para> + + <para> + To generate dependency information for a recipe, run the + following command: + <literallayout class='monospaced'> + $ bitbake -g <replaceable>recipename</replaceable> + </literallayout> + This command writes the following files in the current + directory: + <itemizedlist> + <listitem><para> + <filename>pn-buildlist</filename>: A list of + recipes/targets involved in building + <replaceable>recipename</replaceable>. + "Involved" here means that at least one task from the + recipe needs to run when building + <replaceable>recipename</replaceable> from scratch. + Targets that are in + <ulink url='&YOCTO_DOCS_REF_URL;#var-ASSUME_PROVIDED'><filename>ASSUME_PROVIDED</filename></ulink> + are not listed. + </para></listitem> + <listitem><para> + <filename>task-depends.dot</filename>: A graph showing + dependencies between tasks. + </para></listitem> + </itemizedlist> + </para> + + <para> + The graphs are in + <ulink url='https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29'>DOT</ulink> + format and can be converted to images (e.g. using the + <filename>dot</filename> tool from + <ulink url='http://www.graphviz.org/'>Graphviz</ulink>). + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + DOT files use a plain text format. + The graphs generated using the + <filename>bitbake -g</filename> command are often so + large as to be difficult to read without special + pruning (e.g. with Bitbake's + <filename>-I</filename> option) and processing. + Despite the form and size of the graphs, the + corresponding <filename>.dot</filename> files can + still be possible to read and provide useful + information. + </para> + + <para>As an example, the + <filename>task-depends.dot</filename> file contains + lines such as the following: + <literallayout class='monospaced'> + "libxslt.do_configure" -> "libxml2.do_populate_sysroot" + </literallayout> + The above example line reveals that the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task in <filename>libxslt</filename> depends on the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink> + task in <filename>libxml2</filename>, which is a + normal + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + dependency between the two recipes. + </para></listitem> + <listitem><para> + For an example of how <filename>.dot</filename> + files can be processed, see the + <filename>scripts/contrib/graph-tool</filename> + Python script, which finds and displays paths + between graph nodes. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + You can use a different method to view dependency information + by using the following command: + <literallayout class='monospaced'> + $ bitbake -g -u taskexp <replaceable>recipename</replaceable> + </literallayout> + This command displays a GUI window from which you can view + build-time and runtime dependencies for the recipes involved in + building <replaceable>recipename</replaceable>. + </para> + </section> + + <section id='dev-viewing-task-variable-dependencies'> + <title>Viewing Task Variable Dependencies</title> + + <para> + As mentioned in the + "<ulink url='&YOCTO_DOCS_BB_URL;#checksums'>Checksums (Signatures)</ulink>" + section of the BitBake User Manual, BitBake tries to + automatically determine what variables a task depends on so + that it can rerun the task if any values of the variables + change. + This determination is usually reliable. + However, if you do things like construct variable names at + runtime, then you might have to manually declare dependencies + on those variables using <filename>vardeps</filename> as + described in the + "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" + section of the BitBake User Manual. + </para> + + <para> + If you are unsure whether a variable dependency is being + picked up automatically for a given task, you can list the + variable dependencies BitBake has determined by doing the + following: + <orderedlist> + <listitem><para> + Build the recipe containing the task: + <literallayout class='monospaced'> + $ bitbake <replaceable>recipename</replaceable> + </literallayout> + </para></listitem> + <listitem><para> + Inside the + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink> + directory, find the signature data + (<filename>sigdata</filename>) file that corresponds + to the task. + The <filename>sigdata</filename> files contain a pickled + Python database of all the metadata that went into + creating the input checksum for the task. + As an example, for the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> + task of the <filename>db</filename> recipe, the + <filename>sigdata</filename> file might be found in the + following location: + <literallayout class='monospaced'> + ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 + </literallayout> + For tasks that are accelerated through the shared state + (<ulink url='&YOCTO_DOCS_OM_URL;#shared-state-cache'>sstate</ulink>) + cache, an additional <filename>siginfo</filename> file + is written into + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> + along with the cached task output. + The <filename>siginfo</filename> files contain exactly + the same information as <filename>sigdata</filename> + files. + </para></listitem> + <listitem><para> + Run <filename>bitbake-dumpsig</filename> on the + <filename>sigdata</filename> or + <filename>siginfo</filename> file. + Here is an example: + <literallayout class='monospaced'> + $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 + </literallayout> + In the output of the above command, you will find a + line like the following, which lists all the (inferred) + variable dependencies for the task. + This list also includes indirect dependencies from + variables depending on other variables, recursively. + <literallayout class='monospaced'> + Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch'] + </literallayout> + <note> + Functions (e.g. <filename>base_do_fetch</filename>) + also count as variable dependencies. + These functions in turn depend on the variables they + reference. + </note> + The output of <filename>bitbake-dumpsig</filename> also + includes the value each variable had, a list of + dependencies for each variable, and + <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHBASE_WHITELIST'><filename>BB_HASHBASE_WHITELIST</filename></ulink> + information. + </para></listitem> + </orderedlist> + </para> + + <para> + There is also a <filename>bitbake-diffsigs</filename> command + for comparing two <filename>siginfo</filename> or + <filename>sigdata</filename> files. + This command can be helpful when trying to figure out what + changed between two versions of a task. + If you call <filename>bitbake-diffsigs</filename> with just one + file, the command behaves like + <filename>bitbake-dumpsig</filename>. + </para> + + <para> + You can also use BitBake to dump out the signature construction + information without executing tasks by using either of the + following BitBake command-line options: + <literallayout class='monospaced'> + ‐‐dump-signatures=<replaceable>SIGNATURE_HANDLER</replaceable> + -S <replaceable>SIGNATURE_HANDLER</replaceable> + </literallayout> + <note> + Two common values for + <replaceable>SIGNATURE_HANDLER</replaceable> are "none" and + "printdiff", which dump only the signature or compare the + dumped signature with the cached one, respectively. + </note> + Using BitBake with either of these options causes BitBake to + dump out <filename>sigdata</filename> files in the + <filename>stamps</filename> directory for every task it would + have executed instead of building the specified target package. + </para> + </section> + + <section id='dev-viewing-metadata-used-to-create-the-input-signature-of-a-shared-state-task'> + <title>Viewing Metadata Used to Create the Input Signature of a Shared State Task</title> + + <para> + Seeing what metadata went into creating the input signature + of a shared state (sstate) task can be a useful debugging + aid. + This information is available in signature information + (<filename>siginfo</filename>) files in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>. + For information on how to view and interpret information in + <filename>siginfo</filename> files, see the + "<link linkend='dev-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>" + section. + </para> + + <para> + For conceptual information on shared state, see the + "<ulink url='&YOCTO_DOCS_OM_URL;#shared-state'>Shared State</ulink>" + section in the Yocto Project Overview and Concepts Manual. + </para> + </section> + + <section id='dev-invalidating-shared-state-to-force-a-task-to-run'> + <title>Invalidating Shared State to Force a Task to Run</title> + + <para> + The OpenEmbedded build system uses + <ulink url='&YOCTO_DOCS_OM_URL;#overview-checksums'>checksums</ulink> + and + <ulink url='&YOCTO_DOCS_OM_URL;#shared-state'>shared state</ulink> + cache to avoid unnecessarily rebuilding tasks. + Collectively, this scheme is known as "shared state code." + </para> + + <para> + As with all schemes, this one has some drawbacks. + It is possible that you could make implicit changes to your + code that the checksum calculations do not take into + account. + These implicit changes affect a task's output but do not + trigger the shared state code into rebuilding a recipe. + Consider an example during which a tool changes its output. + Assume that the output of <filename>rpmdeps</filename> + changes. + The result of the change should be that all the + <filename>package</filename> and + <filename>package_write_rpm</filename> shared state cache + items become invalid. + However, because the change to the output is + external to the code and therefore implicit, + the associated shared state cache items do not become + invalidated. + In this case, the build process uses the cached items + rather than running the task again. + Obviously, these types of implicit changes can cause + problems. + </para> + + <para> + To avoid these problems during the build, you need to + understand the effects of any changes you make. + Realize that changes you make directly to a function + are automatically factored into the checksum calculation. + Thus, these explicit changes invalidate the associated + area of shared state cache. + However, you need to be aware of any implicit changes that + are not obvious changes to the code and could affect + the output of a given task. + </para> + + <para> + When you identify an implicit change, you can easily + take steps to invalidate the cache and force the tasks + to run. + The steps you can take are as simple as changing a + function's comments in the source code. + For example, to invalidate package shared state files, + change the comment statements of + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> + or the comments of one of the functions it calls. + Even though the change is purely cosmetic, it causes the + checksum to be recalculated and forces the build system to + run the task again. + <note> + For an example of a commit that makes a cosmetic + change to invalidate shared state, see this + <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. + </note> + </para> + </section> + + <section id='dev-debugging-taskrunning'> + <title>Running Specific Tasks</title> + + <para> + Any given recipe consists of a set of tasks. + The standard BitBake behavior in most cases is: + <filename>do_fetch</filename>, + <filename>do_unpack</filename>, + <filename>do_patch</filename>, + <filename>do_configure</filename>, + <filename>do_compile</filename>, + <filename>do_install</filename>, + <filename>do_package</filename>, + <filename>do_package_write_*</filename>, and + <filename>do_build</filename>. + The default task is <filename>do_build</filename> and any tasks + on which it depends build first. + Some tasks, such as <filename>do_devshell</filename>, are not + part of the default build chain. + If you wish to run a task that is not part of the default build + chain, you can use the <filename>-c</filename> option in + BitBake. + Here is an example: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c devshell + </literallayout> + </para> + + <para> + The <filename>-c</filename> option respects task dependencies, + which means that all other tasks (including tasks from other + recipes) that the specified task depends on will be run before + the task. + Even when you manually specify a task to run with + <filename>-c</filename>, BitBake will only run the task if it + considers it "out of date". + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#stamp-files-and-the-rerunning-of-tasks'>Stamp Files and the Rerunning of Tasks</ulink>" + section in the Yocto Project Overview and Concepts Manual for + how BitBake determines whether a task is "out of date". + </para> + + <para> + If you want to force an up-to-date task to be rerun (e.g. + because you made manual modifications to the recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> + that you want to try out), then you can use the + <filename>-f</filename> option. + <note> + The reason <filename>-f</filename> is never required when + running the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-devshell'><filename>do_devshell</filename></ulink> + task is because the + <filename>[</filename><ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink><filename>]</filename> + variable flag is already set for the task. + </note> + The following example shows one way you can use the + <filename>-f</filename> option: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop + . + . + make some changes to the source code in the work directory + . + . + $ bitbake matchbox-desktop -c compile -f + $ bitbake matchbox-desktop + </literallayout> + </para> + + <para> + This sequence first builds and then recompiles + <filename>matchbox-desktop</filename>. + The last command reruns all tasks (basically the packaging + tasks) after the compile. + BitBake recognizes that the <filename>do_compile</filename> + task was rerun and therefore understands that the other tasks + also need to be run again. + </para> + + <para> + Another, shorter way to rerun a task and all + <ulink url='&YOCTO_DOCS_REF_URL;#normal-recipe-build-tasks'>normal recipe build tasks</ulink> + that depend on it is to use the <filename>-C</filename> + option. + <note> + This option is upper-cased and is separate from the + <filename>-c</filename> option, which is lower-cased. + </note> + Using this option invalidates the given task and then runs the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-build'><filename>do_build</filename></ulink> + task, which is the default task if no task is given, and the + tasks on which it depends. + You could replace the final two commands in the previous example + with the following single command: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -C compile + </literallayout> + Internally, the <filename>-f</filename> and + <filename>-C</filename> options work by tainting (modifying) the + input checksum of the specified task. + This tainting indirectly causes the task and its + dependent tasks to be rerun through the normal task dependency + mechanisms. + <note> + BitBake explicitly keeps track of which tasks have been + tainted in this fashion, and will print warnings such as the + following for builds involving such tasks: + <literallayout class='monospaced'> + WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run + </literallayout> + The purpose of the warning is to let you know that the work + directory and build output might not be in the clean state + they would be in for a "normal" build, depending on what + actions you took. + To get rid of such warnings, you can remove the work + directory and rebuild the recipe, as follows: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c clean + $ bitbake matchbox-desktop + </literallayout> + </note> + </para> + + <para> + You can view a list of tasks in a given package by running the + <filename>do_listtasks</filename> task as follows: + <literallayout class='monospaced'> + $ bitbake matchbox-desktop -c listtasks + </literallayout> + The results appear as output to the console and are also in the + file <filename>${WORKDIR}/temp/log.do_listtasks</filename>. + </para> + </section> + + <section id='dev-debugging-bitbake'> + <title>General BitBake Problems</title> + + <para> + You can see debug output from BitBake by using the + <filename>-D</filename> option. + The debug output gives more information about what BitBake + is doing and the reason behind it. + Each <filename>-D</filename> option you use increases the + logging level. + The most common usage is <filename>-DDD</filename>. + </para> + + <para> + The output from + <filename>bitbake -DDD -v</filename> <replaceable>targetname</replaceable> + can reveal why BitBake chose a certain version of a package or + why BitBake picked a certain provider. + This command could also help you in a situation where you think + BitBake did something unexpected. + </para> + </section> + + <section id='dev-debugging-buildfile'> + <title>Building with No Dependencies</title> + + <para> + To build a specific recipe (<filename>.bb</filename> file), + you can use the following command form: + <literallayout class='monospaced'> + $ bitbake -b <replaceable>somepath</replaceable>/<replaceable>somerecipe</replaceable>.bb + </literallayout> + This command form does not check for dependencies. + Consequently, you should use it only when you know existing + dependencies have been met. + <note> + You can also specify fragments of the filename. + In this case, BitBake checks for a unique match. + </note> + </para> + </section> + + <section id='recipe-logging-mechanisms'> + <title>Recipe Logging Mechanisms</title> + + <para> + The Yocto Project provides several logging functions for + producing debugging output and reporting errors and warnings. + For Python functions, the following logging functions exist. + All of these functions log to + <filename>${T}/log.do_</filename><replaceable>task</replaceable>, + and can also log to standard output (stdout) with the right + settings: + <itemizedlist> + <listitem><para> + <filename>bb.plain(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes <replaceable>msg</replaceable> as is to the + log while also logging to stdout. + </para></listitem> + <listitem><para> + <filename>bb.note(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "NOTE: <replaceable>msg</replaceable>" to the + log. + Also logs to stdout if BitBake is called with "-v". + </para></listitem> + <listitem><para> + <filename>bb.debug(</filename><replaceable>level</replaceable><filename>, </filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "DEBUG: <replaceable>msg</replaceable>" to the + log. + Also logs to stdout if the log level is greater than or + equal to <replaceable>level</replaceable>. + See the + "<ulink url='&YOCTO_DOCS_BB_URL;#usage-and-syntax'>-D</ulink>" + option in the BitBake User Manual for more information. + </para></listitem> + <listitem><para> + <filename>bb.warn(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "WARNING: <replaceable>msg</replaceable>" to the + log while also logging to stdout. + </para></listitem> + <listitem><para> + <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename>: + Writes "ERROR: <replaceable>msg</replaceable>" to the + log while also logging to standard out (stdout). + <note> + Calling this function does not cause the task to fail. + </note> + </para></listitem> + <listitem><para> + <filename>bb.fatal(</filename><replaceable>msg</replaceable><filename>)</filename>: + This logging function is similar to + <filename>bb.error(</filename><replaceable>msg</replaceable><filename>)</filename> + but also causes the calling task to fail. + <note> + <filename>bb.fatal()</filename> raises an exception, + which means you do not need to put a "return" + statement after the function. + </note> + </para></listitem> + </itemizedlist> + </para> + + <para> + The same logging functions are also available in shell + functions, under the names + <filename>bbplain</filename>, <filename>bbnote</filename>, + <filename>bbdebug</filename>, <filename>bbwarn</filename>, + <filename>bberror</filename>, and <filename>bbfatal</filename>. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-logging'><filename>logging</filename></ulink> + class implements these functions. + See that class in the + <filename>meta/classes</filename> folder of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + for information. + </para> + + <section id='logging-with-python'> + <title>Logging With Python</title> + + <para> + When creating recipes using Python and inserting code that + handles build logs, keep in mind the goal is to have + informative logs while keeping the console as "silent" as + possible. + Also, if you want status messages in the log, use the + "debug" loglevel. + </para> + + <para> + Following is an example written in Python. + The code handles logging for a function that determines the + number of tasks needed to be run. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-listtasks'><filename>do_listtasks</filename></ulink>" + section for additional information: + <literallayout class='monospaced'> + python do_listtasks() { + bb.debug(2, "Starting to figure out the task list") + if noteworthy_condition: + bb.note("There are 47 tasks to run") + bb.debug(2, "Got to point xyz") + if warning_trigger: + bb.warn("Detected warning_trigger, this might be a problem later.") + if recoverable_error: + bb.error("Hit recoverable_error, you really need to fix this!") + if fatal_error: + bb.fatal("fatal_error detected, unable to print the task list") + bb.plain("The tasks present are abc") + bb.debug(2, "Finished figuring out the tasklist") + } + </literallayout> + </para> + </section> + + <section id='logging-with-bash'> + <title>Logging With Bash</title> + + <para> + When creating recipes using Bash and inserting code that + handles build logs, you have the same goals - informative + with minimal console output. + The syntax you use for recipes written in Bash is similar + to that of recipes written in Python described in the + previous section. + </para> + + <para> + Following is an example written in Bash. + The code logs the progress of the <filename>do_my_function</filename> function. + <literallayout class='monospaced'> + do_my_function() { + bbdebug 2 "Running do_my_function" + if [ exceptional_condition ]; then + bbnote "Hit exceptional_condition" + fi + bbdebug 2 "Got to point xyz" + if [ warning_trigger ]; then + bbwarn "Detected warning_trigger, this might cause a problem later." + fi + if [ recoverable_error ]; then + bberror "Hit recoverable_error, correcting" + fi + if [ fatal_error ]; then + bbfatal "fatal_error detected" + fi + bbdebug 2 "Completed do_my_function" + } + </literallayout> + </para> + </section> + </section> + + <section id='debugging-parallel-make-races'> + <title>Debugging Parallel Make Races</title> + + <para> + A parallel <filename>make</filename> race occurs when the build + consists of several parts that are run simultaneously and + a situation occurs when the output or result of one + part is not ready for use with a different part of the build + that depends on that output. + Parallel make races are annoying and can sometimes be difficult + to reproduce and fix. + However, some simple tips and tricks exist that can help + you debug and fix them. + This section presents a real-world example of an error + encountered on the Yocto Project autobuilder and the process + used to fix it. + <note> + If you cannot properly fix a <filename>make</filename> race + condition, you can work around it by clearing either the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> + variables. + </note> + </para> + + <section id='the-failure'> + <title>The Failure</title> + + <para> + For this example, assume that you are building an image that + depends on the "neard" package. + And, during the build, BitBake runs into problems and + creates the following output. + <note> + This example log file has longer lines artificially + broken to make the listing easier to read. + </note> + If you examine the output or the log file, you see the + failure during <filename>make</filename>: + <literallayout class='monospaced'> + | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] + | DEBUG: Executing shell function do_compile + | NOTE: make -j 16 + | make --no-print-directory all-am + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/types.h include/near/types.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/log.h include/near/log.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/tag.h include/near/tag.h + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/setting.h include/near/setting.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/device.h include/near/device.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/snep.h include/near/snep.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/version.h include/near/version.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h + | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h + | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/ + build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/ + yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0 + -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/ + lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/ + tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/ + nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/ + yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3 + -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\" + -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c + -o tools/snep-send.o tools/snep-send.c + | In file included from tools/snep-send.c:16:0: + | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory + | #include <near/dbus.h> + | ^ + | compilation terminated. + | make[1]: *** [tools/snep-send.o] Error 1 + | make[1]: *** Waiting for unfinished jobs.... + | make: *** [all] Error 2 + | ERROR: oe_runmake failed + </literallayout> + </para> + </section> + + <section id='reproducing-the-error'> + <title>Reproducing the Error</title> + + <para> + Because race conditions are intermittent, they do not + manifest themselves every time you do the build. + In fact, most times the build will complete without problems + even though the potential race condition exists. + Thus, once the error surfaces, you need a way to reproduce + it. + </para> + + <para> + In this example, compiling the "neard" package is causing + the problem. + So the first thing to do is build "neard" locally. + Before you start the build, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + variable in your <filename>local.conf</filename> file to + a high number (e.g. "-j 20"). + Using a high value for <filename>PARALLEL_MAKE</filename> + increases the chances of the race condition showing up: + <literallayout class='monospaced'> + $ bitbake neard + </literallayout> + </para> + + <para> + Once the local build for "neard" completes, start a + <filename>devshell</filename> build: + <literallayout class='monospaced'> + $ bitbake neard -c devshell + </literallayout> + For information on how to use a + <filename>devshell</filename>, see the + "<link linkend='platdev-appdev-devshell'>Using a Development Shell</link>" + section. + </para> + + <para> + In the <filename>devshell</filename>, do the following: + <literallayout class='monospaced'> + $ make clean + $ make tools/snep-send.o + </literallayout> + The <filename>devshell</filename> commands cause the failure + to clearly be visible. + In this case, a missing dependency exists for the "neard" + Makefile target. + Here is some abbreviated, sample output with the + missing dependency clearly visible at the end: + <literallayout class='monospaced'> + i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... + . + . + . + tools/snep-send.c + In file included from tools/snep-send.c:16:0: + tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory + #include <near/dbus.h> + ^ + compilation terminated. + make: *** [tools/snep-send.o] Error 1 + $ + </literallayout> + </para> + </section> + + <section id='creating-a-patch-for-the-fix'> + <title>Creating a Patch for the Fix</title> + + <para> + Because there is a missing dependency for the Makefile + target, you need to patch the + <filename>Makefile.am</filename> file, which is generated + from <filename>Makefile.in</filename>. + You can use Quilt to create the patch: + <literallayout class='monospaced'> + $ quilt new parallelmake.patch + Patch patches/parallelmake.patch is now on top + $ quilt add Makefile.am + File Makefile.am added to patch patches/parallelmake.patch + </literallayout> + For more information on using Quilt, see the + "<link linkend='using-a-quilt-workflow'>Using Quilt in Your Workflow</link>" + section. + </para> + + <para> + At this point you need to make the edits to + <filename>Makefile.am</filename> to add the missing + dependency. + For our example, you have to add the following line + to the file: + <literallayout class='monospaced'> + tools/snep-send.$(OBJEXT): include/near/dbus.h + </literallayout> + </para> + + <para> + Once you have edited the file, use the + <filename>refresh</filename> command to create the patch: + <literallayout class='monospaced'> + $ quilt refresh + Refreshed patch patches/parallelmake.patch + </literallayout> + Once the patch file exists, you need to add it back to the + originating recipe folder. + Here is an example assuming a top-level + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + named <filename>poky</filename>: + <literallayout class='monospaced'> + $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard + </literallayout> + The final thing you need to do to implement the fix in the + build is to update the "neard" recipe (i.e. + <filename>neard-0.14.bb</filename>) so that the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statement includes the patch file. + The recipe file is in the folder above the patch. + Here is what the edited <filename>SRC_URI</filename> + statement would look like: + <literallayout class='monospaced'> + SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ + file://neard.in \ + file://neard.service.in \ + file://parallelmake.patch \ + " + </literallayout> + </para> + + <para> + With the patch complete and moved to the correct folder and + the <filename>SRC_URI</filename> statement updated, you can + exit the <filename>devshell</filename>: + <literallayout class='monospaced'> + $ exit + </literallayout> + </para> + </section> + + <section id='testing-the-build'> + <title>Testing the Build</title> + + <para> + With everything in place, you can get back to trying the + build again locally: + <literallayout class='monospaced'> + $ bitbake neard + </literallayout> + This build should succeed. + </para> + + <para> + Now you can open up a <filename>devshell</filename> again + and repeat the clean and make operations as follows: + <literallayout class='monospaced'> + $ bitbake neard -c devshell + $ make clean + $ make tools/snep-send.o + </literallayout> + The build should work without issue. + </para> + + <para> + As with all solved problems, if they originated upstream, + you need to submit the fix for the recipe in OE-Core and + upstream so that the problem is taken care of at its + source. + See the + "<link linkend='how-to-submit-a-change'>Submitting a Change to the Yocto Project</link>" + section for more information. + </para> + </section> + </section> + + <section id="platdev-gdb-remotedebug"> + <title>Debugging With the GNU Project Debugger (GDB) Remotely</title> + + <para> + GDB allows you to examine running programs, which in turn helps + you to understand and fix problems. + It also allows you to perform post-mortem style analysis of + program crashes. + GDB is available as a package within the Yocto Project and is + installed in SDK images by default. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + chapter in the Yocto Project Reference Manual for a description of + these images. + You can find information on GDB at + <ulink url="http://sourceware.org/gdb/"/>. + <note><title>Tip</title> + For best results, install debug (<filename>-dbg</filename>) + packages for the applications you are going to debug. + Doing so makes extra debug symbols available that give you + more meaningful output. + </note> + </para> + + <para> + Sometimes, due to memory or disk space constraints, it is not + possible to use GDB directly on the remote target to debug + applications. + These constraints arise because GDB needs to load the debugging + information and the binaries of the process being debugged. + Additionally, GDB needs to perform many computations to locate + information such as function names, variable names and values, + stack traces and so forth - even before starting the debugging + process. + These extra computations place more load on the target system + and can alter the characteristics of the program being debugged. + </para> + + <para> + To help get past the previously mentioned constraints, you can + use gdbserver, which runs on the remote target and does not + load any debugging information from the debugged process. + Instead, a GDB instance processes the debugging information that + is run on a remote computer - the host GDB. + The host GDB then sends control commands to gdbserver to make + it stop or start the debugged program, as well as read or write + memory regions of that debugged program. + All the debugging information loaded and processed as well + as all the heavy debugging is done by the host GDB. + Offloading these processes gives the gdbserver running on the + target a chance to remain small and fast. + </para> + + <para> + Because the host GDB is responsible for loading the debugging + information and for doing the necessary processing to make + actual debugging happen, you have to make sure the host can + access the unstripped binaries complete with their debugging + information and also be sure the target is compiled with no + optimizations. + The host GDB must also have local access to all the libraries + used by the debugged program. + Because gdbserver does not need any local debugging information, + the binaries on the remote target can remain stripped. + However, the binaries must also be compiled without optimization + so they match the host's binaries. + </para> + + <para> + To remain consistent with GDB documentation and terminology, + the binary being debugged on the remote target machine is + referred to as the "inferior" binary. + For documentation on GDB see the + <ulink url="http://sourceware.org/gdb/documentation/">GDB site</ulink>. + </para> + + <para> + The following steps show you how to debug using the GNU project + debugger. + <orderedlist> + <listitem><para> + <emphasis>Configure your build system to construct the + companion debug filesystem:</emphasis></para> + + <para>In your <filename>local.conf</filename> file, set + the following: + <literallayout class='monospaced'> + IMAGE_GEN_DEBUGFS = "1" + IMAGE_FSTYPES_DEBUGFS = "tar.bz2" + </literallayout> + These options cause the OpenEmbedded build system + to generate a special companion filesystem fragment, + which contains the matching source and debug symbols to + your deployable filesystem. + The build system does this by looking at what is in the + deployed filesystem, and pulling the corresponding + <filename>-dbg</filename> packages.</para> + + <para>The companion debug filesystem is not a complete + filesystem, but only contains the debug fragments. + This filesystem must be combined with the full filesystem + for debugging. + Subsequent steps in this procedure show how to combine + the partial filesystem with the full filesystem. + </para></listitem> + <listitem><para> + <emphasis>Configure the system to include gdbserver in + the target filesystem:</emphasis></para> + + <para>Make the following addition in either your + <filename>local.conf</filename> file or in an image + recipe: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = “ gdbserver" + </literallayout> + The change makes sure the <filename>gdbserver</filename> + package is included. + </para></listitem> + <listitem><para> + <emphasis>Build the environment:</emphasis></para> + + <para>Use the following command to construct the image + and the companion Debug Filesystem: + <literallayout class='monospaced'> + $ bitbake <replaceable>image</replaceable> + </literallayout> + Build the cross GDB component and make it available + for debugging. + Build the SDK that matches the image. + Building the SDK is best for a production build + that can be used later for debugging, especially + during long term maintenance: + <literallayout class='monospaced'> + $ bitbake -c populate_sdk <replaceable>image</replaceable> + </literallayout></para> + + <para>Alternatively, you can build the minimal + toolchain components that match the target. + Doing so creates a smaller than typical SDK and only + contains a minimal set of components with which to + build simple test applications, as well as run the + debugger: + <literallayout class='monospaced'> + $ bitbake meta-toolchain + </literallayout></para> + + <para>A final method is to build Gdb itself within + the build system: + <literallayout class='monospaced'> + $ bitbake gdb-cross-<replaceable>architecture</replaceable> + </literallayout> + Doing so produces a temporary copy of + <filename>cross-gdb</filename> you can use for + debugging during development. + While this is the quickest approach, the two previous + methods in this step are better when considering + long-term maintenance strategies. + <note> + If you run + <filename>bitbake gdb-cross</filename>, the + OpenEmbedded build system suggests the actual + image (e.g. <filename>gdb-cross-i586</filename>). + The suggestion is usually the actual name you want + to use. + </note> + </para></listitem> + <listitem><para> + <emphasis>Set up the</emphasis> <filename>debugfs</filename></para> + + <para>Run the following commands to set up the + <filename>debugfs</filename>: + <literallayout class='monospaced'> + $ mkdir debugfs + $ cd debugfs + $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>.rootfs.tar.bz2 + $ tar xvfj <replaceable>build-dir</replaceable>/tmp-glibc/deploy/images/<replaceable>machine</replaceable>/<replaceable>image</replaceable>-dbg.rootfs.tar.bz2 + </literallayout> + </para></listitem> + <listitem><para> + <emphasis>Set up GDB</emphasis></para> + + <para>Install the SDK (if you built one) and then + source the correct environment file. + Sourcing the environment file puts the SDK in your + <filename>PATH</filename> environment variable.</para> + + <para>If you are using the build system, Gdb is + located in + <replaceable>build-dir</replaceable>/tmp/sysroots/<replaceable>host</replaceable>/usr/bin/<replaceable>architecture</replaceable>/<replaceable>architecture</replaceable>-gdb + </para></listitem> + <listitem><para> + <emphasis>Boot the target:</emphasis></para> + + <para>For information on how to run QEMU, see the + <ulink url='http://wiki.qemu.org/Documentation/GettingStartedDevelopers'>QEMU Documentation</ulink>. + <note> + Be sure to verify that your host can access the + target via TCP. + </note> + </para></listitem> + <listitem><para> + <emphasis>Debug a program:</emphasis></para> + + <para>Debugging a program involves running gdbserver + on the target and then running Gdb on the host. + The example in this step debugs + <filename>gzip</filename>: + <literallayout class='monospaced'> + root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help + </literallayout> + For additional gdbserver options, see the + <ulink url='https://www.gnu.org/software/gdb/documentation/'>GDB Server Documentation</ulink>. + </para> + + <para>After running gdbserver on the target, you need + to run Gdb on the host and configure it and connect to + the target. + Use these commands: + <literallayout class='monospaced'> + $ cd <replaceable>directory-holding-the-debugfs-directory</replaceable> + $ <replaceable>arch</replaceable>-gdb + + (gdb) set sysroot debugfs + (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug + (gdb) target remote <replaceable>IP-of-target</replaceable>:1234 + </literallayout> + At this point, everything should automatically load + (i.e. matching binaries, symbols and headers). + <note> + The Gdb <filename>set</filename> commands in the + previous example can be placed into the users + <filename>~/.gdbinit</filename> file. + Upon starting, Gdb automatically runs whatever + commands are in that file. + </note> + </para></listitem> + <listitem><para> + <emphasis>Deploying without a full image + rebuild:</emphasis></para> + + <para>In many cases, during development you want a + quick method to deploy a new binary to the target and + debug it, without waiting for a full image build. + </para> + + <para>One approach to solving this situation is to + just build the component you want to debug. + Once you have built the component, copy the + executable directly to both the target and the + host <filename>debugfs</filename>.</para> + + <para>If the binary is processed through the debug + splitting in OpenEmbedded, you should also + copy the debug items (i.e. <filename>.debug</filename> + contents and corresponding + <filename>/usr/src/debug</filename> files) + from the work directory. + Here is an example: + <literallayout class='monospaced'> + $ bitbake bash + $ bitbake -c devshell bash + $ cd .. + $ scp packages-split/bash/bin/bash <replaceable>target</replaceable>:/bin/bash + $ cp -a packages-split/bash-dbg/* <replaceable>path</replaceable>/debugfs + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='debugging-with-the-gnu-project-debugger-gdb-on-the-target'> + <title>Debugging with the GNU Project Debugger (GDB) on the Target</title> + + <para> + The previous section addressed using GDB remotely for debugging + purposes, which is the most usual case due to the inherent + hardware limitations on many embedded devices. + However, debugging in the target hardware itself is also + possible with more powerful devices. + This section describes what you need to do in order to support + using GDB to debug on the target hardware. + </para> + + <para> + To support this kind of debugging, you need do the following: + <itemizedlist> + <listitem><para> + Ensure that GDB is on the target. + You can do this by adding "gdb" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " gdb" + </literallayout> + Alternatively, you can add "tools-debug" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>: + <literallayout class='monospaced'> + IMAGE_FEATURES_append = " tools-debug" + </literallayout> + </para></listitem> + <listitem><para> + Ensure that debug symbols are present. + You can make sure these symbols are present by + installing <filename>-dbg</filename>: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " <replaceable>packagename</replaceable>-dbg" + </literallayout> + Alternatively, you can do the following to include all + the debug symbols: + <literallayout class='monospaced'> + IMAGE_FEATURES_append = " dbg-pkgs" + </literallayout> + </para></listitem> + </itemizedlist> + <note> + To improve the debug information accuracy, you can reduce + the level of optimization used by the compiler. + For example, when adding the following line to your + <filename>local.conf</filename> file, you will reduce + optimization from + <ulink url='&YOCTO_DOCS_REF_URL;#var-FULL_OPTIMIZATION'><filename>FULL_OPTIMIZATION</filename></ulink> + of "-O2" to + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEBUG_OPTIMIZATION'><filename>DEBUG_OPTIMIZATION</filename></ulink> + of "-O -fno-omit-frame-pointer": + <literallayout class='monospaced'> + DEBUG_BUILD = "1" + </literallayout> + Consider that this will reduce the application's performance + and is recommended only for debugging purposes. + </note> + </para> + </section> + + <section id='dev-other-debugging-others'> + <title>Other Debugging Tips</title> + + <para> + Here are some other tips that you might find useful: + <itemizedlist> + <listitem><para> + When adding new packages, it is worth watching for + undesirable items making their way into compiler command + lines. + For example, you do not want references to local system + files like + <filename>/usr/lib/</filename> or + <filename>/usr/include/</filename>. + </para></listitem> + <listitem><para> + If you want to remove the <filename>psplash</filename> + boot splashscreen, + add <filename>psplash=false</filename> to the kernel + command line. + Doing so prevents <filename>psplash</filename> from + loading and thus allows you to see the console. + It is also possible to switch out of the splashscreen by + switching the virtual console (e.g. Fn+Left or Fn+Right + on a Zaurus). + </para></listitem> + <listitem><para> + Removing + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + (usually <filename>tmp/</filename>, within the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>) + can often fix temporary build issues. + Removing <filename>TMPDIR</filename> is usually a + relatively cheap operation, because task output will be + cached in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> + (usually <filename>sstate-cache/</filename>, which is + also in the Build Directory). + <note> + Removing <filename>TMPDIR</filename> might be a + workaround rather than a fix. + Consequently, trying to determine the underlying + cause of an issue before removing the directory is + a good idea. + </note> + </para></listitem> + <listitem><para> + Understanding how a feature is used in practice within + existing recipes can be very helpful. + It is recommended that you configure some method that + allows you to quickly search through files.</para> + + <para>Using GNU Grep, you can use the following shell + function to recursively search through common + recipe-related files, skipping binary files, + <filename>.git</filename> directories, and the + Build Directory (assuming its name starts with + "build"): + <literallayout class='monospaced'> + g() { + grep -Ir \ + --exclude-dir=.git \ + --exclude-dir='build*' \ + --include='*.bb*' \ + --include='*.inc*' \ + --include='*.conf*' \ + --include='*.py*' \ + "$@" + } + </literallayout> + Following are some usage examples: + <literallayout class='monospaced'> + $ g FOO # Search recursively for "FOO" + $ g -i foo # Search recursively for "foo", ignoring case + $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" + </literallayout> + If figuring out how some feature works requires a lot of + searching, it might indicate that the documentation + should be extended or improved. + In such cases, consider filing a documentation bug using + the Yocto Project implementation of + <ulink url='https://bugzilla.yoctoproject.org/'>Bugzilla</ulink>. + For information on how to submit a bug against + the Yocto Project, see the Yocto Project Bugzilla + <ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>wiki page</ulink> + and the + "<link linkend='submitting-a-defect-against-the-yocto-project'>Submitting a Defect Against the Yocto Project</link>" + section. + <note> + The manuals might not be the right place to document + variables that are purely internal and have a + limited scope (e.g. internal variables used to + implement a single <filename>.bbclass</filename> + file). + </note> + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='making-changes-to-the-yocto-project'> + <title>Making Changes to the Yocto Project</title> + + <para> + Because the Yocto Project is an open-source, community-based + project, you can effect changes to the project. + This section presents procedures that show you how to submit + a defect against the project and how to submit a change. + </para> + + <section id='submitting-a-defect-against-the-yocto-project'> + <title>Submitting a Defect Against the Yocto Project</title> + + <para> + Use the Yocto Project implementation of + <ulink url='http://www.bugzilla.org/about/'>Bugzilla</ulink> + to submit a defect (bug) against the Yocto Project. + For additional information on this implementation of Bugzilla see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-bugtracker'>Yocto Project Bugzilla</ulink>" + section in the Yocto Project Reference Manual. + For more detail on any of the following steps, see the Yocto Project + <ulink url='&YOCTO_WIKI_URL;/wiki/Bugzilla_Configuration_and_Bug_Tracking'>Bugzilla wiki page</ulink>. + </para> + + <para> + Use the following general steps to submit a bug" + + <orderedlist> + <listitem><para> + Open the Yocto Project implementation of + <ulink url='&YOCTO_BUGZILLA_URL;'>Bugzilla</ulink>. + </para></listitem> + <listitem><para> + Click "File a Bug" to enter a new bug. + </para></listitem> + <listitem><para> + Choose the appropriate "Classification", "Product", and + "Component" for which the bug was found. + Bugs for the Yocto Project fall into one of several + classifications, which in turn break down into several + products and components. + For example, for a bug against the + <filename>meta-intel</filename> layer, you would choose + "Build System, Metadata & Runtime", "BSPs", and + "bsps-meta-intel", respectively. + </para></listitem> + <listitem><para> + Choose the "Version" of the Yocto Project for which you found + the bug (e.g. &DISTRO;). + </para></listitem> + <listitem><para> + Determine and select the "Severity" of the bug. + The severity indicates how the bug impacted your work. + </para></listitem> + <listitem><para> + Choose the "Hardware" that the bug impacts. + </para></listitem> + <listitem><para> + Choose the "Architecture" that the bug impacts. + </para></listitem> + <listitem><para> + Choose a "Documentation change" item for the bug. + Fixing a bug might or might not affect the Yocto Project + documentation. + If you are unsure of the impact to the documentation, select + "Don't Know". + </para></listitem> + <listitem><para> + Provide a brief "Summary" of the bug. + Try to limit your summary to just a line or two and be sure + to capture the essence of the bug. + </para></listitem> + <listitem><para> + Provide a detailed "Description" of the bug. + You should provide as much detail as you can about the context, + behavior, output, and so forth that surrounds the bug. + You can even attach supporting files for output from logs by + using the "Add an attachment" button. + </para></listitem> + <listitem><para> + Click the "Submit Bug" button submit the bug. + A new Bugzilla number is assigned to the bug and the defect + is logged in the bug tracking system. + </para></listitem> + </orderedlist> + Once you file a bug, the bug is processed by the Yocto Project Bug + Triage Team and further details concerning the bug are assigned + (e.g. priority and owner). + You are the "Submitter" of the bug and any further categorization, + progress, or comments on the bug result in Bugzilla sending you an + automated email concerning the particular change or progress to the + bug. + </para> + </section> + + <section id='how-to-submit-a-change'> + <title>Submitting a Change to the Yocto Project</title> + + <para> + Contributions to the Yocto Project and OpenEmbedded are very welcome. + Because the system is extremely configurable and flexible, we recognize + that developers will want to extend, configure or optimize it for + their specific uses. + </para> + + <para> + The Yocto Project uses a mailing list and a patch-based workflow + that is similar to the Linux kernel but contains important + differences. + In general, a mailing list exists through which you can submit + patches. + You should send patches to the appropriate mailing list so that they + can be reviewed and merged by the appropriate maintainer. + The specific mailing list you need to use depends on the + location of the code you are changing. + Each component (e.g. layer) should have a + <filename>README</filename> file that indicates where to send + the changes and which process to follow. + </para> + + <para> + You can send the patch to the mailing list using whichever approach + you feel comfortable with to generate the patch. + Once sent, the patch is usually reviewed by the community at large. + If somebody has concerns with the patch, they will usually voice + their concern over the mailing list. + If a patch does not receive any negative reviews, the maintainer of + the affected layer typically takes the patch, tests it, and then + based on successful testing, merges the patch. + </para> + + <para id='figuring-out-the-mailing-list-to-use'> + The "poky" repository, which is the Yocto Project's reference build + environment, is a hybrid repository that contains several + individual pieces (e.g. BitBake, Metadata, documentation, + and so forth) built using the combo-layer tool. + The upstream location used for submitting changes varies by + component: + <itemizedlist> + <listitem><para> + <emphasis>Core Metadata:</emphasis> + Send your patch to the + <ulink url='http://lists.openembedded.org/mailman/listinfo/openembedded-core'>openembedded-core</ulink> + mailing list. For example, a change to anything under + the <filename>meta</filename> or + <filename>scripts</filename> directories should be sent + to this mailing list. + </para></listitem> + <listitem><para> + <emphasis>BitBake:</emphasis> + For changes to BitBake (i.e. anything under the + <filename>bitbake</filename> directory), send your patch + to the + <ulink url='http://lists.openembedded.org/mailman/listinfo/bitbake-devel'>bitbake-devel</ulink> + mailing list. + </para></listitem> + <listitem><para> + <emphasis>"meta-*" trees:</emphasis> + These trees contain Metadata. + Use the + <ulink url='https://lists.yoctoproject.org/listinfo/poky'>poky</ulink> + mailing list. + </para></listitem> + </itemizedlist> + </para> + + <para> + For changes to other layers hosted in the Yocto Project source + repositories (i.e. <filename>yoctoproject.org</filename>), tools, + and the Yocto Project documentation, use the + <ulink url='https://lists.yoctoproject.org/listinfo/yocto'>Yocto Project</ulink> + general mailing list. + <note> + Sometimes a layer's documentation specifies to use a + particular mailing list. + If so, use that list. + </note> + For additional recipes that do not fit into the core Metadata, you + should determine which layer the recipe should go into and submit + the change in the manner recommended by the documentation (e.g. + the <filename>README</filename> file) supplied with the layer. + If in doubt, please ask on the Yocto general mailing list or on + the openembedded-devel mailing list. + </para> + + <para> + You can also push a change upstream and request a maintainer to + pull the change into the component's upstream repository. + You do this by pushing to a contribution repository that is upstream. + See the + "<ulink url='&YOCTO_DOCS_OM_URL;#gs-git-workflows-and-the-yocto-project'>Git Workflows and the Yocto Project</ulink>" + section in the Yocto Project Overview and Concepts Manual for additional + concepts on working in the Yocto Project development environment. + </para> + + <para> + Two commonly used testing repositories exist for + OpenEmbedded-Core: + <itemizedlist> + <listitem><para> + <emphasis>"ross/mut" branch:</emphasis> + The "mut" (master-under-test) tree + exists in the <filename>poky-contrib</filename> repository + in the + <ulink url='&YOCTO_GIT_URL;'>Yocto Project source repositories</ulink>. + </para></listitem> + <listitem><para> + <emphasis>"master-next" branch:</emphasis> + This branch is part of the main + "poky" repository in the Yocto Project source repositories. + </para></listitem> + </itemizedlist> + Maintainers use these branches to test submissions prior to merging + patches. + Thus, you can get an idea of the status of a patch based on + whether the patch has been merged into one of these branches. + <note> + This system is imperfect and changes can sometimes get lost in the + flow. + Asking about the status of a patch or change is reasonable if the + change has been idle for a while with no feedback. + The Yocto Project does have plans to use + <ulink url='https://en.wikipedia.org/wiki/Patchwork_(software)'>Patchwork</ulink> + to track the status of patches and also to automatically preview + patches. + </note> + </para> + + <para> + The following sections provide procedures for submitting a change. + </para> + + <section id='pushing-a-change-upstream'> + <title>Using Scripts to Push a Change Upstream and Request a Pull</title> + + <para> + Follow this procedure to push a change to an upstream "contrib" + Git repository: + <note> + You can find general Git information on how to push a change + upstream in the + <ulink url='http://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows'>Git Community Book</ulink>. + </note> + <orderedlist> + <listitem><para> + <emphasis>Make Your Changes Locally:</emphasis> + Make your changes in your local Git repository. + You should make small, controlled, isolated changes. + Keeping changes small and isolated aids review, + makes merging/rebasing easier and keeps the change + history clean should anyone need to refer to it in + future. + </para></listitem> + <listitem><para> + <emphasis>Stage Your Changes:</emphasis> + Stage your changes by using the <filename>git add</filename> + command on each file you changed. + </para></listitem> + <listitem><para id='making-sure-you-have-correct-commit-information'> + <emphasis>Commit Your Changes:</emphasis> + Commit the change by using the + <filename>git commit</filename> command. + Make sure your commit information follows standards by + following these accepted conventions: + <itemizedlist> + <listitem><para> + Be sure to include a "Signed-off-by:" line in the + same style as required by the Linux kernel. + Adding this line signifies that you, the submitter, + have agreed to the Developer's Certificate of + Origin 1.1 as follows: + <literallayout class='monospaced'> + Developer's Certificate of Origin 1.1 + + By making a contribution to this project, I certify that: + + (a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + + (b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + + (c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + + (d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. + </literallayout> + </para></listitem> + <listitem><para> + Provide a single-line summary of the change. + and, + if more explanation is needed, provide more + detail in the body of the commit. + This summary is typically viewable in the + "shortlist" of changes. + Thus, providing something short and descriptive + that gives the reader a summary of the change is + useful when viewing a list of many commits. + You should prefix this short description with the + recipe name (if changing a recipe), or else with + the short form path to the file being changed. + </para></listitem> + <listitem><para> + For the body of the commit message, provide + detailed information that describes what you + changed, why you made the change, and the approach + you used. + It might also be helpful if you mention how you + tested the change. + Provide as much detail as you can in the body of + the commit message. + <note> + You do not need to provide a more detailed + explanation of a change if the change is + minor to the point of the single line + summary providing all the information. + </note> + </para></listitem> + <listitem><para> + If the change addresses a specific bug or issue + that is associated with a bug-tracking ID, + include a reference to that ID in your detailed + description. + For example, the Yocto Project uses a specific + convention for bug references - any commit that + addresses a specific bug should use the following + form for the detailed description. + Be sure to use the actual bug-tracking ID from + Bugzilla for + <replaceable>bug-id</replaceable>: + <literallayout class='monospaced'> + Fixes [YOCTO #<replaceable>bug-id</replaceable>] + + <replaceable>detailed description of change</replaceable> + </literallayout> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Push Your Commits to a "Contrib" Upstream:</emphasis> + If you have arranged for permissions to push to an + upstream contrib repository, push the change to that + repository: + <literallayout class='monospaced'> + $ git push <replaceable>upstream_remote_repo</replaceable> <replaceable>local_branch_name</replaceable> + </literallayout> + For example, suppose you have permissions to push into the + upstream <filename>meta-intel-contrib</filename> + repository and you are working in a local branch named + <replaceable>your_name</replaceable><filename>/README</filename>. + The following command pushes your local commits to the + <filename>meta-intel-contrib</filename> upstream + repository and puts the commit in a branch named + <replaceable>your_name</replaceable><filename>/README</filename>: + <literallayout class='monospaced'> + $ git push meta-intel-contrib <replaceable>your_name</replaceable>/README + </literallayout> + </para></listitem> + <listitem><para id='push-determine-who-to-notify'> + <emphasis>Determine Who to Notify:</emphasis> + Determine the maintainer or the mailing list + that you need to notify for the change.</para> + + <para>Before submitting any change, you need to be sure + who the maintainer is or what mailing list that you need + to notify. + Use either these methods to find out: + <itemizedlist> + <listitem><para> + <emphasis>Maintenance File:</emphasis> + Examine the <filename>maintainers.inc</filename> + file, which is located in the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + at + <filename>meta/conf/distro/include</filename>, + to see who is responsible for code. + </para></listitem> + <listitem><para> + <emphasis>Search by File:</emphasis> + Using <ulink url='&YOCTO_DOCS_OM_URL;#git'>Git</ulink>, + you can enter the following command to bring up a + short list of all commits against a specific file: + <literallayout class='monospaced'> + git shortlog -- <replaceable>filename</replaceable> + </literallayout> + Just provide the name of the file for which you + are interested. + The information returned is not ordered by history + but does include a list of everyone who has + committed grouped by name. + From the list, you can see who is responsible for + the bulk of the changes against the file. + </para></listitem> + <listitem><para> + <emphasis>Examine the List of Mailing Lists:</emphasis> + For a list of the Yocto Project and related mailing + lists, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing lists</ulink>" + section in the Yocto Project Reference Manual. + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para> + <emphasis>Make a Pull Request:</emphasis> + Notify the maintainer or the mailing list that you have + pushed a change by making a pull request.</para> + + <para>The Yocto Project provides two scripts that + conveniently let you generate and send pull requests to the + Yocto Project. + These scripts are <filename>create-pull-request</filename> + and <filename>send-pull-request</filename>. + You can find these scripts in the + <filename>scripts</filename> directory within the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. <filename>~/poky/scripts</filename>). + </para> + + <para>Using these scripts correctly formats the requests + without introducing any whitespace or HTML formatting. + The maintainer that receives your patches either directly + or through the mailing list needs to be able to save and + apply them directly from your emails. + Using these scripts is the preferred method for sending + patches.</para> + + <para>First, create the pull request. + For example, the following command runs the script, + specifies the upstream repository in the contrib directory + into which you pushed the change, and provides a subject + line in the created patch files: + <literallayout class='monospaced'> + $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README" + </literallayout> + Running this script forms + <filename>*.patch</filename> files in a folder named + <filename>pull-</filename><replaceable>PID</replaceable> + in the current directory. + One of the patch files is a cover letter.</para> + + <para>Before running the + <filename>send-pull-request</filename> script, you must + edit the cover letter patch to insert information about + your change. + After editing the cover letter, send the pull request. + For example, the following command runs the script and + specifies the patch directory and email address. + In this example, the email address is a mailing list: + <literallayout class='monospaced'> + $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org + </literallayout> + You need to follow the prompts as the script is + interactive. + <note> + For help on using these scripts, simply provide the + <filename>-h</filename> argument as follows: + <literallayout class='monospaced'> + $ poky/scripts/create-pull-request -h + $ poky/scripts/send-pull-request -h + </literallayout> + </note> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='submitting-a-patch'> + <title>Using Email to Submit a Patch</title> + + <para> + You can submit patches without using the + <filename>create-pull-request</filename> and + <filename>send-pull-request</filename> scripts described in the + previous section. + However, keep in mind, the preferred method is to use the scripts. + </para> + + <para> + Depending on the components changed, you need to submit the email + to a specific mailing list. + For some guidance on which mailing list to use, see the + <link linkend='figuring-out-the-mailing-list-to-use'>list</link> + at the beginning of this section. + For a description of all the available mailing lists, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing Lists</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <para> + Here is the general procedure on how to submit a patch through + email without using the scripts: + <orderedlist> + <listitem><para> + <emphasis>Make Your Changes Locally:</emphasis> + Make your changes in your local Git repository. + You should make small, controlled, isolated changes. + Keeping changes small and isolated aids review, + makes merging/rebasing easier and keeps the change + history clean should anyone need to refer to it in + future. + </para></listitem> + <listitem><para> + <emphasis>Stage Your Changes:</emphasis> + Stage your changes by using the <filename>git add</filename> + command on each file you changed. + </para></listitem> + <listitem><para> + <emphasis>Commit Your Changes:</emphasis> + Commit the change by using the + <filename>git commit --signoff</filename> command. + Using the <filename>--signoff</filename> option identifies + you as the person making the change and also satisfies + the Developer's Certificate of Origin (DCO) shown earlier. + </para> + + <para>When you form a commit, you must follow certain + standards established by the Yocto Project development + team. + See + <link linkend='making-sure-you-have-correct-commit-information'>Step 3</link> + in the previous section for information on how to + provide commit information that meets Yocto Project + commit message standards. + </para></listitem> + <listitem><para> + <emphasis>Format the Commit:</emphasis> + Format the commit into an email message. + To format commits, use the + <filename>git format-patch</filename> command. + When you provide the command, you must include a revision + list or a number of patches as part of the command. + For example, either of these two commands takes your most + recent single commit and formats it as an email message in + the current directory: + <literallayout class='monospaced'> + $ git format-patch -1 + </literallayout> + or + <literallayout class='monospaced'> + $ git format-patch HEAD~ + </literallayout></para> + + <para>After the command is run, the current directory + contains a numbered <filename>.patch</filename> file for + the commit.</para> + + <para>If you provide several commits as part of the + command, the <filename>git format-patch</filename> command + produces a series of numbered files in the current + directory – one for each commit. + If you have more than one patch, you should also use the + <filename>--cover</filename> option with the command, + which generates a cover letter as the first "patch" in + the series. + You can then edit the cover letter to provide a + description for the series of patches. + For information on the + <filename>git format-patch</filename> command, + see <filename>GIT_FORMAT_PATCH(1)</filename> displayed + using the <filename>man git-format-patch</filename> + command. + <note> + If you are or will be a frequent contributor to the + Yocto Project or to OpenEmbedded, you might consider + requesting a contrib area and the necessary associated + rights. + </note> + </para></listitem> + <listitem><para> + <emphasis>Import the Files Into Your Mail Client:</emphasis> + Import the files into your mail client by using the + <filename>git send-email</filename> command. + <note> + In order to use <filename>git send-email</filename>, + you must have the proper Git packages installed on + your host. + For Ubuntu, Debian, and Fedora the package is + <filename>git-email</filename>. + </note></para> + + <para>The <filename>git send-email</filename> command + sends email by using a local or remote Mail Transport Agent + (MTA) such as <filename>msmtp</filename>, + <filename>sendmail</filename>, or through a direct + <filename>smtp</filename> configuration in your Git + <filename>~/.gitconfig</filename> file. + If you are submitting patches through email only, it is + very important that you submit them without any whitespace + or HTML formatting that either you or your mailer + introduces. + The maintainer that receives your patches needs to be able + to save and apply them directly from your emails. + A good way to verify that what you are sending will be + applicable by the maintainer is to do a dry run and send + them to yourself and then save and apply them as the + maintainer would.</para> + + <para>The <filename>git send-email</filename> command is + the preferred method for sending your patches using + email since there is no risk of compromising whitespace + in the body of the message, which can occur when you use + your own mail client. + The command also has several options that let you + specify recipients and perform further editing of the + email message. + For information on how to use the + <filename>git send-email</filename> command, + see <filename>GIT-SEND-EMAIL(1)</filename> displayed using + the <filename>man git-send-email</filename> command. + </para></listitem> + </orderedlist> + </para> + </section> + </section> + </section> + + <section id='working-with-licenses'> + <title>Working With Licenses</title> + + <para> + As mentioned in the + "<ulink url='&YOCTO_DOCS_OM_URL;#licensing'>Licensing</ulink>" + section in the Yocto Project Overview and Concepts Manual, + open source projects are open to the public and they + consequently have different licensing structures in place. + This section describes the mechanism by which the + <ulink url='&YOCTO_DOCS_REF_URL;#build-system-term'>OpenEmbedded build system</ulink> + tracks changes to licensing text and covers how to maintain open + source license compliance during your project's lifecycle. + The section also describes how to enable commercially licensed + recipes, which by default are disabled. + </para> + + <section id="usingpoky-configuring-LIC_FILES_CHKSUM"> + <title>Tracking License Changes</title> + + <para> + The license of an upstream project might change in the future. + In order to prevent these changes going unnoticed, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + variable tracks changes to the license text. The checksums are + validated at the end of the configure step, and if the + checksums do not match, the build will fail. + </para> + + <section id="usingpoky-specifying-LIC_FILES_CHKSUM"> + <title>Specifying the <filename>LIC_FILES_CHKSUM</filename> Variable</title> + + <para> + The <filename>LIC_FILES_CHKSUM</filename> + variable contains checksums of the license text in the + source code for the recipe. + Following is an example of how to specify + <filename>LIC_FILES_CHKSUM</filename>: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \ + file://licfile1.txt;beginline=5;endline=29;md5=yyyy \ + file://licfile2.txt;endline=50;md5=zzzz \ + ..." + </literallayout> + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + When using "beginline" and "endline", realize + that line numbering begins with one and not + zero. + Also, the included lines are inclusive (i.e. + lines five through and including 29 in the + previous example for + <filename>licfile1.txt</filename>). + </para></listitem> + <listitem><para> + When a license check fails, the selected license + text is included as part of the QA message. + Using this output, you can determine the exact + start and finish for the needed license text. + </para></listitem> + </itemizedlist> + </note> + </para> + + <para> + The build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> + variable as the default directory when searching files + listed in <filename>LIC_FILES_CHKSUM</filename>. + The previous example employs the default directory. + </para> + + <para> + Consider this next example: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\ + md5=bb14ed3c4cda583abc85401304b5cd4e" + LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6" + </literallayout> + </para> + + <para> + The first line locates a file in + <filename>${S}/src/ls.c</filename> and isolates lines five + through 16 as license text. + The second line refers to a file in + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. + </para> + + <para> + Note that <filename>LIC_FILES_CHKSUM</filename> variable is + mandatory for all recipes, unless the + <filename>LICENSE</filename> variable is set to "CLOSED". + </para> + </section> + + <section id="usingpoky-LIC_FILES_CHKSUM-explanation-of-syntax"> + <title>Explanation of Syntax</title> + + <para> + As mentioned in the previous section, the + <filename>LIC_FILES_CHKSUM</filename> variable lists all + the important files that contain the license text for the + source code. + It is possible to specify a checksum for an entire file, + or a specific section of a file (specified by beginning and + ending line numbers with the "beginline" and "endline" + parameters, respectively). + The latter is useful for source files with a license + notice header, README documents, and so forth. + If you do not use the "beginline" parameter, then it is + assumed that the text begins on the first line of the file. + Similarly, if you do not use the "endline" parameter, + it is assumed that the license text ends with the last + line of the file. + </para> + + <para> + The "md5" parameter stores the md5 checksum of the license + text. + If the license text changes in any way as compared to + this parameter then a mismatch occurs. + This mismatch triggers a build failure and notifies + the developer. + Notification allows the developer to review and address + the license text changes. + Also note that if a mismatch occurs during the build, + the correct md5 checksum is placed in the build log and + can be easily copied to the recipe. + </para> + + <para> + There is no limit to how many files you can specify using + the <filename>LIC_FILES_CHKSUM</filename> variable. + Generally, however, every project requires a few + specifications for license tracking. + Many projects have a "COPYING" file that stores the + license information for all the source code files. + This practice allows you to just track the "COPYING" + file as long as it is kept up to date. + <note><title>Tips</title> + <itemizedlist> + <listitem><para> + If you specify an empty or invalid "md5" + parameter, + <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink> + returns an md5 mis-match + error and displays the correct "md5" parameter + value during the build. + The correct parameter is also captured in + the build log. + </para></listitem> + <listitem><para> + If the whole file contains only license text, + you do not need to use the "beginline" and + "endline" parameters. + </para></listitem> + </itemizedlist> + </note> + </para> + </section> + </section> + + <section id="enabling-commercially-licensed-recipes"> + <title>Enabling Commercially Licensed Recipes</title> + + <para> + By default, the OpenEmbedded build system disables + components that have commercial or other special licensing + requirements. + Such requirements are defined on a + recipe-by-recipe basis through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS'><filename>LICENSE_FLAGS</filename></ulink> + variable definition in the affected recipe. + For instance, the + <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> + recipe contains the following statement: + <literallayout class='monospaced'> + LICENSE_FLAGS = "commercial" + </literallayout> + Here is a slightly more complicated example that contains both + an explicit recipe name and version (after variable expansion): + <literallayout class='monospaced'> + LICENSE_FLAGS = "license_${PN}_${PV}" + </literallayout> + In order for a component restricted by a + <filename>LICENSE_FLAGS</filename> definition to be enabled and + included in an image, it needs to have a matching entry in the + global + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_FLAGS_WHITELIST'><filename>LICENSE_FLAGS_WHITELIST</filename></ulink> + variable, which is a variable typically defined in your + <filename>local.conf</filename> file. + For example, to enable the + <filename>poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly</filename> + package, you could add either the string + "commercial_gst-plugins-ugly" or the more general string + "commercial" to <filename>LICENSE_FLAGS_WHITELIST</filename>. + See the + "<link linkend='license-flag-matching'>License Flag Matching</link>" + section for a full + explanation of how <filename>LICENSE_FLAGS</filename> matching + works. + Here is the example: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly" + </literallayout> + Likewise, to additionally enable the package built from the + recipe containing + <filename>LICENSE_FLAGS = "license_${PN}_${PV}"</filename>, + and assuming that the actual recipe name was + <filename>emgd_1.10.bb</filename>, the following string would + enable that package as well as the original + <filename>gst-plugins-ugly</filename> package: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10" + </literallayout> + As a convenience, you do not need to specify the complete + license string in the whitelist for every package. + You can use an abbreviated form, which consists + of just the first portion or portions of the license + string before the initial underscore character or characters. + A partial string will match any license that contains the + given string as the first portion of its license. + For example, the following whitelist string will also match + both of the packages previously mentioned as well as any other + packages that have licenses starting with "commercial" or + "license". + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial license" + </literallayout> + </para> + + <section id="license-flag-matching"> + <title>License Flag Matching</title> + + <para> + License flag matching allows you to control what recipes + the OpenEmbedded build system includes in the build. + Fundamentally, the build system attempts to match + <filename>LICENSE_FLAGS</filename> strings found in recipes + against <filename>LICENSE_FLAGS_WHITELIST</filename> + strings found in the whitelist. + A match causes the build system to include a recipe in the + build, while failure to find a match causes the build + system to exclude a recipe. + </para> + + <para> + In general, license flag matching is simple. + However, understanding some concepts will help you + correctly and effectively use matching. + </para> + + <para> + Before a flag + defined by a particular recipe is tested against the + contents of the whitelist, the expanded string + <filename>_${PN}</filename> is appended to the flag. + This expansion makes each + <filename>LICENSE_FLAGS</filename> value recipe-specific. + After expansion, the string is then matched against the + whitelist. + Thus, specifying + <filename>LICENSE_FLAGS = "commercial"</filename> + in recipe "foo", for example, results in the string + <filename>"commercial_foo"</filename>. + And, to create a match, that string must appear in the + whitelist. + </para> + + <para> + Judicious use of the <filename>LICENSE_FLAGS</filename> + strings and the contents of the + <filename>LICENSE_FLAGS_WHITELIST</filename> variable + allows you a lot of flexibility for including or excluding + recipes based on licensing. + For example, you can broaden the matching capabilities by + using license flags string subsets in the whitelist. + <note> + When using a string subset, be sure to use the part of + the expanded string that precedes the appended + underscore character (e.g. + <filename>usethispart_1.3</filename>, + <filename>usethispart_1.4</filename>, and so forth). + </note> + For example, simply specifying the string "commercial" in + the whitelist matches any expanded + <filename>LICENSE_FLAGS</filename> definition that starts + with the string "commercial" such as "commercial_foo" and + "commercial_bar", which are the strings the build system + automatically generates for hypothetical recipes named + "foo" and "bar" assuming those recipes simply specify the + following: + <literallayout class='monospaced'> + LICENSE_FLAGS = "commercial" + </literallayout> + Thus, you can choose to exhaustively + enumerate each license flag in the whitelist and + allow only specific recipes into the image, or + you can use a string subset that causes a broader range of + matches to allow a range of recipes into the image. + </para> + + <para> + This scheme works even if the + <filename>LICENSE_FLAGS</filename> string already + has <filename>_${PN}</filename> appended. + For example, the build system turns the license flag + "commercial_1.2_foo" into "commercial_1.2_foo_foo" and + would match both the general "commercial" and the specific + "commercial_1.2_foo" strings found in the whitelist, as + expected. + </para> + + <para> + Here are some other scenarios: + <itemizedlist> + <listitem><para> + You can specify a versioned string in the recipe + such as "commercial_foo_1.2" in a "foo" recipe. + The build system expands this string to + "commercial_foo_1.2_foo". + Combine this license flag with a whitelist that has + the string "commercial" and you match the flag + along with any other flag that starts with the + string "commercial". + </para></listitem> + <listitem><para> + Under the same circumstances, you can use + "commercial_foo" in the whitelist and the build + system not only matches "commercial_foo_1.2" but + also matches any license flag with the string + "commercial_foo", regardless of the version. + </para></listitem> + <listitem><para> + You can be very specific and use both the + package and version parts in the whitelist (e.g. + "commercial_foo_1.2") to specifically match a + versioned recipe. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id="other-variables-related-to-commercial-licenses"> + <title>Other Variables Related to Commercial Licenses</title> + + <para> + Other helpful variables related to commercial + license handling exist and are defined in the + <filename>poky/meta/conf/distro/include/default-distrovars.inc</filename> file: + <literallayout class='monospaced'> + COMMERCIAL_AUDIO_PLUGINS ?= "" + COMMERCIAL_VIDEO_PLUGINS ?= "" + </literallayout> + If you want to enable these components, you can do so by + making sure you have statements similar to the following + in your <filename>local.conf</filename> configuration file: + <literallayout class='monospaced'> + COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \ + gst-plugins-ugly-mpegaudioparse" + COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \ + gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse" + LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp" + </literallayout> + Of course, you could also create a matching whitelist + for those components using the more general "commercial" + in the whitelist, but that would also enable all the + other packages with <filename>LICENSE_FLAGS</filename> + containing "commercial", which you may or may not want: + <literallayout class='monospaced'> + LICENSE_FLAGS_WHITELIST = "commercial" + </literallayout> + </para> + + <para> + Specifying audio and video plug-ins as part of the + <filename>COMMERCIAL_AUDIO_PLUGINS</filename> and + <filename>COMMERCIAL_VIDEO_PLUGINS</filename> statements + (along with the enabling + <filename>LICENSE_FLAGS_WHITELIST</filename>) includes the + plug-ins or components into built images, thus adding + support for media formats or components. + </para> + </section> + </section> + + <section id='maintaining-open-source-license-compliance-during-your-products-lifecycle'> + <title>Maintaining Open Source License Compliance During Your Product's Lifecycle</title> + + <para> + One of the concerns for a development organization using open source + software is how to maintain compliance with various open source + licensing during the lifecycle of the product. + While this section does not provide legal advice or + comprehensively cover all scenarios, it does + present methods that you can use to + assist you in meeting the compliance requirements during a software + release. + </para> + + <para> + With hundreds of different open source licenses that the Yocto + Project tracks, it is difficult to know the requirements of each + and every license. + However, the requirements of the major FLOSS licenses can begin + to be covered by + assuming that three main areas of concern exist: + <itemizedlist> + <listitem><para>Source code must be provided.</para></listitem> + <listitem><para>License text for the software must be + provided.</para></listitem> + <listitem><para>Compilation scripts and modifications to the + source code must be provided. + </para></listitem> + </itemizedlist> + There are other requirements beyond the scope of these + three and the methods described in this section + (e.g. the mechanism through which source code is distributed). + </para> + + <para> + As different organizations have different methods of complying with + open source licensing, this section is not meant to imply that + there is only one single way to meet your compliance obligations, + but rather to describe one method of achieving compliance. + The remainder of this section describes methods supported to meet the + previously mentioned three requirements. + Once you take steps to meet these requirements, + and prior to releasing images, sources, and the build system, + you should audit all artifacts to ensure completeness. + <note> + The Yocto Project generates a license manifest during + image creation that is located + in <filename>${DEPLOY_DIR}/licenses/<replaceable>image_name-datestamp</replaceable></filename> + to assist with any audits. + </note> + </para> + + <section id='providing-the-source-code'> + <title>Providing the Source Code</title> + + <para> + Compliance activities should begin before you generate the + final image. + The first thing you should look at is the requirement that + tops the list for most compliance groups - providing + the source. + The Yocto Project has a few ways of meeting this + requirement. + </para> + + <para> + One of the easiest ways to meet this requirement is + to provide the entire + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> + used by the build. + This method, however, has a few issues. + The most obvious is the size of the directory since it includes + all sources used in the build and not just the source used in + the released image. + It will include toolchain source, and other artifacts, which + you would not generally release. + However, the more serious issue for most companies is accidental + release of proprietary software. + The Yocto Project provides an + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-archiver'><filename>archiver</filename></ulink> + class to help avoid some of these concerns. + </para> + + <para> + Before you employ <filename>DL_DIR</filename> or the + <filename>archiver</filename> class, you need to decide how + you choose to provide source. + The source <filename>archiver</filename> class can generate + tarballs and SRPMs and can create them with various levels of + compliance in mind. + </para> + + <para> + One way of doing this (but certainly not the only way) is to + release just the source as a tarball. + You can do this by adding the following to the + <filename>local.conf</filename> file found in the + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + INHERIT += "archiver" + ARCHIVER_MODE[src] = "original" + </literallayout> + During the creation of your image, the source from all + recipes that deploy packages to the image is placed within + subdirectories of + <filename>DEPLOY_DIR/sources</filename> based on the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + for each recipe. + Releasing the entire directory enables you to comply with + requirements concerning providing the unmodified source. + It is important to note that the size of the directory can + get large. + </para> + + <para> + A way to help mitigate the size issue is to only release + tarballs for licenses that require the release of + source. + Let us assume you are only concerned with GPL code as + identified by running the following script: + <literallayout class='monospaced'> + # Script to archive a subset of packages matching specific license(s) + # Source and license files are copied into sub folders of package folder + # Must be run from build folder + #!/bin/bash + src_release_dir="source-release" + mkdir -p $src_release_dir + for a in tmp/deploy/sources/*; do + for d in $a/*; do + # Get package name from path + p=`basename $d` + p=${p%-*} + p=${p%-*} + # Only archive GPL packages (update *GPL* regex for your license check) + numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l` + if [ $numfiles -gt 1 ]; then + echo Archiving $p + mkdir -p $src_release_dir/$p/source + cp $d/* $src_release_dir/$p/source 2> /dev/null + mkdir -p $src_release_dir/$p/license + cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null + fi + done + done + </literallayout> + At this point, you could create a tarball from the + <filename>gpl_source_release</filename> directory and + provide that to the end user. + This method would be a step toward achieving compliance + with section 3a of GPLv2 and with section 6 of GPLv3. + </para> + </section> + + <section id='providing-license-text'> + <title>Providing License Text</title> + + <para> + One requirement that is often overlooked is inclusion + of license text. + This requirement also needs to be dealt with prior to + generating the final image. + Some licenses require the license text to accompany + the binary. + You can achieve this by adding the following to your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + COPY_LIC_MANIFEST = "1" + COPY_LIC_DIRS = "1" + LICENSE_CREATE_PACKAGE = "1" + </literallayout> + Adding these statements to the configuration file ensures + that the licenses collected during package generation + are included on your image. + <note> + <para>Setting all three variables to "1" results in the + image having two copies of the same license file. + One copy resides in + <filename>/usr/share/common-licenses</filename> and + the other resides in + <filename>/usr/share/license</filename>.</para> + + <para>The reason for this behavior is because + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_DIRS'><filename>COPY_LIC_DIRS</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_MANIFEST'><filename>COPY_LIC_MANIFEST</filename></ulink> + add a copy of the license when the image is built but do + not offer a path for adding licenses for newly installed + packages to an image. + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_CREATE_PACKAGE'><filename>LICENSE_CREATE_PACKAGE</filename></ulink> + adds a separate package and an upgrade path for adding + licenses to an image.</para> + </note> + </para> + + <para> + As the source <filename>archiver</filename> class has already + archived the original + unmodified source that contains the license files, + you would have already met the requirements for inclusion + of the license information with source as defined by the GPL + and other open source licenses. + </para> + </section> + + <section id='providing-compilation-scripts-and-source-code-modifications'> + <title>Providing Compilation Scripts and Source Code Modifications</title> + + <para> + At this point, we have addressed all we need to + prior to generating the image. + The next two requirements are addressed during the final + packaging of the release. + </para> + + <para> + By releasing the version of the OpenEmbedded build system + and the layers used during the build, you will be providing both + compilation scripts and the source code modifications in one + step. + </para> + + <para> + If the deployment team has a + <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP layer</ulink> + and a distro layer, and those those layers are used to patch, + compile, package, or modify (in any way) any open source + software included in your released images, you + might be required to release those layers under section 3 of + GPLv2 or section 1 of GPLv3. + One way of doing that is with a clean + checkout of the version of the Yocto Project and layers used + during your build. + Here is an example: + <literallayout class='monospaced'> + # We built using the &DISTRO_NAME_NO_CAP; branch of the poky repo + $ git clone -b &DISTRO_NAME_NO_CAP; git://git.yoctoproject.org/poky + $ cd poky + # We built using the release_branch for our layers + $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer + $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer + # clean up the .git repos + $ find . -name ".git" -type d -exec rm -rf {} \; + </literallayout> + One thing a development organization might want to consider + for end-user convenience is to modify + <filename>meta-poky/conf/bblayers.conf.sample</filename> to + ensure that when the end user utilizes the released build + system to build an image, the development organization's + layers are included in the <filename>bblayers.conf</filename> + file automatically: + <literallayout class='monospaced'> + # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf + # changes incompatibly + POKY_BBLAYERS_CONF_VERSION = "2" + + BBPATH = "${TOPDIR}" + BBFILES ?= "" + + BBLAYERS ?= " \ + ##OEROOT##/meta \ + ##OEROOT##/meta-poky \ + ##OEROOT##/meta-yocto-bsp \ + ##OEROOT##/meta-mylayer \ + " + </literallayout> + Creating and providing an archive of the + <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> + layers (recipes, configuration files, and so forth) + enables you to meet your + requirements to include the scripts to control compilation + as well as any modifications to the original source. + </para> + </section> + </section> + </section> + + <section id='using-the-error-reporting-tool'> + <title>Using the Error Reporting Tool</title> + + <para> + The error reporting tool allows you to + submit errors encountered during builds to a central database. + Outside of the build environment, you can use a web interface to + browse errors, view statistics, and query for errors. + The tool works using a client-server system where the client + portion is integrated with the installed Yocto Project + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink> + (e.g. <filename>poky</filename>). + The server receives the information collected and saves it in a + database. + </para> + + <para> + A live instance of the error reporting server exists at + <ulink url='http://errors.yoctoproject.org'></ulink>. + This server exists so that when you want to get help with + build failures, you can submit all of the information on the + failure easily and then point to the URL in your bug report + or send an email to the mailing list. + <note> + If you send error reports to this server, the reports become + publicly visible. + </note> + </para> + + <section id='enabling-and-using-the-tool'> + <title>Enabling and Using the Tool</title> + + <para> + By default, the error reporting tool is disabled. + You can enable it by inheriting the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-report-error'><filename>report-error</filename></ulink> + class by adding the following statement to the end of + your <filename>local.conf</filename> file in your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + <literallayout class='monospaced'> + INHERIT += "report-error" + </literallayout> + </para> + + <para> + By default, the error reporting feature stores information in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LOG_DIR'><filename>LOG_DIR</filename></ulink><filename>}/error-report</filename>. + However, you can specify a directory to use by adding the following + to your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + ERR_REPORT_DIR = "path" + </literallayout> + Enabling error reporting causes the build process to collect + the errors and store them in a file as previously described. + When the build system encounters an error, it includes a + command as part of the console output. + You can run the command to send the error file to the server. + For example, the following command sends the errors to an + upstream server: + <literallayout class='monospaced'> + $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt + </literallayout> + In the previous example, the errors are sent to a public + database available at + <ulink url='http://errors.yoctoproject.org'></ulink>, which is + used by the entire community. + If you specify a particular server, you can send the errors + to a different database. + Use the following command for more information on available + options: + <literallayout class='monospaced'> + $ send-error-report --help + </literallayout> + </para> + + <para> + When sending the error file, you are prompted to review the + data being sent as well as to provide a name and optional + email address. + Once you satisfy these prompts, the command returns a link + from the server that corresponds to your entry in the database. + For example, here is a typical link: + <literallayout class='monospaced'> + http://errors.yoctoproject.org/Errors/Details/9522/ + </literallayout> + Following the link takes you to a web interface where you can + browse, query the errors, and view statistics. + </para> + </section> + + <section id='disabling-the-tool'> + <title>Disabling the Tool</title> + + <para> + To disable the error reporting feature, simply remove or comment + out the following statement from the end of your + <filename>local.conf</filename> file in your + <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. + <literallayout class='monospaced'> + INHERIT += "report-error" + </literallayout> + </para> + </section> + + <section id='setting-up-your-own-error-reporting-server'> + <title>Setting Up Your Own Error Reporting Server</title> + + <para> + If you want to set up your own error reporting server, you + can obtain the code from the Git repository at + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/'></ulink>. + Instructions on how to set it up are in the README document. + </para> + </section> + </section> + + <section id="dev-using-wayland-and-weston"> + <title>Using Wayland and Weston</title> + + <para> + <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)'>Wayland</ulink> + is a computer display server protocol that + provides a method for compositing window managers to communicate + directly with applications and video hardware and expects them to + communicate with input hardware using other libraries. + Using Wayland with supporting targets can result in better control + over graphics frame rendering than an application might otherwise + achieve. + </para> + + <para> + The Yocto Project provides the Wayland protocol libraries and the + reference + <ulink url='http://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston'>Weston</ulink> + compositor as part of its release. + You can find the integrated packages in the + <filename>meta</filename> layer of the + <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>. + Specifically, you can find the recipes that build both Wayland + and Weston at <filename>meta/recipes-graphics/wayland</filename>. + </para> + + <para> + You can build both the Wayland and Weston packages for use only + with targets that accept the + <ulink url='https://en.wikipedia.org/wiki/Mesa_(computer_graphics)'>Mesa 3D and Direct Rendering Infrastructure</ulink>, + which is also known as Mesa DRI. + This implies that you cannot build and use the packages if your + target uses, for example, the + <trademark class='registered'>Intel</trademark> Embedded Media + and Graphics Driver + (<trademark class='registered'>Intel</trademark> EMGD) that + overrides Mesa DRI. + <note> + Due to lack of EGL support, Weston 1.0.3 will not run + directly on the emulated QEMU hardware. + However, this version of Weston will run under X emulation + without issues. + </note> + </para> + + <para> + This section describes what you need to do to implement Wayland and + use the Weston compositor when building an image for a supporting + target. + </para> + + <section id="enabling-wayland-in-an-image"> + <title>Enabling Wayland in an Image</title> + + <para> + To enable Wayland, you need to enable it to be built and enable + it to be included (installed) in the image. + </para> + + <section id="enable-building"> + <title>Building</title> + + <para> + To cause Mesa to build the <filename>wayland-egl</filename> + platform and Weston to build Wayland with Kernel Mode + Setting + (<ulink url='https://wiki.archlinux.org/index.php/Kernel_Mode_Setting'>KMS</ulink>) + support, include the "wayland" flag in the + <ulink url="&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES"><filename>DISTRO_FEATURES</filename></ulink> + statement in your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " wayland" + </literallayout> + <note> + If X11 has been enabled elsewhere, Weston will build + Wayland with X11 support + </note> + </para> + </section> + + <section id="enable-installation-in-an-image"> + <title>Installing</title> + + <para> + To install the Wayland feature into an image, you must + include the following + <ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'><filename>CORE_IMAGE_EXTRA_INSTALL</filename></ulink> + statement in your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + CORE_IMAGE_EXTRA_INSTALL += "wayland weston" + </literallayout> + </para> + </section> + </section> + + <section id="running-weston"> + <title>Running Weston</title> + + <para> + To run Weston inside X11, enabling it as described earlier and + building a Sato image is sufficient. + If you are running your image under Sato, a Weston Launcher + appears in the "Utility" category. + </para> + + <para> + Alternatively, you can run Weston through the command-line + interpretor (CLI), which is better suited for development work. + To run Weston under the CLI, you need to do the following after + your image is built: + <orderedlist> + <listitem><para> + Run these commands to export + <filename>XDG_RUNTIME_DIR</filename>: + <literallayout class='monospaced'> + mkdir -p /tmp/$USER-weston + chmod 0700 /tmp/$USER-weston + export XDG_RUNTIME_DIR=/tmp/$USER-weston + </literallayout> + </para></listitem> + <listitem><para> + Launch Weston in the shell: + <literallayout class='monospaced'> + weston + </literallayout></para></listitem> + </orderedlist> + </para> + </section> + </section> +</chapter> + +<!-- +vim: expandtab tw=80 ts=4 +--> |
