diff options
Diffstat (limited to 'import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml')
| -rw-r--r-- | import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml | 10370 |
1 files changed, 10370 insertions, 0 deletions
diff --git a/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml b/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml new file mode 100644 index 000000000..f926f1d47 --- /dev/null +++ b/import-layers/yocto-poky/documentation/dev-manual/dev-manual-common-tasks.xml @@ -0,0 +1,10370 @@ +<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" +"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" +[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > + +<chapter id='extendpoky'> + +<title>Common Tasks</title> + <para> + This chapter describes fundamental procedures such as creating layers, + adding new software packages, extending or customizing images, + porting work to new hardware (adding a new machine), and so forth. + You will find that the procedures documented here occur often in the + development cycle using the Yocto Project. + </para> + + <section id="understanding-and-creating-layers"> + <title>Understanding and Creating Layers</title> + + <para> + The OpenEmbedded build system supports organizing + <link linkend='metadata'>Metadata</link> into multiple layers. + Layers allow you to isolate different types of customizations from + each other. + You might find it tempting to keep everything in one layer when + working on a single project. + However, the more modular your Metadata, the easier + it is to cope with future changes. + </para> + + <para> + To illustrate how layers are used to keep things modular, consider + machine customizations. + These types of customizations typically reside in a special layer, + rather than a general layer, called a Board Support Package (BSP) + Layer. + Furthermore, the machine customizations should be isolated from + recipes and Metadata that support a new GUI environment, + for example. + This situation gives you a couple of layers: one for the machine + configurations, and one for the GUI environment. + It is important to understand, however, that the BSP layer can + still make machine-specific additions to recipes within the GUI + environment layer without polluting the GUI layer itself + with those machine-specific changes. + You can accomplish this through a recipe that is a BitBake append + (<filename>.bbappend</filename>) file, which is described later + in this section. + </para> + + <para> + </para> + + <section id='yocto-project-layers'> + <title>Layers</title> + + <para> + The <link linkend='source-directory'>Source Directory</link> + contains both general layers and BSP + layers right out of the box. + You can easily identify layers that ship with a + Yocto Project release in the Source Directory by their + folder names. + Folders that represent layers typically have names that begin with + the string <filename>meta-</filename>. + <note> + It is not a requirement that a layer name begin with the + prefix <filename>meta-</filename>, but it is a commonly + accepted standard in the Yocto Project community. + </note> + For example, when you set up the Source Directory structure, + you will see several layers: + <filename>meta</filename>, + <filename>meta-skeleton</filename>, + <filename>meta-selftest</filename>, + <filename>meta-poky</filename>, and + <filename>meta-yocto-bsp</filename>. + Each of these folders represents a distinct layer. + </para> + + <para> + As another example, if you set up a local copy of the + <filename>meta-intel</filename> Git repository + and then explore the folder of that general layer, + you will discover many Intel-specific BSP layers inside. + For more information on BSP layers, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" + section in the Yocto Project Board Support Package (BSP) + Developer's Guide. + </para> + </section> + + <section id='creating-your-own-layer'> + <title>Creating Your Own Layer</title> + + <para> + It is very easy to create your own layers to use with the + OpenEmbedded build system. + The Yocto Project ships with scripts that speed up creating + general layers and BSP layers. + This section describes the steps you perform by hand to create + a layer so that you can better understand them. + For information about the layer-creation scripts, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-yocto-bsp-script'>Creating a New BSP Layer Using the yocto-bsp Script</ulink>" + section in the Yocto Project Board Support Package (BSP) + Developer's Guide and the + "<link linkend='creating-a-general-layer-using-the-yocto-layer-script'>Creating a General Layer Using the yocto-layer Script</link>" + section further down in this manual. + </para> + + <para> + Follow these general steps to create your layer: + <orderedlist> + <listitem><para><emphasis>Check Existing Layers:</emphasis> + Before creating a new layer, you should be sure someone + has not already created a layer containing the Metadata + you need. + You can see the + <ulink url='http://layers.openembedded.org/layerindex/layers/'><filename>OpenEmbedded Metadata Index</filename></ulink> + for a list of layers from the OpenEmbedded community + that can be used in the Yocto Project. + </para></listitem> + <listitem><para><emphasis>Create a Directory:</emphasis> + Create the directory for your layer. + While not strictly required, prepend the name of the + folder with the string <filename>meta-</filename>. + For example: + <literallayout class='monospaced'> + meta-mylayer + meta-GUI_xyz + meta-mymachine + </literallayout> + </para></listitem> + <listitem><para><emphasis>Create a Layer Configuration + File:</emphasis> + Inside your new layer folder, you need to create a + <filename>conf/layer.conf</filename> file. + It is easiest to take an existing layer configuration + file and copy that to your layer's + <filename>conf</filename> directory and then modify the + file as needed.</para> + <para>The + <filename>meta-yocto-bsp/conf/layer.conf</filename> file + demonstrates the required syntax: + <literallayout class='monospaced'> + # We have a conf and classes directory, add to BBPATH + BBPATH .= ":${LAYERDIR}" + + # We have recipes-* directories, add to BBFILES + BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + + BBFILE_COLLECTIONS += "yoctobsp" + BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/" + BBFILE_PRIORITY_yoctobsp = "5" + LAYERVERSION_yoctobsp = "3" + </literallayout></para> + <para>Here is an explanation of the example: + <itemizedlist> + <listitem><para>The configuration and + classes directory is appended to + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>. + <note> + All non-distro layers, which include all BSP + layers, are expected to append the layer + directory to the + <filename>BBPATH</filename>. + On the other hand, distro layers, such as + <filename>meta-poky</filename>, can choose + to enforce their own precedence over + <filename>BBPATH</filename>. + For an example of that syntax, see the + <filename>layer.conf</filename> file for + the <filename>meta-poky</filename> layer. + </note></para></listitem> + <listitem><para>The recipes for the layers are + appended to + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'>BBFILES</ulink></filename>. + </para></listitem> + <listitem><para>The + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_COLLECTIONS'>BBFILE_COLLECTIONS</ulink></filename> + variable is then appended with the layer name. + </para></listitem> + <listitem><para>The + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PATTERN'>BBFILE_PATTERN</ulink></filename> + variable is set to a regular expression and is + used to match files from + <filename>BBFILES</filename> into a particular + layer. + In this case, + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDIR'>LAYERDIR</ulink></filename> + is used to make <filename>BBFILE_PATTERN</filename> match within the + layer's path.</para></listitem> + <listitem><para>The + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'>BBFILE_PRIORITY</ulink></filename> + variable then assigns a priority to the layer. + Applying priorities is useful in situations + where the same recipe might appear in multiple + layers and allows you to choose the layer + that takes precedence.</para></listitem> + <listitem><para>The + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERVERSION'>LAYERVERSION</ulink></filename> + variable optionally specifies the version of a + layer as a single number.</para></listitem> + </itemizedlist></para> + <para>Note the use of the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LAYERDIR'>LAYERDIR</ulink></filename> + variable, which expands to the directory of the current + layer.</para> + <para>Through the use of the <filename>BBPATH</filename> + variable, BitBake locates class files + (<filename>.bbclass</filename>), + configuration files, and files that are included + with <filename>include</filename> and + <filename>require</filename> statements. + For these cases, BitBake uses the first file that + matches the name found in <filename>BBPATH</filename>. + This is similar to the way the <filename>PATH</filename> + variable is used for binaries. + It is recommended, therefore, that you use unique + class and configuration + filenames in your custom layer.</para></listitem> + <listitem><para><emphasis>Add Content:</emphasis> Depending + on the type of layer, add the content. + If the layer adds support for a machine, add the machine + configuration in a <filename>conf/machine/</filename> + file within the layer. + If the layer adds distro policy, add the distro + configuration in a <filename>conf/distro/</filename> + file within the layer. + If the layer introduces new recipes, put the recipes + you need in <filename>recipes-*</filename> + subdirectories within the layer. + <note>In order to be compliant with the Yocto Project, + a layer must contain a + <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-filelayout-readme'>README file.</ulink> + </note></para></listitem> + </orderedlist> + </para> + </section> + + <section id='best-practices-to-follow-when-creating-layers'> + <title>Best Practices to Follow When Creating Layers</title> + + <para> + To create layers that are easier to maintain and that will + not impact builds for other machines, you should consider the + information in the following sections. + </para> + + <section id='avoid-overlaying-entire-recipes'> + <title>Avoid "Overlaying" Entire Recipes</title> + + <para> + Avoid "overlaying" entire recipes from other layers in your + configuration. + In other words, do not copy an entire recipe into your + layer and then modify it. + Rather, use an append file (<filename>.bbappend</filename>) + to override + only those parts of the original recipe you need to modify. + </para> + </section> + + <section id='avoid-duplicating-include-files'> + <title>Avoid Duplicating Include Files</title> + + <para> + Avoid duplicating include files. + Use append files (<filename>.bbappend</filename>) + for each recipe + that uses an include file. + Or, if you are introducing a new recipe that requires + the included file, use the path relative to the original + layer directory to refer to the file. + For example, use + <filename>require recipes-core/</filename><replaceable>package</replaceable><filename>/</filename><replaceable>file</replaceable><filename>.inc</filename> + instead of <filename>require </filename><replaceable>file</replaceable><filename>.inc</filename>. + If you're finding you have to overlay the include file, + it could indicate a deficiency in the include file in + the layer to which it originally belongs. + If this is the case, you should try to address that + deficiency instead of overlaying the include file. + For example, you could address this by getting the + maintainer of the include file to add a variable or + variables to make it easy to override the parts needing + to be overridden. + </para> + </section> + + <section id='structure-your-layers'> + <title>Structure Your Layers</title> + + <para> + Proper use of overrides within append files and placement + of machine-specific files within your layer can ensure that + a build is not using the wrong Metadata and negatively + impacting a build for a different machine. + Following are some examples: + <itemizedlist> + <listitem><para><emphasis>Modifying Variables to Support + a Different Machine:</emphasis> + Suppose you have a layer named + <filename>meta-one</filename> that adds support + for building machine "one". + To do so, you use an append file named + <filename>base-files.bbappend</filename> and + create a dependency on "foo" by altering the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable: + <literallayout class='monospaced'> + DEPENDS = "foo" + </literallayout> + The dependency is created during any build that + includes the layer + <filename>meta-one</filename>. + However, you might not want this dependency + for all machines. + For example, suppose you are building for + machine "two" but your + <filename>bblayers.conf</filename> file has the + <filename>meta-one</filename> layer included. + During the build, the + <filename>base-files</filename> for machine + "two" will also have the dependency on + <filename>foo</filename>.</para> + <para>To make sure your changes apply only when + building machine "one", use a machine override + with the <filename>DEPENDS</filename> statement: + <literallayout class='monospaced'> + DEPENDS_one = "foo" + </literallayout> + You should follow the same strategy when using + <filename>_append</filename> and + <filename>_prepend</filename> operations: + <literallayout class='monospaced'> + DEPENDS_append_one = " foo" + DEPENDS_prepend_one = "foo " + </literallayout> + As an actual example, here's a line from the recipe + for gnutls, which adds dependencies on + "argp-standalone" when building with the musl C + library: + <literallayout class='monospaced'> + DEPENDS_append_libc-musl = " argp-standalone" + </literallayout> + <note> + Avoiding "+=" and "=+" and using + machine-specific + <filename>_append</filename> + and <filename>_prepend</filename> operations + is recommended as well. + </note></para></listitem> + <listitem><para><emphasis>Place Machine-Specific Files + in Machine-Specific Locations:</emphasis> + When you have a base recipe, such as + <filename>base-files.bb</filename>, that + contains a + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statement to a file, you can use an append file + to cause the build to use your own version of + the file. + For example, an append file in your layer at + <filename>meta-one/recipes-core/base-files/base-files.bbappend</filename> + could extend + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + using + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + as follows: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:" + </literallayout> + The build for machine "one" will pick up your + machine-specific file as long as you have the + file in + <filename>meta-one/recipes-core/base-files/base-files/</filename>. + However, if you are building for a different + machine and the + <filename>bblayers.conf</filename> file includes + the <filename>meta-one</filename> layer and + the location of your machine-specific file is + the first location where that file is found + according to <filename>FILESPATH</filename>, + builds for all machines will also use that + machine-specific file.</para> + <para>You can make sure that a machine-specific + file is used for a particular machine by putting + the file in a subdirectory specific to the + machine. + For example, rather than placing the file in + <filename>meta-one/recipes-core/base-files/base-files/</filename> + as shown above, put it in + <filename>meta-one/recipes-core/base-files/base-files/one/</filename>. + Not only does this make sure the file is used + only when building for machine "one", but the + build process locates the file more quickly.</para> + <para>In summary, you need to place all files + referenced from <filename>SRC_URI</filename> + in a machine-specific subdirectory within the + layer in order to restrict those files to + machine-specific builds.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='other-recommendations'> + <title>Other Recommendations</title> + + <para> + We also recommend the following: + <itemizedlist> + <listitem><para>Store custom layers in a Git repository + that uses the + <filename>meta-<replaceable>layer_name</replaceable></filename> format. + </para></listitem> + <listitem><para>Clone the repository alongside other + <filename>meta</filename> directories in the + <link linkend='source-directory'>Source Directory</link>. + </para></listitem> + </itemizedlist> + Following these recommendations keeps your Source Directory and + its configuration entirely inside the Yocto Project's core + base. + </para> + </section> + </section> + + <section id='enabling-your-layer'> + <title>Enabling Your Layer</title> + + <para> + Before the OpenEmbedded build system can use your new layer, + you need to enable it. + To enable your layer, simply add your layer's path to the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BBLAYERS'>BBLAYERS</ulink></filename> + variable in your <filename>conf/bblayers.conf</filename> file, + which is found in the + <link linkend='build-directory'>Build Directory</link>. + The following example shows how to enable a layer named + <filename>meta-mylayer</filename>: + <literallayout class='monospaced'> + LCONF_VERSION = "6" + + BBPATH = "${TOPDIR}" + BBFILES ?= "" + + BBLAYERS ?= " \ + $HOME/poky/meta \ + $HOME/poky/meta-poky \ + $HOME/poky/meta-yocto-bsp \ + $HOME/poky/meta-mylayer \ + " + </literallayout> + </para> + + <para> + BitBake parses each <filename>conf/layer.conf</filename> file + as specified in the <filename>BBLAYERS</filename> variable + within the <filename>conf/bblayers.conf</filename> file. + During the processing of each + <filename>conf/layer.conf</filename> file, BitBake adds the + recipes, classes and configurations contained within the + particular layer to the source directory. + </para> + </section> + + <section id='using-bbappend-files'> + <title>Using .bbappend Files</title> + + <para> + Recipes used to append Metadata to other recipes are called + BitBake append files. + BitBake append files use the <filename>.bbappend</filename> file + type suffix, while the corresponding recipes to which Metadata + is being appended use the <filename>.bb</filename> file type + suffix. + </para> + + <para> + A <filename>.bbappend</filename> file allows your layer to make + additions or changes to the content of another layer's recipe + without having to copy the other recipe into your layer. + Your <filename>.bbappend</filename> file resides in your layer, + while the main <filename>.bb</filename> recipe file to + which you are appending Metadata resides in a different layer. + </para> + + <para> + Append files must have the same root names as their corresponding + recipes. + For example, the append file + <filename>someapp_&DISTRO;.bbappend</filename> must apply to + <filename>someapp_&DISTRO;.bb</filename>. + This means the original recipe and append file names are version + number-specific. + If the corresponding recipe is renamed to update to a newer + version, the corresponding <filename>.bbappend</filename> file must + be renamed (and possibly updated) as well. + During the build process, BitBake displays an error on starting + if it detects a <filename>.bbappend</filename> file that does + not have a corresponding recipe with a matching name. + See the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_DANGLINGAPPENDS_WARNONLY'><filename>BB_DANGLINGAPPENDS_WARNONLY</filename></ulink> + variable for information on how to handle this error. + </para> + + <para> + Being able to append information to an existing recipe not only + avoids duplication, but also automatically applies recipe + changes in a different layer to your layer. + If you were copying recipes, you would have to manually merge + changes as they occur. + </para> + + <para> + As an example, consider the main formfactor recipe and a + corresponding formfactor append file both from the + <link linkend='source-directory'>Source Directory</link>. + Here is the main formfactor recipe, which is named + <filename>formfactor_0.0.bb</filename> and located in the + "meta" layer at + <filename>meta/recipes-bsp/formfactor</filename>: + <literallayout class='monospaced'> + SUMMARY = "Device formfactor information" + SECTION = "base" + LICENSE = "MIT" + LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \ + file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + PR = "r45" + + SRC_URI = "file://config file://machconfig" + S = "${WORKDIR}" + + PACKAGE_ARCH = "${MACHINE_ARCH}" + INHIBIT_DEFAULT_DEPS = "1" + + do_install() { + # Install file only if it has contents + install -d ${D}${sysconfdir}/formfactor/ + install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/ + if [ -s "${S}/machconfig" ]; then + install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/ + fi + } + </literallayout> + In the main recipe, note the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable, which tells the OpenEmbedded build system where to + find files during the build. + </para> + + <para> + Following is the append file, which is named + <filename>formfactor_0.0.bbappend</filename> and is from the + Raspberry Pi BSP Layer named + <filename>meta-raspberrypi</filename>. + The file is in <filename>recipes-bsp/formfactor</filename>: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" + </literallayout> + </para> + + <para> + By default, the build system uses the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + variable to locate files. + This append file extends the locations by setting the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + variable. + Setting this variable in the <filename>.bbappend</filename> + file is the most reliable and recommended method for adding + directories to the search path used by the build system + to find files. + </para> + + <para> + The statement in this example extends the directories to include + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-THISDIR'><filename>THISDIR</filename></ulink><filename>}/${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename>, + which resolves to a directory named + <filename>formfactor</filename> in the same directory + in which the append file resides (i.e. + <filename>meta-raspberrypi/recipes-bsp/formfactor/formfactor</filename>. + This implies that you must have the supporting directory + structure set up that will contain any files or patches you + will be including from the layer. + </para> + + <para> + Using the immediate expansion assignment operator + <filename>:=</filename> is important because of the reference to + <filename>THISDIR</filename>. + The trailing colon character is important as it ensures that + items in the list remain colon-separated. + <note> + <para> + BitBake automatically defines the + <filename>THISDIR</filename> variable. + You should never set this variable yourself. + Using "_prepend" as part of the + <filename>FILESEXTRAPATHS</filename> ensures your path + will be searched prior to other paths in the final + list. + </para> + + <para> + Also, not all append files add extra files. + Many append files simply exist to add build options + (e.g. <filename>systemd</filename>). + For these cases, your append file would not even + use the <filename>FILESEXTRAPATHS</filename> statement. + </para> + </note> + </para> + </section> + + <section id='prioritizing-your-layer'> + <title>Prioritizing Your Layer</title> + + <para> + Each layer is assigned a priority value. + Priority values control which layer takes precedence if there + are recipe files with the same name in multiple layers. + For these cases, the recipe file from the layer with a higher + priority number takes precedence. + Priority values also affect the order in which multiple + <filename>.bbappend</filename> files for the same recipe are + applied. + You can either specify the priority manually, or allow the + build system to calculate it based on the layer's dependencies. + </para> + + <para> + To specify the layer's priority manually, use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILE_PRIORITY'><filename>BBFILE_PRIORITY</filename></ulink> + variable. + For example: + <literallayout class='monospaced'> + BBFILE_PRIORITY_mylayer = "1" + </literallayout> + </para> + + <note> + <para>It is possible for a recipe with a lower version number + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + in a layer that has a higher priority to take precedence.</para> + <para>Also, the layer priority does not currently affect the + precedence order of <filename>.conf</filename> + or <filename>.bbclass</filename> files. + Future versions of BitBake might address this.</para> + </note> + </section> + + <section id='managing-layers'> + <title>Managing Layers</title> + + <para> + You can use the BitBake layer management tool to provide a view + into the structure of recipes across a multi-layer project. + Being able to generate output that reports on configured layers + with their paths and priorities and on + <filename>.bbappend</filename> files and their applicable + recipes can help to reveal potential problems. + </para> + + <para> + Use the following form when running the layer management tool. + <literallayout class='monospaced'> + $ bitbake-layers <replaceable>command</replaceable> [<replaceable>arguments</replaceable>] + </literallayout> + The following list describes the available commands: + <itemizedlist> + <listitem><para><filename><emphasis>help:</emphasis></filename> + Displays general help or help on a specified command. + </para></listitem> + <listitem><para><filename><emphasis>show-layers:</emphasis></filename> + Shows the current configured layers. + </para></listitem> + <listitem><para><filename><emphasis>show-recipes:</emphasis></filename> + Lists available recipes and the layers that provide them. + </para></listitem> + <listitem><para><filename><emphasis>show-overlayed:</emphasis></filename> + Lists overlayed recipes. + A recipe is overlayed when a recipe with the same name + exists in another layer that has a higher layer + priority. + </para></listitem> + <listitem><para><filename><emphasis>show-appends:</emphasis></filename> + Lists <filename>.bbappend</filename> files and the + recipe files to which they apply. + </para></listitem> + <listitem><para><filename><emphasis>show-cross-depends:</emphasis></filename> + Lists dependency relationships between recipes that + cross layer boundaries. + </para></listitem> + <listitem><para><filename><emphasis>add-layer:</emphasis></filename> + Adds a layer to <filename>bblayers.conf</filename>. + </para></listitem> + <listitem><para><filename><emphasis>remove-layer:</emphasis></filename> + Removes a layer from <filename>bblayers.conf</filename> + </para></listitem> + <listitem><para><filename><emphasis>flatten:</emphasis></filename> + Flattens the layer configuration into a separate output + directory. + Flattening your layer configuration builds a "flattened" + directory that contains the contents of all layers, + with any overlayed recipes removed and any + <filename>.bbappend</filename> files appended to the + corresponding recipes. + You might have to perform some manual cleanup of the + flattened layer as follows: + <itemizedlist> + <listitem><para>Non-recipe files (such as patches) + are overwritten. + The flatten command shows a warning for these + files. + </para></listitem> + <listitem><para>Anything beyond the normal layer + setup has been added to the + <filename>layer.conf</filename> file. + Only the lowest priority layer's + <filename>layer.conf</filename> is used. + </para></listitem> + <listitem><para>Overridden and appended items from + <filename>.bbappend</filename> files need to be + cleaned up. + The contents of each + <filename>.bbappend</filename> end up in the + flattened recipe. + However, if there are appended or changed + variable values, you need to tidy these up + yourself. + Consider the following example. + Here, the <filename>bitbake-layers</filename> + command adds the line + <filename>#### bbappended ...</filename> so that + you know where the following lines originate: + <literallayout class='monospaced'> + ... + DESCRIPTION = "A useful utility" + ... + EXTRA_OECONF = "--enable-something" + ... + + #### bbappended from meta-anotherlayer #### + + DESCRIPTION = "Customized utility" + EXTRA_OECONF += "--enable-somethingelse" + </literallayout> + Ideally, you would tidy up these utilities as + follows: + <literallayout class='monospaced'> + ... + DESCRIPTION = "Customized utility" + ... + EXTRA_OECONF = "--enable-something --enable-somethingelse" + ... + </literallayout></para></listitem> + </itemizedlist></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='creating-a-general-layer-using-the-yocto-layer-script'> + <title>Creating a General Layer Using the yocto-layer Script</title> + + <para> + The <filename>yocto-layer</filename> script simplifies + creating a new general layer. + <note> + For information on BSP layers, see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP Layers</ulink>" + section in the Yocto Project Board Specific (BSP) + Developer's Guide. + </note> + The default mode of the script's operation is to prompt you for + information needed to generate the layer: + <itemizedlist> + <listitem><para>The layer priority. + </para></listitem> + <listitem><para>Whether or not to create a sample recipe. + </para></listitem> + <listitem><para>Whether or not to create a sample + append file. + </para></listitem> + </itemizedlist> + </para> + + <para> + Use the <filename>yocto-layer create</filename> sub-command + to create a new general layer. + In its simplest form, you can create a layer as follows: + <literallayout class='monospaced'> + $ yocto-layer create mylayer + </literallayout> + The previous example creates a layer named + <filename>meta-mylayer</filename> in the current directory. + </para> + + <para> + As the <filename>yocto-layer create</filename> command runs, + default values for the prompts appear in brackets. + Pressing enter without supplying anything for the prompts + or pressing enter and providing an invalid response causes the + script to accept the default value. + Once the script completes, the new layer + is created in the current working directory. + The script names the layer by prepending + <filename>meta-</filename> to the name you provide. + </para> + + <para> + Minimally, the script creates the following within the layer: + <itemizedlist> + <listitem><para><emphasis>The <filename>conf</filename> + directory:</emphasis> + This directory contains the layer's configuration file. + The root name for the file is the same as the root name + your provided for the layer (e.g. + <filename><replaceable>layer</replaceable>.conf</filename>). + </para></listitem> + <listitem><para><emphasis>The + <filename>COPYING.MIT</filename> file:</emphasis> + The copyright and use notice for the software. + </para></listitem> + <listitem><para><emphasis>The <filename>README</filename> + file:</emphasis> + A file describing the contents of your new layer. + </para></listitem> + </itemizedlist> + </para> + + <para> + If you choose to generate a sample recipe file, the script + prompts you for the name for the recipe and then creates it + in <filename><replaceable>layer</replaceable>/recipes-example/example/</filename>. + The script creates a <filename>.bb</filename> file and a + directory, which contains a sample + <filename>helloworld.c</filename> source file, along with + a sample patch file. + If you do not provide a recipe name, the script uses + "example". + </para> + + <para> + If you choose to generate a sample append file, the script + prompts you for the name for the file and then creates it + in <filename><replaceable>layer</replaceable>/recipes-example-bbappend/example-bbappend/</filename>. + The script creates a <filename>.bbappend</filename> file and a + directory, which contains a sample patch file. + If you do not provide a recipe name, the script uses + "example". + The script also prompts you for the version of the append file. + The version should match the recipe to which the append file + is associated. + </para> + + <para> + The easiest way to see how the <filename>yocto-layer</filename> + script works is to experiment with the script. + You can also read the usage information by entering the + following: + <literallayout class='monospaced'> + $ yocto-layer help + </literallayout> + </para> + + <para> + Once you create your general layer, you must add it to your + <filename>bblayers.conf</filename> file. + Here is an example where a layer named + <filename>meta-mylayer</filename> is added: + <literallayout class='monospaced'> + BBLAYERS = ?" \ + /usr/local/src/yocto/meta \ + /usr/local/src/yocto/meta-poky \ + /usr/local/src/yocto/meta-yocto-bsp \ + /usr/local/src/yocto/meta-mylayer \ + " + </literallayout> + Adding the layer to this file enables the build system to + locate the layer during the build. + </para> + </section> + </section> + + <section id='usingpoky-extend-customimage'> + <title>Customizing Images</title> + + <para> + You can customize images to satisfy particular requirements. + This section describes several methods and provides guidelines for each. + </para> + + <section id='usingpoky-extend-customimage-localconf'> + <title>Customizing Images Using <filename>local.conf</filename></title> + + <para> + Probably the easiest way to customize an image is to add a + package by way of the <filename>local.conf</filename> + configuration file. + Because it is limited to local use, this method generally only + allows you to add packages and is not as flexible as creating + your own customized image. + When you add packages using local variables this way, you need + to realize that these variable changes are in effect for every + build and consequently affect all images, which might not + be what you require. + </para> + + <para> + To add a package to your image using the local configuration + file, use the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename> + variable with the <filename>_append</filename> operator: + <literallayout class='monospaced'> + IMAGE_INSTALL_append = " strace" + </literallayout> + Use of the syntax is important - specifically, the space between + the quote and the package name, which is + <filename>strace</filename> in this example. + This space is required since the <filename>_append</filename> + operator does not add the space. + </para> + + <para> + Furthermore, you must use <filename>_append</filename> instead + of the <filename>+=</filename> operator if you want to avoid + ordering issues. + The reason for this is because doing so unconditionally appends + to the variable and avoids ordering problems due to the + variable being set in image recipes and + <filename>.bbclass</filename> files with operators like + <filename>?=</filename>. + Using <filename>_append</filename> ensures the operation takes + affect. + </para> + + <para> + As shown in its simplest use, + <filename>IMAGE_INSTALL_append</filename> affects all images. + It is possible to extend the syntax so that the variable + applies to a specific image only. + Here is an example: + <literallayout class='monospaced'> + IMAGE_INSTALL_append_pn-core-image-minimal = " strace" + </literallayout> + This example adds <filename>strace</filename> to the + <filename>core-image-minimal</filename> image only. + </para> + + <para> + You can add packages using a similar approach through the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-CORE_IMAGE_EXTRA_INSTALL'>CORE_IMAGE_EXTRA_INSTALL</ulink></filename> + variable. + If you use this variable, only + <filename>core-image-*</filename> images are affected. + </para> + </section> + + <section id='usingpoky-extend-customimage-imagefeatures'> + <title>Customizing Images Using Custom <filename>IMAGE_FEATURES</filename> and + <filename>EXTRA_IMAGE_FEATURES</filename></title> + + <para> + Another method for customizing your image is to enable or + disable high-level image features by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> + variables. + Although the functions for both variables are nearly equivalent, + best practices dictate using <filename>IMAGE_FEATURES</filename> + from within a recipe and using + <filename>EXTRA_IMAGE_FEATURES</filename> from within + your <filename>local.conf</filename> file, which is found in the + <link linkend='build-directory'>Build Directory</link>. + </para> + + <para> + To understand how these features work, the best reference is + <filename>meta/classes/core-image.bbclass</filename>. + This class lists out the available + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + of which most map to package groups while some, such as + <filename>debug-tweaks</filename> and + <filename>read-only-rootfs</filename>, resolve as general + configuration settings. + </para> + + <para> + In summary, the file looks at the contents of the + <filename>IMAGE_FEATURES</filename> variable and then maps + or configures the feature accordingly. + Based on this information, the build system automatically + adds the appropriate packages or configurations to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink> + variable. + Effectively, you are enabling extra features by extending the + class or creating a custom class for use with specialized image + <filename>.bb</filename> files. + </para> + + <para> + Use the <filename>EXTRA_IMAGE_FEATURES</filename> variable + from within your local configuration file. + Using a separate area from which to enable features with + this variable helps you avoid overwriting the features in the + image recipe that are enabled with + <filename>IMAGE_FEATURES</filename>. + The value of <filename>EXTRA_IMAGE_FEATURES</filename> is added + to <filename>IMAGE_FEATURES</filename> within + <filename>meta/conf/bitbake.conf</filename>. + </para> + + <para> + To illustrate how you can use these variables to modify your + image, consider an example that selects the SSH server. + The Yocto Project ships with two SSH servers you can use + with your images: Dropbear and OpenSSH. + Dropbear is a minimal SSH server appropriate for + resource-constrained environments, while OpenSSH is a + well-known standard SSH server implementation. + By default, the <filename>core-image-sato</filename> image + is configured to use Dropbear. + The <filename>core-image-full-cmdline</filename> and + <filename>core-image-lsb</filename> images both + include OpenSSH. + The <filename>core-image-minimal</filename> image does not + contain an SSH server. + </para> + + <para> + You can customize your image and change these defaults. + Edit the <filename>IMAGE_FEATURES</filename> variable + in your recipe or use the + <filename>EXTRA_IMAGE_FEATURES</filename> in your + <filename>local.conf</filename> file so that it configures the + image you are working with to include + <filename>ssh-server-dropbear</filename> or + <filename>ssh-server-openssh</filename>. + </para> + + <note> + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" + section in the Yocto Project Reference Manual for a complete + list of image features that ship with the Yocto Project. + </note> + </section> + + <section id='usingpoky-extend-customimage-custombb'> + <title>Customizing Images Using Custom .bb Files</title> + + <para> + You can also customize an image by creating a custom recipe + that defines additional software as part of the image. + The following example shows the form for the two lines you need: + <literallayout class='monospaced'> + IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2" + + inherit core-image + </literallayout> + </para> + + <para> + Defining the software using a custom recipe gives you total + control over the contents of the image. + It is important to use the correct names of packages in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename> + variable. + You must use the OpenEmbedded notation and not the Debian notation for the names + (e.g. <filename>glibc-dev</filename> instead of <filename>libc6-dev</filename>). + </para> + + <para> + The other method for creating a custom image is to base it on an existing image. + For example, if you want to create an image based on <filename>core-image-sato</filename> + but add the additional package <filename>strace</filename> to the image, + copy the <filename>meta/recipes-sato/images/core-image-sato.bb</filename> to a + new <filename>.bb</filename> and add the following line to the end of the copy: + <literallayout class='monospaced'> + IMAGE_INSTALL += "strace" + </literallayout> + </para> + </section> + + <section id='usingpoky-extend-customimage-customtasks'> + <title>Customizing Images Using Custom Package Groups</title> + + <para> + For complex custom images, the best approach for customizing + an image is to create a custom package group recipe that is + used to build the image or images. + A good example of a package group recipe is + <filename>meta/recipes-core/packagegroups/packagegroup-base.bb</filename>. + </para> + + <para> + If you examine that recipe, you see that the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'>PACKAGES</ulink></filename> + variable lists the package group packages to produce. + The <filename>inherit packagegroup</filename> statement + sets appropriate default values and automatically adds + <filename>-dev</filename>, <filename>-dbg</filename>, and + <filename>-ptest</filename> complementary packages for each + package specified in the <filename>PACKAGES</filename> + statement. + <note> + The <filename>inherit packages</filename> should be + located near the top of the recipe, certainly before + the <filename>PACKAGES</filename> statement. + </note> + </para> + + <para> + For each package you specify in <filename>PACKAGES</filename>, + you can use + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'>RDEPENDS</ulink></filename> + and + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'>RRECOMMENDS</ulink></filename> + entries to provide a list of packages the parent task package + should contain. + You can see examples of these further down in the + <filename>packagegroup-base.bb</filename> recipe. + </para> + + <para> + Here is a short, fabricated example showing the same basic + pieces: + <literallayout class='monospaced'> + DESCRIPTION = "My Custom Package Groups" + + inherit packagegroup + + PACKAGES = "\ + packagegroup-custom-apps \ + packagegroup-custom-tools \ + " + + RDEPENDS_packagegroup-custom-apps = "\ + dropbear \ + portmap \ + psplash" + + RDEPENDS_packagegroup-custom-tools = "\ + oprofile \ + oprofileui-server \ + lttng-tools" + + RRECOMMENDS_packagegroup-custom-tools = "\ + kernel-module-oprofile" + </literallayout> + </para> + + <para> + In the previous example, two package group packages are created with their dependencies and their + recommended package dependencies listed: <filename>packagegroup-custom-apps</filename>, and + <filename>packagegroup-custom-tools</filename>. + To build an image using these package group packages, you need to add + <filename>packagegroup-custom-apps</filename> and/or + <filename>packagegroup-custom-tools</filename> to + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'>IMAGE_INSTALL</ulink></filename>. + For other forms of image dependencies see the other areas of this section. + </para> + </section> + + <section id='usingpoky-extend-customimage-image-name'> + <title>Customizing an Image Hostname</title> + + <para> + By default, the configured hostname (i.e. + <filename>/etc/hostname</filename>) in an image is the + same as the machine name. + For example, if + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + equals "qemux86", the configured hostname written to + <filename>/etc/hostname</filename> is "qemux86". + </para> + + <para> + You can customize this name by altering the value of the + "hostname" variable in the + <filename>base-files</filename> recipe using either + an append file or a configuration file. + Use the following in an append file: + <literallayout class='monospaced'> + hostname="myhostname" + </literallayout> + Use the following in a configuration file: + <literallayout class='monospaced'> + hostname_pn-base-files = "myhostname" + </literallayout> + </para> + + <para> + Changing the default value of the variable "hostname" can be + useful in certain situations. + For example, suppose you need to do extensive testing on an + image and you would like to easily identify the image + under test from existing images with typical default + hostnames. + In this situation, you could change the default hostname to + "testme", which results in all the images using the name + "testme". + Once testing is complete and you do not need to rebuild the + image for test any longer, you can easily reset the default + hostname. + </para> + + <para> + Another point of interest is that if you unset the variable, + the image will have no default hostname in the filesystem. + Here is an example that unsets the variable in a + configuration file: + <literallayout class='monospaced'> + hostname_pn-base-files = "" + </literallayout> + Having no default hostname in the filesystem is suitable for + environments that use dynamic hostnames such as virtual + machines. + </para> + </section> + </section> + + <section id='new-recipe-writing-a-new-recipe'> + <title>Writing a New Recipe</title> + + <para> + Recipes (<filename>.bb</filename> files) are fundamental components + in the Yocto Project environment. + Each software component built by the OpenEmbedded build system + requires a recipe to define the component. + This section describes how to create, write, and test a new + recipe. + <note> + For information on variables that are useful for recipes and + for information about recipe naming issues, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-varlocality-recipe-required'>Required</ulink>" + section of the Yocto Project Reference Manual. + </note> + </para> + + <section id='new-recipe-overview'> + <title>Overview</title> + + <para> + The following figure shows the basic process for creating a + new recipe. + The remainder of the section provides details for the steps. + <imagedata fileref="figures/recipe-workflow.png" width="6in" depth="7in" align="center" scalefit="1" /> + </para> + </section> + + <section id='new-recipe-locate-or-automatically-create-a-base-recipe'> + <title>Locate or Automatically Create a Base Recipe</title> + + <para> + You can always write a recipe from scratch. + However, two choices exist that can help you quickly get a + start on a new recipe: + <itemizedlist> + <listitem><para><emphasis><filename>recipetool</filename>:</emphasis> + A tool provided by the Yocto Project that automates + creation of a base recipe based on the source + files. + </para></listitem> + <listitem><para><emphasis>Existing Recipes:</emphasis> + Location and modification of an existing recipe that is + similar in function to the recipe you need. + </para></listitem> + </itemizedlist> + </para> + + <section id='new-recipe-creating-the-base-recipe-using-recipetool'> + <title>Creating the Base Recipe Using <filename>recipetool</filename></title> + + <para> + <filename>recipetool</filename> automates creation of + a base recipe given a set of source code files. + As long as you can extract or point to the source files, + the tool will construct a recipe and automatically + configure all pre-build information into the recipe. + For example, suppose you have an application that builds + using Autotools. + Creating the base recipe using + <filename>recipetool</filename> results in a recipe + that has the pre-build dependencies, license requirements, + and checksums configured. + </para> + + <para> + To run the tool, you just need to be in your + <link linkend='build-directory'>Build Directory</link> + and have sourced the build environment setup script + (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>oe-init-build-env</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>). + Here is the basic <filename>recipetool</filename> syntax: + <note> + Running <filename>recipetool -h</filename> or + <filename>recipetool create -h</filename> produces the + Python-generated help, which presented differently + than what follows here. + </note> + <literallayout class='monospaced'> + recipetool -h + recipetool create [-h] + recipetool [-d] [-q] [--color auto | always | never ] create -o <replaceable>OUTFILE</replaceable> [-m] [-x <replaceable>EXTERNALSRC</replaceable>] <replaceable>source</replaceable> + + -d Enables debug output. + -q Outputs only errors (quiet mode). + --color Colorizes the output automatically, always, or never. + -h Displays Python generated syntax for recipetool. + create Causes recipetool to create a base recipe. The create + command is further defined with these options: + + -o <replaceable>OUTFILE</replaceable> Specifies the full path and filename for the generated + recipe. + -m Causes the recipe to be machine-specific rather than + architecture-specific (default). + -x <replaceable>EXTERNALSRC</replaceable> Fetches and extracts source files from <replaceable>source</replaceable> + and places them in <replaceable>EXTERNALSRC</replaceable>. + <replaceable>source</replaceable> must be a URL. + -h Displays Python-generated syntax for create. + <replaceable>source</replaceable> Specifies the source code on which to base the + recipe. + </literallayout> + </para> + + <para> + Running <filename>recipetool create -o</filename> <replaceable>OUTFILE</replaceable> + creates the base recipe and locates it properly in the + layer that contains your source files. + Following are some syntax examples: + </para> + + <para> + Use this syntax to generate a recipe based on <replaceable>source</replaceable>. + Once generated, the recipe resides in the existing source + code layer: + <literallayout class='monospaced'> + recipetool create -o <replaceable>OUTFILE</replaceable> <replaceable>source</replaceable> + </literallayout> + Use this syntax to generate a recipe using code that you + extract from <replaceable>source</replaceable>. + The extracted code is placed in its own layer defined + by <replaceable>EXTERNALSRC</replaceable>. + <literallayout class='monospaced'> + recipetool create -o <replaceable>OUTFILE</replaceable> -x <replaceable>EXTERNALSRC</replaceable> <replaceable>source</replaceable> + </literallayout> + Use this syntax to generate a recipe based on <replaceable>source</replaceable>. + The options direct <filename>recipetool</filename> to + run in "quiet mode" and to generate debugging information. + Once generated, the recipe resides in the existing source + code layer: + <literallayout class='monospaced'> + recipetool create -o <replaceable>OUTFILE</replaceable> <replaceable>source</replaceable> + </literallayout> + </para> + </section> + + <section id='new-recipe-locating-and-using-a-similar-recipe'> + <title>Locating and Using a Similar Recipe</title> + + <para> + Before writing a recipe from scratch, it is often useful to + discover whether someone else has already written one that + meets (or comes close to meeting) your needs. + The Yocto Project and OpenEmbedded communities maintain many + recipes that might be candidates for what you are doing. + You can find a good central index of these recipes in the + <ulink url='http://layers.openembedded.org'>OpenEmbedded metadata index</ulink>. + </para> + + <para> + Working from an existing recipe or a skeleton recipe is the + best way to get started. + Here are some points on both methods: + <itemizedlist> + <listitem><para><emphasis>Locate and modify a recipe that + is close to what you want to do:</emphasis> + This method works when you are familiar with the + current recipe space. + The method does not work so well for those new to + the Yocto Project or writing recipes.</para> + <para>Some risks associated with this method are + using a recipe that has areas totally unrelated to + what you are trying to accomplish with your recipe, + not recognizing areas of the recipe that you might + have to add from scratch, and so forth. + All these risks stem from unfamiliarity with the + existing recipe space.</para></listitem> + <listitem><para><emphasis>Use and modify the following + skeleton recipe:</emphasis> + If for some reason you do not want to use + <filename>recipetool</filename> and you cannot + find an existing recipe that is close to meeting + your needs, you can use the following structure to + provide the fundamental areas of a new recipe. + <literallayout class='monospaced'> + DESCRIPTION = "" + HOMEPAGE = "" + LICENSE = "" + SECTION = "" + DEPENDS = "" + LIC_FILES_CHKSUM = "" + + SRC_URI = "" + </literallayout> + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='new-recipe-storing-and-naming-the-recipe'> + <title>Storing and Naming the Recipe</title> + + <para> + Once you have your base recipe, you should put it in your + own layer and name it appropriately. + Locating it correctly ensures that the OpenEmbedded build + system can find it when you use BitBake to process the + recipe. + </para> + + <itemizedlist> + <listitem><para><emphasis>Storing Your Recipe:</emphasis> + The OpenEmbedded build system locates your recipe + through the layer's <filename>conf/layer.conf</filename> + file and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBFILES'><filename>BBFILES</filename></ulink> + variable. + This variable sets up a path from which the build system can + locate recipes. + Here is the typical use: + <literallayout class='monospaced'> + BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + </literallayout> + Consequently, you need to be sure you locate your new recipe + inside your layer such that it can be found.</para> + <para>You can find more information on how layers are + structured in the + "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" + section.</para></listitem> + <listitem><para><emphasis>Naming Your Recipe:</emphasis> + When you name your recipe, you need to follow this naming + convention: + <literallayout class='monospaced'> + <replaceable>basename</replaceable>_<replaceable>version</replaceable>.bb + </literallayout> + Use lower-cased characters and do not include the reserved + suffixes <filename>-native</filename>, + <filename>-cross</filename>, <filename>-initial</filename>, + or <filename>-dev</filename> casually (i.e. do not use them + as part of your recipe name unless the string applies). + Here are some examples: + <literallayout class='monospaced'> + cups_1.7.0.bb + gawk_4.0.2.bb + irssi_0.8.16-rc1.bb + </literallayout></para></listitem> + </itemizedlist> + </section> + + <section id='understanding-recipe-syntax'> + <title>Understanding Recipe Syntax</title> + + <para> + Understanding recipe file syntax is important for + writing recipes. + The following list overviews the basic items that make up a + BitBake recipe file. + For more complete BitBake syntax descriptions, see the + "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>" + chapter of the BitBake User Manual. + <itemizedlist> + <listitem><para><emphasis>Variable Assignments and Manipulations:</emphasis> + Variable assignments allow a value to be assigned to a + variable. + The assignment can be static text or might include + the contents of other variables. + In addition to the assignment, appending and prepending + operations are also supported.</para> + <para>The following example shows some of the ways + you can use variables in recipes: + <literallayout class='monospaced'> + S = "${WORKDIR}/postfix-${PV}" + CFLAGS += "-DNO_ASM" + SRC_URI_append = " file://fixup.patch" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Functions:</emphasis> + Functions provide a series of actions to be performed. + You usually use functions to override the default + implementation of a task function or to complement + a default function (i.e. append or prepend to an + existing function). + Standard functions use <filename>sh</filename> shell + syntax, although access to OpenEmbedded variables and + internal methods are also available.</para> + <para>The following is an example function from the + <filename>sed</filename> recipe: + <literallayout class='monospaced'> + do_install () { + autotools_do_install + install -d ${D}${base_bindir} + mv ${D}${bindir}/sed ${D}${base_bindir}/sed + rmdir ${D}${bindir}/ + } + </literallayout> + It is also possible to implement new functions that + are called between existing tasks as long as the + new functions are not replacing or complementing the + default functions. + You can implement functions in Python + instead of shell. + Both of these options are not seen in the majority of + recipes.</para></listitem> + <listitem><para><emphasis>Keywords:</emphasis> + BitBake recipes use only a few keywords. + You use keywords to include common + functions (<filename>inherit</filename>), load parts + of a recipe from other files + (<filename>include</filename> and + <filename>require</filename>) and export variables + to the environment (<filename>export</filename>).</para> + <para>The following example shows the use of some of + these keywords: + <literallayout class='monospaced'> + export POSTCONF = "${STAGING_BINDIR}/postconf" + inherit autoconf + require otherfile.inc + </literallayout> + </para></listitem> + <listitem><para><emphasis>Comments:</emphasis> + Any lines that begin with the hash character + (<filename>#</filename>) are treated as comment lines + and are ignored: + <literallayout class='monospaced'> + # This is a comment + </literallayout> + </para></listitem> + </itemizedlist> + </para> + + <para> + This next list summarizes the most important and most commonly + used parts of the recipe syntax. + For more information on these parts of the syntax, you can + reference the + <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink> + chapter in the BitBake User Manual. + <itemizedlist> + <listitem><para><emphasis>Line Continuation: <filename>\</filename></emphasis> - + Use the backward slash (<filename>\</filename>) + character to split a statement over multiple lines. + Place the slash character at the end of the line that + is to be continued on the next line: + <literallayout class='monospaced'> + VAR = "A really long \ + line" + </literallayout> + <note> + You cannot have any characters including spaces + or tabs after the slash character. + </note> + </para></listitem> + <listitem><para><emphasis>Using Variables: <filename>${...}</filename></emphasis> - + Use the <filename>${<replaceable>varname</replaceable>}</filename> syntax to + access the contents of a variable: + <literallayout class='monospaced'> + SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Quote All Assignments: <filename>"<replaceable>value</replaceable>"</filename></emphasis> - + Use double quotes around the value in all variable + assignments. + <literallayout class='monospaced'> + VAR1 = "${OTHERVAR}" + VAR2 = "The version is ${PV}" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Conditional Assignment: <filename>?=</filename></emphasis> - + Conditional assignment is used to assign a value to + a variable, but only when the variable is currently + unset. + Use the question mark followed by the equal sign + (<filename>?=</filename>) to make a "soft" assignment + used for conditional assignment. + Typically, "soft" assignments are used in the + <filename>local.conf</filename> file for variables + that are allowed to come through from the external + environment. + </para> + <para>Here is an example where + <filename>VAR1</filename> is set to "New value" if + it is currently empty. + However, if <filename>VAR1</filename> has already been + set, it remains unchanged: + <literallayout class='monospaced'> + VAR1 ?= "New value" + </literallayout> + In this next example, <filename>VAR1</filename> + is left with the value "Original value": + <literallayout class='monospaced'> + VAR1 = "Original value" + VAR1 ?= "New value" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Appending: <filename>+=</filename></emphasis> - + Use the plus character followed by the equals sign + (<filename>+=</filename>) to append values to existing + variables. + <note> + This operator adds a space between the existing + content of the variable and the new content. + </note></para> + <para>Here is an example: + <literallayout class='monospaced'> + SRC_URI += "file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Prepending: <filename>=+</filename></emphasis> - + Use the equals sign followed by the plus character + (<filename>=+</filename>) to prepend values to existing + variables. + <note> + This operator adds a space between the new content + and the existing content of the variable. + </note></para> + <para>Here is an example: + <literallayout class='monospaced'> + VAR =+ "Starts" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Appending: <filename>_append</filename></emphasis> - + Use the <filename>_append</filename> operator to + append values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + <para>The following example shows the space being + explicitly added to the start to ensure the appended + value is not merged with the existing value: + <literallayout class='monospaced'> + SRC_URI_append = " file://fix-makefile.patch" + </literallayout> + You can also use the <filename>_append</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + SRC_URI_append_sh4 = " file://fix-makefile.patch" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Prepending: <filename>_prepend</filename></emphasis> - + Use the <filename>_prepend</filename> operator to + prepend values to existing variables. + This operator does not add any additional space. + Also, the operator is applied after all the + <filename>+=</filename>, and + <filename>=+</filename> operators have been applied and + after all <filename>=</filename> assignments have + occurred. + </para> + <para>The following example shows the space being + explicitly added to the end to ensure the prepended + value is not merged with the existing value: + <literallayout class='monospaced'> + CFLAGS_prepend = "-I${S}/myincludes " + </literallayout> + You can also use the <filename>_prepend</filename> + operator with overrides, which results in the actions + only being performed for the specified target or + machine: + <literallayout class='monospaced'> + CFLAGS_prepend_sh4 = "-I${S}/myincludes " + </literallayout> + </para></listitem> + <listitem><para><emphasis>Overrides:</emphasis> - + You can use overrides to set a value conditionally, + typically based on how the recipe is being built. + For example, to set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-KBRANCH'><filename>KBRANCH</filename></ulink> + variable's value to "standard/base" for any target + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + except for qemuarm where it should be set to + "standard/arm-versatile-926ejs", you would do the + following: + <literallayout class='monospaced'> + KBRANCH = "standard/base" + KBRANCH_qemuarm = "standard/arm-versatile-926ejs" + </literallayout> + Overrides are also used to separate alternate values + of a variable in other situations. + For example, when setting variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + that are specific to individual packages produced by + a recipe, you should always use an override that + specifies the name of the package. + </para></listitem> + <listitem><para><emphasis>Indentation:</emphasis> + Use spaces for indentation rather than than tabs. + For shell functions, both currently work. + However, it is a policy decision of the Yocto Project + to use tabs in shell functions. + Realize that some layers have a policy to use spaces + for all indentation. + </para></listitem> + <listitem><para><emphasis>Using Python for Complex Operations: <filename>${@<replaceable>python_code</replaceable>}</filename></emphasis> - + For more advanced processing, it is possible to use + Python code during variable assignments (e.g. + search and replacement on a variable).</para> + <para>You indicate Python code using the + <filename>${@<replaceable>python_code</replaceable>}</filename> + syntax for the variable assignment: + <literallayout class='monospaced'> + SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz + </literallayout> + </para></listitem> + <listitem><para><emphasis>Shell Function Syntax:</emphasis> + Write shell functions as if you were writing a shell + script when you describe a list of actions to take. + You should ensure that your script works with a generic + <filename>sh</filename> and that it does not require + any <filename>bash</filename> or other shell-specific + functionality. + The same considerations apply to various system + utilities (e.g. <filename>sed</filename>, + <filename>grep</filename>, <filename>awk</filename>, + and so forth) that you might wish to use. + If in doubt, you should check with multiple + implementations - including those from BusyBox. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-running-a-build-on-the-recipe'> + <title>Running a Build on the Recipe</title> + + <para> + Creating a new recipe is usually an iterative process that + requires using BitBake to process the recipe multiple times in + order to progressively discover and add information to the + recipe file. + </para> + + <para> + Assuming you have sourced a build environment setup script (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>) + and you are in the + <link linkend='build-directory'>Build Directory</link>, + use BitBake to process your recipe. + All you need to provide is the + <filename><replaceable>basename</replaceable></filename> of the recipe as described + in the previous section: + <literallayout class='monospaced'> + $ bitbake <replaceable>basename</replaceable> + </literallayout> + + </para> + + <para> + During the build, the OpenEmbedded build system creates a + temporary work directory for each recipe + (<filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>) + where it keeps extracted source files, log files, intermediate + compilation and packaging files, and so forth. + </para> + + <para> + The per-recipe temporary work directory is constructed as follows and + depends on several factors: + <literallayout class='monospaced'> + BASE_WORKDIR ?= "${TMPDIR}/work" + WORKDIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}" + </literallayout> + As an example, assume a Source Directory top-level folder named + <filename>poky</filename>, a default Build Directory at + <filename>poky/build</filename>, and a + <filename>qemux86-poky-linux</filename> machine target system. + Furthermore, suppose your recipe is named + <filename>foo_1.3.0.bb</filename>. + In this case, the work directory the build system uses to + build the package would be as follows: + <literallayout class='monospaced'> + poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0 + </literallayout> + Inside this directory you can find sub-directories such as + <filename>image</filename>, <filename>packages-split</filename>, + and <filename>temp</filename>. + After the build, you can examine these to determine how well + the build went. + <note> + You can find log files for each task in the recipe's + <filename>temp</filename> directory (e.g. + <filename>poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp</filename>). + Log files are named <filename>log.<replaceable>taskname</replaceable></filename> + (e.g. <filename>log.do_configure</filename>, + <filename>log.do_fetch</filename>, and + <filename>log.do_compile</filename>). + </note> + </para> + + <para> + You can find more information about the build process in the + "<ulink url='&YOCTO_DOCS_REF_URL;#closer-look'>A Closer Look at the Yocto Project Development Environment</ulink>" + chapter of the Yocto Project Reference Manual. + </para> + + <para> + You can also reference the following variables in the + Yocto Project Reference Manual's glossary for more information: + <itemizedlist> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>: + The top-level build output directory</listitem> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-MULTIMACH_TARGET_SYS'><filename>MULTIMACH_TARGET_SYS</filename></ulink>: + The target system identifier</listitem> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>: + The recipe name</listitem> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-EXTENDPE'><filename>EXTENDPE</filename></ulink>: + The epoch - (if + <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink> + is not specified, which is usually the case for most + recipes, then <filename>EXTENDPE</filename> is blank)</listitem> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>: + The recipe version</listitem> + <listitem><ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>: + The recipe revision</listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-fetching-code'> + <title>Fetching Code</title> + + <para> + The first thing your recipe must do is specify how to fetch + the source files. + Fetching is controlled mainly through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable. + Your recipe must have a <filename>SRC_URI</filename> variable + that points to where the source is located. + For a graphical representation of source locations, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#sources-dev-environment'>Sources</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <para> + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink> + task uses the prefix of each entry in the + <filename>SRC_URI</filename> variable value to determine which + fetcher to use to get your source files. + It is the <filename>SRC_URI</filename> variable that triggers + the fetcher. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + task uses the variable after source is fetched to apply + patches. + The OpenEmbedded build system uses + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESOVERRIDES'><filename>FILESOVERRIDES</filename></ulink> + for scanning directory locations for local files in + <filename>SRC_URI</filename>. + </para> + + <para> + The <filename>SRC_URI</filename> variable in your recipe must + define each unique location for your source files. + It is good practice to not hard-code pathnames in an URL used + in <filename>SRC_URI</filename>. + Rather than hard-code these paths, use + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink><filename>}</filename>, + which causes the fetch process to use the version specified in + the recipe filename. + Specifying the version in this manner means that upgrading the + recipe to a future version is as simple as renaming the recipe + to match the new version. + </para> + + <para> + Here is a simple example from the + <filename>meta/recipes-devtools/cdrtools/cdrtools-native_3.01a20.bb</filename> + recipe where the source comes from a single tarball. + Notice the use of the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + variable: + <literallayout class='monospaced'> + SRC_URI = "ftp://ftp.berlios.de/pub/cdrecord/alpha/cdrtools-${PV}.tar.bz2" + </literallayout> + </para> + + <para> + Files mentioned in <filename>SRC_URI</filename> whose names end + in a typical archive extension (e.g. <filename>.tar</filename>, + <filename>.tar.gz</filename>, <filename>.tar.bz2</filename>, + <filename>.zip</filename>, and so forth), are automatically + extracted during the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> + task. + For another example that specifies these types of files, see + the + "<link linkend='new-recipe-autotooled-package'>Autotooled Package</link>" + section. + </para> + + <para> + Another way of specifying source is from an SCM. + For Git repositories, you must specify + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink> + and you should specify + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + to include the revision with + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCPV'><filename>SRCPV</filename></ulink>. + Here is an example from the recipe + <filename>meta/recipes-kernel/blktrace/blktrace_git.bb</filename>: + <literallayout class='monospaced'> + SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385" + + PR = "r6" + PV = "1.0.5+git${SRCPV}" + + SRC_URI = "git://git.kernel.dk/blktrace.git \ + file://ldflags.patch" + </literallayout> + </para> + + <para> + If your <filename>SRC_URI</filename> statement includes + URLs pointing to individual files fetched from a remote server + other than a version control system, BitBake attempts to + verify the files against checksums defined in your recipe to + ensure they have not been tampered with or otherwise modified + since the recipe was written. + Two checksums are used: + <filename>SRC_URI[md5sum]</filename> and + <filename>SRC_URI[sha256sum]</filename>. + </para> + + <para> + If your <filename>SRC_URI</filename> variable points to + more than a single URL (excluding SCM URLs), you need to + provide the <filename>md5</filename> and + <filename>sha256</filename> checksums for each URL. + For these cases, you provide a name for each URL as part of + the <filename>SRC_URI</filename> and then reference that name + in the subsequent checksum statements. + Here is an example: + <literallayout class='monospaced'> + SRC_URI = "${DEBIAN_MIRROR}/main/a/apmd/apmd_3.2.2.orig.tar.gz;name=tarball \ + ${DEBIAN_MIRROR}/main/a/apmd/apmd_${PV}.diff.gz;name=patch + + SRC_URI[tarball.md5sum] = "b1e6309e8331e0f4e6efd311c2d97fa8" + SRC_URI[tarball.sha256sum] = "7f7d9f60b7766b852881d40b8ff91d8e39fccb0d1d913102a5c75a2dbb52332d" + + SRC_URI[patch.md5sum] = "57e1b689264ea80f78353519eece0c92" + SRC_URI[patch.sha256sum] = "7905ff96be93d725544d0040e425c42f9c05580db3c272f11cff75b9aa89d430" + </literallayout> + </para> + + <para> + Proper values for <filename>md5</filename> and + <filename>sha256</filename> checksums might be available + with other signatures on the download page for the upstream + source (e.g. <filename>md5</filename>, + <filename>sha1</filename>, <filename>sha256</filename>, + <filename>GPG</filename>, and so forth). + Because the OpenEmbedded build system only deals with + <filename>sha256sum</filename> and <filename>md5sum</filename>, + you should verify all the signatures you find by hand. + </para> + + <para> + If no <filename>SRC_URI</filename> checksums are specified + when you attempt to build the recipe, or you provide an + incorrect checksum, the build will produce an error for each + missing or incorrect checksum. + As part of the error message, the build system provides + the checksum string corresponding to the fetched file. + Once you have the correct checksums, you can copy and paste + them into your recipe and then run the build again to continue. + <note> + As mentioned, if the upstream source provides signatures + for verifying the downloaded source code, you should + verify those manually before setting the checksum values + in the recipe and continuing with the build. + </note> + </para> + + <para> + This final example is a bit more complicated and is from the + <filename>meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb</filename> + recipe. + The example's <filename>SRC_URI</filename> statement identifies + multiple files as the source files for the recipe: a tarball, a + patch file, a desktop file, and an icon. + <literallayout class='monospaced'> + SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \ + file://xwc.patch \ + file://rxvt.desktop \ + file://rxvt.png" + </literallayout> + </para> + + <para> + When you specify local files using the + <filename>file://</filename> URI protocol, the build system + fetches files from the local machine. + The path is relative to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + variable and searches specific directories in a certain order: + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BP'><filename>BP</filename></ulink><filename>}</filename>, + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink><filename>}</filename>, + and <filename>files</filename>. + The directories are assumed to be subdirectories of the + directory in which the recipe or append file resides. + For another example that specifies these types of files, see the + "<link linkend='new-recipe-single-c-file-package-hello-world'>Single .c File Package (Hello World!)</link>" + section. + </para> + + <para> + The previous example also specifies a patch file. + Patch files are files whose names usually end in + <filename>.patch</filename> or <filename>.diff</filename> but + can end with compressed suffixes such as + <filename>diff.gz</filename> and + <filename>patch.bz2</filename>, for example. + The build system automatically applies patches as described + in the + "<link linkend='new-recipe-patching-code'>Patching Code</link>" section. + </para> + </section> + + <section id='new-recipe-unpacking-code'> + <title>Unpacking Code</title> + + <para> + During the build, the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink> + task unpacks the source with + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename> + pointing to where it is unpacked. + </para> + + <para> + If you are fetching your source files from an upstream source + archived tarball and the tarball's internal structure matches + the common convention of a top-level subdirectory named + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink><filename>}-${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink><filename>}</filename>, + then you do not need to set <filename>S</filename>. + However, if <filename>SRC_URI</filename> specifies to fetch + source from an archive that does not use this convention, + or from an SCM like Git or Subversion, your recipe needs to + define <filename>S</filename>. + </para> + + <para> + If processing your recipe using BitBake successfully unpacks + the source files, you need to be sure that the directory + pointed to by <filename>${S}</filename> matches the structure + of the source. + </para> + </section> + + <section id='new-recipe-patching-code'> + <title>Patching Code</title> + + <para> + Sometimes it is necessary to patch code after it has been + fetched. + Any files mentioned in <filename>SRC_URI</filename> whose + names end in <filename>.patch</filename> or + <filename>.diff</filename> or compressed versions of these + suffixes (e.g. <filename>diff.gz</filename> are treated as + patches. + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink> + task automatically applies these patches. + </para> + + <para> + The build system should be able to apply patches with the "-p1" + option (i.e. one directory level in the path will be stripped + off). + If your patch needs to have more directory levels stripped off, + specify the number of levels using the "striplevel" option in + the <filename>SRC_URI</filename> entry for the patch. + Alternatively, if your patch needs to be applied in a specific + subdirectory that is not specified in the patch file, use the + "patchdir" option in the entry. + </para> + + <para> + As with all local files referenced in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + using <filename>file://</filename>, you should place + patch files in a directory next to the recipe either + named the same as the base name of the recipe + (<ulink url='&YOCTO_DOCS_REF_URL;#var-BP'><filename>BP</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-BPN'><filename>BPN</filename></ulink>) + or "files". + </para> + </section> + + <section id='new-recipe-licensing'> + <title>Licensing</title> + + <para> + Your recipe needs to have both the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'><filename>LIC_FILES_CHKSUM</filename></ulink> + variables: + <itemizedlist> + <listitem><para><emphasis><filename>LICENSE</filename>:</emphasis> + This variable specifies the license for the software. + If you do not know the license under which the software + you are building is distributed, you should go to the + source code and look for that information. + Typical files containing this information include + <filename>COPYING</filename>, + <filename>LICENSE</filename>, and + <filename>README</filename> files. + You could also find the information near the top of + a source file. + For example, given a piece of software licensed under + the GNU General Public License version 2, you would + set <filename>LICENSE</filename> as follows: + <literallayout class='monospaced'> + LICENSE = "GPLv2" + </literallayout></para> + <para>The licenses you specify within + <filename>LICENSE</filename> can have any name as long + as you do not use spaces, since spaces are used as + separators between license names. + For standard licenses, use the names of the files in + <filename>meta/files/common-licenses/</filename> + or the <filename>SPDXLICENSEMAP</filename> flag names + defined in <filename>meta/conf/licenses.conf</filename>. + </para></listitem> + <listitem><para><emphasis><filename>LIC_FILES_CHKSUM</filename>:</emphasis> + The OpenEmbedded build system uses this variable to + make sure the license text has not changed. + If it has, the build produces an error and it affords + you the chance to figure it out and correct the problem. + </para> + <para>You need to specify all applicable licensing + files for the software. + At the end of the configuration step, the build process + will compare the checksums of the files to be sure + the text has not changed. + Any differences result in an error with the message + containing the current checksum. + For more explanation and examples of how to set the + <filename>LIC_FILES_CHKSUM</filename> variable, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</ulink>" + section in the Yocto Project Reference Manual.</para> + <para>To determine the correct checksum string, you + can list the appropriate files in the + <filename>LIC_FILES_CHKSUM</filename> variable with + incorrect md5 strings, attempt to build the software, + and then note the resulting error messages that will + report the correct md5 strings. + See the + "<link linkend='new-recipe-fetching-code'>Fetching Code</link>" + section for additional information. + </para> + + <para> + Here is an example that assumes the software has a + <filename>COPYING</filename> file: + <literallayout class='monospaced'> + LIC_FILES_CHKSUM = "file://COPYING;md5=xxx" + </literallayout> + When you try to build the software, the build system + will produce an error and give you the correct string + that you can substitute into the recipe file for a + subsequent build. + </para></listitem> + </itemizedlist> + </para> + +<!-- + + <para> + For trying this out I created a new recipe named + <filename>htop_1.0.2.bb</filename> and put it in + <filename>poky/meta/recipes-extended/htop</filename>. + There are two license type statements in my very simple + recipe: + <literallayout class='monospaced'> + LICENSE = "" + + LIC_FILES_CHKSUM = "" + + SRC_URI[md5sum] = "" + SRC_URI[sha256sum] = "" + </literallayout> + Evidently, you need to run a <filename>bitbake -c cleanall htop</filename>. + Next, you delete or comment out the two <filename>SRC_URI</filename> + lines at the end and then attempt to build the software with + <filename>bitbake htop</filename>. + Doing so causes BitBake to report some errors and and give + you the actual strings you need for the last two + <filename>SRC_URI</filename> lines. + Prior to this, you have to dig around in the home page of the + source for <filename>htop</filename> and determine that the + software is released under GPLv2. + You can provide that in the <filename>LICENSE</filename> + statement. + Now you edit your recipe to have those two strings for + the <filename>SRC_URI</filename> statements: + <literallayout class='monospaced'> + LICENSE = "GPLv2" + + LIC_FILES_CHKSUM = "" + + SRC_URI = "${SOURCEFORGE_MIRROR}/htop/htop-${PV}.tar.gz" + SRC_URI[md5sum] = "0d01cca8df3349c74569cefebbd9919e" + SRC_URI[sha256sum] = "ee60657b044ece0df096c053060df7abf3cce3a568ab34d260049e6a37ccd8a1" + </literallayout> + At this point, you can build the software again using the + <filename>bitbake htop</filename> command. + There is just a set of errors now associated with the + empty <filename>LIC_FILES_CHKSUM</filename> variable now. + </para> +--> + + </section> + + <section id='new-recipe-configuring-the-recipe'> + <title>Configuring the Recipe</title> + + <para> + Most software provides some means of setting build-time + configuration options before compilation. + Typically, setting these options is accomplished by running a + configure script with some options, or by modifying a build + configuration file. + <note> + As of Yocto Project Release 7.1, some of the core recipes + that package binary configuration scripts now disable the + scripts due to the scripts previously requiring error-prone + path substitution. + The OpenEmbedded build system uses + <filename>pkg-config</filename> now, which is much more + robust. + You can find a list of the <filename>*-config</filename> + scripts that are disabled list in the + "<ulink url='&YOCTO_DOCS_REF_URL;#migration-1.7-binary-configuration-scripts-disabled'>Binary Configuration Scripts Disabled</ulink>" + section in the Yocto Project Reference Manual. + </note> + </para> + + <para> + A major part of build-time configuration is about checking for + build-time dependencies and possibly enabling optional + functionality as a result. + You need to specify any build-time dependencies for the + software you are building in your recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + value, in terms of other recipes that satisfy those + dependencies. + You can often find build-time or runtime + dependencies described in the software's documentation. + </para> + + <para> + The following list provides configuration items of note based + on how your software is built: + <itemizedlist> + <listitem><para><emphasis>Autotools:</emphasis> + If your source files have a + <filename>configure.ac</filename> file, then your + software is built using Autotools. + If this is the case, you just need to worry about + modifying the configuration.</para> + <para>When using Autotools, your recipe needs to inherit + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + class and your recipe does not have to contain a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task. + However, you might still want to make some adjustments. + For example, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink> + to pass any needed configure options that are specific + to the recipe.</para></listitem> + <listitem><para><emphasis>CMake:</emphasis> + If your source files have a + <filename>CMakeLists.txt</filename> file, then your + software is built using CMake. + If this is the case, you just need to worry about + modifying the configuration.</para> + <para>When you use CMake, your recipe needs to inherit + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-cmake'><filename>cmake</filename></ulink> + class and your recipe does not have to contain a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task. + You can make some adjustments by setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECMAKE'><filename>EXTRA_OECMAKE</filename></ulink> + to pass any needed configure options that are specific + to the recipe.</para></listitem> + <listitem><para><emphasis>Other:</emphasis> + If your source files do not have a + <filename>configure.ac</filename> or + <filename>CMakeLists.txt</filename> file, then your + software is built using some method other than Autotools + or CMake. + If this is the case, you normally need to provide a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + task in your recipe + unless, of course, there is nothing to configure. + </para> + <para>Even if your software is not being built by + Autotools or CMake, you still might not need to deal + with any configuration issues. + You need to determine if configuration is even a required step. + You might need to modify a Makefile or some configuration file + used for the build to specify necessary build options. + Or, perhaps you might need to run a provided, custom + configure script with the appropriate options.</para> + <para>For the case involving a custom configure + script, you would run + <filename>./configure --help</filename> and look for + the options you need to set.</para></listitem> + </itemizedlist> + </para> + + <para> + Once configuration succeeds, it is always good practice to + look at the <filename>log.do_configure</filename> file to + ensure that the appropriate options have been enabled and no + additional build-time dependencies need to be added to + <filename>DEPENDS</filename>. + For example, if the configure script reports that it found + something not mentioned in <filename>DEPENDS</filename>, or + that it did not find something that it needed for some + desired optional functionality, then you would need to add + those to <filename>DEPENDS</filename>. + Looking at the log might also reveal items being checked for, + enabled, or both that you do not want, or items not being found + that are in <filename>DEPENDS</filename>, in which case + you would need to look at passing extra options to the + configure script as needed. + For reference information on configure options specific to the + software you are building, you can consult the output of the + <filename>./configure --help</filename> command within + <filename>${S}</filename> or consult the software's upstream + documentation. + </para> + </section> + + <section id='new-recipe-compilation'> + <title>Compilation</title> + + <para> + During a build, the <filename>do_compile</filename> task + happens after source is fetched, unpacked, and configured. + If the recipe passes through <filename>do_compile</filename> + successfully, nothing needs to be done. + </para> + + <para> + However, if the compile step fails, you need to diagnose the + failure. + Here are some common issues that cause failures. + <note> + For cases where improper paths are detected for + configuration files or for when libraries/headers cannot + be found, be sure you are using the more robust + <filename>pkg-config</filename>. + See the note in section + "<link linkend='new-recipe-configuring-the-recipe'>Configuring the Recipe</link>" + for additional information. + </note> + <itemizedlist> + <listitem><para><emphasis>Parallel build failures:</emphasis> + These failures manifest themselves as intermittent + errors, or errors reporting that a file or directory + that should be created by some other part of the build + process could not be found. + This type of failure can occur even if, upon inspection, + the file or directory does exist after the build has + failed, because that part of the build process happened + in the wrong order.</para> + <para>To fix the problem, you need to either satisfy + the missing dependency in the Makefile or whatever + script produced the Makefile, or (as a workaround) + set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + to an empty string: + <literallayout class='monospaced'> + PARALLEL_MAKE = "" + </literallayout></para> + <para> + For information on parallel Makefile issues, see the + "<link linkend='debugging-parallel-make-races'>Debugging Parallel Make Races</link>" + section. + </para></listitem> + <listitem><para><emphasis>Improper host path usage:</emphasis> + This failure applies to recipes building for the target + or <filename>nativesdk</filename> only. + The failure occurs when the compilation process uses + improper headers, libraries, or other files from the + host system when cross-compiling for the target. + </para> + <para>To fix the problem, examine the + <filename>log.do_compile</filename> file to identify + the host paths being used (e.g. + <filename>/usr/include</filename>, + <filename>/usr/lib</filename>, and so forth) and then + either add configure options, apply a patch, or do both. + </para></listitem> + <listitem><para><emphasis>Failure to find required + libraries/headers:</emphasis> + If a build-time dependency is missing because it has + not been declared in + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + or because the dependency exists but the path used by + the build process to find the file is incorrect and the + configure step did not detect it, the compilation + process could fail. + For either of these failures, the compilation process + notes that files could not be found. + In these cases, you need to go back and add additional + options to the configure script as well as possibly + add additional build-time dependencies to + <filename>DEPENDS</filename>.</para> + <para>Occasionally, it is necessary to apply a patch + to the source to ensure the correct paths are used. + If you need to specify paths to find files staged + into the sysroot from other recipes, use the variables + that the OpenEmbedded build system provides + (e.g. + <filename>STAGING_BINDIR</filename>, + <filename>STAGING_INCDIR</filename>, + <filename>STAGING_DATADIR</filename>, and so forth). +<!-- + (e.g. + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_BINDIR'><filename>STAGING_BINDIR</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_INCDIR'><filename>STAGING_INCDIR</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-STAGING_DATADIR'><filename>STAGING_DATADIR</filename></ulink>, + and so forth). +--> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-installing'> + <title>Installing</title> + + <para> + During <filename>do_install</filename>, the task copies the + built files along with their hierarchy to locations that + would mirror their locations on the target device. + The installation process copies files from the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink><filename>}</filename>, + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink><filename>}</filename>, + and + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename> + directories to the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename> + directory to create the structure as it should appear on the + target system. + </para> + + <para> + How your software is built affects what you must do to be + sure your software is installed correctly. + The following list describes what you must do for installation + depending on the type of build system used by the software + being built: + <itemizedlist> + <listitem><para><emphasis>Autotools and CMake:</emphasis> + If the software your recipe is building uses Autotools + or CMake, the OpenEmbedded build + system understands how to install the software. + Consequently, you do not have to have a + <filename>do_install</filename> task as part of your + recipe. + You just need to make sure the install portion of the + build completes with no issues. + However, if you wish to install additional files not + already being installed by + <filename>make install</filename>, you should do this + using a <filename>do_install_append</filename> function + using the install command as described in + the "Manual" bulleted item later in this list. + </para></listitem> + <listitem><para><emphasis>Other (using + <filename>make install</filename>):</emphasis> + You need to define a + <filename>do_install</filename> function in your + recipe. + The function should call + <filename>oe_runmake install</filename> and will likely + need to pass in the destination directory as well. + How you pass that path is dependent on how the + <filename>Makefile</filename> being run is written + (e.g. <filename>DESTDIR=${D}</filename>, + <filename>PREFIX=${D}</filename>, + <filename>INSTALLROOT=${D}</filename>, and so forth). + </para> + <para>For an example recipe using + <filename>make install</filename>, see the + "<link linkend='new-recipe-makefile-based-package'>Makefile-Based Package</link>" + section.</para></listitem> + <listitem><para><emphasis>Manual:</emphasis> + You need to define a + <filename>do_install</filename> function in your + recipe. + The function must first use + <filename>install -d</filename> to create the + directories under + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink><filename>}</filename>. + Once the directories exist, your function can use + <filename>install</filename> to manually install the + built software into the directories.</para> + <para>You can find more information on + <filename>install</filename> at + <ulink url='http://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html'></ulink>. + </para></listitem> + </itemizedlist> + </para> + + <para> + For the scenarios that do not use Autotools or + CMake, you need to track the installation + and diagnose and fix any issues until everything installs + correctly. + You need to look in the default location of + <filename>${D}</filename>, which is + <filename>${WORKDIR}/image</filename>, to be sure your + files have been installed correctly. + </para> + + <note><title>Notes</title> + <itemizedlist> + <listitem><para> + During the installation process, you might need to + modify some of the installed files to suit the target + layout. + For example, you might need to replace hard-coded paths + in an initscript with values of variables provided by + the build system, such as replacing + <filename>/usr/bin/</filename> with + <filename>${bindir}</filename>. + If you do perform such modifications during + <filename>do_install</filename>, be sure to modify the + destination file after copying rather than before + copying. + Modifying after copying ensures that the build system + can re-execute <filename>do_install</filename> if + needed. + </para></listitem> + <listitem><para> + <filename>oe_runmake install</filename>, which can be + run directly or can be run indirectly by the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-cmake'><filename>cmake</filename></ulink> + classes, runs <filename>make install</filename> in + parallel. + Sometimes, a Makefile can have missing dependencies + between targets that can result in race conditions. + If you experience intermittent failures during + <filename>do_install</filename>, you might be able to + work around them by disabling parallel Makefile + installs by adding the following to the recipe: + <literallayout class='monospaced'> + PARALLEL_MAKEINST = "" + </literallayout> + See + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> + for additional information. + </para></listitem> + </itemizedlist> + </note> + </section> + + <section id='new-recipe-enabling-system-services'> + <title>Enabling System Services</title> + + <para> + If you want to install a service, which is a process that + usually starts on boot and runs in the background, then + you must include some additional definitions in your recipe. + </para> + + <para> + If you are adding services and the service initialization + script or the service file itself is not installed, you must + provide for that installation in your recipe using a + <filename>do_install_append</filename> function. + If your recipe already has a <filename>do_install</filename> + function, update the function near its end rather than + adding an additional <filename>do_install_append</filename> + function. + </para> + + <para> + When you create the installation for your services, you need + to accomplish what is normally done by + <filename>make install</filename>. + In other words, make sure your installation arranges the output + similar to how it is arranged on the target system. + </para> + + <para> + The OpenEmbedded build system provides support for starting + services two different ways: + <itemizedlist> + <listitem><para><emphasis>SysVinit:</emphasis> + SysVinit is a system and service manager that + manages the init system used to control the very basic + functions of your system. + The init program is the first program + started by the Linux kernel when the system boots. + Init then controls the startup, running and shutdown + of all other programs.</para> + <para>To enable a service using SysVinit, your recipe + needs to inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-update-rc.d'><filename>update-rc.d</filename></ulink> + class. + The class helps facilitate safely installing the + package on the target.</para> + <para>You will need to set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_PACKAGES'><filename>INITSCRIPT_PACKAGES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_NAME'><filename>INITSCRIPT_NAME</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-INITSCRIPT_PARAMS'><filename>INITSCRIPT_PARAMS</filename></ulink> + variables within your recipe.</para></listitem> + <listitem><para><emphasis>systemd:</emphasis> + System Management Daemon (systemd) was designed to + replace SysVinit and to provide + enhanced management of services. + For more information on systemd, see the systemd + homepage at + <ulink url='http://freedesktop.org/wiki/Software/systemd/'></ulink>. + </para> + <para>To enable a service using systemd, your recipe + needs to inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-systemd'><filename>systemd</filename></ulink> + class. + See the <filename>systemd.bbclass</filename> file + located in your + <link linkend='source-directory'>Source Directory</link>. + section for more information. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='new-recipe-packaging'> + <title>Packaging</title> + + <para> + Successful packaging is a combination of automated processes + performed by the OpenEmbedded build system and some + specific steps you need to take. + The following list describes the process: + <itemizedlist> + <listitem><para><emphasis>Splitting Files</emphasis>: + The <filename>do_package</filename> task splits the + files produced by the recipe into logical components. + Even software that produces a single binary might + still have debug symbols, documentation, and other + logical components that should be split out. + The <filename>do_package</filename> task ensures + that files are split up and packaged correctly. + </para></listitem> + <listitem><para><emphasis>Running QA Checks</emphasis>: + The + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> + class adds a step to + the package generation process so that output quality + assurance checks are generated by the OpenEmbedded + build system. + This step performs a range of checks to be sure the + build's output is free of common problems that show + up during runtime. + For information on these checks, see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink> + class and the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-qa-checks'>QA Error and Warning Messages</ulink>" + chapter in the Yocto Project Reference Manual. + </para></listitem> + <listitem><para><emphasis>Hand-Checking Your Packages</emphasis>: + After you build your software, you need to be sure + your packages are correct. + Examine the + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}/packages-split</filename> + directory and make sure files are where you expect + them to be. + If you discover problems, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>, + <filename>do_install(_append)</filename>, and so forth as + needed. + </para></listitem> + <listitem><para><emphasis>Splitting an Application into Multiple Packages</emphasis>: + If you need to split an application into several + packages, see the + "<link linkend='splitting-an-application-into-multiple-packages'>Splitting an Application into Multiple Packages</link>" + section for an example. + </para></listitem> + <listitem><para><emphasis>Installing a Post-Installation Script</emphasis>: + For an example showing how to install a + post-installation script, see the + "<link linkend='new-recipe-post-installation-scripts'>Post-Installation Scripts</link>" + section. + </para></listitem> + <listitem><para><emphasis>Marking Package Architecture</emphasis>: + Depending on what your recipe is building and how it + is configured, it might be important to mark the + packages produced as being specific to a particular + machine, or to mark them as not being specific to + a particular machine or architecture at all.</para> + <para>By default, packages apply to any machine with the + same architecture as the target machine. + When a recipe produces packages that are + machine-specific (e.g. the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + value is passed into the configure script or a patch + is applied only for a particular machine), you should + mark them as such by adding the following to the + recipe: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${MACHINE_ARCH}" + </literallayout></para> + <para>On the other hand, if the recipe produces packages + that do not contain anything specific to the target + machine or architecture at all (e.g. recipes + that simply package script files or configuration + files), you should use the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> + class to do this for you by adding this to your + recipe: + <literallayout class='monospaced'> + inherit allarch + </literallayout> + Ensuring that the package architecture is correct is + not critical while you are doing the first few builds + of your recipe. + However, it is important in order + to ensure that your recipe rebuilds (or does not + rebuild) appropriately in response to changes in + configuration, and to ensure that you get the + appropriate packages installed on the target machine, + particularly if you run separate builds for more + than one target machine. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='properly-versioning-pre-release-recipes'> + <title>Properly Versioning Pre-Release Recipes</title> + + <para> + Sometimes the name of a recipe can lead to versioning + problems when the recipe is upgraded to a final release. + For example, consider the + <filename>irssi_0.8.16-rc1.bb</filename> recipe file in + the list of example recipes in the + "<link linkend='new-recipe-storing-and-naming-the-recipe'>Storing and Naming the Recipe</link>" + section. + This recipe is at a release candidate stage (i.e. + "rc1"). + When the recipe is released, the recipe filename becomes + <filename>irssi_0.8.16.bb</filename>. + The version change from <filename>0.8.16-rc1</filename> + to <filename>0.8.16</filename> is seen as a decrease by the + build system and package managers, so the resulting packages + will not correctly trigger an upgrade. + </para> + + <para> + In order to ensure the versions compare properly, the + recommended convention is to set + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + within the recipe to + "<replaceable>previous_version</replaceable>+<replaceable>current_version</replaceable>". + You can use an additional variable so that you can use the + current version elsewhere. + Here is an example: + <literallayout class='monospaced'> + REALPV = "0.8.16-rc1" + PV = "0.8.15+${REALPV}" + </literallayout> + </para> + </section> + + <section id='new-recipe-post-installation-scripts'> + <title>Post-Installation Scripts</title> + + <para> + Post-installation scripts run immediately after installing + a package on the target or during image creation when a + package is included in an image. + To add a post-installation script to a package, add a + <filename>pkg_postinst_PACKAGENAME()</filename> function to + the recipe file (<filename>.bb</filename>) and replace + <filename>PACKAGENAME</filename> with the name of the package + you want to attach to the <filename>postinst</filename> + script. + To apply the post-installation script to the main package + for the recipe, which is usually what is required, specify + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename> + in place of <filename>PACKAGENAME</filename>. + </para> + + <para> + A post-installation function has the following structure: + <literallayout class='monospaced'> + pkg_postinst_PACKAGENAME() { + # Commands to carry out + } + </literallayout> + </para> + + <para> + The script defined in the post-installation function is + called when the root filesystem is created. + If the script succeeds, the package is marked as installed. + If the script fails, the package is marked as unpacked and + the script is executed when the image boots again. + </para> + + <para> + Sometimes it is necessary for the execution of a + post-installation script to be delayed until the first boot. + For example, the script might need to be executed on the + device itself. + To delay script execution until boot time, use the following + structure in the post-installation script: + <literallayout class='monospaced'> + pkg_postinst_PACKAGENAME() { + if [ x"$D" = "x" ]; then + # Actions to carry out on the device go here + else + exit 1 + fi + } + </literallayout> + </para> + + <para> + The previous example delays execution until the image boots + again because the environment variable <filename>D</filename> + points to the directory containing the image when + the root filesystem is created at build time but is unset + when executed on the first boot. + </para> + + <note> + Equivalent support for pre-install, pre-uninstall, and + post-uninstall scripts exist by way of + <filename>pkg_preinst</filename>, + <filename>pkg_prerm</filename>, and + <filename>pkg_postrm</filename>, respectively. + These scrips work in exactly the same way as does + <filename>pkg_postinst</filename> with the exception that they + run at different times. + Also, because of when they run, they are not applicable to + being run at image creation time like + <filename>pkg_postinst</filename>. + </note> + </section> + + <section id='new-recipe-testing'> + <title>Testing</title> + + <para> + The final step for completing your recipe is to be sure that + the software you built runs correctly. + To accomplish runtime testing, add the build's output + packages to your image and test them on the target. + </para> + + <para> + For information on how to customize your image by adding + specific packages, see the + "<link linkend='usingpoky-extend-customimage'>Customizing Images</link>" + section. + </para> + </section> + + <section id='new-recipe-testing-examples'> + <title>Examples</title> + + <para> + To help summarize how to write a recipe, this section provides + some examples given various scenarios: + <itemizedlist> + <listitem><para>Recipes that use local files</para></listitem> + <listitem><para>Using an Autotooled package</para></listitem> + <listitem><para>Using a Makefile-based package</para></listitem> + <listitem><para>Splitting an application into multiple packages</para></listitem> + <listitem><para>Adding binaries to an image</para></listitem> + </itemizedlist> + </para> + + <section id='new-recipe-single-c-file-package-hello-world'> + <title>Single .c File Package (Hello World!)</title> + + <para> + Building an application from a single file that is stored + locally (e.g. under <filename>files</filename>) requires + a recipe that has the file listed in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename> + variable. + Additionally, you need to manually write the + <filename>do_compile</filename> and + <filename>do_install</filename> tasks. + The <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'>S</ulink></filename> + variable defines the directory containing the source code, + which is set to + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> + in this case - the directory BitBake uses for the build. + <literallayout class='monospaced'> + SUMMARY = "Simple helloworld application" + SECTION = "examples" + LICENSE = "MIT" + LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" + + SRC_URI = "file://helloworld.c" + + S = "${WORKDIR}" + + do_compile() { + ${CC} helloworld.c -o helloworld + } + + do_install() { + install -d ${D}${bindir} + install -m 0755 helloworld ${D}${bindir} + } + </literallayout> + </para> + + <para> + By default, the <filename>helloworld</filename>, + <filename>helloworld-dbg</filename>, and + <filename>helloworld-dev</filename> packages are built. + For information on how to customize the packaging process, + see the + "<link linkend='splitting-an-application-into-multiple-packages'>Splitting an Application into Multiple Packages</link>" + section. + </para> + </section> + + <section id='new-recipe-autotooled-package'> + <title>Autotooled Package</title> + <para> + Applications that use Autotools such as <filename>autoconf</filename> and + <filename>automake</filename> require a recipe that has a source archive listed in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename> and + also inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink> + class, which contains the definitions of all the steps + needed to build an Autotool-based application. + The result of the build is automatically packaged. + And, if the application uses NLS for localization, packages with local information are + generated (one package per language). + Following is one example: (<filename>hello_2.3.bb</filename>) + <literallayout class='monospaced'> + SUMMARY = "GNU Helloworld application" + SECTION = "examples" + LICENSE = "GPLv2+" + LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe" + + SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz" + + inherit autotools gettext + </literallayout> + </para> + + <para> + The variable + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LIC_FILES_CHKSUM'>LIC_FILES_CHKSUM</ulink></filename> + is used to track source license changes as described in the + "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-configuring-LIC_FILES_CHKSUM'>Tracking License Changes</ulink>" section. + You can quickly create Autotool-based recipes in a manner similar to the previous example. + </para> + </section> + + <section id='new-recipe-makefile-based-package'> + <title>Makefile-Based Package</title> + + <para> + Applications that use GNU <filename>make</filename> also require a recipe that has + the source archive listed in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename>. + You do not need to add a <filename>do_compile</filename> step since by default BitBake + starts the <filename>make</filename> command to compile the application. + If you need additional <filename>make</filename> options, you should store them in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OEMAKE'>EXTRA_OEMAKE</ulink></filename> + variable. + BitBake passes these options into the GNU <filename>make</filename> invocation. + Note that a <filename>do_install</filename> task is still required. + Otherwise, BitBake runs an empty <filename>do_install</filename> task by default. + </para> + + <para> + Some applications might require extra parameters to be passed to the compiler. + For example, the application might need an additional header path. + You can accomplish this by adding to the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-CFLAGS'>CFLAGS</ulink></filename> variable. + The following example shows this: + <literallayout class='monospaced'> + CFLAGS_prepend = "-I ${S}/include " + </literallayout> + </para> + + <para> + In the following example, <filename>mtd-utils</filename> is a makefile-based package: + <literallayout class='monospaced'> + SUMMARY = "Tools for managing memory technology devices" + SECTION = "base" + DEPENDS = "zlib lzo e2fsprogs util-linux" + HOMEPAGE = "http://www.linux-mtd.infradead.org/" + LICENSE = "GPLv2+" + LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \ + file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c" + + # Use the latest version at 26 Oct, 2013 + SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b" + SRC_URI = "git://git.infradead.org/mtd-utils.git \ + file://add-exclusion-to-mkfs-jffs2-git-2.patch \ + " + + PV = "1.5.1+git${SRCPV}" + + S = "${WORKDIR}/git/" + + EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'" + + do_install () { + oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir} + } + + PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc" + + FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool" + FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*" + FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image" + + PARALLEL_MAKE = "" + + BBCLASSEXTEND = "native" + </literallayout> + </para> + </section> + + <section id='splitting-an-application-into-multiple-packages'> + <title>Splitting an Application into Multiple Packages</title> + + <para> + You can use the variables + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'>PACKAGES</ulink></filename> and + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'>FILES</ulink></filename> + to split an application into multiple packages. + </para> + + <para> + Following is an example that uses the <filename>libxpm</filename> recipe. + By default, this recipe generates a single package that contains the library along + with a few binaries. + You can modify the recipe to split the binaries into separate packages: + <literallayout class='monospaced'> + require xorg-lib-common.inc + + SUMMARY = "Xpm: X Pixmap extension library" + LICENSE = "BSD" + LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7" + DEPENDS += "libxext libsm libxt" + PE = "1" + + XORG_PN = "libXpm" + + PACKAGES =+ "sxpm cxpm" + FILES_cxpm = "${bindir}/cxpm" + FILES_sxpm = "${bindir}/sxpm" + </literallayout> + </para> + + <para> + In the previous example, we want to ship the <filename>sxpm</filename> + and <filename>cxpm</filename> binaries in separate packages. + Since <filename>bindir</filename> would be packaged into the main + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'>PN</ulink></filename> + package by default, we prepend the <filename>PACKAGES</filename> + variable so additional package names are added to the start of list. + This results in the extra <filename>FILES_*</filename> + variables then containing information that define which files and + directories go into which packages. + Files included by earlier packages are skipped by latter packages. + Thus, the main <filename>PN</filename> package + does not include the above listed files. + </para> + </section> + + <section id='packaging-externally-produced-binaries'> + <title>Packaging Externally Produced Binaries</title> + + <para> + Sometimes, you need to add pre-compiled binaries to an + image. + For example, suppose that binaries for proprietary code + exist, which are created by a particular division of a + company. + Your part of the company needs to use those binaries as + part of an image that you are building using the + OpenEmbedded build system. + Since you only have the binaries and not the source code, + you cannot use a typical recipe that expects to fetch the + source specified in + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + and then compile it. + </para> + + <para> + One method is to package the binaries and then install them + as part of the image. + Generally, it is not a good idea to package binaries + since, among other things, it can hinder the ability to + reproduce builds and could lead to compatibility problems + with ABI in the future. + However, sometimes you have no choice. + </para> + + <para> + The easiest solution is to create a recipe that uses + the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-bin-package'><filename>bin_package</filename></ulink> + class and to be sure that you are using default locations + for build artifacts. + In most cases, the <filename>bin_package</filename> class + handles "skipping" the configure and compile steps as well + as sets things up to grab packages from the appropriate + area. + In particular, this class sets <filename>noexec</filename> + on both the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink> + tasks, sets + <filename>FILES_${PN}</filename> to "/" so that it picks + up all files, and sets up a + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> + task, which effectively copies all files from + <filename>${S}</filename> to <filename>${D}</filename>. + The <filename>bin_package</filename> class works well when + the files extracted into <filename>${S}</filename> are + already laid out in the way they should be laid out + on the target. + For more information on these variables, see the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>, + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> + variables in the Yocto Project Reference Manual's variable + glossary. + </para> + + <para> + If you can't use the <filename>bin_package</filename> + class, you need to be sure you are doing the following: + <itemizedlist> + <listitem><para>Create a recipe where the + <filename>do_configure</filename> and + <filename>do_compile</filename> tasks do nothing: + <literallayout class='monospaced'> + do_configure[noexec] = "1" + do_compile[noexec] = "1" + </literallayout> + Alternatively, you can make these tasks an empty + function. + </para></listitem> + <listitem><para>Make sure your + <filename>do_install</filename> task installs the + binaries appropriately. + </para></listitem> + <listitem><para>Ensure that you set up + <filename>FILES</filename> (usually + <filename>FILES_${PN}</filename>) to point to the + files you have installed, which of course depends + on where you have installed them and whether + those files are in different locations than the + defaults. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + </section> + + <section id="platdev-newmachine"> + <title>Adding a New Machine</title> + + <para> + Adding a new machine to the Yocto Project is a straightforward + process. + This section describes how to add machines that are similar + to those that the Yocto Project already supports. + <note> + Although well within the capabilities of the Yocto Project, + adding a totally new architecture might require + changes to <filename>gcc/glibc</filename> and to the site + information, which is beyond the scope of this manual. + </note> + </para> + + <para> + For a complete example that shows how to add a new machine, + see the + "<ulink url='&YOCTO_DOCS_BSP_URL;#creating-a-new-bsp-layer-using-the-yocto-bsp-script'>Creating a New BSP Layer Using the yocto-bsp Script</ulink>" + section in the Yocto Project Board Support Package (BSP) Developer's Guide. + </para> + + <section id="platdev-newmachine-conffile"> + <title>Adding the Machine Configuration File</title> + + <para> + To add a new machine, you need to add a new machine + configuration file to the layer's + <filename>conf/machine</filename> directory. + This configuration file provides details about the device + you are adding. + </para> + + <para> + The OpenEmbedded build system uses the root name of the + machine configuration file to reference the new machine. + For example, given a machine configuration file named + <filename>crownbay.conf</filename>, the build system + recognizes the machine as "crownbay". + </para> + + <para> + The most important variables you must set in your machine + configuration file or include from a lower-level configuration + file are as follows: + <itemizedlist> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_ARCH'>TARGET_ARCH</ulink></filename> + (e.g. "arm")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</ulink>_virtual/kernel</filename> + </para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'>MACHINE_FEATURES</ulink></filename> + (e.g. "apm screen wifi")</para></listitem> + </itemizedlist> + </para> + + <para> + You might also need these variables: + <itemizedlist> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLES'>SERIAL_CONSOLES</ulink></filename> + (e.g. "115200;ttyS0 115200;ttyS1")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'>KERNEL_IMAGETYPE</ulink></filename> + (e.g. "zImage")</para></listitem> + <listitem><para><filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'>IMAGE_FSTYPES</ulink></filename> + (e.g. "tar.gz jffs2")</para></listitem> + </itemizedlist> + </para> + + <para> + You can find full details on these variables in the reference + section. + You can leverage existing machine <filename>.conf</filename> + files from <filename>meta-yocto-bsp/conf/machine/</filename>. + </para> + </section> + + <section id="platdev-newmachine-kernel"> + <title>Adding a Kernel for the Machine</title> + + <para> + The OpenEmbedded build system needs to be able to build a kernel + for the machine. + You need to either create a new kernel recipe for this machine, + or extend an existing kernel recipe. + You can find several kernel recipe examples in the + Source Directory at + <filename>meta/recipes-kernel/linux</filename> + that you can use as references. + </para> + + <para> + If you are creating a new kernel recipe, normal recipe-writing + rules apply for setting up a + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'>SRC_URI</ulink></filename>. + Thus, you need to specify any necessary patches and set + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-S'>S</ulink></filename> + to point at the source code. + You need to create a <filename>do_configure</filename> task that + configures the unpacked kernel with a + <filename>defconfig</filename> file. + You can do this by using a <filename>make defconfig</filename> + command or, more commonly, by copying in a suitable + <filename>defconfig</filename> file and then running + <filename>make oldconfig</filename>. + By making use of <filename>inherit kernel</filename> and + potentially some of the <filename>linux-*.inc</filename> files, + most other functionality is centralized and the defaults of the + class normally work well. + </para> + + <para> + If you are extending an existing kernel recipe, it is usually + a matter of adding a suitable <filename>defconfig</filename> + file. + The file needs to be added into a location similar to + <filename>defconfig</filename> files used for other machines + in a given kernel recipe. + A possible way to do this is by listing the file in the + <filename>SRC_URI</filename> and adding the machine to the + expression in + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-COMPATIBLE_MACHINE'>COMPATIBLE_MACHINE</ulink></filename>: + <literallayout class='monospaced'> + COMPATIBLE_MACHINE = '(qemux86|qemumips)' + </literallayout> + For more information on <filename>defconfig</filename> files, + see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>" + section in the Yocto Project Linux Kernel Development Manual. + </para> + </section> + + <section id="platdev-newmachine-formfactor"> + <title>Adding a Formfactor Configuration File</title> + + <para> + A formfactor configuration file provides information about the + target hardware for which the image is being built and information that + the build system cannot obtain from other sources such as the kernel. + Some examples of information contained in a formfactor configuration file include + framebuffer orientation, whether or not the system has a keyboard, + the positioning of the keyboard in relation to the screen, and + the screen resolution. + </para> + + <para> + The build system uses reasonable defaults in most cases. + However, if customization is + necessary, you need to create a <filename>machconfig</filename> file + in the <filename>meta/recipes-bsp/formfactor/files</filename> + directory. + This directory contains directories for specific machines such as + <filename>qemuarm</filename> and <filename>qemux86</filename>. + For information about the settings available and the defaults, see the + <filename>meta/recipes-bsp/formfactor/files/config</filename> file found in the + same area. + </para> + + <para> + Following is an example for "qemuarm" machine: + <literallayout class='monospaced'> + HAVE_TOUCHSCREEN=1 + HAVE_KEYBOARD=1 + + DISPLAY_CAN_ROTATE=0 + DISPLAY_ORIENTATION=0 + #DISPLAY_WIDTH_PIXELS=640 + #DISPLAY_HEIGHT_PIXELS=480 + #DISPLAY_BPP=16 + DISPLAY_DPI=150 + DISPLAY_SUBPIXEL_ORDER=vrgb + </literallayout> + </para> + </section> + </section> + + <section id="platdev-working-with-libraries"> + <title>Working With Libraries</title> + + <para> + Libraries are an integral part of your system. + This section describes some common practices you might find + helpful when working with libraries to build your system: + <itemizedlist> + <listitem><para><link linkend='including-static-library-files'>How to include static library files</link> + </para></listitem> + <listitem><para><link linkend='combining-multiple-versions-library-files-into-one-image'>How to use the Multilib feature to combine multiple versions of library files into a single image</link> + </para></listitem> + <listitem><para><link linkend='installing-multiple-versions-of-the-same-library'>How to install multiple versions of the same library in parallel on the same system</link> + </para></listitem> + </itemizedlist> + </para> + + <section id='including-static-library-files'> + <title>Including Static Library Files</title> + + <para> + If you are building a library and the library offers static linking, you can control + which static library files (<filename>*.a</filename> files) get included in the + built library. + </para> + + <para> + The <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES_*</filename></ulink> + variables in the + <filename>meta/conf/bitbake.conf</filename> configuration file define how files installed + by the <filename>do_install</filename> task are packaged. + By default, the <filename>PACKAGES</filename> variable includes + <filename>${PN}-staticdev</filename>, which represents all static library files. + <note> + Some previously released versions of the Yocto Project + defined the static library files through + <filename>${PN}-dev</filename>. + </note> + Following is part of the BitBake configuration file, where + you can see how the static library files are defined: + <literallayout class='monospaced'> + PACKAGE_BEFORE_PN ?= "" + PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}" + PACKAGES_DYNAMIC = "^${PN}-locale-.*" + FILES = "" + + FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \ + ${sysconfdir} ${sharedstatedir} ${localstatedir} \ + ${base_bindir}/* ${base_sbindir}/* \ + ${base_libdir}/*${SOLIBS} \ + ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \ + ${datadir}/${BPN} ${libdir}/${BPN}/* \ + ${datadir}/pixmaps ${datadir}/applications \ + ${datadir}/idl ${datadir}/omf ${datadir}/sounds \ + ${libdir}/bonobo/servers" + + FILES_${PN}-bin = "${bindir}/* ${sbindir}/*" + + FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \ + ${datadir}/gnome/help" + SECTION_${PN}-doc = "doc" + + FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}" + FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \ + ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \ + ${datadir}/aclocal ${base_libdir}/*.o \ + ${libdir}/${BPN}/*.la ${base_libdir}/*.la" + SECTION_${PN}-dev = "devel" + ALLOW_EMPTY_${PN}-dev = "1" + RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})" + + FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a" + SECTION_${PN}-staticdev = "devel" + RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})" + </literallayout> + </para> + </section> + + <section id="combining-multiple-versions-library-files-into-one-image"> + <title>Combining Multiple Versions of Library Files into One Image</title> + + <para> + The build system offers the ability to build libraries with different + target optimizations or architecture formats and combine these together + into one system image. + You can link different binaries in the image + against the different libraries as needed for specific use cases. + This feature is called "Multilib." + </para> + + <para> + An example would be where you have most of a system compiled in 32-bit + mode using 32-bit libraries, but you have something large, like a database + engine, that needs to be a 64-bit application and uses 64-bit libraries. + Multilib allows you to get the best of both 32-bit and 64-bit libraries. + </para> + + <para> + While the Multilib feature is most commonly used for 32 and 64-bit differences, + the approach the build system uses facilitates different target optimizations. + You could compile some binaries to use one set of libraries and other binaries + to use a different set of libraries. + The libraries could differ in architecture, compiler options, or other + optimizations. + </para> + + <para> + Several examples exist in the + <filename>meta-skeleton</filename> layer found in the + <link linkend='source-directory'>Source Directory</link>: + <itemizedlist> + <listitem><para><filename>conf/multilib-example.conf</filename> + configuration file</para></listitem> + <listitem><para><filename>conf/multilib-example2.conf</filename> + configuration file</para></listitem> + <listitem><para><filename>recipes-multilib/images/core-image-multilib-example.bb</filename> + recipe</para></listitem> + </itemizedlist> + </para> + + <section id='preparing-to-use-multilib'> + <title>Preparing to Use Multilib</title> + + <para> + User-specific requirements drive the Multilib feature. + Consequently, there is no one "out-of-the-box" configuration that likely + exists to meet your needs. + </para> + + <para> + In order to enable Multilib, you first need to ensure your recipe is + extended to support multiple libraries. + Many standard recipes are already extended and support multiple libraries. + You can check in the <filename>meta/conf/multilib.conf</filename> + configuration file in the + <link linkend='source-directory'>Source Directory</link> to see how this is + done using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBCLASSEXTEND'><filename>BBCLASSEXTEND</filename></ulink> + variable. + Eventually, all recipes will be covered and this list will + not be needed. + </para> + + <para> + For the most part, the Multilib class extension works automatically to + extend the package name from <filename>${PN}</filename> to + <filename>${MLPREFIX}${PN}</filename>, where <filename>MLPREFIX</filename> + is the particular multilib (e.g. "lib32-" or "lib64-"). + Standard variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RPROVIDES'><filename>RPROVIDES</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES_DYNAMIC'><filename>PACKAGES_DYNAMIC</filename></ulink> + are automatically extended by the system. + If you are extending any manual code in the recipe, you can use the + <filename>${MLPREFIX}</filename> variable to ensure those names are extended + correctly. + This automatic extension code resides in <filename>multilib.bbclass</filename>. + </para> + </section> + + <section id='using-multilib'> + <title>Using Multilib</title> + + <para> + After you have set up the recipes, you need to define the actual + combination of multiple libraries you want to build. + You accomplish this through your <filename>local.conf</filename> + configuration file in the + <link linkend='build-directory'>Build Directory</link>. + An example configuration would be as follows: + <literallayout class='monospaced'> + MACHINE = "qemux86-64" + require conf/multilib.conf + MULTILIBS = "multilib:lib32" + DEFAULTTUNE_virtclass-multilib-lib32 = "x86" + IMAGE_INSTALL_append = " lib32-glib-2.0" + </literallayout> + This example enables an + additional library named <filename>lib32</filename> alongside the + normal target packages. + When combining these "lib32" alternatives, the example uses "x86" for tuning. + For information on this particular tuning, see + <filename>meta/conf/machine/include/ia32/arch-ia32.inc</filename>. + </para> + + <para> + The example then includes <filename>lib32-glib-2.0</filename> + in all the images, which illustrates one method of including a + multiple library dependency. + You can use a normal image build to include this dependency, + for example: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + You can also build Multilib packages specifically with a command like this: + <literallayout class='monospaced'> + $ bitbake lib32-glib-2.0 + </literallayout> + </para> + </section> + + <section id='additional-implementation-details'> + <title>Additional Implementation Details</title> + + <para> + Generic implementation details as well as details that are + specific to package management systems exist. + Following are implementation details that exist regardless + of the package management system: + <itemizedlist> + <listitem><para>The typical convention used for the + class extension code as used by + Multilib assumes that all package names specified + in + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + that contain <filename>${PN}</filename> have + <filename>${PN}</filename> at the start of the name. + When that convention is not followed and + <filename>${PN}</filename> appears at + the middle or the end of a name, problems occur. + </para></listitem> + <listitem><para>The + <ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_VENDOR'><filename>TARGET_VENDOR</filename></ulink> + value under Multilib will be extended to + "-<replaceable>vendor</replaceable>ml<replaceable>multilib</replaceable>" + (e.g. "-pokymllib32" for a "lib32" Multilib with + Poky). + The reason for this slightly unwieldy contraction + is that any "-" characters in the vendor + string presently break Autoconf's + <filename>config.sub</filename>, and + other separators are problematic for different + reasons. + </para></listitem> + </itemizedlist> + </para> +' + <para> + For the RPM Package Management System, the following implementation details + exist: + <itemizedlist> + <listitem><para>A unique architecture is defined for the Multilib packages, + along with creating a unique deploy folder under + <filename>tmp/deploy/rpm</filename> in the + <link linkend='build-directory'>Build Directory</link>. + For example, consider <filename>lib32</filename> in a + <filename>qemux86-64</filename> image. + The possible architectures in the system are "all", "qemux86_64", + "lib32_qemux86_64", and "lib32_x86".</para></listitem> + <listitem><para>The <filename>${MLPREFIX}</filename> variable is stripped from + <filename>${PN}</filename> during RPM packaging. + The naming for a normal RPM package and a Multilib RPM package in a + <filename>qemux86-64</filename> system resolves to something similar to + <filename>bash-4.1-r2.x86_64.rpm</filename> and + <filename>bash-4.1.r2.lib32_x86.rpm</filename>, respectively. + </para></listitem> + <listitem><para>When installing a Multilib image, the RPM backend first + installs the base image and then installs the Multilib libraries. + </para></listitem> + <listitem><para>The build system relies on RPM to resolve the identical files in the + two (or more) Multilib packages.</para></listitem> + </itemizedlist> + </para> + + <para> + For the IPK Package Management System, the following implementation details exist: + <itemizedlist> + <listitem><para>The <filename>${MLPREFIX}</filename> is not stripped from + <filename>${PN}</filename> during IPK packaging. + The naming for a normal RPM package and a Multilib IPK package in a + <filename>qemux86-64</filename> system resolves to something like + <filename>bash_4.1-r2.x86_64.ipk</filename> and + <filename>lib32-bash_4.1-rw_x86.ipk</filename>, respectively. + </para></listitem> + <listitem><para>The IPK deploy folder is not modified with + <filename>${MLPREFIX}</filename> because packages with and without + the Multilib feature can exist in the same folder due to the + <filename>${PN}</filename> differences.</para></listitem> + <listitem><para>IPK defines a sanity check for Multilib installation + using certain rules for file comparison, overridden, etc. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + + <section id='installing-multiple-versions-of-the-same-library'> + <title>Installing Multiple Versions of the Same Library</title> + + <para> + Situations can exist where you need to install and use + multiple versions of the same library on the same system + at the same time. + These situations almost always exist when a library API + changes and you have multiple pieces of software that + depend on the separate versions of the library. + To accommodate these situations, you can install multiple + versions of the same library in parallel on the same system. + </para> + + <para> + The process is straightforward as long as the libraries use + proper versioning. + With properly versioned libraries, all you need to do to + individually specify the libraries is create separate, + appropriately named recipes where the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> part of the + name includes a portion that differentiates each library version + (e.g.the major part of the version number). + Thus, instead of having a single recipe that loads one version + of a library (e.g. <filename>clutter</filename>), you provide + multiple recipes that result in different versions + of the libraries you want. + As an example, the following two recipes would allow the + two separate versions of the <filename>clutter</filename> + library to co-exist on the same system: + <literallayout class='monospaced'> + clutter-1.6_1.6.20.bb + clutter-1.8_1.8.4.bb + </literallayout> + Additionally, if you have other recipes that depend on a given + library, you need to use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + variable to create the dependency. + Continuing with the same example, if you want to have a recipe + depend on the 1.8 version of the <filename>clutter</filename> + library, use the following in your recipe: + <literallayout class='monospaced'> + DEPENDS = "clutter-1.8" + </literallayout> + </para> + </section> + </section> + + <section id='dev-optionally-using-an-external-toolchain'> + <title>Optionally Using an External Toolchain</title> + + <para> + You might want to use an external toolchain as part of your + development. + If this is the case, the fundamental steps you need to accomplish + are as follows: + <itemizedlist> + <listitem><para> + Understand where the installed toolchain resides. + For cases where you need to build the external toolchain, + you would need to take separate steps to build and install + the toolchain. + </para></listitem> + <listitem><para> + Make sure you add the layer that contains the toolchain to + your <filename>bblayers.conf</filename> file through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBLAYERS'><filename>BBLAYERS</filename></ulink> + variable. + </para></listitem> + <listitem><para> + Set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNAL_TOOLCHAIN'><filename>EXTERNAL_TOOLCHAIN</filename></ulink> + variable in your <filename>local.conf</filename> file + to the location in which you installed the toolchain. + </para></listitem> + </itemizedlist> + A good example of an external toolchain used with the Yocto Project + is <trademark class='registered'>Mentor Graphics</trademark> + Sourcery G++ Toolchain. + You can see information on how to use that particular layer in the + <filename>README</filename> file at + <ulink url='http://github.com/MentorEmbedded/meta-sourcery/'></ulink>. + You can find further information by reading about the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TCMODE'><filename>TCMODE</filename></ulink> + variable in the Yocto Project Reference Manual's variable glossary. + </para> + </section> + + <section id='creating-partitioned-images'> + <title>Creating Partitioned Images</title> + + <para> + Creating an image for a particular hardware target using the + OpenEmbedded build system does not necessarily mean you can boot + that image as is on your device. + Physical devices accept and boot images in various ways depending + on the specifics of the device. + Usually, information about the hardware can tell you what image + format the device requires. + Should your device require multiple partitions on an SD card, flash, + or an HDD, you can use the OpenEmbedded Image Creator, + <filename>wic</filename>, to create the properly partitioned image. + </para> + + <para> + The <filename>wic</filename> command generates partitioned images + from existing OpenEmbedded build artifacts. + Image generation is driven by partitioning commands contained + in an Openembedded kickstart file (<filename>.wks</filename>) + specified either directly on the command line or as one of a + selection of canned <filename>.wks</filename> files as shown + with the <filename>wic list images</filename> command in the + "<link linkend='using-a-provided-kickstart_file'>Using an Existing Kickstart File</link>" + section. + When applied to a given set of build artifacts, the result is an + image or set of images that can be directly written onto media and + used on a particular system. + </para> + + <para> + The <filename>wic</filename> command and the infrastructure + it is based on is by definition incomplete. + Its purpose is to allow the generation of customized images, + and as such was designed to be completely extensible through a + plugin interface. + See the + "<link linkend='openembedded-kickstart-plugins'>Plugins</link>" + section for information on these plugins. + </para> + + <para> + This section provides some background information on + <filename>wic</filename>, describes what you need to have in + place to run the tool, provides instruction on how to use + <filename>wic</filename>, and provides several examples. + </para> + + <section id='wic-background'> + <title>Background</title> + + <para> + This section provides some background on the + <filename>wic</filename> utility. + While none of this information is required to use + <filename>wic</filename>, you might find it interesting. + <itemizedlist> + <listitem><para> + The name "wic" is derived from OpenEmbedded + Image Creator (oeic). + The "oe" diphthong in "oeic" was promoted to the + letter "w", because "oeic" is both difficult to remember and + pronounce.</para></listitem> + <listitem><para> + <filename>wic</filename> is loosely based on the + Meego Image Creator (<filename>mic</filename>) + framework. + The <filename>wic</filename> implementation has been + heavily modified to make direct use of OpenEmbedded + build artifacts instead of package installation and + configuration, which are already incorporated within + the OpenEmbedded artifacts.</para></listitem> + <listitem><para> + <filename>wic</filename> is a completely independent + standalone utility that initially provides + easier-to-use and more flexible replacements for a + couple bits of existing functionality in OE Core's + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-image-live'><filename>image-live</filename></ulink> + class and <filename>mkefidisk.sh</filename> script. + The difference between + <filename>wic</filename> and those examples is + that with <filename>wic</filename> the + functionality of those scripts is implemented + by a general-purpose partitioning language, which is + based on Redhat kickstart syntax.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='wic-requirements'> + <title>Requirements</title> + + <para> + In order to use the <filename>wic</filename> utility + with the OpenEmbedded Build system, your system needs + to meet the following requirements: + <itemizedlist> + <listitem><para>The Linux distribution on your + development host must support the Yocto Project. + See the + "<ulink url='&YOCTO_DOCS_REF_URL;#detailed-supported-distros'>Supported Linux Distributions</ulink>" + section in the Yocto Project Reference Manual for this + list of distributions.</para></listitem> + <listitem><para> + The standard system utilities, such as + <filename>cp</filename>, must be installed on your + development host system. + </para></listitem> + <listitem><para> + You need to have the build artifacts already + available, which typically means that you must + have already created an image using the + Openembedded build system (e.g. + <filename>core-image-minimal</filename>). + While it might seem redundant to generate an image in + order to create an image using + <filename>wic</filename>, the current version of + <filename>wic</filename> requires the artifacts + in the form generated by the build system. + </para></listitem> + <listitem><para> + You must build several native tools, which are tools + built to run on the build system: + <literallayout class='monospaced'> + $ bitbake parted-native dosfstools-native mtools-native + </literallayout> + </para></listitem> + <listitem><para> + You must have sourced one of the build environment + setup scripts (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink>) + found in the + <link linkend='build-directory'>Build Directory</link>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='wic-getting-help'> + <title>Getting Help</title> + + <para> + You can get general help for the <filename>wic</filename> + by entering the <filename>wic</filename> command by itself + or by entering the command with a help argument as follows: + <literallayout class='monospaced'> + $ wic -h + $ wic --help + </literallayout> + </para> + + <para> + Currently, <filename>wic</filename> supports two commands: + <filename>create</filename> and <filename>list</filename>. + You can get help for these commands as follows: + <literallayout class='monospaced'> + $ wic help <replaceable>command</replaceable> + </literallayout> + </para> + + <para> + You can also get detailed help on a number of topics + from the help system. + The output of <filename>wic --help</filename> + displays a list of available help + topics under a "Help topics" heading. + You can have the help system display the help text for + a given topic by prefacing the topic with + <filename>wic help</filename>: + <literallayout class='monospaced'> + $ wic help <replaceable>help_topic</replaceable> + </literallayout> + </para> + + <para> + You can find out more about the images + <filename>wic</filename> creates using the existing + kickstart files with the following form of the command: + <literallayout class='monospaced'> + $ wic list <replaceable>image</replaceable> help + </literallayout> + where <filename><replaceable>image</replaceable></filename> is either + <filename>directdisk</filename> or + <filename>mkefidisk</filename>. + </para> + </section> + + <section id='operational-modes'> + <title>Operational Modes</title> + + <para> + You can use <filename>wic</filename> in two different + modes, depending on how much control you need for + specifying the Openembedded build artifacts that are + used for creating the image: Raw and Cooked: + <itemizedlist> + <listitem><para><emphasis>Raw Mode:</emphasis> + You explicitly specify build artifacts through + command-line arguments.</para></listitem> + <listitem><para><emphasis>Cooked Mode:</emphasis> + The current + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + setting and image name are used to automatically locate + and provide the build artifacts.</para></listitem> + </itemizedlist> + </para> + + <para> + Regardless of the mode you use, you need to have the build + artifacts ready and available. + Additionally, the environment must be set up using the + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink> + script found in the + <link linkend='build-directory'>Build Directory</link>. + </para> + + <section id='raw-mode'> + <title>Raw Mode</title> + + <para> + The general form of the 'wic' command in raw mode is: + <literallayout class='monospaced'> + $ wic create <replaceable>image_name</replaceable>.wks [<replaceable>options</replaceable>] [...] + + Where: + + <replaceable>image_name</replaceable>.wks + An OpenEmbedded kickstart file. You can provide + your own custom file or use a file from a set of + existing files as described by further options. + + -o <replaceable>OUTDIR</replaceable>, --outdir=<replaceable>OUTDIR</replaceable> + The name of a directory in which to create image. + + -i <replaceable>PROPERTIES_FILE</replaceable>, --infile=<replaceable>PROPERTIES_FILE</replaceable> + The name of a file containing the values for image + properties as a JSON file. + + -e <replaceable>IMAGE_NAME</replaceable>, --image-name=<replaceable>IMAGE_NAME</replaceable> + The name of the image from which to use the artifacts + (e.g. <filename>core-image-sato</filename>). + + -r <replaceable>ROOTFS_DIR</replaceable>, --rootfs-dir=<replaceable>ROOTFS_DIR</replaceable> + The path to the <filename>/rootfs</filename> directory to use as the + <filename>.wks</filename> rootfs source. + + -b <replaceable>BOOTIMG_DIR</replaceable>, --bootimg-dir=<replaceable>BOOTIMG_DIR</replaceable> + The path to the directory containing the boot artifacts + (e.g. <filename>/EFI</filename> or <filename>/syslinux</filename>) to use as the <filename>.wks</filename> bootimg + source. + + -k <replaceable>KERNEL_DIR</replaceable>, --kernel-dir=<replaceable>KERNEL_DIR</replaceable> + The path to the directory containing the kernel to use + in the <filename>.wks</filename> boot image. + + -n <replaceable>NATIVE_SYSROOT</replaceable>, --native-sysroot=<replaceable>NATIVE_SYSROOT</replaceable> + The path to the native sysroot containing the tools to use + to build the image. + + -s, --skip-build-check + Skips the build check. + + -D, --debug + Output debug information. + </literallayout> + <note> + You do not need root privileges to run + <filename>wic</filename>. + In fact, you should not run as root when using the + utility. + </note> + </para> + </section> + + <section id='cooked-mode'> + <title>Cooked Mode</title> + + <para> + The general form of the <filename>wic</filename> command + using Cooked Mode is: + <literallayout class='monospaced'> + $ wic create <replaceable>kickstart_file</replaceable> -e <replaceable>image_name</replaceable> + + Where: + + <replaceable>kickstart_file</replaceable> + An OpenEmbedded kickstart file. You can provide your own + custom file or supplied file. + + <replaceable>image_name</replaceable> + Specifies the image built using the OpenEmbedded build + system. + </literallayout> + This form is the simplest and most user-friendly, as it + does not require specifying all individual parameters. + All you need to provide is your own + <filename>.wks</filename> file or one provided with the + release. + </para> + </section> + </section> + + <section id='using-a-provided-kickstart_file'> + <title>Using an Existing Kickstart File</title> + + <para> + If you do not want to create your own + <filename>.wks</filename> file, you can use an existing + file provided by the <filename>wic</filename> installation. + Use the following command to list the available files: + <literallayout class='monospaced'> + $ wic list images + directdisk Create a 'pcbios' direct disk image + mkefidisk Create an EFI disk image + </literallayout> + When you use an existing file, you do not have to use the + <filename>.wks</filename> extension. + Here is an example in Raw Mode that uses the + <filename>directdisk</filename> file: + <literallayout class='monospaced'> + $ wic create directdisk -r <replaceable>rootfs_dir</replaceable> -b <replaceable>bootimg_dir</replaceable> \ + -k <replaceable>kernel_dir</replaceable> -n <replaceable>native_sysroot</replaceable> + </literallayout> + </para> + + <para> + Here are the actual partition language commands + used in the <filename>mkefidisk.wks</filename> file to generate + an image: + <literallayout class='monospaced'> + # short-description: Create an EFI disk image + # long-description: Creates a partitioned EFI disk image that the user + # can directly dd to boot media. + + part /boot --source bootimg-efi --ondisk sda --label msdos --active --align 1024 + + part / --source rootfs --ondisk sda --fstype=ext3 --label platform --align 1024 + + part swap --ondisk sda --size 44 --label swap1 --fstype=swap + + bootloader --timeout=10 --append="rootwait rootfstype=ext3 console=ttyPCH0,115200 console=tty0 vmalloc=256MB snd-hda-intel.enable_msi=0" + </literallayout> + </para> + </section> + + <section id='wic-usage-examples'> + <title>Examples</title> + + <para> + This section provides several examples that show how to use + the <filename>wic</filename> utility. + All the examples assume the list of requirements in the + "<link linkend='wic-requirements'>Requirements</link>" section + have been met. + The examples assume the previously generated image is + <filename>core-image-minimal</filename>. + </para> + + <section id='generate-an-image-using-a-provided-kickstart-file'> + <title>Generate an Image using an Existing Kickstart File</title> + + <para> + This example runs in Cooked Mode and uses the + <filename>mkefidisk</filename> kickstart file: + <literallayout class='monospaced'> + $ wic create mkefidisk -e core-image-minimal + Checking basic build environment... + Done. + + Creating image(s)... + + Info: The new image(s) can be found here: + /var/tmp/wic/build/mkefidisk-201310230946-sda.direct + + The following build artifacts were used to create the image(s): + ROOTFS_DIR: /home/trz/yocto/yocto-image/build/tmp/work/minnow-poky-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/trz/yocto/yocto-image/build/tmp/work/minnow-poky-linux/core-image-minimal/1.0-r0/core-image-minimal-1.0/hddimg + KERNEL_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/minnow/usr/src/kernel + NATIVE_SYSROOT: /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux + + + The image(s) were created using OE kickstart file: + /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/mkefidisk.wks + </literallayout> + This example shows the easiest way to create an image + by running in Cooked Mode and using the + <filename>-e</filename> option with an existing kickstart + file. + All that is necessary is to specify the image used to + generate the artifacts. + Your <filename>local.conf</filename> needs to have the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + variable set to the machine you are using, which is + "minnow" in this example. + </para> + + <para> + The output specifies the exact image created as well as + where it was created. + The output also names the artifacts used and the exact + <filename>.wks</filename> script that was used to generate + the image. + <note> + You should always verify the details provided in the + output to make sure that the image was indeed created + exactly as expected. + </note> + </para> + + <para> + Continuing with the example, you can now directly + <filename>dd</filename> the image to a USB stick, or + whatever media for which you built your image, + and boot the resulting media: + <literallayout class='monospaced'> + $ sudo dd if=/var/tmp/wic/build/mkefidisk-201310230946-sda.direct of=/dev/sdb + [sudo] password for trz: + 182274+0 records in + 182274+0 records out + 93324288 bytes (93 MB) copied, 14.4777 s, 6.4 MB/s + [trz@empanada ~]$ sudo eject /dev/sdb + </literallayout> + </para> + </section> + + <section id='using-a-modified-kickstart-file'> + <title>Using a Modified Kickstart File</title> + + <para> + Because <filename>wic</filename> image creation is driven + by the kickstart file, it is easy to affect image creation + by changing the parameters in the file. + This next example demonstrates that through modification + of the <filename>directdisk</filename> kickstart file. + </para> + + <para> + As mentioned earlier, you can use the command + <filename>wic list images</filename> to show the list + of existing kickstart files. + The directory in which these files reside is + <filename>scripts/lib/image/canned-wks/</filename> + located in the + <link linkend='source-directory'>Source Directory</link>. + Because the available files reside in this directory, you + can create and add your own custom files to the directory. + Subsequent use of the <filename>wic list images</filename> + command would then include your kickstart files. + </para> + + <para> + In this example, the existing + <filename>directdisk</filename> file already does most + of what is needed. + However, for the hardware in this example, the image will + need to boot from <filename>sdb</filename> instead of + <filename>sda</filename>, which is what the + <filename>directdisk</filename> kickstart file uses. + </para> + + <para> + The example begins by making a copy of the + <filename>directdisk.wks</filename> file in the + <filename>scripts/lib/image/canned-wks</filename> + directory and then changing the lines that specify the + target disk from which to boot. + <literallayout class='monospaced'> + $ cp /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisk.wks \ + /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisksdb.wks + </literallayout> + Next, the example modifies the + <filename>directdisksdb.wks</filename> file and changes all + instances of "<filename>--ondisk sda</filename>" + to "<filename>--ondisk sdb</filename>". + The example changes the following two lines and leaves the + remaining lines untouched: + <literallayout class='monospaced'> + part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024 + part / --source rootfs --ondisk sdb --fstype=ext3 --label platform --align 1024 + </literallayout> + Once the lines are changed, the example generates the + <filename>directdisksdb</filename> image. + The command points the process at the + <filename>core-image-minimal</filename> artifacts for the + Next Unit of Computing (nuc) + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + the <filename>local.conf</filename>. + <literallayout class='monospaced'> + $ wic create directdisksdb -e core-image-minimal + Checking basic build environment... + Done. + + Creating image(s)... + + Info: The new image(s) can be found here: + /var/tmp/wic/build/directdisksdb-201310231131-sdb.direct + + The following build artifacts were used to create the image(s): + ROOTFS_DIR: /home/trz/yocto/yocto-image/build/tmp/work/nuc-poky-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/nuc/usr/share + KERNEL_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/nuc/usr/src/kernel + NATIVE_SYSROOT: /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux + + + The image(s) were created using OE kickstart file: + /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisksdb.wks + </literallayout> + Continuing with the example, you can now directly + <filename>dd</filename> the image to a USB stick, or + whatever media for which you built your image, + and boot the resulting media: + <literallayout class='monospaced'> + $ sudo dd if=/var/tmp/wic/build/directdisksdb-201310231131-sdb.direct of=/dev/sdb + 86018+0 records in + 86018+0 records out + 44041216 bytes (44 MB) copied, 13.0734 s, 3.4 MB/s + [trz@empanada tmp]$ sudo eject /dev/sdb + </literallayout> + </para> + </section> + + <section id='creating-an-image-based-on-core-image-minimal-and-crownbay-noemgd'> + <title>Creating an Image Based on <filename>core-image-minimal</filename> and <filename>crownbay-noemgd</filename></title> + + <para> + This example creates an image based on + <filename>core-image-minimal</filename> and a + <filename>crownbay-noemgd</filename> + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + that works right out of the box. + <literallayout class='monospaced'> + $ wic create directdisk -e core-image-minimal + + Checking basic build environment... + Done. + + Creating image(s)... + + Info: The new image(s) can be found here: + /var/tmp/wic/build/directdisk-201309252350-sda.direct + + The following build artifacts were used to create the image(s): + + ROOTFS_DIR: /home/trz/yocto/yocto-image/build/tmp/work/crownbay_noemgd-poky-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/share + KERNEL_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/src/kernel + NATIVE_SYSROOT: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/src/kernel + + The image(s) were created using OE kickstart file: + /home/trz/yocto/yocto-image/scripts/lib/image/canned-wks/directdisk.wks + </literallayout> + </para> + </section> + + <section id='using-a-modified-kickstart-file-and-running-in-raw-mode'> + <title>Using a Modified Kickstart File and Running in Raw Mode</title> + + <para> + This next example manually specifies each build artifact + (runs in Raw Mode) and uses a modified kickstart file. + The example also uses the <filename>-o</filename> option + to cause <filename>wic</filename> to create the output + somewhere other than the default + <filename>/var/tmp/wic</filename> directory: + <literallayout class='monospaced'> + $ wic create ~/test.wks -o /home/trz/testwic --rootfs-dir \ + /home/trz/yocto/yocto-image/build/tmp/work/crownbay_noemgd-poky-linux/core-image-minimal/1.0-r0/rootfs \ + --bootimg-dir /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/share \ + --kernel-dir /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/src/kernel \ + --native-sysroot /home/trz/yocto/yocto-image/build/tmp/sysroots/x86_64-linux + + Creating image(s)... + + Info: The new image(s) can be found here: + /home/trz/testwic/build/test-201309260032-sda.direct + + The following build artifacts were used to create the image(s): + + ROOTFS_DIR: /home/trz/yocto/yocto-image/build/tmp/work/crownbay_noemgd-poky-linux/core-image-minimal/1.0-r0/rootfs + BOOTIMG_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/share + KERNEL_DIR: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/src/kernel + NATIVE_SYSROOT: /home/trz/yocto/yocto-image/build/tmp/sysroots/crownbay-noemgd/usr/src/kernel + + The image(s) were created using OE kickstart file: + /home/trz/test.wks + </literallayout> + For this example, + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + did not have to be specified in the + <filename>local.conf</filename> file since the artifact is + manually specified. + </para> + </section> + </section> + + <section id='openembedded-kickstart-plugins'> + <title>Plugins</title> + + <para> + Plugins allow <filename>wic</filename> functionality to + be extended and specialized by users. + This section documents the plugin interface, which is + currently restricted to source plugins. + </para> + + <para> + Source plugins provide a mechanism to customize + various aspects of the image generation process in + <filename>wic</filename>, mainly the contents of + partitions. + The plugins provide a mechanism for mapping values + specified in <filename>.wks</filename> files using the + <filename>--source</filename> keyword to a + particular plugin implementation that populates a + corresponding partition. + </para> + + <para> + A source plugin is created as a subclass of + <filename>SourcePlugin</filename>. + The plugin file containing it is added to + <filename>scripts/lib/wic/plugins/source/</filename> to + make the plugin implementation available to the + <filename>wic</filename> implementation. + For more information, see + <filename>scripts/lib/wic/pluginbase.py</filename>. + </para> + + <para> + Source plugins can also be implemented and added by + external layers. + As such, any plugins found in a + <filename>scripts/lib/wic/plugins/source/</filename> + directory in an external layer are also made + available. + </para> + + <para> + When the <filename>wic</filename> implementation needs + to invoke a partition-specific implementation, it looks + for the plugin that has the same name as the + <filename>--source</filename> parameter given to + that partition. + For example, if the partition is set up as follows: + <literallayout class='monospaced'> + part /boot --source bootimg-pcbios ... + </literallayout> + The methods defined as class members of the plugin + having the matching <filename>bootimg-pcbios.name</filename> + class member are used. + </para> + + <para> + To be more concrete, here is the plugin definition that + matches a + <filename>--source bootimg-pcbios</filename> usage, + along with an example + method called by the <filename>wic</filename> implementation + when it needs to invoke an implementation-specific + partition-preparation function: + <literallayout class='monospaced'> + class BootimgPcbiosPlugin(SourcePlugin): + name = 'bootimg-pcbios' + + @classmethod + def do_prepare_partition(self, part, ...) + </literallayout> + If the subclass itself does not implement a function, a + default version in a superclass is located and + used, which is why all plugins must be derived from + <filename>SourcePlugin</filename>. + </para> + + <para> + The <filename>SourcePlugin</filename> class defines the + following methods, which is the current set of methods + that can be implemented or overridden by + <filename>--source</filename> plugins. + Any methods not implemented by a + <filename>SourcePlugin</filename> subclass inherit the + implementations present in the + <filename>SourcePlugin</filename> class. + For more information, see the + <filename>SourcePlugin</filename> source for details: + </para> + + <para> + <itemizedlist> + <listitem><para><emphasis><filename>do_prepare_partition()</filename>:</emphasis> + Called to do the actual content population for a + partition. + In other words, the method prepares the final + partition image that is incorporated into the + disk image. + </para></listitem> + <listitem><para><emphasis><filename>do_configure_partition()</filename>:</emphasis> + Called before + <filename>do_prepare_partition()</filename>. + This method is typically used to create custom + configuration files for a partition (e.g. syslinux or + grub configuration files). + </para></listitem> + <listitem><para><emphasis><filename>do_install_disk()</filename>:</emphasis> + Called after all partitions have been prepared and + assembled into a disk image. + This method provides a hook to allow finalization of a + disk image, (e.g. writing an MBR). + </para></listitem> + <listitem><para><emphasis><filename>do_stage_partition()</filename>:</emphasis> + Special content-staging hook called before + <filename>do_prepare_partition()</filename>. + This method is normally empty.</para> + <para>Typically, a partition just uses the passed-in + parameters (e.g. the unmodified value of + <filename>bootimg_dir</filename>). + However, in some cases things might need to be + more tailored. + As an example, certain files might additionally + need to be taken from + <filename>bootimg_dir + /boot</filename>. + This hook allows those files to be staged in a + customized fashion. + <note> + <filename>get_bitbake_var()</filename> + allows you to access non-standard variables + that you might want to use for this. + </note> + </para></listitem> + </itemizedlist> + </para> + + <para> + This scheme is extensible. + Adding more hooks is a simple matter of adding more + plugin methods to <filename>SourcePlugin</filename> and + derived classes. + The code that then needs to call the plugin methods uses + <filename>plugin.get_source_plugin_methods()</filename> + to find the method or methods needed by the call. + Retrieval of those methods is accomplished + by filling up a dict with keys + containing the method names of interest. + On success, these will be filled in with the actual + methods. + Please see the <filename>wic</filename> + implementation for examples and details. + </para> + </section> + + <section id='openembedded-kickstart-wks-reference'> + <title>OpenEmbedded Kickstart (.wks) Reference</title> + + <para> + The current <filename>wic</filename> implementation supports + only the basic kickstart partitioning commands: + <filename>partition</filename> (or <filename>part</filename> + for short) and <filename>bootloader</filename>. + <note> + Future updates will implement more commands and options. + If you use anything that is not specifically + supported, results can be unpredictable. + </note> + </para> + + <para> + The following is a list of the commands, their syntax, + and meanings. + The commands are based on the Fedora + kickstart versions but with modifications to + reflect <filename>wic</filename> capabilities. + You can see the original documentation for those commands + at the following links: + <itemizedlist> + <listitem><para> + <ulink url='http://fedoraproject.org/wiki/Anaconda/Kickstart#part_or_partition'>http://fedoraproject.org/wiki/Anaconda/Kickstart#part_or_partition</ulink> + </para></listitem> + <listitem><para> + <ulink url='http://fedoraproject.org/wiki/Anaconda/Kickstart#bootloader'>http://fedoraproject.org/wiki/Anaconda/Kickstart#bootloader</ulink> + </para></listitem> + </itemizedlist> + </para> + + <section id='command-part-or-partition'> + <title>Command: part or partition</title> + + <para> + Either of these commands create a partition on the system + and uses the following syntax: + <literallayout class='monospaced'> + part [<replaceable>mntpoint</replaceable>] + partition [<replaceable>mntpoint</replaceable>] + </literallayout> + If you do not provide + <replaceable>mntpoint</replaceable>, wic creates a partition + but does not mount it. + </para> + + <para> + The <filename><replaceable>mntpoint</replaceable></filename> + is where the + partition will be mounted and must be of one of the + following forms: + <itemizedlist> + <listitem><para><filename>/<replaceable>path</replaceable></filename>: + For example, <filename>/</filename>, + <filename>/usr</filename>, or + <filename>/home</filename></para></listitem> + <listitem><para><filename>swap</filename>: + The created partition is used as swap space. + </para></listitem> + </itemizedlist> + </para> + + <para> + Specifying a <replaceable>mntpoint</replaceable> causes + the partition to automatically be mounted. + Wic achieves this by adding entries to the filesystem + table (fstab) during image generation. + In order for wic to generate a valid fstab, you must + also provide one of the <filename>--ondrive</filename>, + <filename>--ondisk</filename>, or + <filename>--use-uuid</filename> partition options as part + of the command. + Here is an example using "/" as the mountpoint. + The command uses "--ondisk" to force the partition onto + the <filename>sdb</filename> disk: + <literallayout class='monospaced'> + part / --source rootfs --ondisk sdb --fstype=ext3 --label platform --align 1024 + </literallayout> + </para> + + <para> + Here is a list that describes other supported options you + can use with the <filename>part</filename> and + <filename>partition</filename> commands: + <itemizedlist> + <listitem><para><emphasis><filename>--size</filename>:</emphasis> + The minimum partition size in MBytes. + Specify an integer value such as 500. + Do not append the number with "MB". + You do not need this option if you use + <filename>--source</filename>.</para></listitem> + <listitem><para><emphasis><filename>--source</filename>:</emphasis> + This option is a + <filename>wic</filename>-specific option that + names the source of the data that populates + the partition. + The most common value for this option is + "rootfs", but you can use any value that maps to + a valid source plugin. + For information on the source plugins, see the + "<link linkend='openembedded-kickstart-plugins'>Plugins</link>" + section.</para> + <para>If you use + <filename>--source rootfs</filename>, + <filename>wic</filename> creates a partition as + large as needed and to fill it with the contents of + the root filesystem pointed to by the + <filename>-r</filename> command-line option + or the equivalent rootfs derived from the + <filename>-e</filename> command-line + option. + The filesystem type used to create the + partition is driven by the value of the + <filename>--fstype</filename> option + specified for the partition. + See the entry on + <filename>--fstype</filename> that + follows for more information. + </para> + <para>If you use + <filename>--source <replaceable>plugin-name</replaceable></filename>, + <filename>wic</filename> creates a partition as + large as needed and fills it with the contents of + the partition that is generated by the + specified plugin name using the data pointed + to by the <filename>-r</filename> command-line + option or the equivalent rootfs derived from the + <filename>-e</filename> command-line + option. + Exactly what those contents and filesystem type end + up being are dependent on the given plugin + implementation. + </para> + <para>If you do not use the + <filename>--source</filename> option, the + <filename>wic</filename> command creates an empty + partition. + Consequently, you must use the + <filename>--size</filename> option to specify the + size of the empty partition. + </para></listitem> + <listitem><para><emphasis><filename>--ondisk</filename> or <filename>--ondrive</filename>:</emphasis> + Forces the partition to be created on a particular + disk.</para></listitem> + <listitem><para><emphasis><filename>--fstype</filename>:</emphasis> + Sets the file system type for the partition. + Valid values are: + <itemizedlist> + <listitem><para><filename>ext4</filename> + </para></listitem> + <listitem><para><filename>ext3</filename> + </para></listitem> + <listitem><para><filename>ext2</filename> + </para></listitem> + <listitem><para><filename>btrfs</filename> + </para></listitem> + <listitem><para><filename>squashfs</filename> + </para></listitem> + <listitem><para><filename>swap</filename> + </para></listitem> + </itemizedlist></para></listitem> + <listitem><para><emphasis><filename>--fsoptions</filename>:</emphasis> + Specifies a free-form string of options to be + used when mounting the filesystem. + This string will be copied into the + <filename>/etc/fstab</filename> file of the + installed system and should be enclosed in + quotes. + If not specified, the default string + is "defaults". + </para></listitem> + <listitem><para><emphasis><filename>--label label</filename>:</emphasis> + Specifies the label to give to the filesystem to + be made on the partition. + If the given label is already in use by another + filesystem, a new label is created for the + partition.</para></listitem> + <listitem><para><emphasis><filename>--active</filename>:</emphasis> + Marks the partition as active.</para></listitem> + <listitem><para><emphasis><filename>--align (in KBytes)</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that says to start a partition on an + x KBytes boundary.</para></listitem> + <listitem><para><emphasis><filename>--no-table</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option. + Using the option reserves space for the partition + and causes it to become populated. + However, the partition is not added to the + partition table. + </para></listitem> + <listitem><para><emphasis><filename>--extra-space</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that adds extra space after the space + filled by the content of the partition. + The final size can go beyond the size specified + by the <filename>--size</filename> option. + The default value is 10 Mbytes. + </para></listitem> + <listitem><para><emphasis><filename>--overhead-factor</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that multiplies the size of the partition by + the option's value. + You must supply a value greater than or equal to + "1". + The default value is "1.3". + </para></listitem> + <listitem><para><emphasis><filename>--part-type</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that specifies the partition type globally + unique identifier (GUID) for GPT partitions. + You can find the list of partition type GUIDs + at + <ulink url='http://en.wikipedia.org/wiki/GUID_Partition_Table#Partition_type_GUIDs'></ulink>. + </para></listitem> + <listitem><para><emphasis><filename>--use-uuid</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that causes <filename>wic</filename> to + generate a random GUID for the partition. + The generated identifier is used in the bootloader + configuration to specify the root partition. + </para></listitem> + <listitem><para><emphasis><filename>--uuid</filename>:</emphasis> + This option is a <filename>wic</filename>-specific + option that specifies the partition UUID. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='command-bootloader'> + <title>Command: bootloader</title> + + <para> + This command specifies how the boot loader should be + configured and supports the following options: + <note> + Bootloader functionality and boot partitions are + implemented by the various + <filename>--source</filename> + plugins that implement bootloader functionality. + The bootloader command essentially provides a means of + modifying bootloader configuration. + </note> + <itemizedlist> + <listitem><para><emphasis><filename>--timeout</filename>:</emphasis> + Specifies the number of seconds before the + bootloader times out and boots the default option. + </para></listitem> + <listitem><para><emphasis><filename>--append</filename>:</emphasis> + Specifies kernel parameters. + These parameters will be added to the syslinux + <filename>APPEND</filename> or + <filename>grub</filename> kernel command line. + </para></listitem> + <listitem><para><emphasis><filename>--configfile</filename>:</emphasis> + Specifies a user-defined configuration file for + the bootloader. + You can provide a full pathname for the file or + a file that exists in the + <filename>canned-wks</filename> folder. + This option overrides all other bootloader options. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + </section> + + <section id='configuring-the-kernel'> + <title>Configuring the Kernel</title> + + <para> + Configuring the Yocto Project kernel consists of making sure the + <filename>.config</filename> file has all the right information + in it for the image you are building. + You can use the <filename>menuconfig</filename> tool and + configuration fragments to make sure your + <filename>.config</filename> file is just how you need it. + You can also save known configurations in a + <filename>defconfig</filename> file that the build system can use + for kernel configuration. + </para> + + <para> + This section describes how to use <filename>menuconfig</filename>, + create and use configuration fragments, and how to interactively + modify your <filename>.config</filename> file to create the + leanest kernel configuration file possible. + </para> + + <para> + For more information on kernel configuration, see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>" + section in the Yocto Project Linux Kernel Development Manual. + </para> + + <section id='using-menuconfig'> + <title>Using <filename>menuconfig</filename></title> + + <para> + The easiest way to define kernel configurations is to set them through the + <filename>menuconfig</filename> tool. + This tool provides an interactive method with which + to set kernel configurations. + For general information on <filename>menuconfig</filename>, see + <ulink url='http://en.wikipedia.org/wiki/Menuconfig'></ulink>. + </para> + + <para> + To use the <filename>menuconfig</filename> tool in the Yocto Project development + environment, you must launch it using BitBake. + Thus, the environment must be set up using the + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#structure-memres-core-script'><filename>oe-init-build-env-memres</filename></ulink> + script found in the + <link linkend='build-directory'>Build Directory</link>. + You must also be sure of the state of your build in the + <link linkend='source-directory'>Source Directory</link>. + The following commands run <filename>menuconfig</filename> + assuming the Source Directory's top-level folder is + <filename>~/poky</filename>: + <literallayout class='monospaced'> + $ cd poky + $ source oe-init-build-env + $ bitbake linux-yocto -c kernel_configme -f + $ bitbake linux-yocto -c menuconfig + </literallayout> + Once <filename>menuconfig</filename> comes up, its standard + interface allows you to interactively examine and configure + all the kernel configuration parameters. + After making your changes, simply exit the tool and save your + changes to create an updated version of the + <filename>.config</filename> configuration file. + </para> + + <para> + Consider an example that configures the <filename>linux-yocto-3.14</filename> + kernel. + The OpenEmbedded build system recognizes this kernel as + <filename>linux-yocto</filename>. + Thus, the following commands from the shell in which you previously sourced the + environment initialization script cleans the shared state cache and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> + directory and then runs <filename>menuconfig</filename>: + <literallayout class='monospaced'> + $ bitbake linux-yocto -c menuconfig + </literallayout> + </para> + + <para> + Once <filename>menuconfig</filename> launches, use the interface + to navigate through the selections to find the configuration settings in + which you are interested. + For example, consider the <filename>CONFIG_SMP</filename> configuration setting. + You can find it at <filename>Processor Type and Features</filename> under + the configuration selection <filename>Symmetric Multi-processing Support</filename>. + After highlighting the selection, use the arrow keys to select or deselect + the setting. + When you are finished with all your selections, exit out and save them. + </para> + + <para> + Saving the selections updates the <filename>.config</filename> configuration file. + This is the file that the OpenEmbedded build system uses to configure the + kernel during the build. + You can find and examine this file in the Build Directory in + <filename>tmp/work/</filename>. + The actual <filename>.config</filename> is located in the area where the + specific kernel is built. + For example, if you were building a Linux Yocto kernel based on the + Linux 3.14 kernel and you were building a QEMU image targeted for + <filename>x86</filename> architecture, the + <filename>.config</filename> file would be located here: + <literallayout class='monospaced'> + poky/build/tmp/work/qemux86-poky-linux/linux-yocto-3.14.11+git1+84f... + ...656ed30-r1/linux-qemux86-standard-build + </literallayout> + <note> + The previous example directory is artificially split and many of the characters + in the actual filename are omitted in order to make it more readable. + Also, depending on the kernel you are using, the exact pathname + for <filename>linux-yocto-3.14...</filename> might differ. + </note> + </para> + + <para> + Within the <filename>.config</filename> file, you can see the kernel settings. + For example, the following entry shows that symmetric multi-processor support + is not set: + <literallayout class='monospaced'> + # CONFIG_SMP is not set + </literallayout> + </para> + + <para> + A good method to isolate changed configurations is to use a combination of the + <filename>menuconfig</filename> tool and simple shell commands. + Before changing configurations with <filename>menuconfig</filename>, copy the + existing <filename>.config</filename> and rename it to something else, + use <filename>menuconfig</filename> to make + as many changes as you want and save them, then compare the renamed configuration + file against the newly created file. + You can use the resulting differences as your base to create configuration fragments + to permanently save in your kernel layer. + <note> + Be sure to make a copy of the <filename>.config</filename> and don't just + rename it. + The build system needs an existing <filename>.config</filename> + from which to work. + </note> + </para> + </section> + + <section id='creating-a-defconfig-file'> + <title>Creating a <filename>defconfig</filename> File</title> + + <para> + A <filename>defconfig</filename> file is simply a + <filename>.config</filename> renamed to "defconfig". + You can use a <filename>defconfig</filename> file + to retain a known set of kernel configurations from which the + OpenEmbedded build system can draw to create the final + <filename>.config</filename> file. + <note> + Out-of-the-box, the Yocto Project never ships a + <filename>defconfig</filename> or + <filename>.config</filename> file. + The OpenEmbedded build system creates the final + <filename>.config</filename> file used to configure the + kernel. + </note> + </para> + + <para> + To create a <filename>defconfig</filename>, start with a + complete, working Linux kernel <filename>.config</filename> + file. + Copy that file to the appropriate + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink><filename>}</filename> + directory in your layer's + <filename>recipes-kernel/linux</filename> directory, and rename + the copied file to "defconfig". + Then, add the following lines to the linux-yocto + <filename>.bbappend</filename> file in your layer: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" + SRC_URI += "file://defconfig" + </literallayout> + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + tells the build system how to search for the file, while the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + extends the + <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESPATH'><filename>FILESPATH</filename></ulink> + variable (search directories) to include the + <filename>${PN}</filename> directory you created to hold the + configuration changes. + <note> + The build system applies the configurations from the + <filename>defconfig</filename> file before applying any + subsequent configuration fragments. + The final kernel configuration is a combination of the + configurations in the <filename>defconfig</filename> + file and any configuration fragments you provide. + You need to realize that if you have any configuration + fragments, the build system applies these on top of and + after applying the existing defconfig file configurations. + </note> + For more information on configuring the kernel, see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#changing-the-configuration'>Changing the Configuration</ulink>" + and + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>" + sections, both in the Yocto Project Linux Kernel Development + Manual. + </para> + </section> + + <section id='creating-config-fragments'> + <title>Creating Configuration Fragments</title> + + <para> + Configuration fragments are simply kernel options that appear in a file + placed where the OpenEmbedded build system can find and apply them. + Syntactically, the configuration statement is identical to what would appear + in the <filename>.config</filename> file, which is in the + <link linkend='build-directory'>Build Directory</link>: + <literallayout class='monospaced'> + tmp/work/<replaceable>arch</replaceable>-poky-linux/linux-yocto-<replaceable>release_specific_string</replaceable>/linux-<replaceable>arch</replaceable>-<replaceable>build_type</replaceable> + </literallayout> + </para> + + <para> + It is simple to create a configuration fragment. + For example, issuing the following from the shell creates a configuration fragment + file named <filename>my_smp.cfg</filename> that enables multi-processor support + within the kernel: + <literallayout class='monospaced'> + $ echo "CONFIG_SMP=y" >> my_smp.cfg + </literallayout> + <note> + All configuration fragment files must use the + <filename>.cfg</filename> extension in order for the + OpenEmbedded build system to recognize them as a + configuration fragment. + </note> + </para> + + <para> + Where do you put your configuration fragment files? + You can place these files in the same area pointed to by + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>. + The OpenEmbedded build system picks up the configuration and + adds it to the kernel's configuration. + For example, suppose you had a set of configuration options + in a file called <filename>myconfig.cfg</filename>. + If you put that file inside a directory named + <filename>linux-yocto</filename> that resides in the same + directory as the kernel's append file and then add a + <filename>SRC_URI</filename> statement such as the following + to the kernel's append file, those configuration options + will be picked up and applied when the kernel is built. + <literallayout class='monospaced'> + SRC_URI += "file://myconfig.cfg" + </literallayout> + </para> + + <para> + As mentioned earlier, you can group related configurations into multiple files and + name them all in the <filename>SRC_URI</filename> statement as well. + For example, you could group separate configurations specifically for Ethernet and graphics + into their own files and add those by using a <filename>SRC_URI</filename> statement like the + following in your append file: + <literallayout class='monospaced'> + SRC_URI += "file://myconfig.cfg \ + file://eth.cfg \ + file://gfx.cfg" + </literallayout> + </para> + </section> + + <section id='fine-tuning-the-kernel-configuration-file'> + <title>Fine-Tuning the Kernel Configuration File</title> + + <para> + You can make sure the <filename>.config</filename> file is as lean or efficient as + possible by reading the output of the kernel configuration fragment audit, + noting any issues, making changes to correct the issues, and then repeating. + </para> + + <para> + As part of the kernel build process, the + <filename>do_kernel_configcheck</filename> task runs. + This task validates the kernel configuration by checking the final + <filename>.config</filename> file against the input files. + During the check, the task produces warning messages for the following + issues: + <itemizedlist> + <listitem><para>Requested options that did not make the final + <filename>.config</filename> file.</para></listitem> + <listitem><para>Configuration items that appear twice in the same + configuration fragment.</para></listitem> + <listitem><para>Configuration items tagged as "required" that were overridden. + </para></listitem> + <listitem><para>A board overrides a non-board specific option.</para></listitem> + <listitem><para>Listed options not valid for the kernel being processed. + In other words, the option does not appear anywhere.</para></listitem> + </itemizedlist> + <note> + The <filename>do_kernel_configcheck</filename> task can + also optionally report if an option is overridden during + processing. + </note> + </para> + + <para> + For each output warning, a message points to the file + that contains a list of the options and a pointer to the + configuration fragment that defines them. + Collectively, the files are the key to streamlining the + configuration. + </para> + + <para> + To streamline the configuration, do the following: + <orderedlist> + <listitem><para>Start with a full configuration that you + know works - it builds and boots successfully. + This configuration file will be your baseline. + </para></listitem> + <listitem><para>Separately run the + <filename>do_kernel_configme</filename> and + <filename>do_kernel_configcheck</filename> tasks. + </para></listitem> + <listitem><para>Take the resulting list of files from the + <filename>do_kernel_configcheck</filename> task + warnings and do the following: + <itemizedlist> + <listitem><para> + Drop values that are redefined in the fragment + but do not change the final + <filename>.config</filename> file. + </para></listitem> + <listitem><para> + Analyze and potentially drop values from the + <filename>.config</filename> file that override + required configurations. + </para></listitem> + <listitem><para> + Analyze and potentially remove non-board + specific options. + </para></listitem> + <listitem><para> + Remove repeated and invalid options. + </para></listitem> + </itemizedlist></para></listitem> + <listitem><para> + After you have worked through the output of the kernel + configuration audit, you can re-run the + <filename>do_kernel_configme</filename> and + <filename>do_kernel_configcheck</filename> tasks to + see the results of your changes. + If you have more issues, you can deal with them as + described in the previous step. + </para></listitem> + </orderedlist> + </para> + + <para> + Iteratively working through steps two through four eventually yields + a minimal, streamlined configuration file. + Once you have the best <filename>.config</filename>, you can build the Linux + Yocto kernel. + </para> + </section> + + <section id='determining-hardware-and-non-hardware-features-for-the-kernel-configuration-audit-phase'> + <title>Determining Hardware and Non-Hardware Features for the Kernel Configuration Audit Phase</title> + + <para> + This section describes part of the kernel configuration audit + phase that most developers can ignore. + During this part of the audit phase, the contents of the final + <filename>.config</filename> file are compared against the + fragments specified by the system. + These fragments can be system fragments, distro fragments, + or user specified configuration elements. + Regardless of their origin, the OpenEmbedded build system + warns the user if a specific option is not included in the + final kernel configuration. + </para> + + <para> + In order to not overwhelm the user with configuration warnings, + by default the system only reports on missing "hardware" + options because a missing hardware option could mean a boot + failure or that important hardware is not available. + </para> + + <para> + To determine whether or not a given option is "hardware" or + "non-hardware", the kernel Metadata contains files that + classify individual or groups of options as either hardware + or non-hardware. + To better show this, consider a situation where the + Yocto Project kernel cache contains the following files: + <literallayout class='monospaced'> + kernel-cache/features/drm-psb/hardware.cfg + kernel-cache/features/kgdb/hardware.cfg + kernel-cache/ktypes/base/hardware.cfg + kernel-cache/bsp/mti-malta32/hardware.cfg + kernel-cache/bsp/fsl-mpc8315e-rdb/hardware.cfg + kernel-cache/bsp/qemu-ppc32/hardware.cfg + kernel-cache/bsp/qemuarma9/hardware.cfg + kernel-cache/bsp/mti-malta64/hardware.cfg + kernel-cache/bsp/arm-versatile-926ejs/hardware.cfg + kernel-cache/bsp/common-pc/hardware.cfg + kernel-cache/bsp/common-pc-64/hardware.cfg + kernel-cache/features/rfkill/non-hardware.cfg + kernel-cache/ktypes/base/non-hardware.cfg + kernel-cache/features/aufs/non-hardware.kcf + kernel-cache/features/ocf/non-hardware.kcf + kernel-cache/ktypes/base/non-hardware.kcf + kernel-cache/ktypes/base/hardware.kcf + kernel-cache/bsp/qemu-ppc32/hardware.kcf + </literallayout> + The following list provides explanations for the various + files: + <itemizedlist> + <listitem><para><filename>hardware.kcf</filename>: + Specifies a list of kernel Kconfig files that contain + hardware options only. + </para></listitem> + <listitem><para><filename>non-hardware.kcf</filename>: + Specifies a list of kernel Kconfig files that contain + non-hardware options only. + </para></listitem> + <listitem><para><filename>hardware.cfg</filename>: + Specifies a list of kernel + <filename>CONFIG_</filename> options that are hardware, + regardless of whether or not they are within a Kconfig + file specified by a hardware or non-hardware + Kconfig file (i.e. <filename>hardware.kcf</filename> or + <filename>non-hardware.kcf</filename>). + </para></listitem> + <listitem><para><filename>non-hardware.cfg</filename>: + Specifies a list of kernel + <filename>CONFIG_</filename> options that are + not hardware, regardless of whether or not they are + within a Kconfig file specified by a hardware or + non-hardware Kconfig file (i.e. + <filename>hardware.kcf</filename> or + <filename>non-hardware.kcf</filename>). + </para></listitem> + </itemizedlist> + Here is a specific example using the + <filename>kernel-cache/bsp/mti-malta32/hardware.cfg</filename>: + <literallayout class='monospaced'> + CONFIG_SERIAL_8250 + CONFIG_SERIAL_8250_CONSOLE + CONFIG_SERIAL_8250_NR_UARTS + CONFIG_SERIAL_8250_PCI + CONFIG_SERIAL_CORE + CONFIG_SERIAL_CORE_CONSOLE + CONFIG_VGA_ARB + </literallayout> + The kernel configuration audit automatically detects these + files (hence the names must be exactly the ones discussed here), + and uses them as inputs when generating warnings about the + final <filename>.config</filename> file. + </para> + + <para> + A user-specified kernel Metadata repository, or recipe space + feature, can use these same files to classify options that are + found within its <filename>.cfg</filename> files as hardware + or non-hardware, to prevent the OpenEmbedded build system from + producing an error or warning when an option is not in the + final <filename>.config</filename> file. + </para> + </section> + </section> + + <section id="patching-the-kernel"> + <title>Patching the Kernel</title> + + <para> + Patching the kernel involves changing or adding configurations to an existing kernel, + changing or adding recipes to the kernel that are needed to support specific hardware features, + or even altering the source code itself. + <note> + You can use the <filename>yocto-kernel</filename> script + found in the <link linkend='source-directory'>Source Directory</link> + under <filename>scripts</filename> to manage kernel patches and configuration. + See the "<ulink url='&YOCTO_DOCS_BSP_URL;#managing-kernel-patches-and-config-items-with-yocto-kernel'>Managing kernel Patches and Config Items with yocto-kernel</ulink>" + section in the Yocto Project Board Support Packages (BSP) Developer's Guide for + more information.</note> + </para> + + <para> + This example creates a simple patch by adding some QEMU emulator console + output at boot time through <filename>printk</filename> statements in the kernel's + <filename>calibrate.c</filename> source code file. + Applying the patch and booting the modified image causes the added + messages to appear on the emulator's console. + </para> + + <para> + The example assumes a clean build exists for the <filename>qemux86</filename> + machine in a + <link linkend='source-directory'>Source Directory</link> + named <filename>poky</filename>. + Furthermore, the <link linkend='build-directory'>Build Directory</link> is + <filename>build</filename> and is located in <filename>poky</filename> and + the kernel is based on the Linux 3.4 kernel. + </para> + + <para> + Also, for more information on patching the kernel, see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#applying-patches'>Applying Patches</ulink>" + section in the Yocto Project Linux Kernel Development Manual. + </para> + + <section id='create-a-layer-for-your-changes'> + <title>Create a Layer for your Changes</title> + + <para> + The first step is to create a layer so you can isolate your + changes. + Rather than use the <filename>yocto-layer</filename> script + to create the layer, this example steps through the process + by hand. + If you want information on the script that creates a general + layer, see the + "<link linkend='creating-a-general-layer-using-the-yocto-layer-script'>Creating a General Layer Using the yocto-layer Script</link>" + section. + </para> + + <para> + These two commands create a directory you can use for your + layer: + <literallayout class='monospaced'> + $ cd ~/poky + $ mkdir meta-mylayer + </literallayout> + Creating a directory that follows the Yocto Project layer naming + conventions sets up the layer for your changes. + The layer is where you place your configuration files, append + files, and patch files. + To learn more about creating a layer and filling it with the + files you need, see the "<link linkend='understanding-and-creating-layers'>Understanding + and Creating Layers</link>" section. + </para> + </section> + + <section id='finding-the-kernel-source-code'> + <title>Finding the Kernel Source Code</title> + + <para> + Each time you build a kernel image, the kernel source code is fetched + and unpacked into the following directory: + <literallayout class='monospaced'> + ${S}/linux + </literallayout> + See the "<link linkend='finding-the-temporary-source-code'>Finding Temporary Source Code</link>" + section and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> variable + for more information about where source is kept during a build. + </para> + + <para> + For this example, we are going to patch the + <filename>init/calibrate.c</filename> file + by adding some simple console <filename>printk</filename> statements that we can + see when we boot the image using QEMU. + </para> + </section> + + <section id='creating-the-patch'> + <title>Creating the Patch</title> + + <para> + Two methods exist by which you can create the patch: + <link linkend='using-devtool-in-your-workflow'><filename>devtool</filename></link> and + <link linkend='using-a-quilt-workflow'>Quilt</link>. + For kernel patches, the Git workflow is more appropriate. + This section assumes the Git workflow and shows the steps specific to + this example. + <orderedlist> + <listitem><para><emphasis>Change the working directory</emphasis>: + Change to where the kernel source code is before making + your edits to the <filename>calibrate.c</filename> file: + <literallayout class='monospaced'> + $ cd ~/poky/build/tmp/work/qemux86-poky-linux/linux-yocto-${PV}-${PR}/linux + </literallayout> + Because you are working in an established Git repository, + you must be in this directory in order to commit your changes + and create the patch file. + <note>The <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> variables + represent the version and revision for the + <filename>linux-yocto</filename> recipe. + The <filename>PV</filename> variable includes the Git meta and machine + hashes, which make the directory name longer than you might + expect. + </note></para></listitem> + <listitem><para><emphasis>Edit the source file</emphasis>: + Edit the <filename>init/calibrate.c</filename> file to have the + following changes: + <literallayout class='monospaced'> + void calibrate_delay(void) + { + unsigned long lpj; + static bool printed; + int this_cpu = smp_processor_id(); + + printk("*************************************\n"); + printk("* *\n"); + printk("* HELLO YOCTO KERNEL *\n"); + printk("* *\n"); + printk("*************************************\n"); + + if (per_cpu(cpu_loops_per_jiffy, this_cpu)) { + . + . + . + </literallayout></para></listitem> + <listitem><para><emphasis>Stage and commit your changes</emphasis>: + These Git commands display the modified file, stage it, and then + commit the file: + <literallayout class='monospaced'> + $ git status + $ git add init/calibrate.c + $ git commit -m "calibrate: Add printk example" + </literallayout></para></listitem> + <listitem><para><emphasis>Generate the patch file</emphasis>: + This Git command creates the a patch file named + <filename>0001-calibrate-Add-printk-example.patch</filename> + in the current directory. + <literallayout class='monospaced'> + $ git format-patch -1 + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='set-up-your-layer-for-the-build'> + <title>Set Up Your Layer for the Build</title> + + <para>These steps get your layer set up for the build: + <orderedlist> + <listitem><para><emphasis>Create additional structure</emphasis>: + Create the additional layer structure: + <literallayout class='monospaced'> + $ cd ~/poky/meta-mylayer + $ mkdir conf + $ mkdir recipes-kernel + $ mkdir recipes-kernel/linux + $ mkdir recipes-kernel/linux/linux-yocto + </literallayout> + The <filename>conf</filename> directory holds your configuration files, while the + <filename>recipes-kernel</filename> directory holds your append file and + your patch file.</para></listitem> + <listitem><para><emphasis>Create the layer configuration file</emphasis>: + Move to the <filename>meta-mylayer/conf</filename> directory and create + the <filename>layer.conf</filename> file as follows: + <literallayout class='monospaced'> + # We have a conf and classes directory, add to BBPATH + BBPATH .= ":${LAYERDIR}" + + # We have recipes-* directories, add to BBFILES + BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \ + ${LAYERDIR}/recipes-*/*/*.bbappend" + + BBFILE_COLLECTIONS += "mylayer" + BBFILE_PATTERN_mylayer = "^${LAYERDIR}/" + BBFILE_PRIORITY_mylayer = "5" + </literallayout> + Notice <filename>mylayer</filename> as part of the last three + statements.</para></listitem> + <listitem><para><emphasis>Create the kernel recipe append file</emphasis>: + Move to the <filename>meta-mylayer/recipes-kernel/linux</filename> directory and create + the <filename>linux-yocto_3.4.bbappend</filename> file as follows: + <literallayout class='monospaced'> + FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:" + + SRC_URI += "file://0001-calibrate-Add-printk-example.patch" + </literallayout> + The <ulink url='&YOCTO_DOCS_REF_URL;#var-FILESEXTRAPATHS'><filename>FILESEXTRAPATHS</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statements enable the OpenEmbedded build system to find the patch file. + For more information on using append files, see the + "<link linkend='using-bbappend-files'>Using .bbappend Files</link>" + section. + </para></listitem> + <listitem><para><emphasis>Put the patch file in your layer</emphasis>: + Move the <filename>0001-calibrate-Add-printk-example.patch</filename> file to + the <filename>meta-mylayer/recipes-kernel/linux/linux-yocto</filename> + directory.</para></listitem> + </orderedlist> + </para> + </section> + + <section id='set-up-for-the-build'> + <title>Set Up for the Build</title> + + <para> + Do the following to make sure the build parameters are set up for the example. + Once you set up these build parameters, they do not have to change unless you + change the target architecture of the machine you are building: + <itemizedlist> + <listitem><para><emphasis>Build for the correct target architecture:</emphasis> Your + selected <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + definition within the <filename>local.conf</filename> file in the + <link linkend='build-directory'>Build Directory</link> + specifies the target architecture used when building the Linux kernel. + By default, <filename>MACHINE</filename> is set to + <filename>qemux86</filename>, which specifies a 32-bit + <trademark class='registered'>Intel</trademark> Architecture + target machine suitable for the QEMU emulator.</para></listitem> + <listitem><para><emphasis>Identify your <filename>meta-mylayer</filename> + layer:</emphasis> The + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBLAYERS'><filename>BBLAYERS</filename></ulink> + variable in the + <filename>bblayers.conf</filename> file found in the + <filename>poky/build/conf</filename> directory needs to have the path to your local + <filename>meta-mylayer</filename> layer. + By default, the <filename>BBLAYERS</filename> variable contains paths to + <filename>meta</filename>, <filename>meta-poky</filename>, and + <filename>meta-yocto-bsp</filename> in the + <filename>poky</filename> Git repository. + Add the path to your <filename>meta-mylayer</filename> location: + <literallayout class='monospaced'> + BBLAYERS ?= " \ + $HOME/poky/meta \ + $HOME/poky/meta-poky \ + $HOME/poky/meta-yocto-bsp \ + $HOME/poky/meta-mylayer \ + " + </literallayout></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='build-the-modified-qemu-kernel-image'> + <title>Build the Modified QEMU Kernel Image</title> + + <para> + The following steps build your modified kernel image: + <orderedlist> + <listitem><para><emphasis>Be sure your build environment is initialized</emphasis>: + Your environment should be set up since you previously sourced + the + <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink> + script. + If it is not, source the script again from <filename>poky</filename>. + <literallayout class='monospaced'> + $ cd ~/poky + $ source &OE_INIT_FILE; + </literallayout> + </para></listitem> + <listitem><para><emphasis>Clean up</emphasis>: + Be sure to clean the shared state out by using BitBake + to run from within the Build Directory the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-cleansstate'><filename>do_cleansstate</filename></ulink> + task as follows: + <literallayout class='monospaced'> + $ bitbake -c cleansstate linux-yocto + </literallayout></para> + <para> + <note> + Never remove any files by hand from the + <filename>tmp/deploy</filename> + directory inside the + <link linkend='build-directory'>Build Directory</link>. + Always use the various BitBake clean tasks to + clear out previous build artifacts. + For information on the clean tasks, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-clean'><filename>do_clean</filename></ulink>", + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-cleanall'><filename>do_cleanall</filename></ulink>", + and + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-cleansstate'><filename>do_cleansstate</filename></ulink>" + sections all in the Yocto Project Reference + Manual. + </note> + </para></listitem> + <listitem><para><emphasis>Build the image</emphasis>: + Next, build the kernel image using this command: + <literallayout class='monospaced'> + $ bitbake -k linux-yocto + </literallayout></para></listitem> + </orderedlist> + </para> + </section> + + <section id='boot-the-image-and-verify-your-changes'> + <title>Boot the Image and Verify Your Changes</title> + + <para> + These steps boot the image and allow you to see the changes + <orderedlist> + <listitem><para><emphasis>Boot the image</emphasis>: + Boot the modified image in the QEMU emulator + using this command: + <literallayout class='monospaced'> + $ runqemu qemux86 + </literallayout></para></listitem> + <listitem><para><emphasis>Verify the changes</emphasis>: + Log into the machine using <filename>root</filename> with no password and then + use the following shell command to scroll through the console's boot output. + <literallayout class='monospaced'> + # dmesg | less + </literallayout> + You should see the results of your <filename>printk</filename> statements + as part of the output.</para></listitem> + </orderedlist> + </para> + </section> + </section> + + <section id='making-images-more-secure'> + <title>Making Images More Secure</title> + + <para> + Security is of increasing concern for embedded devices. + Consider the issues and problems discussed in just this + sampling of work found across the Internet: + <itemizedlist> + <listitem><para><emphasis> + "<ulink url='https://www.schneier.com/blog/archives/2014/01/security_risks_9.html'>Security Risks of Embedded Systems</ulink>"</emphasis> + by Bruce Schneier + </para></listitem> + <listitem><para><emphasis> + "<ulink url='http://internetcensus2012.bitbucket.org/paper.html'>Internet Census 2012</ulink>"</emphasis> + by Carna Botnet</para></listitem> + <listitem><para><emphasis> + "<ulink url='http://elinux.org/images/6/6f/Security-issues.pdf'>Security Issues for Embedded Devices</ulink>"</emphasis> + by Jake Edge + </para></listitem> + </itemizedlist> + </para> + + <para> + When securing your image is of concern, there are steps, tools, + and variables that you can consider to help you reach the + security goals you need for your particular device. + Not all situations are identical when it comes to making an + image secure. + Consequently, this section provides some guidance and suggestions + for consideration when you want to make your image more secure. + <note> + Because the security requirements and risks are + different for every type of device, this section cannot + provide a complete reference on securing your custom OS. + It is strongly recommended that you also consult other sources + of information on embedded Linux system hardening and on + security. + </note> + </para> + + <section id='general-considerations'> + <title>General Considerations</title> + + <para> + General considerations exist that help you create more + secure images. + You should consider the following suggestions to help + make your device more secure: + <itemizedlist> + <listitem><para> + Scan additional code you are adding to the system + (e.g. application code) by using static analysis + tools. + Look for buffer overflows and other potential + security problems. + </para></listitem> + <listitem><para> + Pay particular attention to the security for + any web-based administration interface. + </para> + <para>Web interfaces typically need to perform + administrative functions and tend to need to run with + elevated privileges. + Thus, the consequences resulting from the interface's + security becoming compromised can be serious. + Look for common web vulnerabilities such as + cross-site-scripting (XSS), unvalidated inputs, + and so forth.</para> + <para>As with system passwords, the default credentials + for accessing a web-based interface should not be the + same across all devices. + This is particularly true if the interface is enabled + by default as it can be assumed that many end-users + will not change the credentials. + </para></listitem> + <listitem><para> + Ensure you can update the software on the device to + mitigate vulnerabilities discovered in the future. + This consideration especially applies when your + device is network-enabled. + </para></listitem> + <listitem><para> + Ensure you remove or disable debugging functionality + before producing the final image. + For information on how to do this, see the + "<link linkend='considerations-specific-to-the-openembedded-build-system'>Considerations Specific to the OpenEmbedded Build System</link>" + section. + </para></listitem> + <listitem><para> + Ensure you have no network services listening that + are not needed. + </para></listitem> + <listitem><para> + Remove any software from the image that is not needed. + </para></listitem> + <listitem><para> + Enable hardware support for secure boot functionality + when your device supports this functionality. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='security-flags'> + <title>Security Flags</title> + + <para> + The Yocto Project has security flags that you can enable that + help make your build output more secure. + The security flags are in the + <filename>meta/conf/distro/include/security_flags.inc</filename> + file in your + <link linkend='source-directory'>Source Directory</link> + (e.g. <filename>poky</filename>). + <note> + Depending on the recipe, certain security flags are enabled + and disabled by default. + </note> + </para> + + <para> +<!-- + The GCC/LD flags in <filename>security_flags.inc</filename> + enable more secure code generation. + By including the <filename>security_flags.inc</filename> + file, you enable flags to the compiler and linker that cause + them to generate more secure code. + <note> + The GCC/LD flags are enabled by default in the + <filename>poky-lsb</filename> distribution. + </note> +--> + Use the following line in your + <filename>local.conf</filename> file or in your custom + distribution configuration file to enable the security + compiler and linker flags for your build: + <literallayout class='monospaced'> + require conf/distro/include/security_flags.inc + </literallayout> + </para> + </section> + + <section id='considerations-specific-to-the-openembedded-build-system'> + <title>Considerations Specific to the OpenEmbedded Build System</title> + + <para> + You can take some steps that are specific to the + OpenEmbedded build system to make your images more secure: + <itemizedlist> + <listitem><para> + Ensure "debug-tweaks" is not one of your selected + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>. + When creating a new project, the default is to provide you + with an initial <filename>local.conf</filename> file that + enables this feature using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> variable with the line: + <literallayout class='monospaced'> + EXTRA_IMAGE_FEATURES = "debug-tweaks" + </literallayout> + To disable that feature, simply comment out that line in your + <filename>local.conf</filename> file, or + make sure <filename>IMAGE_FEATURES</filename> does not contain + "debug-tweaks" before producing your final image. + Among other things, leaving this in place sets the + root password as blank, which makes logging in for + debugging or inspection easy during + development but also means anyone can easily log in + during production. + </para></listitem> + <listitem><para> + It is possible to set a root password for the image + and also to set passwords for any extra users you might + add (e.g. administrative or service type users). + When you set up passwords for multiple images or + users, you should not duplicate passwords. + </para> + <para> + To set up passwords, use the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-extrausers'><filename>extrausers</filename></ulink> + class, which is the preferred method. + For an example on how to set up both root and user + passwords, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-extrausers'><filename>extrausers.bbclass</filename></ulink>" + section. + <note> + When adding extra user accounts or setting a + root password, be cautious about setting the + same password on every device. + If you do this, and the password you have set + is exposed, then every device is now potentially + compromised. + If you need this access but want to ensure + security, consider setting a different, + random password for each device. + Typically, you do this as a separate step after + you deploy the image onto the device. + </note> + </para></listitem> + <listitem><para> + Consider enabling a Mandatory Access Control (MAC) + framework such as SMACK or SELinux and tuning it + appropriately for your device's usage. + You can find more information in the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/meta-selinux/'><filename>meta-selinux</filename></ulink> + layer. + </para></listitem> + </itemizedlist> + </para> + + <para> + </para> + </section> + + <section id='tools-for-hardening-your-image'> + <title>Tools for Hardening Your Image</title> + + <para> + The Yocto Project provides tools for making your image + more secure. + You can find these tools in the + <filename>meta-security</filename> layer of the + <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'>Yocto Project Source Repositories</ulink>. + </para> + </section> + </section> + + <section id='creating-your-own-distribution'> + <title>Creating Your Own Distribution</title> + + <para> + When you build an image using the Yocto Project and + do not alter any distribution + <link linkend='metadata'>Metadata</link>, you are creating a + Poky distribution. + If you wish to gain more control over package alternative + selections, compile-time options, and other low-level + configurations, you can create your own distribution. + </para> + + <para> + To create your own distribution, the basic steps consist of + creating your own distribution layer, creating your own + distribution configuration file, and then adding any needed + code and Metadata to the layer. + The following steps provide some more detail: + <itemizedlist> + <listitem><para><emphasis>Create a layer for your new distro:</emphasis> + Create your distribution layer so that you can keep your + Metadata and code for the distribution separate. + It is strongly recommended that you create and use your own + layer for configuration and code. + Using your own layer as compared to just placing + configurations in a <filename>local.conf</filename> + configuration file makes it easier to reproduce the same + build configuration when using multiple build machines. + See the + "<link linkend='creating-a-general-layer-using-the-yocto-layer-script'>Creating a General Layer Using the yocto-layer Script</link>" + section for information on how to quickly set up a layer. + </para></listitem> + <listitem><para><emphasis>Create the distribution configuration file:</emphasis> + The distribution configuration file needs to be created in + the <filename>conf/distro</filename> directory of your + layer. + You need to name it using your distribution name + (e.g. <filename>mydistro.conf</filename>). + <note> + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable in your + <filename>local.conf</filename> file determines the + name of your distribution. + </note></para> + <para>You can split out parts of your configuration file + into include files and then "require" them from within + your distribution configuration file. + Be sure to place the include files in the + <filename>conf/distro/include</filename> directory of + your layer. + A common example usage of include files would be to + separate out the selection of desired version and revisions + for individual recipes. +</para> + <para>Your configuration file needs to set the following + required variables: + <literallayout class='monospaced'> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_NAME'><filename>DISTRO_NAME</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_VERSION'><filename>DISTRO_VERSION</filename></ulink> + </literallayout> + These following variables are optional and you typically + set them from the distribution configuration file: + <literallayout class='monospaced'> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_EXTRA_RDEPENDS'><filename>DISTRO_EXTRA_RDEPENDS</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_EXTRA_RRECOMMENDS'><filename>DISTRO_EXTRA_RRECOMMENDS</filename></ulink> + <ulink url='&YOCTO_DOCS_REF_URL;#var-TCLIBC'><filename>TCLIBC</filename></ulink> + </literallayout> + <tip> + If you want to base your distribution configuration file + on the very basic configuration from OE-Core, you + can use + <filename>conf/distro/defaultsetup.conf</filename> as + a reference and just include variables that differ + as compared to <filename>defaultsetup.conf</filename>. + Alternatively, you can create a distribution + configuration file from scratch using the + <filename>defaultsetup.conf</filename> file + or configuration files from other distributions + such as Poky or Angstrom as references. + </tip></para></listitem> + <listitem><para><emphasis>Provide miscellaneous variables:</emphasis> + Be sure to define any other variables for which you want to + create a default or enforce as part of the distribution + configuration. + You can include nearly any variable from the + <filename>local.conf</filename> file. + The variables you use are not limited to the list in the + previous bulleted item.</para></listitem> + <listitem><para><emphasis>Point to Your distribution configuration file:</emphasis> + In your <filename>local.conf</filename> file in the + <link linkend='build-directory'>Build Directory</link>, + set your + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable to point to your distribution's configuration file. + For example, if your distribution's configuration file is + named <filename>mydistro.conf</filename>, then you point + to it as follows: + <literallayout class='monospaced'> + DISTRO = "mydistro" + </literallayout></para></listitem> + <listitem><para><emphasis>Add more to the layer if necessary:</emphasis> + Use your layer to hold other information needed for the + distribution: + <itemizedlist> + <listitem><para>Add recipes for installing + distro-specific configuration files that are not + already installed by another recipe. + If you have distro-specific configuration files + that are included by an existing recipe, you should + add an append file (<filename>.bbappend</filename>) + for those. + For general information and recommendations + on how to add recipes to your layer, see the + "<link linkend='creating-your-own-layer'>Creating Your Own Layer</link>" + and + "<link linkend='best-practices-to-follow-when-creating-layers'>Best Practices to Follow When Creating Layers</link>" + sections.</para></listitem> + <listitem><para>Add any image recipes that are specific + to your distribution.</para></listitem> + <listitem><para>Add a <filename>psplash</filename> + append file for a branded splash screen. + For information on append files, see the + "<link linkend='using-bbappend-files'>Using .bbappend Files</link>" + section.</para></listitem> + <listitem><para>Add any other append files to make + custom changes that are specific to individual + recipes.</para></listitem> + </itemizedlist></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='creating-a-custom-template-configuration-directory'> + <title>Creating a Custom Template Configuration Directory</title> + + <para> + If you are producing your own customized version + of the build system for use by other users, you might + want to customize the message shown by the setup script or + you might want to change the template configuration files (i.e. + <filename>local.conf</filename> and + <filename>bblayers.conf</filename>) that are created in + a new build directory. + </para> + + <para> + The OpenEmbedded build system uses the environment variable + <filename>TEMPLATECONF</filename> to locate the directory + from which it gathers configuration information that ultimately + ends up in the + <link linkend='build-directory'>Build Directory's</link> + <filename>conf</filename> directory. + By default, <filename>TEMPLATECONF</filename> is set as + follows in the <filename>poky</filename> repository: + <literallayout class='monospaced'> + TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf} + </literallayout> + This is the directory used by the build system to find templates + from which to build some key configuration files. + If you look at this directory, you will see the + <filename>bblayers.conf.sample</filename>, + <filename>local.conf.sample</filename>, and + <filename>conf-notes.txt</filename> files. + The build system uses these files to form the respective + <filename>bblayers.conf</filename> file, + <filename>local.conf</filename> file, and display the list of + BitBake targets when running the setup script. + </para> + + <para> + To override these default configuration files with + configurations you want used within every new + Build Directory, simply set the + <filename>TEMPLATECONF</filename> variable to your directory. + The <filename>TEMPLATECONF</filename> variable is set in the + <filename>.templateconf</filename> file, which is in the + top-level + <link linkend='source-directory'>Source Directory</link> + folder (e.g. <filename>poky</filename>). + Edit the <filename>.templateconf</filename> so that it can locate + your directory. + </para> + + <para> + Best practices dictate that you should keep your + template configuration directory in your custom distribution layer. + For example, suppose you have a layer named + <filename>meta-mylayer</filename> located in your home directory + and you want your template configuration directory named + <filename>myconf</filename>. + Changing the <filename>.templateconf</filename> as follows + causes the OpenEmbedded build system to look in your directory + and base its configuration files on the + <filename>*.sample</filename> configuration files it finds. + The final configuration files (i.e. + <filename>local.conf</filename> and + <filename>bblayers.conf</filename> ultimately still end up in + your Build Directory, but they are based on your + <filename>*.sample</filename> files. + <literallayout class='monospaced'> + TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf} + </literallayout> + </para> + + <para> + Aside from the <filename>*.sample</filename> configuration files, + the <filename>conf-notes.txt</filename> also resides in the + default <filename>meta-poky/conf</filename> directory. + The scripts that set up the build environment + (i.e. + <ulink url="&YOCTO_DOCS_REF_URL;#structure-core-script"><filename>&OE_INIT_FILE;</filename></ulink> + and + <ulink url="&YOCTO_DOCS_REF_URL;#structure-memres-core-script"><filename>oe-init-build-env-memres</filename></ulink>) + use this file to display BitBake targets as part of the script + output. + Customizing this <filename>conf-notes.txt</filename> file is a + good way to make sure your list of custom targets appears + as part of the script's output. + </para> + + <para> + Here is the default list of targets displayed as a result of + running either of the setup scripts: + <literallayout class='monospaced'> + You can now run 'bitbake <target>' + + Common targets are: + core-image-minimal + core-image-sato + meta-toolchain + meta-ide-support + </literallayout> + </para> + + <para> + Changing the listed common targets is as easy as editing your + version of <filename>conf-notes.txt</filename> in your + custom template configuration directory and making sure you + have <filename>TEMPLATECONF</filename> set to your directory. + </para> + </section> + + <section id='building-a-tiny-system'> + <title>Building a Tiny System</title> + + <para> + Very small distributions have some significant advantages such + as requiring less on-die or in-package memory (cheaper), better + performance through efficient cache usage, lower power requirements + due to less memory, faster boot times, and reduced development + overhead. + Some real-world examples where a very small distribution gives + you distinct advantages are digital cameras, medical devices, + and small headless systems. + </para> + + <para> + This section presents information that shows you how you can + trim your distribution to even smaller sizes than the + <filename>poky-tiny</filename> distribution, which is around + 5 Mbytes, that can be built out-of-the-box using the Yocto Project. + </para> + + <section id='tiny-system-overview'> + <title>Overview</title> + + <para> + The following list presents the overall steps you need to + consider and perform to create distributions with smaller + root filesystems, achieve faster boot times, maintain your critical + functionality, and avoid initial RAM disks: + <itemizedlist> + <listitem><para> + <link linkend='goals-and-guiding-principles'>Determine your goals and guiding principles.</link> + </para></listitem> + <listitem><para> + <link linkend='understand-what-gives-your-image-size'>Understand what contributes to your image size.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-root-filesystem'>Reduce the size of the root filesystem.</link> + </para></listitem> + <listitem><para> + <link linkend='trim-the-kernel'>Reduce the size of the kernel.</link> + </para></listitem> + <listitem><para> + <link linkend='remove-package-management-requirements'>Eliminate packaging requirements.</link> + </para></listitem> + <listitem><para> + <link linkend='look-for-other-ways-to-minimize-size'>Look for other ways to minimize size.</link> + </para></listitem> + <listitem><para> + <link linkend='iterate-on-the-process'>Iterate on the process.</link> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='goals-and-guiding-principles'> + <title>Goals and Guiding Principles</title> + + <para> + Before you can reach your destination, you need to know + where you are going. + Here is an example list that you can use as a guide when + creating very small distributions: + <itemizedlist> + <listitem><para>Determine how much space you need + (e.g. a kernel that is 1 Mbyte or less and + a root filesystem that is 3 Mbytes or less). + </para></listitem> + <listitem><para>Find the areas that are currently + taking 90% of the space and concentrate on reducing + those areas. + </para></listitem> + <listitem><para>Do not create any difficult "hacks" + to achieve your goals.</para></listitem> + <listitem><para>Leverage the device-specific + options.</para></listitem> + <listitem><para>Work in a separate layer so that you + keep changes isolated. + For information on how to create layers, see + the "<link linkend='understanding-and-creating-layers'>Understanding and Creating Layers</link>" section. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='understand-what-gives-your-image-size'> + <title>Understand What Contributes to Your Image Size</title> + + <para> + It is easiest to have something to start with when creating + your own distribution. + You can use the Yocto Project out-of-the-box to create the + <filename>poky-tiny</filename> distribution. + Ultimately, you will want to make changes in your own + distribution that are likely modeled after + <filename>poky-tiny</filename>. + <note> + To use <filename>poky-tiny</filename> in your build, + set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + variable in your + <filename>local.conf</filename> file to "poky-tiny" + as described in the + "<link linkend='creating-your-own-distribution'>Creating Your Own Distribution</link>" + section. + </note> + </para> + + <para> + Understanding some memory concepts will help you reduce the + system size. + Memory consists of static, dynamic, and temporary memory. + Static memory is the TEXT (code), DATA (initialized data + in the code), and BSS (uninitialized data) sections. + Dynamic memory represents memory that is allocated at runtime: + stacks, hash tables, and so forth. + Temporary memory is recovered after the boot process. + This memory consists of memory used for decompressing + the kernel and for the <filename>__init__</filename> + functions. + </para> + + <para> + To help you see where you currently are with kernel and root + filesystem sizes, you can use two tools found in the + <link linkend='source-directory'>Source Directory</link> in + the <filename>scripts/tiny/</filename> directory: + <itemizedlist> + <listitem><para><filename>ksize.py</filename>: Reports + component sizes for the kernel build objects. + </para></listitem> + <listitem><para><filename>dirsize.py</filename>: Reports + component sizes for the root filesystem.</para></listitem> + </itemizedlist> + This next tool and command help you organize configuration + fragments and view file dependencies in a human-readable form: + <itemizedlist> + <listitem><para><filename>merge_config.sh</filename>: + Helps you manage configuration files and fragments + within the kernel. + With this tool, you can merge individual configuration + fragments together. + The tool allows you to make overrides and warns you + of any missing configuration options. + The tool is ideal for allowing you to iterate on + configurations, create minimal configurations, and + create configuration files for different machines + without having to duplicate your process.</para> + <para>The <filename>merge_config.sh</filename> script is + part of the Linux Yocto kernel Git repositories + (i.e. <filename>linux-yocto-3.14</filename>, + <filename>linux-yocto-3.10</filename>, + <filename>linux-yocto-3.8</filename>, and so forth) + in the + <filename>scripts/kconfig</filename> directory.</para> + <para>For more information on configuration fragments, + see the + "<ulink url='&YOCTO_DOCS_KERNEL_DEV_URL;#generating-configuration-files'>Generating Configuration Files</ulink>" + section of the Yocto Project Linux Kernel Development + Manual and the "<link linkend='creating-config-fragments'>Creating Configuration Fragments</link>" + section, which is in this manual.</para></listitem> + <listitem><para><filename>bitbake -u depexp -g <replaceable>bitbake_target</replaceable></filename>: + Using the BitBake command with these options brings up + a Dependency Explorer from which you can view file + dependencies. + Understanding these dependencies allows you to make + informed decisions when cutting out various pieces of the + kernel and root filesystem.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='trim-the-root-filesystem'> + <title>Trim the Root Filesystem</title> + + <para> + The root filesystem is made up of packages for booting, + libraries, and applications. + To change things, you can configure how the packaging happens, + which changes the way you build them. + You can also modify the filesystem itself or select a different + filesystem. + </para> + + <para> + First, find out what is hogging your root filesystem by running the + <filename>dirsize.py</filename> script from your root directory: + <literallayout class='monospaced'> + $ cd <replaceable>root-directory-of-image</replaceable> + $ dirsize.py 100000 > dirsize-100k.log + $ cat dirsize-100k.log + </literallayout> + You can apply a filter to the script to ignore files under + a certain size. + The previous example filters out any files below 100 Kbytes. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a + compressed root filesystem. + When you examine your log file, you can focus on areas of the + root filesystem that take up large amounts of memory. + </para> + + <para> + You need to be sure that what you eliminate does not cripple + the functionality you need. + One way to see how packages relate to each other is by using + the Dependency Explorer UI with the BitBake command: + <literallayout class='monospaced'> + $ cd <replaceable>image-directory</replaceable> + $ bitbake -u depexp -g <replaceable>image</replaceable> + </literallayout> + Use the interface to select potential packages you wish to + eliminate and see their dependency relationships. + </para> + + <para> + When deciding how to reduce the size, get rid of packages that + result in minimal impact on the feature set. + For example, you might not need a VGA display. + Or, you might be able to get by with <filename>devtmpfs</filename> + and <filename>mdev</filename> instead of + <filename>udev</filename>. + </para> + + <para> + Use your <filename>local.conf</filename> file to make changes. + For example, to eliminate <filename>udev</filename> and + <filename>glib</filename>, set the following in the + local configuration file: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_dev_manager = "" + </literallayout> + </para> + + <para> + Finally, you should consider exactly the type of root + filesystem you need to meet your needs while also reducing + its size. + For example, consider <filename>cramfs</filename>, + <filename>squashfs</filename>, <filename>ubifs</filename>, + <filename>ext2</filename>, or an <filename>initramfs</filename> + using <filename>initramfs</filename>. + Be aware that <filename>ext3</filename> requires a 1 Mbyte + journal. + If you are okay with running read-only, you do not need this + journal. + </para> + + <note> + After each round of elimination, you need to rebuild your + system and then use the tools to see the effects of your + reductions. + </note> + + + </section> + + <section id='trim-the-kernel'> + <title>Trim the Kernel</title> + + <para> + The kernel is built by including policies for hardware-independent + aspects. + What subsystems do you enable? + For what architecture are you building? + Which drivers do you build by default? + <note>You can modify the kernel source if you want to help + with boot time. + </note> + </para> + + <para> + Run the <filename>ksize.py</filename> script from the top-level + Linux build directory to get an idea of what is making up + the kernel: + <literallayout class='monospaced'> + $ cd <replaceable>top-level-linux-build-directory</replaceable> + $ ksize.py > ksize.log + $ cat ksize.log + </literallayout> + When you examine the log, you will see how much space is + taken up with the built-in <filename>.o</filename> files for + drivers, networking, core kernel files, filesystem, sound, + and so forth. + The sizes reported by the tool are uncompressed, and thus + will be smaller by a relatively constant factor in a compressed + kernel image. + Look to reduce the areas that are large and taking up around + the "90% rule." + </para> + + <para> + To examine, or drill down, into any particular area, use the + <filename>-d</filename> option with the script: + <literallayout class='monospaced'> + $ ksize.py -d > ksize.log + </literallayout> + Using this option breaks out the individual file information + for each area of the kernel (e.g. drivers, networking, and + so forth). + </para> + + <para> + Use your log file to see what you can eliminate from the kernel + based on features you can let go. + For example, if you are not going to need sound, you do not + need any drivers that support sound. + </para> + + <para> + After figuring out what to eliminate, you need to reconfigure + the kernel to reflect those changes during the next build. + You could run <filename>menuconfig</filename> and make all your + changes at once. + However, that makes it difficult to see the effects of your + individual eliminations and also makes it difficult to replicate + the changes for perhaps another target device. + A better method is to start with no configurations using + <filename>allnoconfig</filename>, create configuration + fragments for individual changes, and then manage the + fragments into a single configuration file using + <filename>merge_config.sh</filename>. + The tool makes it easy for you to iterate using the + configuration change and build cycle. + </para> + + <para> + Each time you make configuration changes, you need to rebuild + the kernel and check to see what impact your changes had on + the overall size. + </para> + </section> + + <section id='remove-package-management-requirements'> + <title>Remove Package Management Requirements</title> + + <para> + Packaging requirements add size to the image. + One way to reduce the size of the image is to remove all the + packaging requirements from the image. + This reduction includes both removing the package manager + and its unique dependencies as well as removing the package + management data itself. + </para> + + <para> + To eliminate all the packaging requirements for an image, + be sure that "package-management" is not part of your + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + statement for the image. + When you remove this feature, you are removing the package + manager as well as its dependencies from the root filesystem. + </para> + </section> + + <section id='look-for-other-ways-to-minimize-size'> + <title>Look for Other Ways to Minimize Size</title> + + <para> + Depending on your particular circumstances, other areas that you + can trim likely exist. + The key to finding these areas is through tools and methods + described here combined with experimentation and iteration. + Here are a couple of areas to experiment with: + <itemizedlist> + <listitem><para><filename>glibc</filename>: + In general, follow this process: + <orderedlist> + <listitem><para>Remove <filename>glibc</filename> + features from + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + that you think you do not need.</para></listitem> + <listitem><para>Build your distribution. + </para></listitem> + <listitem><para>If the build fails due to missing + symbols in a package, determine if you can + reconfigure the package to not need those + features. + For example, change the configuration to not + support wide character support as is done for + <filename>ncurses</filename>. + Or, if support for those characters is needed, + determine what <filename>glibc</filename> + features provide the support and restore the + configuration. + </para></listitem> + <listitem><para>Rebuild and repeat the process. + </para></listitem> + </orderedlist></para></listitem> + <listitem><para><filename>busybox</filename>: + For BusyBox, use a process similar as described for + <filename>glibc</filename>. + A difference is you will need to boot the resulting + system to see if you are able to do everything you + expect from the running system. + You need to be sure to integrate configuration fragments + into Busybox because BusyBox handles its own core + features and then allows you to add configuration + fragments on top. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='iterate-on-the-process'> + <title>Iterate on the Process</title> + + <para> + If you have not reached your goals on system size, you need + to iterate on the process. + The process is the same. + Use the tools and see just what is taking up 90% of the root + filesystem and the kernel. + Decide what you can eliminate without limiting your device + beyond what you need. + </para> + + <para> + Depending on your system, a good place to look might be + Busybox, which provides a stripped down + version of Unix tools in a single, executable file. + You might be able to drop virtual terminal services or perhaps + ipv6. + </para> + </section> + </section> + + <section id='building-images-for-more-than-one-machine'> + <title>Building Images for More than One Machine</title> + + <para> + A common scenario developers face is creating images for several + different machines that use the same software environment. + In this situation, it is tempting to set the + tunings and optimization flags for each build specifically for + the targeted hardware (i.e. "maxing out" the tunings). + Doing so can considerably add to build times and package feed + maintenance collectively for the machines. + For example, selecting tunes that are extremely specific to a + CPU core used in a system might enable some micro optimizations + in GCC for that particular system but would otherwise not gain + you much of a performance difference across the other systems + as compared to using a more general tuning across all the builds + (e.g. setting + <ulink url='var-DEFAULTTUNE'><filename>DEFAULTTUNE</filename></ulink> + specifically for each machine's build). + Rather than "max out" each build's tunings, you can take steps that + cause the OpenEmbedded build system to reuse software across the + various machines where it makes sense. + </para> + <para> + If build speed and package feed maintenance are considerations, + you should consider the points in this section that can help you + optimize your tunings to best consider build times and package + feed maintenance. + <itemizedlist> + <listitem><para><emphasis>Share the Build Directory:</emphasis> + If at all possible, share the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> + across builds. + The Yocto Project supports switching between different + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink> + values in the same <filename>TMPDIR</filename>. + This practice is well supported and regularly used by + developers when building for multiple machines. + When you use the same <filename>TMPDIR</filename> for + multiple machine builds, the OpenEmbedded build system can + reuse the existing native and often cross-recipes for + multiple machines. + Thus, build time decreases. + <note> + If + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO'><filename>DISTRO</filename></ulink> + settings change or fundamental configuration settings + such as the filesystem layout, you need to work with + a clean <filename>TMPDIR</filename>. + Sharing <filename>TMPDIR</filename> under these + circumstances might work but since it is not + guaranteed, you should use a clean + <filename>TMPDIR</filename>. + </note> + </para></listitem> + <listitem><para><emphasis>Enable the Appropriate Package Architecture:</emphasis> + By default, the OpenEmbedded build system enables three + levels of package architectures: "all", "tune" or "package", + and "machine". + Any given recipe usually selects one of these package + architectures (types) for its output. + Depending for what a given recipe creates packages, making + sure you enable the appropriate package architecture can + directly impact the build time.</para> + <para>A recipe that just generates scripts can enable + "all" architecture because there are no binaries to build. + To specifically enable "all" architecture, be sure your + recipe inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-allarch'><filename>allarch</filename></ulink> + class. + This class is useful for "all" architectures because it + configures many variables so packages can be used across + multiple architectures.</para> + <para>If your recipe needs to generate packages that are + machine-specific or when one of the build or runtime + dependencies is already machine-architecture dependent, + which makes your recipe also machine-architecture dependent, + make sure your recipe enables the "machine" package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_ARCH'><filename>MACHINE_ARCH</filename></ulink> + variable: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${MACHINE_ARCH}" + </literallayout> + When you do not specifically enable a package + architecture through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>, + The OpenEmbedded build system defaults to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TUNE_PKGARCH'><filename>TUNE_PKGARCH</filename></ulink> + setting: + <literallayout class='monospaced'> + PACKAGE_ARCH = "${TUNE_PKGARCH}" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Choose a Generic Tuning File if Possible:</emphasis> + Some tunes are more generic and can run on multiple targets + (e.g. an <filename>armv5</filename> set of packages could + run on <filename>armv6</filename> and + <filename>armv7</filename> processors in most cases). + Similarly, <filename>i486</filename> binaries could work + on <filename>i586</filename> and higher processors. + You should realize, however, that advances on newer + processor versions would not be used.</para> + <para>If you select the same tune for several different + machines, the OpenEmbedded build system reuses software + previously built, thus speeding up the overall build time. + Realize that even though a new sysroot for each machine is + generated, the software is not recompiled and only one + package feed exists. + </para></listitem> + <listitem><para><emphasis>Manage Granular Level Packaging:</emphasis> + Sometimes cases exist where injecting another level + of package architecture beyond the three higher levels + noted earlier can be useful. + For example, consider the <filename>emgd</filename> + graphics stack in the + <filename>meta-intel</filename> layer. + In this layer, a subset of software exists that is + compiled against something different from the rest of the + generic packages. + You can examine the key code in the + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi'>Source Repositories</ulink> + "daisy" branch in + <filename>classes/emgd-gl.bbclass</filename>. + For a specific set of packages, the code redefines + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>. + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXTRA_ARCHS'><filename>PACKAGE_EXTRA_ARCHS</filename></ulink> + is then appended with this extra tune name in + <filename>meta-intel-emgd.inc</filename>. + The result is that when searching for packages, the + build system uses a four-level search and the packages + in this new level are preferred as compared to the standard + tune. + The overall result is that the build system reuses most + software from the common tune except for specific cases + as needed. + </para></listitem> + <listitem><para><emphasis>Use Tools to Debug Issues:</emphasis> + Sometimes you can run into situations where software is + being rebuilt when you think it should not be. + For example, the OpenEmbedded build system might not be + using shared state between machines when you think it + should be. + These types of situations are usually due to references + to machine-specific variables such as + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-SERIAL_CONSOLE'><filename>SERIAL_CONSOLE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-XSERVER'><filename>XSERVER</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE_FEATURES'><filename>MACHINE_FEATURES</filename></ulink>, + and so forth in code that is supposed to only be + tune-specific or when the recipe depends + (<ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-RSUGGESTS'><filename>RSUGGESTS</filename></ulink>, + and so forth) on some other recipe that already has + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> + defined as "${MACHINE_ARCH}". + <note> + Patches to fix any issues identified are most welcome + as these issues occasionally do occur. + </note></para> + <para>For such cases, you can use some tools to help you + sort out the situation: + <itemizedlist> + <listitem><para><emphasis><filename>sstate-diff-machines.sh</filename>:</emphasis> + You can find this tool in the + <filename>scripts</filename> directory of the + Source Repositories. + See the comments in the script for information on + how to use the tool. + </para></listitem> + <listitem><para><emphasis>BitBake's "-S printdiff" Option:</emphasis> + Using this option causes BitBake to try to + establish the closest signature match it can + (e.g. in the shared state cache) and then run + <filename>bitbake-diffsigs</filename> over the + matches to determine the stamps and delta where + these two stamp trees diverge. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='working-with-packages'> + <title>Working with Packages</title> + + <para> + This section describes a few tasks that involve packages: + <itemizedlist> + <listitem><para> + <link linkend='excluding-packages-from-an-image'>Excluding packages from an image</link> + </para></listitem> + <listitem><para> + <link linkend='incrementing-a-package-revision-number'>Incrementing a package revision number</link> + </para></listitem> + <listitem><para> + <link linkend='handling-optional-module-packaging'>Handling optional module packaging</link> + </para></listitem> + <listitem><para> + <link linkend='using-runtime-package-management'>Using Runtime Package Management</link> + </para></listitem> + <listitem><para> + <link linkend='testing-packages-with-ptest'>Setting up and running package test (ptest)</link> + </para></listitem> + </itemizedlist> + </para> + + <section id='excluding-packages-from-an-image'> + <title>Excluding Packages from an Image</title> + + <para> + You might find it necessary to prevent specific packages + from being installed into an image. + If so, you can use several variables to direct the build + system to essentially ignore installing recommended packages + or to not install a package at all. + </para> + + <para> + The following list introduces variables you can use to + prevent packages from being installed into your image. + Each of these variables only works with IPK and RPM + package types. + Support for Debian packages does not exist. + Also, you can use these variables from your + <filename>local.conf</filename> file or attach them to a + specific image recipe by using a recipe name override. + For more detail on the variables, see the descriptions in the + Yocto Project Reference Manual's glossary chapter. + <itemizedlist> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-BAD_RECOMMENDATIONS'><filename>BAD_RECOMMENDATIONS</filename></ulink>: + Use this variable to specify "recommended-only" + packages that you do not want installed. + </para></listitem> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-NO_RECOMMENDATIONS'><filename>NO_RECOMMENDATIONS</filename></ulink>: + Use this variable to prevent all "recommended-only" + packages from being installed. + </para></listitem> + <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>: + Use this variable to prevent specific packages from + being installed regardless of whether they are + "recommended-only" or not. + You need to realize that the build process could + fail with an error when you + prevent the installation of a package whose presence + is required by an installed package. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='incrementing-a-package-revision-number'> + <title>Incrementing a Package Revision Number</title> + + <para> + If a committed change results in changing the package output, + then the value of the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + variable needs to be increased (or "bumped"). + Increasing <filename>PR</filename> occurs one of two ways: + <itemizedlist> + <listitem><para>Automatically using a Package Revision + Service (PR Service).</para></listitem> + <listitem><para>Manually incrementing the + <filename>PR</filename> variable.</para></listitem> + </itemizedlist> + </para> + + <para> + Given that one of the challenges any build system and its + users face is how to maintain a package feed that is compatible + with existing package manager applications such as + RPM, APT, and OPKG, using an automated system is much + preferred over a manual system. + In either system, the main requirement is that version + numbering increases in a linear fashion and that a number of + version components exist that support that linear progression. + </para> + + <para> + The following two sections provide information on the PR Service + and on manual <filename>PR</filename> bumping. + </para> + + <section id='working-with-a-pr-service'> + <title>Working With a PR Service</title> + + <para> + As mentioned, attempting to maintain revision numbers in the + <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> + is error prone, inaccurate, and causes problems for people + submitting recipes. + Conversely, the PR Service automatically generates + increasing numbers, particularly the revision field, + which removes the human element. + <note> + For additional information on using a PR Service, you + can see the + <ulink url='&YOCTO_WIKI_URL;/wiki/PR_Service'>PR Service</ulink> + wiki page. + </note> + </para> + + <para> + The Yocto Project uses variables in order of + decreasing priority to facilitate revision numbering (i.e. + <ulink url='&YOCTO_DOCS_REF_URL;#var-PE'><filename>PE</filename></ulink>, + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink>, and + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + for epoch, version, and revision, respectively). + The values are highly dependent on the policies and + procedures of a given distribution and package feed. + </para> + + <para> + Because the OpenEmbedded build system uses + "<ulink url='&YOCTO_DOCS_REF_URL;#checksums'>signatures</ulink>", + which are unique to a given build, the build system + knows when to rebuild packages. + All the inputs into a given task are represented by a + signature, which can trigger a rebuild when different. + Thus, the build system itself does not rely on the + <filename>PR</filename> numbers to trigger a rebuild. + The signatures, however, can be used to generate + <filename>PR</filename> values. + </para> + + <para> + The PR Service works with both + <filename>OEBasic</filename> and + <filename>OEBasicHash</filename> generators. + The value of <filename>PR</filename> bumps when the + checksum changes and the different generator mechanisms + change signatures under different circumstances. + </para> + + <para> + As implemented, the build system includes values from + the PR Service into the <filename>PR</filename> field as + an addition using the form "<filename>.x</filename>" so + <filename>r0</filename> becomes <filename>r0.1</filename>, + <filename>r0.2</filename> and so forth. + This scheme allows existing <filename>PR</filename> values + to be used for whatever reasons, which include manual + <filename>PR</filename> bumps, should it be necessary. + </para> + + <para> + By default, the PR Service is not enabled or running. + Thus, the packages generated are just "self consistent". + The build system adds and removes packages and + there are no guarantees about upgrade paths but images + will be consistent and correct with the latest changes. + </para> + + <para> + The simplest form for a PR Service is for it to exist + for a single host development system that builds the + package feed (building system). + For this scenario, you can enable a local PR Service by + setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-PRSERV_HOST'><filename>PRSERV_HOST</filename></ulink> + in your <filename>local.conf</filename> file in the + <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + PRSERV_HOST = "localhost:0" + </literallayout> + Once the service is started, packages will automatically + get increasing <filename>PR</filename> values and + BitBake will take care of starting and stopping the server. + </para> + + <para> + If you have a more complex setup where multiple host + development systems work against a common, shared package + feed, you have a single PR Service running and it is + connected to each building system. + For this scenario, you need to start the PR Service using + the <filename>bitbake-prserv</filename> command: + <literallayout class='monospaced'> + bitbake-prserv --host <replaceable>ip</replaceable> --port <replaceable>port</replaceable> --start + </literallayout> + In addition to hand-starting the service, you need to + update the <filename>local.conf</filename> file of each + building system as described earlier so each system + points to the server and port. + </para> + + <para> + It is also recommended you use build history, which adds + some sanity checks to package versions, in conjunction with + the server that is running the PR Service. + To enable build history, add the following to each building + system's <filename>local.conf</filename> file: + <literallayout class='monospaced'> + # It is recommended to activate "buildhistory" for testing the PR service + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "1" + </literallayout> + For information on build history, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <note> + <para>The OpenEmbedded build system does not maintain + <filename>PR</filename> information as part of the + shared state (sstate) packages. + If you maintain an sstate feed, its expected that either + all your building systems that contribute to the sstate + feed use a shared PR Service, or you do not run a PR + Service on any of your building systems. + Having some systems use a PR Service while others do + not leads to obvious problems.</para> + <para>For more information on shared state, see the + "<ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink>" + section in the Yocto Project Reference Manual.</para> + </note> + </section> + + <section id='manually-bumping-pr'> + <title>Manually Bumping PR</title> + + <para> + The alternative to setting up a PR Service is to manually + bump the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> + variable. + </para> + + <para> + If a committed change results in changing the package output, + then the value of the PR variable needs to be increased + (or "bumped") as part of that commit. + For new recipes you should add the <filename>PR</filename> + variable and set its initial value equal to "r0", which is the default. + Even though the default value is "r0", the practice of adding it to a new recipe makes + it harder to forget to bump the variable when you make changes + to the recipe in future. + </para> + + <para> + If you are sharing a common <filename>.inc</filename> file with multiple recipes, + you can also use the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-INC_PR'>INC_PR</ulink></filename> + variable to ensure that + the recipes sharing the <filename>.inc</filename> file are rebuilt when the + <filename>.inc</filename> file itself is changed. + The <filename>.inc</filename> file must set <filename>INC_PR</filename> + (initially to "r0"), and all recipes referring to it should set <filename>PR</filename> + to "$(INC_PR).0" initially, incrementing the last number when the recipe is changed. + If the <filename>.inc</filename> file is changed then its + <filename>INC_PR</filename> should be incremented. + </para> + + <para> + When upgrading the version of a package, assuming the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'>PV</ulink></filename> + changes, the <filename>PR</filename> variable should be + reset to "r0" (or "$(INC_PR).0" if you are using + <filename>INC_PR</filename>). + </para> + + <para> + Usually, version increases occur only to packages. + However, if for some reason <filename>PV</filename> changes but does not + increase, you can increase the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-PE'>PE</ulink></filename> + variable (Package Epoch). + The <filename>PE</filename> variable defaults to "0". + </para> + + <para> + Version numbering strives to follow the + <ulink url='http://www.debian.org/doc/debian-policy/ch-controlfields.html'> + Debian Version Field Policy Guidelines</ulink>. + These guidelines define how versions are compared and what "increasing" a version means. + </para> + </section> + </section> + + <section id='handling-optional-module-packaging'> + <title>Handling Optional Module Packaging</title> + + <para> + Many pieces of software split functionality into optional + modules (or plug-ins) and the plug-ins that are built + might depend on configuration options. + To avoid having to duplicate the logic that determines what + modules are available in your recipe or to avoid having + to package each module by hand, the OpenEmbedded build system + provides functionality to handle module packaging dynamically. + </para> + + <para> + To handle optional module packaging, you need to do two things: + <itemizedlist> + <listitem><para>Ensure the module packaging is actually + done.</para></listitem> + <listitem><para>Ensure that any dependencies on optional + modules from other recipes are satisfied by your recipe. + </para></listitem> + </itemizedlist> + </para> + + <section id='making-sure-the-packaging-is-done'> + <title>Making Sure the Packaging is Done</title> + + <para> + To ensure the module packaging actually gets done, you use + the <filename>do_split_packages</filename> function within + the <filename>populate_packages</filename> Python function + in your recipe. + The <filename>do_split_packages</filename> function + searches for a pattern of files or directories under a + specified path and creates a package for each one it finds + by appending to the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink> + variable and setting the appropriate values for + <filename>FILES_packagename</filename>, + <filename>RDEPENDS_packagename</filename>, + <filename>DESCRIPTION_packagename</filename>, and so forth. + Here is an example from the <filename>lighttpd</filename> + recipe: + <literallayout class='monospaced'> + python populate_packages_prepend () { + lighttpd_libdir = d.expand('${libdir}') + do_split_packages(d, lighttpd_libdir, '^mod_(.*)\.so$', + 'lighttpd-module-%s', 'Lighttpd module for %s', + extra_depends='') + } + </literallayout> + The previous example specifies a number of things in the + call to <filename>do_split_packages</filename>. + <itemizedlist> + <listitem><para>A directory within the files installed + by your recipe through <filename>do_install</filename> + in which to search.</para></listitem> + <listitem><para>A regular expression used to match module + files in that directory. + In the example, note the parentheses () that mark + the part of the expression from which the module + name should be derived.</para></listitem> + <listitem><para>A pattern to use for the package names. + </para></listitem> + <listitem><para>A description for each package. + </para></listitem> + <listitem><para>An empty string for + <filename>extra_depends</filename>, which disables + the default dependency on the main + <filename>lighttpd</filename> package. + Thus, if a file in <filename>${libdir}</filename> + called <filename>mod_alias.so</filename> is found, + a package called <filename>lighttpd-module-alias</filename> + is created for it and the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DESCRIPTION'><filename>DESCRIPTION</filename></ulink> + is set to "Lighttpd module for alias".</para></listitem> + </itemizedlist> + </para> + + <para> + Often, packaging modules is as simple as the previous + example. + However, more advanced options exist that you can use + within <filename>do_split_packages</filename> to modify its + behavior. + And, if you need to, you can add more logic by specifying + a hook function that is called for each package. + It is also perfectly acceptable to call + <filename>do_split_packages</filename> multiple times if + you have more than one set of modules to package. + </para> + + <para> + For more examples that show how to use + <filename>do_split_packages</filename>, see the + <filename>connman.inc</filename> file in the + <filename>meta/recipes-connectivity/connman/</filename> + directory of the <filename>poky</filename> + <link linkend='yocto-project-repositories'>source repository</link>. + You can also find examples in + <filename>meta/classes/kernel.bbclass</filename>. + </para> + + <para> + Following is a reference that shows + <filename>do_split_packages</filename> mandatory and + optional arguments: + <literallayout class='monospaced'> + Mandatory arguments + + root + The path in which to search + file_regex + Regular expression to match searched files. + Use parentheses () to mark the part of this + expression that should be used to derive the + module name (to be substituted where %s is + used in other function arguments as noted below) + output_pattern + Pattern to use for the package names. Must + include %s. + description + Description to set for each package. Must + include %s. + + Optional arguments + + postinst + Postinstall script to use for all packages + (as a string) + recursive + True to perform a recursive search - default + False + hook + A hook function to be called for every match. + The function will be called with the following + arguments (in the order listed): + + f + Full path to the file/directory match + pkg + The package name + file_regex + As above + output_pattern + As above + modulename + The module name derived using file_regex + + extra_depends + Extra runtime dependencies (RDEPENDS) to be + set for all packages. The default value of None + causes a dependency on the main package + (${PN}) - if you do not want this, pass empty + string '' for this parameter. + aux_files_pattern + Extra item(s) to be added to FILES for each + package. Can be a single string item or a list + of strings for multiple items. Must include %s. + postrm + postrm script to use for all packages (as a + string) + allow_dirs + True to allow directories to be matched - + default False + prepend + If True, prepend created packages to PACKAGES + instead of the default False which appends them + match_path + match file_regex on the whole relative path to + the root rather than just the file name + aux_files_pattern_verbatim + Extra item(s) to be added to FILES for each + package, using the actual derived module name + rather than converting it to something legal + for a package name. Can be a single string item + or a list of strings for multiple items. Must + include %s. + allow_links + True to allow symlinks to be matched - default + False + summary + Summary to set for each package. Must include %s; + defaults to description if not set. + </literallayout> + </para> + </section> + + <section id='satisfying-dependencies'> + <title>Satisfying Dependencies</title> + + <para> + The second part for handling optional module packaging + is to ensure that any dependencies on optional modules + from other recipes are satisfied by your recipe. + You can be sure these dependencies are satisfied by + using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES_DYNAMIC'><filename>PACKAGES_DYNAMIC</filename></ulink> variable. + Here is an example that continues with the + <filename>lighttpd</filename> recipe shown earlier: + <literallayout class='monospaced'> + PACKAGES_DYNAMIC = "lighttpd-module-.*" + </literallayout> + The name specified in the regular expression can of + course be anything. + In this example, it is <filename>lighttpd-module-</filename> + and is specified as the prefix to ensure that any + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-RRECOMMENDS'><filename>RRECOMMENDS</filename></ulink> + on a package name starting with the prefix are satisfied + during build time. + If you are using <filename>do_split_packages</filename> + as described in the previous section, the value you put in + <filename>PACKAGES_DYNAMIC</filename> should correspond to + the name pattern specified in the call to + <filename>do_split_packages</filename>. + </para> + </section> + </section> + + <section id='using-runtime-package-management'> + <title>Using Runtime Package Management</title> + + <para> + During a build, BitBake always transforms a recipe into one or + more packages. + For example, BitBake takes the <filename>bash</filename> recipe + and currently produces the <filename>bash-dbg</filename>, + <filename>bash-staticdev</filename>, + <filename>bash-dev</filename>, <filename>bash-doc</filename>, + <filename>bash-locale</filename>, and + <filename>bash</filename> packages. + Not all generated packages are included in an image. + </para> + + <para> + In several situations, you might need to update, add, remove, + or query the packages on a target device at runtime + (i.e. without having to generate a new image). + Examples of such situations include: + <itemizedlist> + <listitem><para> + You want to provide in-the-field updates to deployed + devices (e.g. security updates). + </para></listitem> + <listitem><para> + You want to have a fast turn-around development cycle + for one or more applications that run on your device. + </para></listitem> + <listitem><para> + You want to temporarily install the "debug" packages + of various applications on your device so that + debugging can be greatly improved by allowing + access to symbols and source debugging. + </para></listitem> + <listitem><para> + You want to deploy a more minimal package selection of + your device but allow in-the-field updates to add a + larger selection for customization. + </para></listitem> + </itemizedlist> + </para> + + <para> + In all these situations, you have something similar to a more + traditional Linux distribution in that in-field devices + are able to receive pre-compiled packages from a server for + installation or update. + Being able to install these packages on a running, + in-field device is what is termed "runtime package + management". + </para> + + <para> + In order to use runtime package management, you + need a host/server machine that serves up the pre-compiled + packages plus the required metadata. + You also need package manipulation tools on the target. + The build machine is a likely candidate to act as the server. + However, that machine does not necessarily have to be the + package server. + The build machine could push its artifacts to another machine + that acts as the server (e.g. Internet-facing). + </para> + + <para> + A simple build that targets just one device produces + more than one package database. + In other words, the packages produced by a build are separated + out into a couple of different package groupings based on + criteria such as the target's CPU architecture, the target + board, or the C library used on the target. + For example, a build targeting the <filename>qemuarm</filename> + device produces the following three package databases: + <filename>all</filename>, <filename>armv5te</filename>, and + <filename>qemuarm</filename>. + If you wanted your <filename>qemuarm</filename> device to be + aware of all the packages that were available to it, + you would need to point it to each of these databases + individually. + In a similar way, a traditional Linux distribution usually is + configured to be aware of a number of software repositories + from which it retrieves packages. + </para> + + <para> + Using runtime package management is completely optional and + not required for a successful build or deployment in any + way. + But if you want to make use of runtime package management, + you need to do a couple things above and beyond the basics. + The remainder of this section describes what you need to do. + </para> + + <section id='runtime-package-management-build'> + <title>Build Considerations</title> + + <para> + This section describes build considerations that you need + to be aware of in order to provide support for runtime + package management. + </para> + + <para> + When BitBake generates packages it needs to know + what format or formats to use. + In your configuration, you use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink> + variable to specify the format. + <note> + You can choose to have more than one format but you must + provide at least one. + </note> + </para> + + <para> + If you would like your image to start off with a basic + package database of the packages in your current build + as well as have the relevant tools available on the + target for runtime package management, you can include + "package-management" in the + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + variable. + Including "package-management" in this + configuration variable ensures that when the image + is assembled for your target, the image includes + the currently-known package databases as well as + the target-specific tools required for runtime + package management to be performed on the target. + However, this is not strictly necessary. + You could start your image off without any databases + but only include the required on-target package + tool(s). + As an example, you could include "opkg" in your + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink> + variable if you are using the IPK package format. + You can then initialize your target's package database(s) + later once your image is up and running. + </para> + + <para> + Whenever you perform any sort of build step that can + potentially generate a package or modify an existing + package, it is always a good idea to re-generate the + package index with: + <literallayout class='monospaced'> + $ bitbake package-index + </literallayout> + Realize that it is not sufficient to simply do the + following: + <literallayout class='monospaced'> + $ bitbake <replaceable>some-package</replaceable> package-index + </literallayout> + This is because BitBake does not properly schedule the + <filename>package-index</filename> target fully after any + other target has completed. + Thus, be sure to run the package update step separately. + </para> + + <para> + As described below in the + "<link linkend='runtime-package-management-target-ipk'>Using IPK</link>" + section, if you are using IPK as your package format, you + can make use of the + <filename>distro-feed-configs</filename> recipe provided + by <filename>meta-oe</filename> in order to configure your + target to use your IPK databases. + </para> + + <para> + When your build is complete, your packages reside in the + <filename>${TMPDIR}/deploy/<replaceable>package-format</replaceable></filename> + directory. + For example, if <filename>${TMPDIR}</filename> + is <filename>tmp</filename> and your selected package type + is IPK, then your IPK packages are available in + <filename>tmp/deploy/ipk</filename>. + </para> + </section> + + <section id='runtime-package-management-server'> + <title>Host or Server Machine Setup</title> + + <para> + Typically, packages are served from a server using + HTTP. + However, other protocols are possible. + If you want to use HTTP, then setup and configure a + web server, such as Apache 2 or lighttpd, on the machine + serving the packages. + </para> + + <para> + As previously mentioned, the build machine can act as the + package server. + In the following sections that describe server machine + setups, the build machine is assumed to also be the server. + </para> + + <section id='package-server-apache'> + <title>Serving Packages via Apache 2</title> + + <para> + This example assumes you are using the Apache 2 + server: + <orderedlist> + <listitem><para> + Add the directory to your Apache + configuration, which you can find at + <filename>/etc/httpd/conf/httpd.conf</filename>. + Use commands similar to these on the + development system. + These example commands assume a top-level + <link linkend='source-directory'>Source Directory</link> + named <filename>poky</filename> in your home + directory. + The example also assumes an RPM package type. + If you are using a different package type, such + as IPK, use "ipk" in the pathnames: + <literallayout class='monospaced'> + <VirtualHost *:80> + .... + Alias /rpm ~/poky/build/tmp/deploy/rpm + <Directory "~/poky/build/tmp/deploy/rpm"> + Options +Indexes + </Directory> + </VirtualHost> + </literallayout></para></listitem> + <listitem><para> + Reload the Apache configuration as described + in this step. + For all commands, be sure you have root + privileges. + </para> + + <para> + If your development system is using Fedora or + CentOS, use the following: + <literallayout class='monospaced'> + # service httpd reload + </literallayout> + For Ubuntu and Debian, use the following: + <literallayout class='monospaced'> + # /etc/init.d/apache2 reload + </literallayout> + For OpenSUSE, use the following: + <literallayout class='monospaced'> + # /etc/init.d/apache2 reload + </literallayout></para></listitem> + <listitem><para> + If you are using Security-Enhanced Linux + (SELinux), you need to label the files as + being accessible through Apache. + Use the following command from the development + host. + This example assumes RPM package types: + <literallayout class='monospaced'> + # chcon -R -h -t httpd_sys_content_t tmp/deploy/rpm + </literallayout></para></listitem> + </orderedlist> + </para> + </section> + + <section id='package-server-lighttpd'> + <title>Serving Packages via lighttpd</title> + + <para> + If you are using lighttpd, all you need + to do is to provide a link from your + <filename>${TMPDIR}/deploy/<replaceable>package-format</replaceable></filename> + directory to lighttpd's document-root. + You can determine the specifics of your lighttpd + installation by looking through its configuration file, + which is usually found at: + <filename>/etc/lighttpd/lighttpd.conf</filename>. + </para> + + <para> + For example, if you are using IPK, lighttpd's + document-root is set to + <filename>/var/www/lighttpd</filename>, and you had + packages for a target named "BOARD", + then you might create a link from your build location + to lighttpd's document-root as follows: + <literallayout class='monospaced'> + # ln -s $(PWD)/tmp/deploy/ipk /var/www/lighttpd/BOARD-dir + </literallayout> + </para> + + <para> + At this point, you need to start the lighttpd server. + The method used to start the server varies by + distribution. + However, one basic method that starts it by hand is: + <literallayout class='monospaced'> + # lighttpd -f /etc/lighttpd/lighttpd.conf + </literallayout> + </para> + </section> + </section> + + <section id='runtime-package-management-target'> + <title>Target Setup</title> + + <para> + Setting up the target differs depending on the + package management system. + This section provides information for RPM and IPK. + </para> + + <section id='runtime-package-management-target-rpm'> + <title>Using RPM</title> + + <para> + The application for performing runtime package + management of RPM packages on the target is called + <filename>smart</filename>. + </para> + + <para> + On the target machine, you need to inform + <filename>smart</filename> of every package database + you want to use. + As an example, suppose your target device can use the + following three package databases from a server named + <filename>server.name</filename>: + <filename>all</filename>, <filename>i586</filename>, + and <filename>qemux86</filename>. + Given this example, issue the following commands on the + target: + <literallayout class='monospaced'> + # smart channel --add all type=rpm-md baseurl=http://server.name/rpm/all + # smart channel --add i585 type=rpm-md baseurl=http://server.name/rpm/i586 + # smart channel --add qemux86 type=rpm-md baseurl=http://server.name/rpm/qemux86 + </literallayout> + Also from the target machine, fetch the repository + information using this command: + <literallayout class='monospaced'> + # smart update + </literallayout> + You can now use the <filename>smart query</filename> + and <filename>smart install</filename> commands to + find and install packages from the repositories. + </para> + </section> + + <section id='runtime-package-management-target-ipk'> + <title>Using IPK</title> + + <para> + The application for performing runtime package + management of IPK packages on the target is called + <filename>opkg</filename>. + </para> + + <para> + In order to inform <filename>opkg</filename> of the + package databases you want to use, simply create one + or more <filename>*.conf</filename> files in the + <filename>/etc/opkg</filename> directory on the target. + The <filename>opkg</filename> application uses them + to find its available package databases. + As an example, suppose you configured your HTTP server + on your machine named + <filename>www.mysite.com</filename> to serve files + from a <filename>BOARD-dir</filename> directory under + its document-root. + In this case, you might create a configuration + file on the target called + <filename>/etc/opkg/base-feeds.conf</filename> that + contains: + <literallayout class='monospaced'> + src/gz all http://www.mysite.com/BOARD-dir/all + src/gz armv7a http://www.mysite.com/BOARD-dir/armv7a + src/gz beaglebone http://www.mysite.com/BOARD-dir/beaglebone + </literallayout> + </para> + + <para> + As a way of making it easier to generate and make + these IPK configuration files available on your + target, simply define + <ulink url='&YOCTO_DOCS_REF_URL;#var-FEED_DEPLOYDIR_BASE_URI'><filename>FEED_DEPLOYDIR_BASE_URI</filename></ulink> + to point to your server and the location within the + document-root which contains the databases. + For example: if you are serving your packages over + HTTP, your server's IP address is 192.168.7.1, and + your databases are located in a directory called + <filename>BOARD-dir</filename> underneath your HTTP + server's document-root, you need to set + <filename>FEED_DEPLOYDIR_BASE_URI</filename> to + <filename>http://192.168.7.1/BOARD-dir</filename> and + a set of configuration files will be generated for you + in your target to work with this feed. + </para> + + <para> + On the target machine, fetch (or refresh) the + repository information using this command: + <literallayout class='monospaced'> + # opkg update + </literallayout> + You can now use the <filename>opkg list</filename> and + <filename>opkg install</filename> commands to find and + install packages from the repositories. + </para> + </section> + </section> + </section> + + <section id='testing-packages-with-ptest'> + <title>Testing Packages With ptest</title> + + <para> + A Package Test (ptest) runs tests against packages built + by the OpenEmbedded build system on the target machine. + A ptest contains at least two items: the actual test, and + a shell script (<filename>run-ptest</filename>) that starts + the test. + The shell script that starts the test must not contain + the actual test - the script only starts the test. + On the other hand, the test can be anything from a simple + shell script that runs a binary and checks the output to + an elaborate system of test binaries and data files. + </para> + + <para> + The test generates output in the format used by + Automake: + <literallayout class='monospaced'> + <replaceable>result</replaceable>: <replaceable>testname</replaceable> + </literallayout> + where the result can be <filename>PASS</filename>, + <filename>FAIL</filename>, or <filename>SKIP</filename>, + and the testname can be any identifying string. + </para> + + <para> + For a list of Yocto Project recipes that are already + enabled with ptest, see the + <ulink url='https://wiki.yoctoproject.org/wiki/Ptest'>Ptest</ulink> + wiki page. + <note> + A recipe is "ptest-enabled" if it inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-ptest'><filename>ptest</filename></ulink> + class. + </note> + </para> + + <section id='adding-ptest-to-your-build'> + <title>Adding ptest to Your Build</title> + + <para> + To add package testing to your build, add the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink> + variables to your <filename>local.conf</filename> file, + which is found in the + <link linkend='build-directory'>Build Directory</link>: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " ptest" + EXTRA_IMAGE_FEATURES += "ptest-pkgs" + </literallayout> + Once your build is complete, the ptest files are installed + into the + <filename>/usr/lib/<replaceable>package</replaceable>/ptest</filename> + directory within the image, where + <filename><replaceable>package</replaceable></filename> + is the name of the package. + </para> + </section> + + <section id='running-ptest'> + <title>Running ptest</title> + + <para> + The <filename>ptest-runner</filename> package installs a + shell script that loops through all installed ptest test + suites and runs them in sequence. + Consequently, you might want to add this package to + your image. + </para> + </section> + + <section id='getting-your-package-ready'> + <title>Getting Your Package Ready</title> + + <para> + In order to enable a recipe to run installed ptests + on target hardware, + you need to prepare the recipes that build the packages + you want to test. + Here is what you have to do for each recipe: + <itemizedlist> + <listitem><para><emphasis>Be sure the recipe + inherits the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-ptest'><filename>ptest</filename></ulink> + class:</emphasis> + Include the following line in each recipe: + <literallayout class='monospaced'> + inherit ptest + </literallayout> + </para></listitem> + <listitem><para><emphasis>Create <filename>run-ptest</filename>:</emphasis> + This script starts your test. + Locate the script where you will refer to it + using + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>. + Here is an example that starts a test for + <filename>dbus</filename>: + <literallayout class='monospaced'> + #!/bin/sh + cd test + make -k runtest-TESTS + </literallayout> + </para></listitem> + <listitem><para><emphasis>Ensure dependencies are + met:</emphasis> + If the test adds build or runtime dependencies + that normally do not exist for the package + (such as requiring "make" to run the test suite), + use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPENDS'><filename>DEPENDS</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink> + variables in your recipe in order for the package + to meet the dependencies. + Here is an example where the package has a runtime + dependency on "make": + <literallayout class='monospaced'> + RDEPENDS_${PN}-ptest += "make" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Add a function to build the + test suite:</emphasis> + Not many packages support cross-compilation of + their test suites. + Consequently, you usually need to add a + cross-compilation function to the package. + </para> + + <para>Many packages based on Automake compile and + run the test suite by using a single command + such as <filename>make check</filename>. + However, the host <filename>make check</filename> + builds and runs on the same computer, while + cross-compiling requires that the package is built + on the host but executed for the target + architecture (though often, as in the case for + ptest, the execution occurs on the host). + The built version of Automake that ships with the + Yocto Project includes a patch that separates + building and execution. + Consequently, packages that use the unaltered, + patched version of <filename>make check</filename> + automatically cross-compiles.</para> + <para>Regardless, you still must add a + <filename>do_compile_ptest</filename> function to + build the test suite. + Add a function similar to the following to your + recipe: + <literallayout class='monospaced'> + do_compile_ptest() { + oe_runmake buildtest-TESTS + } + </literallayout> + </para></listitem> + <listitem><para><emphasis>Ensure special configurations + are set:</emphasis> + If the package requires special configurations + prior to compiling the test code, you must + insert a <filename>do_configure_ptest</filename> + function into the recipe. + </para></listitem> + <listitem><para><emphasis>Install the test + suite:</emphasis> + The <filename>ptest</filename> class + automatically copies the file + <filename>run-ptest</filename> to the target and + then runs make <filename>install-ptest</filename> + to run the tests. + If this is not enough, you need to create a + <filename>do_install_ptest</filename> function and + make sure it gets called after the + "make install-ptest" completes. + </para></listitem> + </itemizedlist> + </para> + </section> + </section> + </section> + + <section id='working-with-source-files'> + <title>Working with Source Files</title> + + <para> + The OpenEmbedded build system works with source files located + through the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable. + When you build something using BitBake, a big part of the operation + is locating and downloading all the source tarballs. + For images, downloading all the source for various packages can + take a significant amount of time. + </para> + + <para> + This section presents information for working with source + files that can lead to more efficient use of resources and + time. + </para> + + <section id='setting-up-effective-mirrors'> + <title>Setting up Effective Mirrors</title> + + <para> + As mentioned, a good deal that goes into a Yocto Project + build is simply downloading all of the source tarballs. + Maybe you have been working with another build system + (OpenEmbedded or Angstrom) for which you have built up a + sizable directory of source tarballs. + Or, perhaps someone else has such a directory for which you + have read access. + If so, you can save time by adding statements to your + configuration file so that the build process checks local + directories first for existing tarballs before checking the + Internet. + </para> + + <para> + Here is an efficient way to set it up in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/" + INHERIT += "own-mirrors" + BB_GENERATE_MIRROR_TARBALLS = "1" + # BB_NO_NETWORK = "1" + </literallayout> + </para> + + <para> + In the previous example, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink> + variable causes the OpenEmbedded build system to generate + tarballs of the Git repositories and store them in the + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> + directory. + Due to performance reasons, generating and storing these + tarballs is not the build system's default behavior. + </para> + + <para> + You can also use the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink> + variable. + For an example, see the variable's glossary entry in the + Yocto Project Reference Manual. + </para> + </section> + + <section id='getting-source-files-and-suppressing-the-build'> + <title>Getting Source Files and Suppressing the Build</title> + + <para> + Another technique you can use to ready yourself for a + successive string of build operations, is to pre-fetch + all the source files without actually starting a build. + This technique lets you work through any download issues + and ultimately gathers all the source files into your + download directory + <ulink url='&YOCTO_DOCS_REF_URL;#structure-build-downloads'><filename>build/downloads</filename></ulink>, + which is located with + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>. + </para> + + <para> + Use the following BitBake command form to fetch all the + necessary sources without starting the build: + <literallayout class='monospaced'> + $ bitbake -c fetchall <replaceable>target</replaceable> + </literallayout> + This variation of the BitBake command guarantees that you + have all the sources for that BitBake target should you + disconnect from the Internet and want to do the build + later offline. + </para> + </section> + </section> + + <section id="building-software-from-an-external-source"> + <title>Building Software from an External Source</title> + + <para> + By default, the OpenEmbedded build system uses the + <link linkend='build-directory'>Build Directory</link> when + building source code. + The build process involves fetching the source files, unpacking + them, and then patching them if necessary before the build takes + place. + </para> + + <para> + Situations exist where you might want to build software from source + files that are external to and thus outside of the + OpenEmbedded build system. + For example, suppose you have a project that includes a new BSP with + a heavily customized kernel. + And, you want to minimize exposing the build system to the + development team so that they can focus on their project and + maintain everyone's workflow as much as possible. + In this case, you want a kernel source directory on the development + machine where the development occurs. + You want the recipe's + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + variable to point to the external directory and use it as is, not + copy it. + </para> + + <para> + To build from software that comes from an external source, all you + need to do is inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class and then set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink> + variable to point to your external source code. + Here are the statements to put in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + INHERIT += "externalsrc" + EXTERNALSRC_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + + <para> + This next example shows how to accomplish the same thing by setting + <filename>EXTERNALSRC</filename> in the recipe itself or in the + recipe's append file: + <literallayout class='monospaced'> + EXTERNALSRC = "<replaceable>path</replaceable>" + EXTERNALSRC_BUILD = "<replaceable>path</replaceable>" + </literallayout> + <note> + In order for these settings to take effect, you must globally + or locally inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink> + class. + </note> + </para> + + <para> + By default, <filename>externalsrc.bbclass</filename> builds + the source code in a directory separate from the external source + directory as specified by + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC'><filename>EXTERNALSRC</filename></ulink>. + If you need to have the source built in the same directory in + which it resides, or some other nominated directory, you can set + <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTERNALSRC_BUILD'><filename>EXTERNALSRC_BUILD</filename></ulink> + to point to that directory: + <literallayout class='monospaced'> + EXTERNALSRC_BUILD_pn-<replaceable>myrecipe</replaceable> = "<replaceable>path-to-your-source-tree</replaceable>" + </literallayout> + </para> + </section> + + <section id="selecting-an-initialization-manager"> + <title>Selecting an Initialization Manager</title> + + <para> + By default, the Yocto Project uses SysVinit as the initialization + manager. + However, support also exists for systemd, + which is a full replacement for init with + parallel starting of services, reduced shell overhead and other + features that are used by many distributions. + </para> + + <para> + If you want to use SysVinit, you do + not have to do anything. + But, if you want to use systemd, you must + take some steps as described in the following sections. + </para> + + <section id='using-systemd-exclusively'> + <title>Using systemd Exclusively</title> + + <para> + Set the these variables in your distribution configuration + file as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " systemd" + VIRTUAL-RUNTIME_init_manager = "systemd" + </literallayout> + You can also prevent the SysVinit + distribution feature from + being automatically enabled as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit" + </literallayout> + Doing so removes any redundant SysVinit scripts. + </para> + + <para> + To remove initscripts from your image altogether, + set this variable also: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_initscripts = "" + </literallayout> + </para> + + <para> + For information on the backfill variable, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES_BACKFILL_CONSIDERED'><filename>DISTRO_FEATURES_BACKFILL_CONSIDERED</filename></ulink>. + </para> + </section> + + <section id='using-systemd-for-the-main-image-and-using-sysvinit-for-the-rescue-image'> + <title>Using systemd for the Main Image and Using SysVinit for the Rescue Image</title> + + <para> + Set these variables in your distribution configuration + file as follows: + <literallayout class='monospaced'> + DISTRO_FEATURES_append = " systemd" + VIRTUAL-RUNTIME_init_manager = "systemd" + </literallayout> + Doing so causes your main image to use the + <filename>packagegroup-core-boot.bb</filename> recipe and + systemd. + The rescue/minimal image cannot use this package group. + However, it can install SysVinit + and the appropriate packages will have support for both + systemd and SysVinit. + </para> + </section> + </section> + + <section id="selecting-dev-manager"> + <title>Selecting a Device Manager</title> + + <para> + The Yocto Project provides multiple ways to manage the device + manager (<filename>/dev</filename>): + <itemizedlist> + <listitem><para><emphasis>Persistent and Pre-Populated<filename>/dev</filename>:</emphasis> + For this case, the <filename>/dev</filename> directory + is persistent and the required device nodes are created + during the build. + </para></listitem> + <listitem><para><emphasis>Use <filename>devtmpfs</filename> with a Device Manager:</emphasis> + For this case, the <filename>/dev</filename> directory + is provided by the kernel as an in-memory file system and + is automatically populated by the kernel at runtime. + Additional configuration of device nodes is done in user + space by a device manager like + <filename>udev</filename> or + <filename>busybox-mdev</filename>. + </para></listitem> + </itemizedlist> + </para> + + <section id="static-dev-management"> + <title>Using Persistent and Pre-Populated<filename>/dev</filename></title> + + <para> + To use the static method for device population, you need to + set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-USE_DEVFS'><filename>USE_DEVFS</filename></ulink> + variable to "0" as follows: + <literallayout class='monospaced'> + USE_DEVFS = "0" + </literallayout> + </para> + + <para> + The content of the resulting <filename>/dev</filename> + directory is defined in a Device Table file. + The + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_DEVICE_TABLES'><filename>IMAGE_DEVICE_TABLES</filename></ulink> + variable defines the Device Table to use and should be set + in the machine or distro configuration file. + Alternatively, you can set this variable in your + <filename>local.conf</filename> configuration file. + </para> + + <para> + If you do not define the + <filename>IMAGE_DEVICE_TABLES</filename> variable, the default + <filename>device_table-minimal.txt</filename> is used: + <literallayout class='monospaced'> + IMAGE_DEVICE_TABLES = "device_table-mymachine.txt" + </literallayout> + </para> + + <para> + The population is handled by the <filename>makedevs</filename> + utility during image creation: + </para> + </section> + + <section id="devtmpfs-dev-management"> + <title>Using <filename>devtmpfs</filename> and a Device Manager</title> + + <para> + To use the dynamic method for device population, you need to + use (or be sure to set) the + <ulink url='&YOCTO_DOCS_REF_URL;#var-USE_DEVFS'><filename>USE_DEVFS</filename></ulink> + variable to "1", which is the default: + <literallayout class='monospaced'> + USE_DEVFS = "1" + </literallayout> + With this setting, the resulting <filename>/dev</filename> + directory is populated by the kernel using + <filename>devtmpfs</filename>. + Make sure the corresponding kernel configuration variable + <filename>CONFIG_DEVTMPFS</filename> is set when building + you build a Linux kernel. + </para> + + <para> + All devices created by <filename>devtmpfs</filename> will be + owned by <filename>root</filename> and have permissions + <filename>0600</filename>. + </para> + + <para> + To have more control over the device nodes, you can use a + device manager like <filename>udev</filename> or + <filename>busybox-mdev</filename>. + You choose the device manager by defining the + <filename>VIRTUAL-RUNTIME_dev_manager</filename> variable + in your machine or distro configuration file. + Alternatively, you can set this variable in your + <filename>local.conf</filename> configuration file: + <literallayout class='monospaced'> + VIRTUAL-RUNTIME_dev_manager = "udev" + + # Some alternative values + # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev" + # VIRTUAL-RUNTIME_dev_manager = "systemd" + </literallayout> + </para> + </section> + </section> + + <section id="platdev-appdev-srcrev"> + <title>Using an External SCM</title> + + <para> + If you're working on a recipe that pulls from an external Source + Code Manager (SCM), it is possible to have the OpenEmbedded build + system notice new recipe changes added to the SCM and then build + the resulting packages that depend on the new recipes by using + the latest versions. + This only works for SCMs from which it is possible to get a + sensible revision number for changes. + Currently, you can do this with Apache Subversion (SVN), Git, and + Bazaar (BZR) repositories. + </para> + + <para> + To enable this behavior, the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> + of the recipe needs to reference + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCPV'><filename>SRCPV</filename></ulink>. + Here is an example: + <literallayout class='monospaced'> + PV = "1.2.3+git${SRCPV}" + </literallayout> + Then, you can add the following to your + <filename>local.conf</filename>: + <literallayout class='monospaced'> + SRCREV_pn-<replaceable>PN</replaceable> = "${AUTOREV}" + </literallayout> + <ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> + is the name of the recipe for which you want to enable automatic source + revision updating. + </para> + + <para> + If you do not want to update your local configuration file, you can + add the following directly to the recipe to finish enabling + the feature: + <literallayout class='monospaced'> + SRCREV = "${AUTOREV}" + </literallayout> + </para> + + <para> + The Yocto Project provides a distribution named + <filename>poky-bleeding</filename>, whose configuration + file contains the line: + <literallayout class='monospaced'> + require conf/distro/include/poky-floating-revisions.inc + </literallayout> + This line pulls in the listed include file that contains + numerous lines of exactly that form: + <literallayout class='monospaced'> + #SRCREV_pn-opkg-native ?= "${AUTOREV}" + #SRCREV_pn-opkg-sdk ?= "${AUTOREV}" + #SRCREV_pn-opkg ?= "${AUTOREV}" + #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}" + #SRCREV_pn-opkg-utils ?= "${AUTOREV}" + SRCREV_pn-gconf-dbus ?= "${AUTOREV}" + SRCREV_pn-matchbox-common ?= "${AUTOREV}" + SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}" + SRCREV_pn-matchbox-desktop ?= "${AUTOREV}" + SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}" + SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}" + SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}" + SRCREV_pn-matchbox-terminal ?= "${AUTOREV}" + SRCREV_pn-matchbox-wm ?= "${AUTOREV}" + SRCREV_pn-settings-daemon ?= "${AUTOREV}" + SRCREV_pn-screenshot ?= "${AUTOREV}" + . + . + . + </literallayout> + These lines allow you to experiment with building a + distribution that tracks the latest development source + for numerous packages. + <note><title>Caution</title> + The <filename>poky-bleeding</filename> distribution + is not tested on a regular basis. + Keep this in mind if you use it. + </note> + </para> + </section> + + <section id='creating-a-read-only-root-filesystem'> + <title>Creating a Read-Only Root Filesystem</title> + + <para> + Suppose, for security reasons, you need to disable + your target device's root filesystem's write permissions + (i.e. you need a read-only root filesystem). + Or, perhaps you are running the device's operating system + from a read-only storage device. + For either case, you can customize your image for + that behavior. + </para> + + <note> + Supporting a read-only root filesystem requires that the system and + applications do not try to write to the root filesystem. + You must configure all parts of the target system to write + elsewhere, or to gracefully fail in the event of attempting to + write to the root filesystem. + </note> + + <section id='creating-the-root-filesystem'> + <title>Creating the Root Filesystem</title> + + <para> + To create the read-only root filesystem, simply add the + "read-only-rootfs" feature to your image. + Using either of the following statements in your + image recipe or from within the + <filename>local.conf</filename> file found in the + <link linkend='build-directory'>Build Directory</link> + causes the build system to create a read-only root filesystem: + <literallayout class='monospaced'> + IMAGE_FEATURES = "read-only-rootfs" + </literallayout> + or + <literallayout class='monospaced'> + EXTRA_IMAGE_FEATURES += "read-only-rootfs" + </literallayout> + </para> + + <para> + For more information on how to use these variables, see the + "<link linkend='usingpoky-extend-customimage-imagefeatures'>Customizing Images Using Custom <filename>IMAGE_FEATURES</filename> and <filename>EXTRA_IMAGE_FEATURES</filename></link>" + section. + For information on the variables, see + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + and <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_IMAGE_FEATURES'><filename>EXTRA_IMAGE_FEATURES</filename></ulink>. + </para> + </section> + + <section id='post-installation-scripts'> + <title>Post-Installation Scripts</title> + + <para> + It is very important that you make sure all + post-Installation (<filename>pkg_postinst</filename>) scripts + for packages that are installed into the image can be run + at the time when the root filesystem is created during the + build on the host system. + These scripts cannot attempt to run during first-boot on the + target device. + With the "read-only-rootfs" feature enabled, + the build system checks during root filesystem creation to make + sure all post-installation scripts succeed. + If any of these scripts still need to be run after the root + filesystem is created, the build immediately fails. + These build-time checks ensure that the build fails + rather than the target device fails later during its + initial boot operation. + </para> + + <para> + Most of the common post-installation scripts generated by the + build system for the out-of-the-box Yocto Project are engineered + so that they can run during root filesystem creation + (e.g. post-installation scripts for caching fonts). + However, if you create and add custom scripts, you need + to be sure they can be run during this file system creation. + </para> + + <para> + Here are some common problems that prevent + post-installation scripts from running during root filesystem + creation: + <itemizedlist> + <listitem><para> + <emphasis>Not using $D in front of absolute + paths:</emphasis> + The build system defines + <filename>$</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> + when the root filesystem is created. + Furthermore, <filename>$D</filename> is blank when the + script is run on the target device. + This implies two purposes for <filename>$D</filename>: + ensuring paths are valid in both the host and target + environments, and checking to determine which + environment is being used as a method for taking + appropriate actions. + </para></listitem> + <listitem><para> + <emphasis>Attempting to run processes that are + specific to or dependent on the target + architecture:</emphasis> + You can work around these attempts by using native + tools, which run on the host system, + to accomplish the same tasks, or + by alternatively running the processes under QEMU, + which has the <filename>qemu_run_binary</filename> + function. + For more information, see the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-qemu'><filename>qemu</filename></ulink> + class.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id='areas-with-write-access'> + <title>Areas With Write Access</title> + + <para> + With the "read-only-rootfs" feature enabled, + any attempt by the target to write to the root filesystem at + runtime fails. + Consequently, you must make sure that you configure processes + and applications that attempt these types of writes do so + to directories with write access (e.g. + <filename>/tmp</filename> or <filename>/var/run</filename>). + </para> + </section> + </section> + + <section id="performing-automated-runtime-testing"> + <title>Performing Automated Runtime Testing</title> + + <para> + The OpenEmbedded build system makes available a series of automated + tests for images to verify runtime functionality. + You can run these tests on either QEMU or actual target hardware. + Tests are written in Python making use of the + <filename>unittest</filename> module, and the majority of them + run commands on the target system over SSH. + This section describes how you set up the environment to use these + tests, run available tests, and write and add your own tests. + </para> + + <section id='enabling-tests'> + <title>Enabling Tests</title> + + <para> + Depending on whether you are planning to run tests using + QEMU or on the hardware, you have to take + different steps to enable the tests. + See the following subsections for information on how to + enable both types of tests. + </para> + + <section id='qemu-image-enabling-tests'> + <title>Enabling Runtime Tests on QEMU</title> + + <para> + In order to run tests, you need to do the following: + <itemizedlist> + <listitem><para><emphasis>Set up to avoid interaction + with <filename>sudo</filename> for networking:</emphasis> + To accomplish this, you must do one of the + following: + <itemizedlist> + <listitem><para>Add + <filename>NOPASSWD</filename> for your user + in <filename>/etc/sudoers</filename> either for + all commands or just for + <filename>runqemu-ifup</filename>. + You must provide the full path as that can + change if you are using multiple clones of the + source repository. + <note> + On some distributions, you also need to + comment out "Defaults requiretty" in + <filename>/etc/sudoers</filename>. + </note></para></listitem> + <listitem><para>Manually configure a tap interface + for your system.</para></listitem> + <listitem><para>Run as root the script in + <filename>scripts/runqemu-gen-tapdevs</filename>, + which should generate a list of tap devices. + This is the option typically chosen for + Autobuilder-type environments. + </para></listitem> + </itemizedlist></para></listitem> + <listitem><para><emphasis>Set the + <filename>DISPLAY</filename> variable:</emphasis> + You need to set this variable so that you have an X + server available (e.g. start + <filename>vncserver</filename> for a headless machine). + </para></listitem> + <listitem><para><emphasis>Be sure your host's firewall + accepts incoming connections from + 192.168.7.0/24:</emphasis> + Some of the tests (in particular smart tests) start an + HTTP server on a random high number port, which is + used to serve files to the target. + The smart module serves + <filename>${DEPLOY_DIR}/rpm</filename> so it can run + smart channel commands. That means your host's firewall + must accept incoming connections from 192.168.7.0/24, + which is the default IP range used for tap devices + by <filename>runqemu</filename>.</para></listitem> + <listitem><para><emphasis>Be sure your host has the + correct packages installed:</emphasis> + Depending your host's distribution, you need + to have the following packages installed: + <itemizedlist> + <listitem><para>Ubuntu and Debian: + <filename>sysstat</filename> and + <filename>iproute2</filename> + </para></listitem> + <listitem><para>OpenSUSE: + <filename>sysstat</filename> and + <filename>iproute2</filename> + </para></listitem> + <listitem><para>Fedora: + <filename>sysstat</filename> and + <filename>iproute</filename> + </para></listitem> + <listitem><para>CentOS: + <filename>sysstat</filename> and + <filename>iproute</filename> + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para> + + <para> + Once you start running the tests, the following happens: + <orderedlist> + <listitem><para>A copy of the root filesystem is written + to <filename>${WORKDIR}/testimage</filename>. + </para></listitem> + <listitem><para>The image is booted under QEMU using the + standard <filename>runqemu</filename> script. + </para></listitem> + <listitem><para>A default timeout of 500 seconds occurs + to allow for the boot process to reach the login prompt. + You can change the timeout period by setting + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_QEMUBOOT_TIMEOUT'><filename>TEST_QEMUBOOT_TIMEOUT</filename></ulink> + in the <filename>local.conf</filename> file. + </para></listitem> + <listitem><para>Once the boot process is reached and the + login prompt appears, the tests run. + The full boot log is written to + <filename>${WORKDIR}/testimage/qemu_boot_log</filename>. + </para></listitem> + <listitem><para>Each test module loads in the order found + in <filename>TEST_SUITES</filename>. + You can find the full output of the commands run over + SSH in + <filename>${WORKDIR}/testimgage/ssh_target_log</filename>. + </para></listitem> + <listitem><para>If no failures occur, the task running the + tests ends successfully. + You can find the output from the + <filename>unittest</filename> in the task log at + <filename>${WORKDIR}/temp/log.do_testimage</filename>. + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='hardware-image-enabling-tests'> + <title>Enabling Runtime Tests on Hardware</title> + + <para> + The OpenEmbedded build system can run tests on real + hardware, and for certain devices it can also deploy + the image to be tested onto the device beforehand. + </para> + + <para> + For automated deployment, a "master image" is installed + onto the hardware once as part of setup. + Then, each time tests are to be run, the following + occurs: + <orderedlist> + <listitem><para>The master image is booted into and + used to write the image to be tested to + a second partition. + </para></listitem> + <listitem><para>The device is then rebooted using an + external script that you need to provide. + </para></listitem> + <listitem><para>The device boots into the image to be + tested. + </para></listitem> + </orderedlist> + </para> + + <para> + When running tests (independent of whether the image + has been deployed automatically or not), the device is + expected to be connected to a network on a + pre-determined IP address. + You can either use static IP addresses written into + the image, or set the image to use DHCP and have your + DHCP server on the test network assign a known IP address + based on the MAC address of the device. + </para> + + <para> + In order to run tests on hardware, you need to set + <filename>TEST_TARGET</filename> to an appropriate value. + For QEMU, you do not have to change anything, the default + value is "QemuTarget". + For running tests on hardware, the following options exist: + <itemizedlist> + <listitem><para><emphasis>"SimpleRemoteTarget":</emphasis> + Choose "SimpleRemoteTarget" if you are going to + run tests on a target system that is already + running the image to be tested and is available + on the network. + You can use "SimpleRemoteTarget" in conjunction + with either real hardware or an image running + within a separately started QEMU or any + other virtual machine manager. + </para></listitem> + <listitem><para><emphasis>"GummibootTarget":</emphasis> + Choose "GummibootTarget" if your hardware is + an EFI-based machine with + <filename>gummiboot</filename> as bootloader and + <filename>core-image-testmaster</filename> + (or something similar) is installed. + Also, your hardware under test must be in a + DHCP-enabled network that gives it the same IP + address for each reboot.</para> + <para>If you choose "GummibootTarget", there are + additional requirements and considerations. + See the + "<link linkend='selecting-gummiboottarget'>Selecting GummibootTarget</link>" + section, which follows, for more information. + </para></listitem> + <listitem><para><emphasis>"BeagleBoneTarget":</emphasis> + Choose "BeagleBoneTarget" if you are deploying + images and running tests on the BeagleBone + "Black" or original "White" hardware. + For information on how to use these tests, see the + comments at the top of the BeagleBoneTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"EdgeRouterTarget":</emphasis> + Choose "EdgeRouterTarget" is you are deploying + images and running tests on the Ubiquiti Networks + EdgeRouter Lite. + For information on how to use these tests, see the + comments at the top of the EdgeRouterTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"GrubTarget":</emphasis> + Choose the "supports deploying images and running + tests on any generic PC that boots using GRUB. + For information on how to use these tests, see the + comments at the top of the GrubTarget + <filename>meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py</filename> + file. + </para></listitem> + <listitem><para><emphasis>"<replaceable>your-target</replaceable>":</emphasis> + Create your own custom target if you want to run + tests when you are deploying images and running + tests on a custom machine within your BSP layer. + To do this, you need to add a Python unit that + defines the target class under + <filename>lib/oeqa/controllers/</filename> within + your layer. + You must also provide an empty + <filename>__init__.py</filename>. + For examples, see files in + <filename>meta-yocto-bsp/lib/oeqa/controllers/</filename>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='selecting-gummiboottarget'> + <title>Selecting GummibootTarget</title> + + <para> + If you did not set <filename>TEST_TARGET</filename> to + "GummibootTarget", then you do not need any information + in this section. + You can skip down to the + "<link linkend='qemu-image-running-tests'>Running Tests</link>" + section. + </para> + + <para> + If you did set <filename>TEST_TARGET</filename> to + "GummibootTarget", you also need to perform a one-time + setup of your master image by doing the following: + <orderedlist> + <listitem><para><emphasis>Set <filename>EFI_PROVIDER</filename>:</emphasis> + Be sure that <filename>EFI_PROVIDER</filename> + is as follows: + <literallayout class='monospaced'> + EFI_PROVIDER = "gummiboot" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Build the master image:</emphasis> + Build the <filename>core-image-testmaster</filename> + image. + The <filename>core-image-testmaster</filename> + recipe is provided as an example for a + "master" image and you can customize the image + recipe as you would any other recipe. + </para> + <para>Here are the image recipe requirements: + <itemizedlist> + <listitem><para>Inherits + <filename>core-image</filename> + so that kernel modules are installed. + </para></listitem> + <listitem><para>Installs normal linux utilities + not busybox ones (e.g. + <filename>bash</filename>, + <filename>coreutils</filename>, + <filename>tar</filename>, + <filename>gzip</filename>, and + <filename>kmod</filename>). + </para></listitem> + <listitem><para>Uses a custom + Initial RAM Disk (initramfs) image with a + custom installer. + A normal image that you can install usually + creates a single rootfs partition. + This image uses another installer that + creates a specific partition layout. + Not all Board Support Packages (BSPs) + can use an installer. + For such cases, you need to manually create + the following partition layout on the + target: + <itemizedlist> + <listitem><para>First partition mounted + under <filename>/boot</filename>, + labeled "boot". + </para></listitem> + <listitem><para>The main rootfs + partition where this image gets + installed, which is mounted under + <filename>/</filename>. + </para></listitem> + <listitem><para>Another partition + labeled "testrootfs" where test + images get deployed. + </para></listitem> + </itemizedlist> + </para></listitem> + </itemizedlist> + </para></listitem> + <listitem><para><emphasis>Install image:</emphasis> + Install the image that you just built on the target + system. + </para></listitem> + </orderedlist> + </para> + + <para> + The final thing you need to do when setting + <filename>TEST_TARGET</filename> to "GummibootTarget" is + to set up the test image: + <orderedlist> + <listitem><para><emphasis>Set up your <filename>local.conf</filename> file:</emphasis> + Make sure you have the following statements in + your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + IMAGE_FSTYPES += "tar.gz" + INHERIT += "testimage" + TEST_TARGET = "GummibootTarget" + TEST_TARGET_IP = "192.168.2.3" + </literallayout> + </para></listitem> + <listitem><para><emphasis>Build your test image:</emphasis> + Use BitBake to build the image: + <literallayout class='monospaced'> + $ bitbake core-image-sato + </literallayout> + </para></listitem> + </orderedlist> + </para> + </section> + + <section id='power-control'> + <title>Power Control</title> + + <para> + For most hardware targets other than SimpleRemoteTarget, + you can control power: + <itemizedlist> + <listitem><para> + You can use + <filename>TEST_POWERCONTROL_CMD</filename> + together with + <filename>TEST_POWERCONTROL_EXTRA_ARGS</filename> + as a command that runs on the host and does power + cycling. + The test code passes one argument to that command: + off, on or cycle (off then on). + Here is an example that could appear in your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" + </literallayout> + In this example, the expect script does the + following: + <literallayout class='monospaced'> + ssh test@10.11.12.1 "pyctl nuc1 <replaceable>arg</replaceable>" + </literallayout> + It then runs a Python script that controls power + for a label called <filename>nuc1</filename>. + <note> + You need to customize + <filename>TEST_POWERCONTROL_CMD</filename> + and + <filename>TEST_POWERCONTROL_EXTRA_ARGS</filename> + for your own setup. + The one requirement is that it accepts + "on", "off", and "cycle" as the last argument. + </note> + </para></listitem> + <listitem><para> + When no command is defined, it connects to the + device over SSH and uses the classic reboot command + to reboot the device. + Classic reboot is fine as long as the machine + actually reboots (i.e. the SSH test has not + failed). + It is useful for scenarios where you have a simple + setup, typically with a single board, and where + some manual interaction is okay from time to time. + </para></listitem> + </itemizedlist> + If you have no hardware to automatically perform power + control but still wish to experiment with automated + hardware testing, you can use the dialog-power-control + script that shows a dialog prompting you to perform the + required power action. + This script requires either KDialog or Zenity to be + installed. + To use this script, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_POWERCONTROL_CMD'><filename>TEST_POWERCONTROL_CMD</filename></ulink> + variable as follows: + <literallayout class='monospaced'> + TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" + </literallayout> + </para> + </section> + + <section id='serial-console-connection'> + <title>Serial Console Connection</title> + + <para> + For test target classes requiring a serial console + to interact with the bootloader (e.g. BeagleBoneTarget, + EdgeRouterTarget, and GrubTarget), you need to + specify a command to use to connect to the serial console + of the target machine by using the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SERIALCONTROL_CMD'><filename>TEST_SERIALCONTROL_CMD</filename></ulink> + variable and optionally the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SERIALCONTROL_EXTRA_ARGS'><filename>TEST_SERIALCONTROL_EXTRA_ARGS</filename></ulink> + variable. + </para> + + <para> + These cases could be a serial terminal program if the + machine is connected to a local serial port, or a + <filename>telnet</filename> or + <filename>ssh</filename> command connecting to a remote + console server. + Regardless of the case, the command simply needs to + connect to the serial console and forward that connection + to standard input and output as any normal terminal + program does. + For example, to use the picocom terminal program on + serial device <filename>/dev/ttyUSB0</filename> + at 115200bps, you would set the variable as follows: + <literallayout class='monospaced'> + TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" + </literallayout> + For local devices where the serial port device disappears + when the device reboots, an additional "serdevtry" wrapper + script is provided. + To use this wrapper, simply prefix the terminal command + with + <filename>${COREBASE}/scripts/contrib/serdevtry</filename>: + <literallayout class='monospaced'> + TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b +115200 /dev/ttyUSB0" + </literallayout> + </para> + </section> + </section> + + <section id="qemu-image-running-tests"> + <title>Running Tests</title> + + <para> + You can start the tests automatically or manually: + <itemizedlist> + <listitem><para><emphasis>Automatically running tests:</emphasis> + To run the tests automatically after the + OpenEmbedded build system successfully creates an image, + first set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink> + variable to "1" in your <filename>local.conf</filename> + file in the + <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>: + <literallayout class='monospaced'> + TEST_IMAGE = "1" + </literallayout> + Next, build your image. + If the image successfully builds, the tests will be + run: + <literallayout class='monospaced'> + bitbake core-image-sato + </literallayout></para></listitem> + <listitem><para><emphasis>Manually running tests:</emphasis> + To manually run the tests, first globally inherit the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink> + class by editing your <filename>local.conf</filename> + file: + <literallayout class='monospaced'> + INHERIT += "testimage" + </literallayout> + Next, use BitBake to run the tests: + <literallayout class='monospaced'> + bitbake -c testimage <replaceable>image</replaceable> + </literallayout></para></listitem> + </itemizedlist> + </para> + + <para> + All test files reside in + <filename>meta/lib/oeqa/runtime</filename> in the + <link linkend='source-directory'>Source Directory</link>. + A test name maps directly to a Python module. + Each test module may contain a number of individual tests. + Tests are usually grouped together by the area + tested (e.g tests for systemd reside in + <filename>meta/lib/oeqa/runtime/systemd.py</filename>). + </para> + + <para> + You can add tests to any layer provided you place them in the + proper area and you extend + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> + in the <filename>local.conf</filename> file as normal. + Be sure that tests reside in + <filename><replaceable>layer</replaceable>/lib/oeqa/runtime</filename>. + <note> + Be sure that module names do not collide with module names + used in the default set of test modules in + <filename>meta/lib/oeqa/runtime</filename>. + </note> + </para> + + <para> + You can change the set of tests run by appending or overriding + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SUITES'><filename>TEST_SUITES</filename></ulink> + variable in <filename>local.conf</filename>. + Each name in <filename>TEST_SUITES</filename> represents a + required test for the image. + Test modules named within <filename>TEST_SUITES</filename> + cannot be skipped even if a test is not suitable for an image + (e.g. running the RPM tests on an image without + <filename>rpm</filename>). + Appending "auto" to <filename>TEST_SUITES</filename> causes the + build system to try to run all tests that are suitable for the + image (i.e. each test module may elect to skip itself). + </para> + + <para> + The order you list tests in <filename>TEST_SUITES</filename> + is important and influences test dependencies. + Consequently, tests that depend on other tests should be added + after the test on which they depend. + For example, since the <filename>ssh</filename> test + depends on the + <filename>ping</filename> test, "ssh" needs to come after + "ping" in the list. + The test class provides no re-ordering or dependency handling. + <note> + Each module can have multiple classes with multiple test + methods. + And, Python <filename>unittest</filename> rules apply. + </note> + </para> + + <para> + Here are some things to keep in mind when running tests: + <itemizedlist> + <listitem><para>The default tests for the image are defined + as: + <literallayout class='monospaced'> + DEFAULT_TEST_SUITES_pn-<replaceable>image</replaceable> = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" + </literallayout></para></listitem> + <listitem><para>Add your own test to the list of the + by using the following: + <literallayout class='monospaced'> + TEST_SUITES_append = " mytest" + </literallayout></para></listitem> + <listitem><para>Run a specific list of tests as follows: + <literallayout class='monospaced'> + TEST_SUITES = "test1 test2 test3" + </literallayout> + Remember, order is important. + Be sure to place a test that is dependent on another test + later in the order.</para></listitem> + </itemizedlist> + </para> + </section> + + <section id="exporting-tests"> + <title>Exporting Tests</title> + + <para> + You can export tests so that they can run independently of + the build system. + Exporting tests is required if you want to be able to hand + the test execution off to a scheduler. + You can only export tests that are defined in + <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_SUITES'><filename>TEST_SUITES</filename></ulink>. + </para> + + <para> + If your image is already built, make sure the following are set + in your <filename>local.conf</filename> file. + Be sure to provide the IP address you need: + <literallayout class='monospaced'> + TEST_EXPORT_ONLY = "1" + TEST_TARGET = "simpleremote" + TEST_TARGET_IP = "192.168.7.2" + TEST_SERVER_IP = "192.168.7.1" + </literallayout> + You can then export the tests with the following: + <literallayout class='monospaced'> + $ bitbake core-image-sato -c testimage + </literallayout> + Exporting the tests places them in the + <link linkend='build-directory'>Build Directory</link> in + <filename>tmp/testimage/core-image-sato</filename>, which + is controlled by the + <filename>TEST_EXPORT_DIR</filename> variable. + </para> + + <para> + You can now run the tests outside of the build environment: + <literallayout class='monospaced'> + $ cd tmp/testimage/core-image-sato + $ ./runexported.py testdata.json + </literallayout> + <note> + This "export" feature does not deploy or boot the target + image. + Your target (be it a Qemu or hardware one) + has to already be up and running when you call + <filename>runexported.py</filename> + </note> + </para> + + <para> + The exported data (i.e. <filename>testdata.json</filename>) + contains paths to the Build Directory. + Thus, the contents of the directory can be moved + to another machine as long as you update some paths in the + JSON. + Usually, you only care about the + <filename>${DEPLOY_DIR}/rpm</filename> directory + (assuming the RPM and Smart tests are enabled). + Consequently, running the tests on other machine + means that you have to move the contents and call + <filename>runexported.py</filename> with + "--deploy-dir <replaceable>path</replaceable>" as + follows: + <literallayout class='monospaced'> + ./runexported.py --deploy-dir /new/path/on/this/machine testdata.json + </literallayout> + <filename>runexported.py</filename> accepts other arguments + as well as described using <filename>--help</filename>. + </para> + </section> + + <section id="qemu-image-writing-new-tests"> + <title>Writing New Tests</title> + + <para> + As mentioned previously, all new test files need to be in the + proper place for the build system to find them. + New tests for additional functionality outside of the core + should be added to the layer that adds the functionality, in + <filename><replaceable>layer</replaceable>/lib/oeqa/runtime</filename> + (as long as + <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> + is extended in the layer's + <filename>layer.conf</filename> file as normal). + Just remember the following: + <itemizedlist> + <listitem><para>Filenames need to map directly to test + (module) names. + </para></listitem> + <listitem><para>Do not use module names that + collide with existing core tests. + </para></listitem> + <listitem><para>Minimally, an empty + <filename>__init__.py</filename> file must exist + in the runtime directory. + </para></listitem> + </itemizedlist> + </para> + + <para> + To create a new test, start by copying an existing module + (e.g. <filename>syslog.py</filename> or + <filename>gcc.py</filename> are good ones to use). + Test modules can use code from + <filename>meta/lib/oeqa/utils</filename>, which are helper + classes. + </para> + + <note> + Structure shell commands such that you rely on them and they + return a single code for success. + Be aware that sometimes you will need to parse the output. + See the <filename>df.py</filename> and + <filename>date.py</filename> modules for examples. + </note> + + <para> + You will notice that all test classes inherit + <filename>oeRuntimeTest</filename>, which is found in + <filename>meta/lib/oetest.py</filename>. + This base class offers some helper attributes, which are + described in the following sections: + </para> + + <section id='qemu-image-writing-tests-class-methods'> + <title>Class Methods</title> + + <para> + Class methods are as follows: + <itemizedlist> + <listitem><para><emphasis><filename>hasPackage(pkg)</filename>:</emphasis> + Returns "True" if <filename>pkg</filename> is in the + installed package list of the image, which is based + on the manifest file that is generated during the + <filename>do_rootfs</filename> task. + </para></listitem> + <listitem><para><emphasis><filename>hasFeature(feature)</filename>:</emphasis> + Returns "True" if the feature is in + <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-DISTRO_FEATURES'><filename>DISTRO_FEATURES</filename></ulink>. + </para></listitem> + </itemizedlist> + </para> + </section> + + <section id='qemu-image-writing-tests-class-attributes'> + <title>Class Attributes</title> + + <para> + Class attributes are as follows: + <itemizedlist> + <listitem><para><emphasis><filename>pscmd</filename>:</emphasis> + Equals "ps -ef" if <filename>procps</filename> is + installed in the image. + Otherwise, <filename>pscmd</filename> equals + "ps" (busybox). + </para></listitem> + <listitem><para><emphasis><filename>tc</filename>:</emphasis> + The called test context, which gives access to the + following attributes: + <itemizedlist> + <listitem><para><emphasis><filename>d</filename>:</emphasis> + The BitBake datastore, which allows you to + use stuff such as + <filename>oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")</filename>. + </para></listitem> + <listitem><para><emphasis><filename>testslist</filename> and <filename>testsrequired</filename>:</emphasis> + Used internally. + The tests do not need these. + </para></listitem> + <listitem><para><emphasis><filename>filesdir</filename>:</emphasis> + The absolute path to + <filename>meta/lib/oeqa/runtime/files</filename>, + which contains helper files for tests meant + for copying on the target such as small + files written in C for compilation. + </para></listitem> + <listitem><para><emphasis><filename>target</filename>:</emphasis> + The target controller object used to deploy + and start an image on a particular target + (e.g. QemuTarget, SimpleRemote, and + GummibootTarget). + Tests usually use the following: + <itemizedlist> + <listitem><para><emphasis><filename>ip</filename>:</emphasis> + The target's IP address. + </para></listitem> + <listitem><para><emphasis><filename>server_ip</filename>:</emphasis> + The host's IP address, which is + usually used by the "smart" test + suite. + </para></listitem> + <listitem><para><emphasis><filename>run(cmd, timeout=None)</filename>:</emphasis> + The single, most used method. + This command is a wrapper for: + <filename>ssh root@host "cmd"</filename>. + The command returns a tuple: + (status, output), which are what + their names imply - the return code + of "cmd" and whatever output + it produces. + The optional timeout argument + represents the number of seconds the + test should wait for "cmd" to + return. + If the argument is "None", the + test uses the default instance's + timeout period, which is 300 + seconds. + If the argument is "0", the test + runs until the command returns. + </para></listitem> + <listitem><para><emphasis><filename>copy_to(localpath, remotepath)</filename>:</emphasis> + <filename>scp localpath root@ip:remotepath</filename>. + </para></listitem> + <listitem><para><emphasis><filename>copy_from(remotepath, localpath)</filename>:</emphasis> + <filename>scp root@host:remotepath localpath</filename>. + </para></listitem> + </itemizedlist></para></listitem> + </itemizedlist></para></listitem> + </itemizedlist> + </para> + </section> + + <section id='qemu-image-writing-tests-instance-attributes'> + <title>Instance Attributes</title> + + <para> + A single instance attribute exists, which is + <filename>target</filename>. + The <filename>target</filename> instance attribute is + identical to the class attribute of the same name, which + is described in the previous section. + This attribute exists as both an instance and class + attribute so tests can use + <filename>self.target.run(cmd)</filename> in instance + methods instead of + <filename>oeRuntimeTest.tc.target.run(cmd)</filename>. + </para> + </section> + </section> + </section> + + <section id="platdev-gdb-remotedebug"> + <title>Debugging With the GNU Project Debugger (GDB) Remotely</title> + + <para> + GDB allows you to examine running programs, which in turn helps you to understand and fix problems. + It also allows you to perform post-mortem style analysis of program crashes. + GDB is available as a package within the Yocto Project and is + installed in SDK images by default. + See the "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>" chapter + in the Yocto Project Reference Manual for a description of these images. + You can find information on GDB at <ulink url="http://sourceware.org/gdb/"/>. + </para> + + <tip> + For best results, install debug (<filename>-dbg</filename>) packages + for the applications you are going to debug. + Doing so makes extra debug symbols available that give you more + meaningful output. + </tip> + + <para> + Sometimes, due to memory or disk space constraints, it is not possible + to use GDB directly on the remote target to debug applications. + These constraints arise because GDB needs to load the debugging information and the + binaries of the process being debugged. + Additionally, GDB needs to perform many computations to locate information such as function + names, variable names and values, stack traces and so forth - even before starting the + debugging process. + These extra computations place more load on the target system and can alter the + characteristics of the program being debugged. + </para> + + <para> + To help get past the previously mentioned constraints, you can use Gdbserver. + Gdbserver runs on the remote target and does not load any debugging information + from the debugged process. + Instead, a GDB instance processes the debugging information that is run on a + remote computer - the host GDB. + The host GDB then sends control commands to Gdbserver to make it stop or start the debugged + program, as well as read or write memory regions of that debugged program. + All the debugging information loaded and processed as well + as all the heavy debugging is done by the host GDB. + Offloading these processes gives the Gdbserver running on the target a chance to remain + small and fast. + <note> + By default, source files are part of the + <filename>*-dbg</filename> packages in order to enable GDB + to show source lines in its output. + You can save further space on the target by setting the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_DEBUG_SPLIT_STYLE'><filename>PACKAGE_DEBUG_SPLIT_STYLE</filename></ulink> + variable to "debug-without-src" so that these packages do not + include the source files. + </note> + </para> + + <para> + Because the host GDB is responsible for loading the debugging information and + for doing the necessary processing to make actual debugging happen, + you have to make sure the host can access the unstripped binaries complete + with their debugging information and also be sure the target is compiled with no optimizations. + The host GDB must also have local access to all the libraries used by the + debugged program. + Because Gdbserver does not need any local debugging information, the binaries on + the remote target can remain stripped. + However, the binaries must also be compiled without optimization + so they match the host's binaries. + </para> + + <para> + To remain consistent with GDB documentation and terminology, the binary being debugged + on the remote target machine is referred to as the "inferior" binary. + For documentation on GDB see the + <ulink url="http://sourceware.org/gdb/documentation/">GDB site</ulink>. + </para> + + <para> + The remainder of this section describes the steps you need to take + to debug using the GNU project debugger. + </para> + + <section id='platdev-gdb-remotedebug-setup'> + <title>Set Up the Cross-Development Debugging Environment</title> + + <para> + Before you can initiate a remote debugging session, you need + to be sure you have set up the cross-development environment, + toolchain, and sysroot. + The <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-intro'>Yocto Project Software Development Kit (SDK) Developer's Guide</ulink> + describes this process. + </para> + </section> + + <section id="platdev-gdb-remotedebug-launch-gdbserver"> + <title>Launch Gdbserver on the Target</title> + + <para> + Make sure Gdbserver is installed on the target. + If it is not, install the package + <filename>gdbserver</filename>, which needs the + <filename>libthread-db1</filename> package. + </para> + + <para> + Here is an example, that when entered from the host, + connects to the target and launches Gdbserver in order to + "debug" a binary named <filename>helloworld</filename>: + <literallayout class='monospaced'> + $ gdbserver localhost:2345 /usr/bin/helloworld + </literallayout> + Gdbserver should now be listening on port 2345 for debugging + commands coming from a remote GDB process that is running on + the host computer. + Communication between Gdbserver and the host GDB are done + using TCP. + To use other communication protocols, please refer to the + <ulink url='http://www.gnu.org/software/gdb/'>Gdbserver documentation</ulink>. + </para> + </section> + + <section id="platdev-gdb-remotedebug-launch-gdb"> + <title>Launch GDB on the Host Computer</title> + + <para> + Running GDB on the host computer takes a number of stages, which + this section describes. + </para> + + <section id="platdev-gdb-remotedebug-launch-gdb-buildcross"> + <title>Build the Cross-GDB Package</title> + <para> + A suitable GDB cross-binary is required that runs on your + host computer but also knows about the the ABI of the + remote target. + You can get this binary from the + <link linkend='cross-development-toolchain'>Cross-Development Toolchain</link>. + Here is an example where the toolchain has been installed + in the default directory + <filename>/opt/poky/&DISTRO;</filename>: + <literallayout class='monospaced'> + /opt/poky/&DISTRO;/sysroots/i686-pokysdk-linux/usr/bin/armv7a-vfp-neon-poky-linux-gnueabi/arm-poky-linux-gnueabi-gdb + </literallayout> + where <filename>arm</filename> is the target architecture + and <filename>linux-gnueabi</filename> is the target ABI. + </para> + + <para> + Alternatively, you can use BitBake to build the + <filename>gdb-cross</filename> binary. + Here is an example: + <literallayout class='monospaced'> + $ bitbake gdb-cross + </literallayout> + Once the binary is built, you can find it here: + <literallayout class='monospaced'> + tmp/sysroots/<replaceable>host-arch</replaceable>/usr/bin/<replaceable>target-platform</replaceable>/<replaceable>target-abi</replaceable>-gdb + </literallayout> + </para> + </section> + + <section id='create-the-gdb-initialization-file'> + <title>Create the GDB Initialization File and Point to Your Root Filesystem</title> + + <para> + Aside from the GDB cross-binary, you also need a GDB + initialization file in the same top directory in which + your binary resides. + When you start GDB on your host development system, GDB + finds this initialization file and executes all the + commands within. + For information on the <filename>.gdbinit</filename>, see + "<ulink url='http://sourceware.org/gdb/onlinedocs/gdb/'>Debugging with GDB</ulink>", + which is maintained by + <ulink url='http://www.sourceware.org'>sourceware.org</ulink>. + </para> + + <para> + You need to add a statement in the + <filename>~/.gdbinit</filename> file that points to your + root filesystem. + Here is an example that points to the root filesystem for + an ARM-based target device: + <literallayout class='monospaced'> + set sysroot ~/sysroot_arm + </literallayout> + </para> + </section> + + <section id="platdev-gdb-remotedebug-launch-gdb-launchhost"> + <title>Launch the Host GDB</title> + + <para> + Before launching the host GDB, you need to be sure + you have sourced the cross-debugging environment script, + which if you installed the root filesystem in the default + location is at <filename>/opt/poky/&DISTRO;</filename> + and begins with the string "environment-setup". + For more information, see the + <ulink url='&YOCTO_DOCS_SDK_URL;#sdk-manual'>Yocto Project Software Development Kit (SDK) Developer's + Guide</ulink>. + </para> + + <para> + Finally, switch to the directory where the binary resides + and run the <filename>cross-gdb</filename> binary. + Provide the binary file you are going to debug. + For example, the following command continues with the + example used in the previous section by loading + the <filename>helloworld</filename> binary as well as the + debugging information: + <literallayout class='monospaced'> + $ arm-poky-linux-gnuabi-gdb helloworld + </literallayout> + The commands in your <filename>.gdbinit</filename> execute + and the GDB prompt appears. + </para> + </section> + </section> + + <section id='platdev-gdb-connect-to-the-remote-gdb-server'> + <title>Connect to the Remote GDB Server</title> + + <para> + From the target, you need to connect to the remote GDB + server that is running on the host. + You need to specify the remote host and port. + Here is the command continuing with the example: + <literallayout class='monospaced'> + target remote 192.168.7.2:2345 + </literallayout> + </para> + </section> + + <section id="platdev-gdb-remotedebug-launch-gdb-using"> + <title>Use the Debugger</title> + + <para> + You can now proceed with debugging as normal - as if you were debugging + on the local machine. + For example, to instruct GDB to break in the "main" function and then + continue with execution of the inferior binary use the following commands + from within GDB: + <literallayout class='monospaced'> + (gdb) break main + (gdb) continue + </literallayout> + </para> + + <para> + For more information about using GDB, see the project's online documentation at + <ulink url="http://sourceware.org/gdb/download/onlinedocs/"/>. + </para> + </section> + </section> + + <section id='debugging-parallel-make-races'> + <title>Debugging Parallel Make Races</title> + + <para> + A parallel <filename>make</filename> race occurs when the build + consists of several parts that are run simultaneously and + a situation occurs when the output or result of one + part is not ready for use with a different part of the build that + depends on that output. + Parallel make races are annoying and can sometimes be difficult + to reproduce and fix. + However, some simple tips and tricks exist that can help + you debug and fix them. + This section presents a real-world example of an error encountered + on the Yocto Project autobuilder and the process used to fix it. + <note> + If you cannot properly fix a <filename>make</filename> race + condition, you can work around it by clearing either the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + or + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKEINST'><filename>PARALLEL_MAKEINST</filename></ulink> + variables. + </note> + </para> + + <section id='the-failure'> + <title>The Failure</title> + + <para> + For this example, assume that you are building an image that + depends on the "neard" package. + And, during the build, BitBake runs into problems and + creates the following output. + <note> + This example log file has longer lines artificially + broken to make the listing easier to read. + </note> + If you examine the output or the log file, you see the + failure during <filename>make</filename>: + <literallayout class='monospaced'> + | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] + | DEBUG: Executing shell function do_compile + | NOTE: make -j 16 + | make --no-print-directory all-am + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/types.h include/near/types.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/log.h include/near/log.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/tag.h include/near/tag.h + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/setting.h include/near/setting.h + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | /bin/mkdir -p include/near + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/device.h include/near/device.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/snep.h include/near/snep.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/version.h include/near/version.h + | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ + 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h + | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h + | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/ + build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/ + yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0 + -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/ + lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/ + tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/ + nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/ + yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3 + -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\" + -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c + -o tools/snep-send.o tools/snep-send.c + | In file included from tools/snep-send.c:16:0: + | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory + | #include <near/dbus.h> + | ^ + | compilation terminated. + | make[1]: *** [tools/snep-send.o] Error 1 + | make[1]: *** Waiting for unfinished jobs.... + | make: *** [all] Error 2 + | ERROR: oe_runmake failed + </literallayout> + </para> + </section> + + <section id='reproducing-the-error'> + <title>Reproducing the Error</title> + + <para> + Because race conditions are intermittent, they do not + manifest themselves every time you do the build. + In fact, most times the build will complete without problems + even though the potential race condition exists. + Thus, once the error surfaces, you need a way to reproduce it. + </para> + + <para> + In this example, compiling the "neard" package is causing the + problem. + So the first thing to do is build "neard" locally. + Before you start the build, set the + <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink> + variable in your <filename>local.conf</filename> file to + a high number (e.g. "-j 20"). + Using a high value for <filename>PARALLEL_MAKE</filename> + increases the chances of the race condition showing up: + <literallayout class='monospaced'> + $ bitbake neard + </literallayout> + </para> + + <para> + Once the local build for "neard" completes, start a + <filename>devshell</filename> build: + <literallayout class='monospaced'> + $ bitbake neard -c devshell + </literallayout> + For information on how to use a + <filename>devshell</filename>, see the + "<link linkend='platdev-appdev-devshell'>Using a Development Shell</link>" + section. + </para> + + <para> + In the <filename>devshell</filename>, do the following: + <literallayout class='monospaced'> + $ make clean + $ make tools/snep-send.o + </literallayout> + The <filename>devshell</filename> commands cause the failure + to clearly be visible. + In this case, a missing dependency exists for the "neard" + Makefile target. + Here is some abbreviated, sample output with the + missing dependency clearly visible at the end: + <literallayout class='monospaced'> + i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... + . + . + . + tools/snep-send.c + In file included from tools/snep-send.c:16:0: + tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory + #include <near/dbus.h> + ^ + compilation terminated. + make: *** [tools/snep-send.o] Error 1 + $ + </literallayout> + </para> + </section> + + <section id='creating-a-patch-for-the-fix'> + <title>Creating a Patch for the Fix</title> + + <para> + Because there is a missing dependency for the Makefile + target, you need to patch the + <filename>Makefile.am</filename> file, which is generated + from <filename>Makefile.in</filename>. + You can use Quilt to create the patch: + <literallayout class='monospaced'> + $ quilt new parallelmake.patch + Patch patches/parallelmake.patch is now on top + $ quilt add Makefile.am + File Makefile.am added to patch patches/parallelmake.patch + </literallayout> + For more information on using Quilt, see the + "<link linkend='using-a-quilt-workflow'>Using Quilt in Your Workflow</link>" + section. + </para> + + <para> + At this point you need to make the edits to + <filename>Makefile.am</filename> to add the missing + dependency. + For our example, you have to add the following line + to the file: + <literallayout class='monospaced'> + tools/snep-send.$(OBJEXT): include/near/dbus.h + </literallayout> + </para> + + <para> + Once you have edited the file, use the + <filename>refresh</filename> command to create the patch: + <literallayout class='monospaced'> + $ quilt refresh + Refreshed patch patches/parallelmake.patch + </literallayout> + Once the patch file exists, you need to add it back to the + originating recipe folder. + Here is an example assuming a top-level + <link linkend='source-directory'>Source Directory</link> + named <filename>poky</filename>: + <literallayout class='monospaced'> + $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard + </literallayout> + The final thing you need to do to implement the fix in the + build is to update the "neard" recipe (i.e. + <filename>neard-0.14.bb</filename>) so that the + <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink> + statement includes the patch file. + The recipe file is in the folder above the patch. + Here is what the edited <filename>SRC_URI</filename> + statement would look like: + <literallayout class='monospaced'> + SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ + file://neard.in \ + file://neard.service.in \ + file://parallelmake.patch \ + " + </literallayout> + </para> + + <para> + With the patch complete and moved to the correct folder and + the <filename>SRC_URI</filename> statement updated, you can + exit the <filename>devshell</filename>: + <literallayout class='monospaced'> + $ exit + </literallayout> + </para> + </section> + + <section id='testing-the-build'> + <title>Testing the Build</title> + + <para> + With everything in place, you can get back to trying the + build again locally: + <literallayout class='monospaced'> + $ bitbake neard + </literallayout> + This build should succeed. + </para> + + <para> + Now you can open up a <filename>devshell</filename> again + and repeat the clean and make operations as follows: + <literallayout class='monospaced'> + $ bitbake neard -c devshell + $ make clean + $ make tools/snep-send.o + </literallayout> + The build should work without issue. + </para> + + <para> + As with all solved problems, if they originated upstream, you + need to submit the fix for the recipe in OE-Core and upstream + so that the problem is taken care of at its source. + See the + "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" + section for more information. + </para> + </section> + </section> + +<!-- + <section id="platdev-oprofile"> + <title>Profiling with OProfile</title> + + <para> + <ulink url="http://oprofile.sourceforge.net/">OProfile</ulink> is a + statistical profiler well suited for finding performance + bottlenecks in both user-space software and in the kernel. + This profiler provides answers to questions like "Which functions does my application spend + the most time in when doing X?" + Because the OpenEmbedded build system is well integrated with OProfile, it makes profiling + applications on target hardware straight forward. + <note> + For more information on how to set up and run OProfile, see the + "<ulink url='&YOCTO_DOCS_PROF_URL;#profile-manual-oprofile'>oprofile</ulink>" + section in the Yocto Project Profiling and Tracing Manual. + </note> + </para> + + <para> + To use OProfile, you need an image that has OProfile installed. + The easiest way to do this is with "tools-profile" in the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'>IMAGE_FEATURES</ulink></filename> variable. + You also need debugging symbols to be available on the system where the analysis + takes place. + You can gain access to the symbols by using "dbg-pkgs" in the + <filename>IMAGE_FEATURES</filename> variable or by + installing the appropriate debug (<filename>-dbg</filename>) + packages. + </para> + + <para> + For successful call graph analysis, the binaries must preserve the frame + pointer register and should also be compiled with the + <filename>-fno-omit-framepointer</filename> flag. + You can achieve this by setting the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-SELECTED_OPTIMIZATION'>SELECTED_OPTIMIZATION</ulink></filename> + variable with the following options: + <literallayout class='monospaced'> + -fexpensive-optimizations + -fno-omit-framepointer + -frename-registers + -O2 + </literallayout> + You can also achieve it by setting the + <filename><ulink url='&YOCTO_DOCS_REF_URL;#var-DEBUG_BUILD'>DEBUG_BUILD</ulink></filename> + variable to "1" in the <filename>local.conf</filename> configuration file. + If you use the <filename>DEBUG_BUILD</filename> variable, + you also add extra debugging information that can make the debug + packages large. + </para> + + <section id="platdev-oprofile-target"> + <title>Profiling on the Target</title> + + <para> + Using OProfile, you can perform all the profiling work on the target device. + A simple OProfile session might look like the following: + </para> + + <para> + <literallayout class='monospaced'> + # opcontrol ‐‐reset + # opcontrol ‐‐start ‐‐separate=lib ‐‐no-vmlinux -c 5 + . + . + [do whatever is being profiled] + . + . + # opcontrol ‐‐stop + $ opreport -cl + </literallayout> + </para> + + <para> + In this example, the <filename>reset</filename> command clears any previously profiled data. + The next command starts OProfile. + The options used when starting the profiler separate dynamic library data + within applications, disable kernel profiling, and enable callgraphing up to + five levels deep. + <note> + To profile the kernel, you would specify the + <filename>‐‐vmlinux=/path/to/vmlinux</filename> option. + The <filename>vmlinux</filename> file is usually in the source directory in the + <filename>/boot/</filename> directory and must match the running kernel. + </note> + </para> + + <para> + After you perform your profiling tasks, the next command stops the profiler. + After that, you can view results with the <filename>opreport</filename> command with options + to see the separate library symbols and callgraph information. + </para> + + <para> + Callgraphing logs information about time spent in functions and about a function's + calling function (parent) and called functions (children). + The higher the callgraphing depth, the more accurate the results. + However, higher depths also increase the logging overhead. + Consequently, you should take care when setting the callgraphing depth. + <note> + On ARM, binaries need to have the frame pointer enabled for callgraphing to work. + To accomplish this use the <filename>-fno-omit-framepointer</filename> option + with <filename>gcc</filename>. + </note> + </para> + + <para> + For more information on using OProfile, see the OProfile + online documentation at + <ulink url="http://oprofile.sourceforge.net/docs/"/>. + </para> + </section> + + <section id="platdev-oprofile-oprofileui"> + <title>Using OProfileUI</title> + + <para> + A graphical user interface for OProfile is also available. + You can download and build this interface from the Yocto Project at + <ulink url="&YOCTO_GIT_URL;/cgit.cgi/oprofileui/"></ulink>. + If the "tools-profile" image feature is selected, all necessary binaries + are installed onto the target device for OProfileUI interaction. + For a list of image features that ship with the Yocto Project, + see the + "<ulink url='&YOCTO_DOCS_REF_URL;#ref-features-image'>Image Features</ulink>" + section in the Yocto Project Reference Manual. + </para> + + <para> + Even though the source directory usually includes all needed patches on the target device, you + might find you need other OProfile patches for recent OProfileUI features. + If so, see the <ulink url='&YOCTO_GIT_URL;/cgit.cgi/oprofileui/tree/README'> + OProfileUI README</ulink> for the most recent information. + </para> + + <section id="platdev-oprofile-oprofileui-online"> + <title>Online Mode</title> + + <para> + Using OProfile in online mode assumes a working network connection with the target + hardware. + With this connection, you just need to run "oprofile-server" on the device. + By default, OProfile listens on port 4224. + <note> + You can change the port using the <filename>‐‐port</filename> command-line + option. + </note> + </para> + + <para> + The client program is called <filename>oprofile-viewer</filename> and its UI is relatively + straight forward. + You access key functionality through the buttons on the toolbar, which + are duplicated in the menus. + Here are the buttons: + <itemizedlist> + <listitem><para><emphasis>Connect:</emphasis> Connects to the remote host. + You can also supply the IP address or hostname.</para></listitem> + <listitem><para><emphasis>Disconnect:</emphasis> Disconnects from the target. + </para></listitem> + <listitem><para><emphasis>Start:</emphasis> Starts profiling on the device. + </para></listitem> + <listitem><para><emphasis>Stop:</emphasis> Stops profiling on the device and + downloads the data to the local host. + Stopping the profiler generates the profile and displays it in the viewer. + </para></listitem> + <listitem><para><emphasis>Download:</emphasis> Downloads the data from the + target and generates the profile, which appears in the viewer.</para></listitem> + <listitem><para><emphasis>Reset:</emphasis> Resets the sample data on the device. + Resetting the data removes sample information collected from previous + sampling runs. + Be sure you reset the data if you do not want to include old sample information. + </para></listitem> + <listitem><para><emphasis>Save:</emphasis> Saves the data downloaded from the + target to another directory for later examination.</para></listitem> + <listitem><para><emphasis>Open:</emphasis> Loads previously saved data. + </para></listitem> + </itemizedlist> + </para> + + <para> + The client downloads the complete profile archive from + the target to the host for processing. + This archive is a directory that contains the sample data, the object files, + and the debug information for the object files. + The archive is then converted using the <filename>oparchconv</filename> script, which is + included in this distribution. + The script uses <filename>opimport</filename> to convert the archive from + the target to something that can be processed on the host. + </para> + + <para> + Downloaded archives reside in the + <link linkend='build-directory'>Build Directory</link> in + <filename>tmp</filename> and are cleared up when they are no longer in use. + </para> + + <para> + If you wish to perform kernel profiling, you need to be sure + a <filename>vmlinux</filename> file that matches the running kernel is available. + In the source directory, that file is usually located in + <filename>/boot/vmlinux-<replaceable>kernelversion</replaceable></filename>, where + <filename><replaceable>kernelversion</replaceable></filename> is the version of the kernel. + The OpenEmbedded build system generates separate <filename>vmlinux</filename> + packages for each kernel it builds. + Thus, it should just be a question of making sure a matching package is + installed (e.g. <filename>opkg install kernel-vmlinux</filename>). + The files are automatically installed into development and profiling images + alongside OProfile. + A configuration option exists within the OProfileUI settings page that you can use to + enter the location of the <filename>vmlinux</filename> file. + </para> + + <para> + Waiting for debug symbols to transfer from the device can be slow, and it + is not always necessary to actually have them on the device for OProfile use. + All that is needed is a copy of the filesystem with the debug symbols present + on the viewer system. + The "<link linkend='platdev-gdb-remotedebug-launch-gdb'>Launch GDB on the Host Computer</link>" + section covers how to create such a directory within + the source directory and how to use the OProfileUI Settings + Dialog to specify the location. + If you specify the directory, it will be used when the file checksums + match those on the system you are profiling. + </para> + </section> + + <section id="platdev-oprofile-oprofileui-offline"> + <title>Offline Mode</title> + + <para> + If network access to the target is unavailable, you can generate + an archive for processing in <filename>oprofile-viewer</filename> as follows: + <literallayout class='monospaced'> + # opcontrol ‐‐reset + # opcontrol ‐‐start ‐‐separate=lib ‐‐no-vmlinux -c 5 + . + . + [do whatever is being profiled] + . + . + # opcontrol ‐‐stop + # oparchive -o my_archive + </literallayout> + </para> + + <para> + In the above example, <filename>my_archive</filename> is the name of the + archive directory where you would like the profile archive to be kept. + After the directory is created, you can copy it to another host and load it + using <filename>oprofile-viewer</filename> open functionality. + If necessary, the archive is converted. + </para> + </section> + </section> + </section> +--> + + <section id='maintaining-open-source-license-compliance-during-your-products-lifecycle'> + <title>Maintaining Open Source License Compliance During Your Product's Lifecycle</title> + + <para> + One of the concerns for a development organization using open source + software is how to maintain compliance with various open source + licensing during the lifecycle of the product. + While this section does not provide legal advice or + comprehensively cover all scenarios, it does + present methods that you can use to + assist you in meeting the compliance requirements during a software + release. + </para> + + <para> + With hundreds of different open source licenses that the Yocto + Project tracks, it is difficult to know the requirements of each + and every license. + However, the requirements of the major FLOSS licenses can begin + to be covered by + assuming that three main areas of concern exist: + <itemizedlist> + <listitem><para>Source code must be provided.</para></listitem> + <listitem><para>License text for the software must be + provided.</para></listitem> + <listitem><para>Compilation scripts and modifications to the + source code must be provided. + </para></listitem> + </itemizedlist> + There are other requirements beyond the scope of these + three and the methods described in this section + (e.g. the mechanism through which source code is distributed). + </para> + + <para> + As different organizations have different methods of complying with + open source licensing, this section is not meant to imply that + there is only one single way to meet your compliance obligations, + but rather to describe one method of achieving compliance. + The remainder of this section describes methods supported to meet the + previously mentioned three requirements. + Once you take steps to meet these requirements, + and prior to releasing images, sources, and the build system, + you should audit all artifacts to ensure completeness. + <note> + The Yocto Project generates a license manifest during + image creation that is located + in <filename>${DEPLOY_DIR}/licenses/<replaceable>image_name-datestamp</replaceable></filename> + to assist with any audits. + </note> + </para> + + <section id='providing-the-source-code'> + <title>Providing the Source Code</title> + + <para> + Compliance activities should begin before you generate the + final image. + The first thing you should look at is the requirement that + tops the list for most compliance groups - providing + the source. + The Yocto Project has a few ways of meeting this + requirement. + </para> + + <para> + One of the easiest ways to meet this requirement is + to provide the entire + <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink> + used by the build. + This method, however, has a few issues. + The most obvious is the size of the directory since it includes + all sources used in the build and not just the source used in + the released image. + It will include toolchain source, and other artifacts, which + you would not generally release. + However, the more serious issue for most companies is accidental + release of proprietary software. + The Yocto Project provides an + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-archiver'><filename>archiver</filename></ulink> + class to help avoid some of these concerns. + </para> + + <para> + Before you employ <filename>DL_DIR</filename> or the + archiver class, you need to decide how you choose to + provide source. + The source archiver class can generate tarballs and SRPMs + and can create them with various levels of compliance in mind. + </para> + + <para> + One way of doing this (but certainly not the only way) is to + release just the source as a tarball. + You can do this by adding the following to the + <filename>local.conf</filename> file found in the + <link linkend='build-directory'>Build Directory</link>: + <literallayout class='monospaced'> + INHERIT += "archiver" + ARCHIVER_MODE[src] = "original" + </literallayout> + During the creation of your image, the source from all + recipes that deploy packages to the image is placed within + subdirectories of + <filename>DEPLOY_DIR/sources</filename> based on the + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE'><filename>LICENSE</filename></ulink> + for each recipe. + Releasing the entire directory enables you to comply with + requirements concerning providing the unmodified source. + It is important to note that the size of the directory can + get large. + </para> + + <para> + A way to help mitigate the size issue is to only release + tarballs for licenses that require the release of + source. + Let us assume you are only concerned with GPL code as + identified with the following: + <literallayout class='monospaced'> + $ cd poky/build/tmp/deploy/sources + $ mkdir ~/gpl_source_release + $ for dir in */*GPL*; do cp -r $dir ~/gpl_source_release; done + </literallayout> + At this point, you could create a tarball from the + <filename>gpl_source_release</filename> directory and + provide that to the end user. + This method would be a step toward achieving compliance + with section 3a of GPLv2 and with section 6 of GPLv3. + </para> + </section> + + <section id='providing-license-text'> + <title>Providing License Text</title> + + <para> + One requirement that is often overlooked is inclusion + of license text. + This requirement also needs to be dealt with prior to + generating the final image. + Some licenses require the license text to accompany + the binary. + You can achieve this by adding the following to your + <filename>local.conf</filename> file: + <literallayout class='monospaced'> + COPY_LIC_MANIFEST = "1" + COPY_LIC_DIRS = "1" + LICENSE_CREATE_PACKAGE = "1" + </literallayout> + Adding these statements to the configuration file ensures + that the licenses collected during package generation + are included on your image. + <note> + <para>Setting all three variables to "1" results in the + image having two copies of the same license file. + One copy resides in + <filename>/usr/share/common-licenses</filename> and + the other resides in + <filename>/usr/share/license</filename>.</para> + + <para>The reason for this behavior is because + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_DIRS'><filename>COPY_LIC_DIRS</filename></ulink> + and + <ulink url='&YOCTO_DOCS_REF_URL;#var-COPY_LIC_MANIFEST'><filename>COPY_LIC_MANIFEST</filename></ulink> + add a copy of the license when the image is built but do not + offer a path for adding licenses for newly installed packages + to an image. + <ulink url='&YOCTO_DOCS_REF_URL;#var-LICENSE_CREATE_PACKAGE'><filename>LICENSE_CREATE_PACKAGE</filename></ulink> + adds a separate package and an upgrade path for adding + licenses to an image.</para> + </note> + </para> + + <para> + As the source archiver has already archived the original + unmodified source that contains the license files, + you would have already met the requirements for inclusion + of the license information with source as defined by the GPL + and other open source licenses. + </para> + </section> + + <section id='providing-compilation-scripts-and-source-code-modifications'> + <title>Providing Compilation Scripts and Source Code Modifications</title> + + <para> + At this point, we have addressed all we need to address + prior to generating the image. + The next two requirements are addressed during the final + packaging of the release. + </para> + + <para> + By releasing the version of the OpenEmbedded build system + and the layers used during the build, you will be providing both + compilation scripts and the source code modifications in one + step. + </para> + + <para> + If the deployment team has a + <ulink url='&YOCTO_DOCS_BSP_URL;#bsp-layers'>BSP layer</ulink> + and a distro layer, and those those layers are used to patch, + compile, package, or modify (in any way) any open source + software included in your released images, you + might be required to to release those layers under section 3 of + GPLv2 or section 1 of GPLv3. + One way of doing that is with a clean + checkout of the version of the Yocto Project and layers used + during your build. + Here is an example: + <literallayout class='monospaced'> + # We built using the &DISTRO_NAME_NO_CAP; branch of the poky repo + $ git clone -b &DISTRO_NAME_NO_CAP; git://git.yoctoproject.org/poky + $ cd poky + # We built using the release_branch for our layers + $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer + $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer + # clean up the .git repos + $ find . -name ".git" -type d -exec rm -rf {} \; + </literallayout> + One thing a development organization might want to consider + for end-user convenience is to modify + <filename>meta-poky/conf/bblayers.conf.sample</filename> to + ensure that when the end user utilizes the released build + system to build an image, the development organization's + layers are included in the <filename>bblayers.conf</filename> + file automatically: + <literallayout class='monospaced'> + # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf + # changes incompatibly + LCONF_VERSION = "6" + + BBPATH = "${TOPDIR}" + BBFILES ?= "" + + BBLAYERS ?= " \ + ##OEROOT##/meta \ + ##OEROOT##/meta-poky \ + ##OEROOT##/meta-yocto-bsp \ + ##OEROOT##/meta-mylayer \ + " + </literallayout> + Creating and providing an archive of the + <link linkend='metadata'>Metadata</link> layers + (recipes, configuration files, and so forth) + enables you to meet your + requirements to include the scripts to control compilation + as well as any modifications to the original source. + </para> + </section> + </section> + + <section id='using-the-error-reporting-tool'> + <title>Using the Error Reporting Tool</title> + + <para> + The error reporting tool allows you to + submit errors encountered during builds to a central database. + Outside of the build environment, you can use a web interface to + browse errors, view statistics, and query for errors. + The tool works using a client-server system where the client + portion is integrated with the installed Yocto Project + <link linkend='source-directory'>Source Directory</link> + (e.g. <filename>poky</filename>). + The server receives the information collected and saves it in a + database. + </para> + + <para> + A live instance of the error reporting server exists at + <ulink url='http://errors.yoctoproject.org'></ulink>. + This server exists so that when you want to get help with + build failures, you can submit all of the information on the + failure easily and then point to the URL in your bug report + or send an email to the mailing list. + <note> + If you send error reports to this server, the reports become + publicly visible. + </note> + </para> + + <section id='enabling-and-using-the-tool'> + <title>Enabling and Using the Tool</title> + + <para> + By default, the error reporting tool is disabled. + You can enable it by inheriting the + <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-report-error'><filename>report-error</filename></ulink> + class by adding the following statement to the end of + your <filename>local.conf</filename> file in your + <link linkend='build-directory'>Build Directory</link>. + <literallayout class='monospaced'> + INHERIT += "report-error" + </literallayout> + </para> + + <para> + By default, the error reporting feature stores information in + <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-LOG_DIR'><filename>LOG_DIR</filename></ulink><filename>}/error-report</filename>. + However, you can specify a directory to use by adding the following + to your <filename>local.conf</filename> file: + <literallayout class='monospaced'> + ERR_REPORT_DIR = "path" + </literallayout> + Enabling error reporting causes the build process to collect + the errors and store them in a file as previously described. + When the build system encounters an error, it includes a + command as part of the console output. + You can run the command to send the error file to the server. + For example, the following command sends the errors to an + upstream server: + <literallayout class='monospaced'> + $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt + </literallayout> + In the previous example, the errors are sent to a public + database available at + <ulink url='http://errors.yoctoproject.org'></ulink>, which is + used by the entire community. + If you specify a particular server, you can send the errors + to a different database. + Use the following command for more information on available + options: + <literallayout class='monospaced'> + $ send-error-report --help + </literallayout> + </para> + + <para> + When sending the error file, you are prompted to review the + data being sent as well as to provide a name and optional + email address. + Once you satisfy these prompts, the command returns a link + from the server that corresponds to your entry in the database. + For example, here is a typical link: + <literallayout class='monospaced'> + http://errors.yoctoproject.org/Errors/Details/9522/ + </literallayout> + Following the link takes you to a web interface where you can + browse, query the errors, and view statistics. + </para> + </section> + + <section id='disabling-the-tool'> + <title>Disabling the Tool</title> + + <para> + To disable the error reporting feature, simply remove or comment + out the following statement from the end of your + <filename>local.conf</filename> file in your + <link linkend='build-directory'>Build Directory</link>. + <literallayout class='monospaced'> + INHERIT += "report-error" + </literallayout> + </para> + </section> + + <section id='setting-up-your-own-error-reporting-server'> + <title>Setting Up Your Own Error Reporting Server</title> + + <para> + If you want to set up your own error reporting server, you + can obtain the code from the Git repository at + <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/error-report-web/'></ulink>. + Instructions on how to set it up are in the README document. + </para> + </section> + </section> +</chapter> + +<!-- +vim: expandtab tw=80 ts=4 +--> |
