buildroot/support/scripts, branch 2019.02-op-build

support/scripts/pkg-stats: fix flake8 errors

2019-02-07T21:09:55+00:00

Fixes the following flake8 warnings: support/scripts/pkg-stats:34:2: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:4: W605 invalid escape sequence '$' support/scripts/pkg-stats:34:11: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:13: W605 invalid escape sequence '\(' support/scripts/pkg-stats:34:32: W605 invalid escape sequence '$' support/scripts/pkg-stats:34:34: W605 invalid escape sequence '\)' support/scripts/pkg-stats:35:2: W605 invalid escape sequence '\s' support/scripts/pkg-stats:35:14: W605 invalid escape sequence '\S' support/scripts/pkg-stats:35:17: W605 invalid escape sequence '\s' support/scripts/pkg-stats:42:1: E302 expected 2 blank lines, found 1 support/scripts/pkg-stats:587:133: E501 line too long (157 > 132 characters) Note that the "invalid escape sequence" errors work because Python leaves the \ in place if it doesn't recognise the escape sequence. But it's better practice to use a raw string for regular expressions. Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

skeleton-custom: use a script to check merged usr structure

2019-02-06T16:03:30+00:00

Introduce support/scripts/check-merged-usr.sh, a script that check if a given path complies to the merged /usr requirements: / /bin -> usr/bin /lib -> usr/lib /sbin -> usr/sbin /usr/bin/ /usr/lib/ /usr/sbin/ Use this script in skeleton-custom.mk instead of a bunch of variables filled by $(shell ...) macros. The same script will be used to check rootfs overlays, in a forthcoming change. Signed-off-by: Carlos Santos Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

support/scripts/pkg-stats: add latest upstream version information

2019-02-06T14:57:09+00:00

This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of https://release-monitoring.org/project/15037/. If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we hope to be able to use a database dump of release-monitoring.org instead. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni Tested-by: Matthew Weber Signed-off-by: Thomas Petazzoni

.gitlab-ci.yml: use "extends" keyword

2019-02-06T10:40:28+00:00

Replace all YAML anchors with the new "extends" keyword because it is more readable and more flexible (it works across configuration files combined with the new "include" keyword). Readability is more meaningful in .gitlab-ci.yml.in. In the part of .gitlab-ci.yml that is auto-generated by 'make .gitlab-ci.yml' keep the keyword in the same line of the job name. So instead of this: zynqmp_zcu106_defconfig: extends: .defconfig tests.boot.test_atf.TestATFAllwinner: extends: .runtime_test Use this: zynqmp_zcu106_defconfig: { extends: .defconfig } tests.boot.test_atf.TestATFAllwinner: { extends: .runtime_test } Do this to to keep .gitlab-ci.yml easier to be post-processed by a script. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle Cc: Thomas Petazzoni Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

support/scripts/setlocalversion: ignore user settings for Mercurial

2019-02-04T20:25:33+00:00

setlocalversion will use 'hg id' to determine whether or not the current revision is tagged. If there is no tag, the Mercurial revision is printed, otherwise nothing is printed. The problem is that the user may have custom configuration settings (in their ~/.hgrc file or similar) that changes the output of 'hg id' in a way that the script does not expect. In such cases, the Mercurial revision may not be printed or printed incorrectly. It is good practice to ignore the user environment when calling Mercurial commands from a well-defined script, by setting the environment variable HGRCPATH to the empty string. See also 'hg help environment'. In the particular case of Nokia, a custom extension adds dynamic tags in the repository, i.e. tags that are stored in a file external to the repository and only visible when the extension is active. These tags should not influence the behavior of setlocalversion as they are not official Buildroot tags, i.e. even if a revision is tagged, the Mercurial revision should still be printed. Note that this still does not solve the problem where an organization adds _real_ tags in their Buildroot repository. For example, there might be a moving tag 'last-validated' or tags indicating in which product release that Buildroot revision was used. In these cases, setlocalversion will still not behave as expected, i.e. show the Mercurial revision. Signed-off-by: Thomas De Schampheleire Signed-off-by: Thomas Petazzoni

support/scripts/setlocalversion: fix detection of hg revision when _not_ on branch 'default'

2019-02-04T20:25:33+00:00

When Buildroot is stored in a Mercurial repository on a branch other than 'default' ('master' in git terms), setlocalversion (used to populate /etc/os-release) will incorrectly think that this is a tagged version and will NOT print out the revision hash. This is due to the fact that the output of 'hg id' is assumed to be "

" but when on a branch it actually is: " () " To let setlocalversion receive the output it expects, explicitly ask 'hg id' to retrieve only the revision hash and any tags, ommitting any branch information. Signed-off-by: Thomas De Schampheleire Signed-off-by: Thomas Petazzoni

support/scripts/graph-build-time: replace confusing colors

2019-02-04T14:23:27+00:00

The color for 'extract' is very similar to the one for 'install-images'. Both are cyan-like. Replace the former by a pale blue to make all colors sufficiently distinct. Signed-off-by: Thomas De Schampheleire Tested-by: Matthew Weber Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

support/scripts/graph-build-time: add download times

2019-02-04T14:22:59+00:00

Total build time also involves download. Getting a visibility on the impact of that step can be important for users/admins, e.g. to evaluate different methods of BR2_PRIMARY_SITE. Colors used are some kind of purple (primary scheme) and light orange (alternate scheme). Signed-off-by: Mathias De Maré [ThomasDS: rebase and update colors to avoid confusion] Signed-off-by: Thomas De Schampheleire Tested-by: Matthew Weber Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

support/scripts/check-host-rpath: document existing functions

2019-01-17T21:38:34+00:00

As suggested by Arnout Vandecappelle, let's document the elf_needs_rpath() and check_elf_has_rpath() functions, before we make them a bit more complicated with per-package directory support. Suggested-by: Arnout Vandecappelle (Essensium/Mind) Signed-off-by: Thomas Petazzoni Reviewed-by: "Yann E. MORIN" Signed-off-by: Peter Korsgaard

Makefile: offload .gitlab-ci.yml generation

2018-12-09T20:30:24+00:00

GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: | only: | - /-$CI_JOB_NAME$/ | - /-\$CI_JOB_NAME$/ | - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: | only: | refs: | - branches | - tags | variables: | - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: |.runtime_test: &runtime_test | only: | - tags |tests.package.test_python_txaio.TestPythonPy2Txaio: | <<: *runtime_test | only: | - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni