summaryrefslogtreecommitdiffstats
path: root/support/scripts
Commit message (Collapse)AuthorAgeFilesLines
* support/scripts/pkg-stats: fix flake8 errorsArnout Vandecappelle (Essensium/Mind)2019-02-071-3/+5
| | | | | | | | | | | | | | | | | | | | | | Fixes the following flake8 warnings: support/scripts/pkg-stats:34:2: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:4: W605 invalid escape sequence '\(' support/scripts/pkg-stats:34:11: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:13: W605 invalid escape sequence '\(' support/scripts/pkg-stats:34:32: W605 invalid escape sequence '\)' support/scripts/pkg-stats:34:34: W605 invalid escape sequence '\)' support/scripts/pkg-stats:35:2: W605 invalid escape sequence '\s' support/scripts/pkg-stats:35:14: W605 invalid escape sequence '\S' support/scripts/pkg-stats:35:17: W605 invalid escape sequence '\s' support/scripts/pkg-stats:42:1: E302 expected 2 blank lines, found 1 support/scripts/pkg-stats:587:133: E501 line too long (157 > 132 characters) Note that the "invalid escape sequence" errors work because Python leaves the \ in place if it doesn't recognise the escape sequence. But it's better practice to use a raw string for regular expressions. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* skeleton-custom: use a script to check merged usr structureCarlos Santos2019-02-061-0/+39
| | | | | | | | | | | | | | | | | | | | Introduce support/scripts/check-merged-usr.sh, a script that check if a given path complies to the merged /usr requirements: / /bin -> usr/bin /lib -> usr/lib /sbin -> usr/sbin /usr/bin/ /usr/lib/ /usr/sbin/ Use this script in skeleton-custom.mk instead of a bunch of variables filled by $(shell ...) macros. The same script will be used to check rootfs overlays, in a forthcoming change. Signed-off-by: Carlos Santos <casantos@datacom.ind.br> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/scripts/pkg-stats: add latest upstream version informationThomas Petazzoni2019-02-061-0/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of https://release-monitoring.org/project/15037/. If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we hope to be able to use a database dump of release-monitoring.org instead. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* .gitlab-ci.yml: use "extends" keywordRicardo Martincoski2019-02-061-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | Replace all YAML anchors with the new "extends" keyword because it is more readable and more flexible (it works across configuration files combined with the new "include" keyword). Readability is more meaningful in .gitlab-ci.yml.in. In the part of .gitlab-ci.yml that is auto-generated by 'make .gitlab-ci.yml' keep the keyword in the same line of the job name. So instead of this: zynqmp_zcu106_defconfig: extends: .defconfig tests.boot.test_atf.TestATFAllwinner: extends: .runtime_test Use this: zynqmp_zcu106_defconfig: { extends: .defconfig } tests.boot.test_atf.TestATFAllwinner: { extends: .runtime_test } Do this to to keep .gitlab-ci.yml easier to be post-processed by a script. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/scripts/setlocalversion: ignore user settings for MercurialThomas De Schampheleire2019-02-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | setlocalversion will use 'hg id' to determine whether or not the current revision is tagged. If there is no tag, the Mercurial revision is printed, otherwise nothing is printed. The problem is that the user may have custom configuration settings (in their ~/.hgrc file or similar) that changes the output of 'hg id' in a way that the script does not expect. In such cases, the Mercurial revision may not be printed or printed incorrectly. It is good practice to ignore the user environment when calling Mercurial commands from a well-defined script, by setting the environment variable HGRCPATH to the empty string. See also 'hg help environment'. In the particular case of Nokia, a custom extension adds dynamic tags in the repository, i.e. tags that are stored in a file external to the repository and only visible when the extension is active. These tags should not influence the behavior of setlocalversion as they are not official Buildroot tags, i.e. even if a revision is tagged, the Mercurial revision should still be printed. Note that this still does not solve the problem where an organization adds _real_ tags in their Buildroot repository. For example, there might be a moving tag 'last-validated' or tags indicating in which product release that Buildroot revision was used. In these cases, setlocalversion will still not behave as expected, i.e. show the Mercurial revision. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/setlocalversion: fix detection of hg revision when _not_ on ↵Thomas De Schampheleire2019-02-041-1/+1
| | | | | | | | | | | | | | | | | | | | | branch 'default' When Buildroot is stored in a Mercurial repository on a branch other than 'default' ('master' in git terms), setlocalversion (used to populate /etc/os-release) will incorrectly think that this is a tagged version and will NOT print out the revision hash. This is due to the fact that the output of 'hg id' is assumed to be "<revision> <tags-if-any>" but when on a branch it actually is: "<revision> (<branch>) <tags-if-any>" To let setlocalversion receive the output it expects, explicitly ask 'hg id' to retrieve only the revision hash and any tags, ommitting any branch information. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/graph-build-time: replace confusing colorsThomas De Schampheleire2019-02-041-1/+1
| | | | | | | | | | | The color for 'extract' is very similar to the one for 'install-images'. Both are cyan-like. Replace the former by a pale blue to make all colors sufficiently distinct. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/scripts/graph-build-time: add download timesMathias De Mare2019-02-041-3/+3
| | | | | | | | | | | | | | | Total build time also involves download. Getting a visibility on the impact of that step can be important for users/admins, e.g. to evaluate different methods of BR2_PRIMARY_SITE. Colors used are some kind of purple (primary scheme) and light orange (alternate scheme). Signed-off-by: Mathias De Maré <mathias.de_mare@nokia.com> [ThomasDS: rebase and update colors to avoid confusion] Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/scripts/check-host-rpath: document existing functionsThomas Petazzoni2019-01-171-0/+12
| | | | | | | | | | | As suggested by Arnout Vandecappelle, let's document the elf_needs_rpath() and check_elf_has_rpath() functions, before we make them a bit more complicated with per-package directory support. Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* Makefile: offload .gitlab-ci.yml generationRicardo Martincoski2018-12-091-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GitLab has severe limitations imposed to triggers. Using a variable in a regexp is not allowed: | only: | - /-$CI_JOB_NAME$/ | - /-\$CI_JOB_NAME$/ | - /-%CI_JOB_NAME%$/ Using the key 'variables' always lead to an AND with 'refs', so: | only: | refs: | - branches | - tags | variables: | - $CI_JOB_NAME == $CI_COMMIT_REF_NAME would make the push of a tag not to trigger all jobs anymore. Inheritance is used only for the second level of keys, so: |.runtime_test: &runtime_test | only: | - tags |tests.package.test_python_txaio.TestPythonPy2Txaio: | <<: *runtime_test | only: | - /-TestPythonPy2Txaio$/ would override the entire key 'only', making the push of a tag not to trigger all jobs anymore. So, in order to have a trigger per job and still allow the push of a tag to trigger all jobs (all this in a follow up patch), the regexp for each job must be hardcoded in the .gitlab-ci.yml and also the inherited values for key 'only' must be repeated for every job. This is not a big issue, .gitlab-ci.yml is already automatically generated from a template and there will be no need to hand-editing it when jobs are added or removed. Since the logic to generate the yaml file from the template will become more complex, move the commands from the main Makefile to a script. Using Python or other advanced scripting language for that script would be the most versatile solution, but that would bring another dependency on the host machine, pyyaml if Python is used. So every developer that needs to run 'make .gitlab-ci.yml' and also the docker image used in the GitLab pipelines would need to have pyyaml pre-installed. Instead of adding the mentioned dependency, keep using a bash script. While moving the commands to the script: - mimic the behavior of the previous make target and fail on any command that fails, by using 'set -e'; - break the original lines in one command per line, making the diff for any patch to be applied to this file to look nicer; - keep the script as simple as possible, without functions, just a script that executes from the top to bottom; - do not perform validations on the input parameters, any command that fails already makes the script to fail; - do not add an usage message, the script is not intended to be called directly. This patch does not change functionality. Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> [Thomas: make the script output on stdout rather than take the output file name as second argument.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/graph-depends: introduce MANDATORY_DEPS arrayThomas Petazzoni2018-12-061-1/+6
| | | | | | | | | This array will be re-used in another function in a follow-up commit, so it makes sense to factor it out. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/graph-depends: use proper rootpkg in remove_extra_deps()Thomas Petazzoni2018-12-061-4/+4
| | | | | | | | | | | | | | | | | | | | | The remove_extra_deps() function removes dependencies that we are not interested in seeing in the dependency graph. It does this for all packages, except the 'all' package, which on full dependency graphs is the root of the tree. However, this doesn't take into account package-specific dependency graphs (i.e make <pkg>-graph-depends) where the root is not 'all', but '<pkg>'. Due to this, dependencies on "mandatory deps" were not visible at all, i.e the toolchain package (and its dependencies) and the skeleton package (and its dependencies) were not displayed in package-specific dependency graphs. To fix this, we use the existing rootpkg variable instead of hardcoding 'all'. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/graph-depends: drop legacy target-exceptionsYann E. MORIN2018-12-061-9/+0
| | | | | | | | | | | | | | | | | | | | | | | | Currently, we avoid drawing the dependencies that we call 'target exceptions', becasue they initially were returned by 'show-targets', when they in fact were not really packages and thus should not be on the graph. However, those two exceptions have no longer been reported in the output of show-targets since we merged very old initial top-level parallel build way back in 2014, with commit a24877586a56 (Makefile: add support for top-level parallel make), where they had been converted into purely internal rules. 4 years have passed, we can now drop those exceptions from the graph-depends script. This concludes the cleanup initiated three years ago with commit 0b32791f0076 (graph-depends: remove absent targets from TARGET_EXCEPTIONS). Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* Merge branch 'next'Peter Korsgaard2018-12-022-1/+6
|\ | | | | | | Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
| * boot/grub2: add option to install toolsErico Nunes2018-11-261-0/+4
| | | | | | | | | | | | | | | | | | | | | | Add an option to install grub2 support tools to the target. In the context of Buildroot, some useful target tools provided are grub2-editenv, grub2-reboot, which provide means to manage the grub2, environment, boot order, and others. Signed-off-by: Erico Nunes <nunes.erico@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
| * support/scripts/check-host-rpath: split condition on two statementsThomas Petazzoni2018-11-261-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Inside the check_elf_has_rpath(), we check if the host binary has a correct RPATH, which should be either an absolute path to $(HOST_DIR)/lib, or a relative path using $ORIGIN. Those two conditions are checked in a single statements, but as we are going to add a third condition, let's split this up a bit: - If we have a RPATH to $(HOST_DIR)/lib -> we're good, return 0 - If we have a RPATH to $ORIGIN/../lib -> we're good, return 0 - Otherwise, we will exit the loop, and return 1 Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* | support/graph-depends: fix package names starting with a non-alphaYann E. MORIN2018-11-241-2/+3
|/ | | | | | | | | | | | | | | | Graphviz' dot utility does not like nodes which names does not start with an ^[[:alpha:]], i.e. 18xx-ti-utils would cause grievance: Warning: syntax ambiguity - badly delimited number '18x' in line 4 [...]/graph-depends.dot splits into two tokens Warning: syntax ambiguity - badly delimited number '18x' in line 5 [...]/graph-depends.dot splits into two tokens Warning: syntax ambiguity - badly delimited number '18x' in line 6 [...]/graph-depends.dot splits into two tokens Warning: syntax ambiguity - badly delimited number '18x' in line 7 [...]/graph-depends.dot splits into two tokens Prefix nodes with an underscore to fix that. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/mkmakefile: make wrapper silent by defaultSerj Kalichev2018-11-021-1/+8
| | | | | | | | | | | | | | | Suppose we use Makefile wrapper and build some project out of buildroot tree (O=...). A command like "make busybox-all-external-deps" will output the string "uname 022 && make ..." to stdout before the usefull information. It pollutes stdout. At the same time if we use the same command in the buildroot source-tree then we don't get the additional output. This patch makes wrapper silent by default. People who prefer to see more verbose output can use V=1. Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* package/pkg-generic.mk: increase precision of timestampsThomas Petazzoni2018-10-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | Currently, the timestamps that we keep in build-time.log use a second-level precision. However, as we are going to introduce a new type of graph to draw the time line of a build, this precision is going to be insufficient, as a number of steps are so short that they are not even one second long, and generally the rounding to the second gives a not so great looking graph. Therefore, we add to the timestamps the nanoseconds using the %N date specifier. A milli-second precision would have been sufficient, but %N is all what date(1) provides at the sub-second level. Since this is changing the format of the build-time.log file, this commit adjusts the support/scripts/graph-build-time script accordingly, to account for the floating point numbers that we have as timestamps. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats: URL check using threadsMatt Weber2018-10-091-6/+19
| | | | | | | | | | | | | Adds a pool of worker threads to accelerate connection testing. ~7.5MB and 2% CPU per thread on a Intel i5-3230M CPU @ 2.60GHz. Runtime is ~3min in parallel vs ~15min. CC: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats: URL checking supportMatt Weber2018-10-091-0/+57
| | | | | | | | | | | | | | | | | | | - Adds support to check if a package has a URL and if that URL is valid by doing a header request. - Reports this information as part of the generated html output The URL data is currently gathered from the URL string provided in the Kconfig help sections for each package. This check helps ensure the URLs are valid and can be used for other scripting purposes as the product's home site/URL. CPE XML generation is an example of a case that could use this product URL as part of an automated update generation script. CC: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/mkusers: preserve group membersJohan Oudinet2018-09-121-2/+11
| | | | | | | | | | | | When the function add_one_group is called on an existing group, make sure the members of this group are not removed in the process of deleting then re-adding the group. Signed-off-by: Johan Oudinet <johan.oudinet@gmail.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> [Thomas: add curly braces when referencing ${members}, as suggested by Yann.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats: use parenthesis for printMatt Weber2018-05-191-6/+6
| | | | | | | | | Use Python 3 style print calls, in order to make pkg-stats Python 3 compliant. Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts: drop xorg-release scriptThomas Petazzoni2018-04-271-180/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This script causes a large number of flake8 warnings, is rarely used (but even never used), and is going to be replaced at some point by the improved pkg-stats that will give details about the upstream version available for all packages, not just X.org packages. Therefore, let's drop the xorg-release script in order to silence all those flake8 warnings: support/scripts/xorg-release:36:1: E302 expected 2 blank lines, found 1 support/scripts/xorg-release:58:27: E201 whitespace after '{' support/scripts/xorg-release:58:44: E203 whitespace before ':' support/scripts/xorg-release:58:54: E202 whitespace before '}' support/scripts/xorg-release:63:1: E305 expected 2 blank lines after class or function definition, found 1 support/scripts/xorg-release:64:15: E261 at least two spaces before inline comment support/scripts/xorg-release:67:32: E261 at least two spaces before inline comment support/scripts/xorg-release:86:1: E302 expected 2 blank lines, found 1 support/scripts/xorg-release:95:1: E302 expected 2 blank lines, found 1 support/scripts/xorg-release:107:1: E302 expected 2 blank lines, found 1 support/scripts/xorg-release:115:20: W601 .has_key() is deprecated, use 'in' support/scripts/xorg-release:123:34: E201 whitespace after '{' support/scripts/xorg-release:124:46: E203 whitespace before ':' support/scripts/xorg-release:124:50: E202 whitespace before '}' support/scripts/xorg-release:127:1: E302 expected 2 blank lines, found 1 support/scripts/xorg-release:141:15: W601 .has_key() is deprecated, use 'in' support/scripts/xorg-release:146:21: W601 .has_key() is deprecated, use 'in' support/scripts/xorg-release:176:1: E305 expected 2 blank lines after class or function definition, found 1 support/scripts/xorg-release:180:1: W391 blank line at end of file Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: Bernd Kuhls <bernd.kuhls@t-online.de> Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/check-uniq-files: remove csv module importThomas Petazzoni2018-04-251-1/+0
| | | | | | | | | | | | | | Since commit 5563a1c6a48716debe2983869ddb757318094dce ("support/check-uniq-files: support weird locales and filenames"), the 'csv' Python module is no longer used by the check-uniq-files. Due to this, flake8 complains with: support/scripts/check-uniq-files:4:1: F401 'csv' imported but unused Fix this by dropping the useless csv import. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/fix-rpath: exclude /lib/firmware in the targetThomas Petazzoni2018-04-121-0/+6
| | | | | | | | | | | | | | | | | | | | The /lib/firmware directory contains random firmware for various devices. It happens that some of them might be or appear to be ELF files, but they shouldn't be checked by fix-rpath. For example, one of the Qualcomm VPU firmware file appears to be an ELF file, but patchelf isn't happy about it: $ ./output/host/bin/patchelf --print-rpath output/target/lib/firmware/qcom/venus-4.2/venus.b00 patchelf: patchelf.cc:387: void ElfFile<Elf_Ehdr, Elf_Phdr, Elf_Shdr, Elf_Addr, Elf_Off, Elf_Dyn, Elf_Sym>::parse() [with Elf_Ehdr = Elf32_Ehdr; Elf_Phdr = Elf32_Phdr; Elf_Shdr = Elf32_Shdr; Elf_Addr = unsigned int; Elf_Off = unsigned int; Elf_Dyn = Elf32_Dyn; Elf_Sym = Elf32_Sym]: Assertion `shstrtabIndex < shdrs.size()' failed. Aborted (core dumped) Even though patchelf definitely shouldn't crash, it anyway doesn't make sense to check ELF files in /lib/firmware, so let's exclude this directory from our check. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats: replace with new Python versionThomas Petazzoni2018-04-042-920/+465
| | | | | | Rename pkg-stats-new to pkg-stats. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats-new: add current version informationThomas Petazzoni2018-04-041-0/+46
| | | | | | | | | | | This commit adds a new column in the HTML output containing the current version of a package in Buildroot. As such, it isn't terribly useful, but combined with the latest upstream version added in a follow-up commit, it will become very useful. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats-new: add -n and -p optionsThomas Petazzoni2018-04-041-2/+23
| | | | | | | | | | | | | | | | | | This commit adds the following options to the pkg-stats-new script: -n, to specify a number of packages to parse instead of all packages -p, to specify a list of packages (comma-separated) to parse instead of all packages These options are basically only useful when debugging/developing this script, but they are very useful, because the script is rather slow to run completely with all 2000+ packages, especially once upstream versions will be fetched from release-monitoring.org. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* support/scripts/pkg-stats-new: rewrite in PythonThomas Petazzoni2018-04-041-0/+459
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit adds a new version of the pkg-stats script, rewritten in Python. It is for now implemented in a separate file called, pkg-stats-new, in order to make the diff easily readable. A future commit will rename it to pkg-stats. Compared to the existing shell-based pkg-stats script, the functionality and output is basically the same. The main difference is that the output no longer goes to stdout, but to the file passed as argument using the -o option. This allows stdout to be used for more debugging related information. The way the script works is that a first function get_pkglist() returns a list of Package objects. Then, the function package_init_make_info() uses 'make printvars' to gather information about all packages, stored as class variables in the Package class. Then, we iterate over all packages, and use various methods of the Package class to retrieve all details about the package: infrastructure, presence of hash file, presence of license information, etc. calculate_stats() then calculates global statistics (how packages have license information, how packages have a hash file, etc.). Finally, dump_html() produces the HTML output, using a number of sub-functions. One improvement over the shell-based version is that we can use regexps to exclude some .mk files. Thanks to this, we can exclude all linux-ext-*.mk files, avoiding incorrect matches. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
* support/scripts/graph-depends: add --flat-list optionGeorge Redivo2018-04-011-7/+19
| | | | | | | | | | | | | | | | | | graph-depends currently spits out a graph in .dot format. However, as part of the upcoming introduction of <pkg>-show-recursive-depends and <pkg>-show-recursive-rdepends, we need graph-depends to be able to display a flat list. Signed-off-by: George Redivo <george.redivo@datacom.ind.br> [Thomas: - Rebase on top of graph-depends changes - Do not display the package name itself in the list, only its dependencies (or reverse dependencies) - Display the result on a single line, instead of one package per line, in order to match what <pkg>-show-depends and <pkg>-show-rdepends are doing today.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/scripts/graph-depends: add --quiet optionPeter Korsgaard2018-04-011-1/+3
| | | | | | | | | This will be useful for the upcoming recursive show-depends and show-rdepends features. Signed-off-by: Peter Korsgaard <peter@korsgaard.com> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/scripts/graph-depends: use the standard python logging modulePeter Korsgaard2018-04-012-12/+17
| | | | | | | | | Instead of hardcoded sys.stderr.write() calls. No functional change, but allows us to easily implement a quiet option. Signed-off-by: Peter Korsgaard <peter@korsgaard.com> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/scripts/graph-depends: use colors instead of coloursThomas Petazzoni2018-04-011-8/+8
| | | | | | | | | | | | | | | | The graph-depends was not very consistent in colors vs. colours: some parts were using colours, some parts were using colors. Let's settle on the US spelling, colors. This change the user-visble option --colours to --colors, but it is unlikely that a lot of users customize the colors through BR2_GRAPH_DEPS_OPTS, so this user interface change is considered reasonable. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/scripts/graph-depends: remove global code and most global variablesThomas Petazzoni2018-04-011-145/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The graph-depends script had no main() function, and the main code was actually spread between the function definitions, which was a real mess. This commit moves the global code into a main() function, which allows to more easily follow the flow of the script. The argument parsing code is moved into a parse_args() function. Most of the global variables are removed, and are instead passed as argument when appropriate. This has the side-effect that the print_pkg_deps() function takes a lot of argument, but this is considered better than tons of global variables. The global variables that are removed are: max_depth, transitive, mode, root_colour, target_colour, host_colour, outfile, dict_deps, dict_version, stop_list, exclude_list, arrow_dir. The root_colour/target_colour/host_colour variables are entirely removed, and instead a single colours array is passed, and it's the function using the colors that actually uses the different entries in the array. The way the print_attrs() function determines if we're display the root node is not is changed. Instead of relying on the package name and the mode (which requires passing the root package name, and the mode), it relies on the depth: when the depth is 0, we're at the root node. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/check-uniq-files: support weird locales and filenamesYann E. MORIN2018-03-311-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, when a filename contains characters not representable in the user's locale, we fail hard, especially when the host python is python3. This is because python2 and python3 handle encoding/decoding strings differently, with python3 presumable doing the right thing, but it breaks on some systems, while python2 presumable does the wrong thing, but it works everywhere. (Just joking, obviously...) Part of the issue being that the csv reader in python2 is broken with UTF8. We fix the issue by ditching the csv reader, and simply read the file in binary mode, manually partitioning the lines on the first comma. Then, we use the binary-encoded (really, un-encoded) package names and filenames as values and keys, respectively. Finally, for each filename or package we need to print, we try to decode them with the defaults for the user settings, but catch any decoding exception and fall back to dumping the raw, binary values. Which codec is used by default differs between Python version, but in all cases something sane is printed at least. Thanks a lot to Arnout for the live help doing this patch. :-) Reported-by: Jaap Crezee <jaap@jcz.nl> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Jaap Crezee <jaap@jcz.nl> [Arnout: commit log improvement] Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* support/check-bin-arch: exclude kernel modules for merged /usrYann E. MORIN2018-03-071-0/+1
| | | | | | | | | | | When using a merged /usr, the kernel module path is really /usr/lib/modules, as /lib is a symlink to usr/lib . Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Cc: Peter Korsgaard <peter@korsgaard.com> Reviewed-by: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* spport/check-bin-arch: accept arbitrary per-package ignore pathsYann E. MORIN2018-03-071-22/+34
| | | | | | | | | | | | | | | Some packages (mostly, out-of-tree) may want to install binary blobs for another architecture, outside the locations we currently exclude, like in /opt or whatever... Add support in check-bin-arch to accept any arbitrary location, that individual package can each request to excude from the check, when they are installed. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* support/scripts/check-uniq-files: add indices in format stringThomas Petazzoni2018-03-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Using {} in format strings is only supported in sufficiently recent Python versions. Python 2.6 doesn't support this, and only format strings with numbered arguments: {0}, {1}, etc. Python 2.7: $ python -c 'print("foo {}".format(12))' foo 12 $ python -c 'print("foo {0}".format(12))' foo 12 Python 2.6: $ python -c 'print("foo {}".format(12))' Traceback (most recent call last): File "<string>", line 1, in <module> ValueError: zero length field name in format $ python -c 'print("foo {0}".format(12))' foo 12 Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* check-bin-arch: skip /lib/modules to allow 32-bit userland on 64-bit archThomas De Schampheleire2018-02-251-0/+8
| | | | | | | | | | | | | | | | | | The script check-bin-arch fails as follows on a config for PowerPC e6500 (64-bit CPU) with BR2_ARCH="powerpc" (32-bit userland desired): ERROR: architecture for "/lib/modules/..../lib/libcrc32c.ko" is "PowerPC64", should be "PowerPC" This situation is perfectly acceptable: the kernel is 64-bit and so are its modules, even though userland is 32-bit. To keep check-bin-arch and its caller simple, just skip /lib/modules/ entirely, like is done for /lib/firmware and some others. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
* size-stats: fix code styleRicardo Martincoski2018-01-291-1/+7
| | | | | | | | | | | | Fix these warnings: E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 E713 test for membership should be 'not in' Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* pycompile: fix code styleRicardo Martincoski2018-01-291-0/+3
| | | | | | | | | | | | Fix these warnings: E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yegor Yefremov <yegorslists@googlemail.com> Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* graph-build-time: fix code styleRicardo Martincoski2018-01-291-9/+15
| | | | | | | | | | | | | | | | Fix these warnings: E201 whitespace after '[' E202 whitespace before ']' E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 Ignore these warnings: E402 module level import not at top of file Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* check-uniq-files: fix code styleRicardo Martincoski2018-01-291-1/+3
| | | | | | | | | | | Fix these warnings: E128 continuation line under-indented for visual indent E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Yann E. MORIN <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* graph-depends: fix code styleRicardo Martincoski2018-01-292-28/+48
| | | | | | | | | | | | | | | | | | | Fix these warnings: E122 continuation line missing indentation or outdented E127 continuation line over-indented for visual indent E128 continuation line under-indented for visual indent E202 whitespace before ']' E221 multiple spaces before operator E225 missing whitespace around operator E231 missing whitespace after ',' E302 expected 2 blank lines, found 1 E305 expected 2 blank lines after class or function definition, found 1 E502 the backslash is redundant between brackets E713 test for membership should be 'not in' Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/scripts: sort pie chartsThomas De Schampheleire2018-01-022-2/+2
| | | | | | | | | | Make sure that the pie charts produced by 'graph-build' and 'graph-size' targets are sorted on the size of each piece of the pie. Otherwise, making visual analysis is difficult, as one needs to look at the legends of each piece and do the sorting manually in their head. Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* support/scripts/genimage.sh: show usage when invoked incorrectlyLuca Ceresoli2017-12-111-1/+5
| | | | | | [Peter: use ${0}] Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
* Merge branch 'next'Thomas Petazzoni2017-12-011-0/+40
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This merges the next branch accumulated during the 2017.11 release cycle back into the master branch. A few conflicts had to be resolved: - In the DEVELOPERS file, because Fabrice Fontaine was added as a developer for libupnp in master, and for libupnp18 in next. Resolution is simple: add him for both. - linux/Config.in, because we updated the 4.13.x release used by default in master, while we moved to 4.14 in next. Resolution: use 4.14. - package/libupnp/libupnp.hash: a hash for the license file was added in master, while the package was bumped into next. Resolution: keep the hash for the license file, and keep the hash for the newest version of libupnp. - package/linux-headers/Config.in.host: default version of the kernel headers for 4.13 was bumped to the latest 4.13.x in master, but was changed to 4.14 in next. Resolution: use 4.14. - package/samba4/: samba was bumped to 4.6.11 in master for security reasons, but was bumped to 4.7.3 in next. Resolution: keep 4.7.3. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
| * core: check files are not touched by more than one packageYann E. MORIN2017-11-271-0/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
* | support/scripts/size-stats: avoid divide-by-zeroAndrey Yurovsky2017-11-291-2/+10
|/ | | | | | | | | | Some packages (ex: skeleton-init-systemd) have a zero size so we cannot divide by the package size. In that case make their percent zero explicitly and avoid a ZeroDivisionError exception. Signed-off-by: Andrey Yurovsky <yurovsky@gmail.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
OpenPOWER on IntegriCloud