buildroot/support/scripts/pkg-stats, branch 2019.02-op-build

support/scripts/pkg-stats: fix flake8 errors

2019-02-07T21:09:55+00:00

Fixes the following flake8 warnings: support/scripts/pkg-stats:34:2: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:4: W605 invalid escape sequence '$' support/scripts/pkg-stats:34:11: W605 invalid escape sequence '\$' support/scripts/pkg-stats:34:13: W605 invalid escape sequence '\(' support/scripts/pkg-stats:34:32: W605 invalid escape sequence '$' support/scripts/pkg-stats:34:34: W605 invalid escape sequence '\)' support/scripts/pkg-stats:35:2: W605 invalid escape sequence '\s' support/scripts/pkg-stats:35:14: W605 invalid escape sequence '\S' support/scripts/pkg-stats:35:17: W605 invalid escape sequence '\s' support/scripts/pkg-stats:42:1: E302 expected 2 blank lines, found 1 support/scripts/pkg-stats:587:133: E501 line too long (157 > 132 characters) Note that the "invalid escape sequence" errors work because Python leaves the \ in place if it doesn't recognise the escape sequence. But it's better practice to use a raw string for regular expressions. Signed-off-by: Arnout Vandecappelle (Essensium/Mind)

support/scripts/pkg-stats: add latest upstream version information

2019-02-06T14:57:09+00:00

This commit adds fetching the latest upstream version of each package from release-monitoring.org. The fetching process first tries to use the package mappings of the "Buildroot" distribution [1]. This mapping mechanism allows to tell release-monitoring.org what is the name of a package in a given distribution/build-system. For example, the package xutil_util-macros in Buildroot is named xorg-util-macros on release-monitoring.org. This mapping can be seen in the section "Mappings" of https://release-monitoring.org/project/15037/. If there is no mapping, then it does a regular search, and within the search results, looks for a package whose name matches the Buildroot name. Even though fetching from release-monitoring.org is a bit slow, using multiprocessing.Pool has proven to not be reliable, with some requests ending up with an exception. So we keep a serialized approach, but with a single HTTPSConnectionPool() for all queries. Long term, we hope to be able to use a database dump of release-monitoring.org instead. From an output point of view, the latest version column: - Is green when the version in Buildroot matches the latest upstream version - Is orange when the latest upstream version is unknown because the package was not found on release-monitoring.org - Is red when the version in Buildroot doesn't match the latest upstream version. Note that we are not doing anything smart here: we are just testing if the strings are equal or not. - The cell contains the link to the project on release-monitoring.org if found. - The cell indicates if the match was done using a distro mapping, or through a regular search. [1] https://release-monitoring.org/distro/Buildroot/ Signed-off-by: Thomas Petazzoni Tested-by: Matthew Weber Signed-off-by: Thomas Petazzoni

support/scripts/pkg-stats: URL check using threads

2018-10-09T08:11:47+00:00

Adds a pool of worker threads to accelerate connection testing. ~7.5MB and 2% CPU per thread on a Intel i5-3230M CPU @ 2.60GHz. Runtime is ~3min in parallel vs ~15min. CC: Ricardo Martincoski Signed-off-by: Matthew Weber Reviewed-by: Ricardo Martincoski Signed-off-by: Thomas Petazzoni

support/scripts/pkg-stats: URL checking support

2018-10-09T08:05:46+00:00

- Adds support to check if a package has a URL and if that URL is valid by doing a header request. - Reports this information as part of the generated html output The URL data is currently gathered from the URL string provided in the Kconfig help sections for each package. This check helps ensure the URLs are valid and can be used for other scripting purposes as the product's home site/URL. CPE XML generation is an example of a case that could use this product URL as part of an automated update generation script. CC: Ricardo Martincoski Signed-off-by: Matt Weber Reviewed-by: Ricardo Martincoski Signed-off-by: Thomas Petazzoni

support/scripts/pkg-stats: use parenthesis for print

2018-05-19T11:48:21+00:00

Use Python 3 style print calls, in order to make pkg-stats Python 3 compliant. Signed-off-by: Matthew Weber Reviewed-by: Ricardo Martincoski Signed-off-by: Thomas Petazzoni

support/scripts/pkg-stats: replace with new Python version

2018-04-04T20:05:33+00:00

Rename pkg-stats-new to pkg-stats. Signed-off-by: Thomas Petazzoni

pkg-stats: add exception for nvidia-tegra23.mk

2017-07-06T19:46:04+00:00

This file is not a package per-se, it includes other .mk files that are packages. Signed-off-by: Thomas Petazzoni

pkg-stats: take into account waf packages

2017-07-06T19:45:41+00:00

The waf package infrastructure was not known by the pkg-stats script, so let's add it. Signed-off-by: Thomas Petazzoni

pkg-stats: allow to sort by column

2017-07-06T19:38:43+00:00

With 2000+ packages it's not trivial to identify i.e.: - all packages that don't have a hash file; - all packages that have patches; - all packages that have code style warnings; User experience can be improved by dynamically sorting the resulting table. There is an open-source solution that does that in the client-side and requires minimal changes to our script: sorttable.js. The script is MIT licensed as stated in its website. Also add a hint to the user that the table can be sorted. Signed-off-by: Ricardo Martincoski Cc: Arnout Vandecappelle Signed-off-by: Thomas Petazzoni

tools: rename to 'utils'

2017-07-01T16:07:00+00:00

After some discussion, we found out that "tools" has the four first letters identical to the "toolchain" subfolder, which makes it a bit unpractical with tab-completion. So, this commit renames "tools" to "utils", which is more tab-completion-friendly. This has been discussed with Arnout and Yann. Signed-off-by: Thomas Petazzoni