buildroot/support/download/dl-wrapper, branch 2019.02-op-build

support/download/dl-wrapper: fix urlencode option never being sent to backend

2018-12-09T16:16:04+00:00

Since commit 38de434123 ("download: fix file:// BR2_PRIMARY_SITE (download cache)"), the urlencode option is no longer passed to the download backend, because we use ${backend} instead of ${backend_urlencode}. We must get the urlencode information from backend_urlencode. Signed-off-by: Damien Thébault Reviewed-by: "Yann E. MORIN" [Thomas: rework commit log] Signed-off-by: Thomas Petazzoni

Merge branch 'next'

2018-09-07T11:13:17+00:00

Signed-off-by: Peter Korsgaard

download: fix file:// BR2_PRIMARY_SITE (download cache)

2018-08-20T22:24:29+00:00

wget is the only downloader currently usable with BR2_PRIMARY_SITE, and that doesn't work at all for file:// URLs. The symptoms are these: support/download/dl-wrapper -c '2.4.47' -d '/PATH/build/sw/source/attr' -D '/PATH/build/sw/source' -f 'attr-2.4.47.src.tar.gz' -H 'package/attr//attr.hash' -n 'attr-2.4.47' -N 'attr' -o '/PATH/build/sw/source/attr/attr-2.4.47.src.tar.gz' -u file\|urlencode+file:///NFS/buildroot_dl_cache/attr -u file\|urlencode+file:///NFS/buildroot_dl_cache -u http+http://download.savannah.gnu.org/releases/attr -u http\|urlencode+http://sources.buildroot.net/attr -u http\|urlencode+http://sources.buildroot.net -- file:///NFS/buildroot_dl_cache/attr/attr-2.4.47.src.tar.gz: Unsupported scheme `file'. ERROR: attr-2.4.47.src.tar.gz has wrong sha256 hash: ERROR: expected: 25772f653ac5b2e3ceeb89df50e4688891e21f723c460636548971652af0a859 ERROR: got : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ERROR: Incomplete download, or man-in-the-middle (MITM) attack In the case of custom Linux kernel versions, this is fatal, because there isn't necessarily a hash file to indicate that wget's empty tarball is wrong. This seems to have been broken by commit c8ef0c03b0b, because: 1. BR2_PRIMARY_SITE always appends "urlencode" (package/pkg-download.mk) 2. Anything with the "|urlencode" suffix in $uri will end up using wget due to the backend case wildcarding. 3. The wget backend rejects file:/// URLs ("unsupported scheme"), and we end up with an empty .tar.gz file in the downloads directory. Fix that by shell-extracting the backend name from the left of "|". I'm not positive if all URLs will have a "|", so this code only looks for a "|" left of the "+". Signed-off-by: Hollis Blanchard Reviewed-by: "Yann E. MORIN" Tested-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

support/download: remove help from wrapper

2018-08-09T21:47:54+00:00

The download wrapper is a purely internal helper, and is not supposed to be callable manually. No need to offer some help. Besides, the help text was way out-dated. Signed-off-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

dl-wrapper: Fix support for URIs containing '+'

2018-06-04T19:59:55+00:00

'+' is a valid character in a url. The current dl-wrapper gets the URI scheme by dropping everything after the last '+' character, with the intension of finding 'git' from e.g. 'git+https://uri'. If a uri has a '+' anywhere in it, it ends up using too much of the string as a scheme, and fails to match the handler properly. An example of where this form of URI is used is when using deploy tokens in gitlab. It uses a form like https://:@gitlab.com// where username for deploy token is of the form 'gitlab+deploy-token-'. Use the %% operator to search backwards until the last '+' character when dropping the rest of the string as we know that the first '+' in the string should be the scheme. Signed-off-by: Robert Beckett Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

support/download/dl-wrapper: pass the correct -N option

2018-04-12T20:54:00+00:00

${raw_name} is never defined in dl-wrapper, and therefore the value passed to the -N option is always empty. This causes a problem for the 'cvs' backend, which uses the value of this option as the CVS module to be downloaded. If the name of the CVS module is omitted, all the CVS modules from that CVS repository are downloaded, which creates a tarball with a lot more contents, and the actual useful contents in a sub-directory, obviously breaking patches that should be applied, and the entire build process that follows. Fixes: http://autobuild.buildroot.net/results/fcee0e3d7eeeb373313b1794092c729b1b052348/ Signed-off-by: Thomas Petazzoni Tested-by: "Yann E. MORIN" Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

support/download/dl-wrapper: fix passing remaining options to helper scripts

2018-04-12T20:53:55+00:00

When calling the backend-specific helper scripts, the remaining options are in ${@}. However, in order to let the helper script know that those remaining options should not be parsed, but instead passed as-is to the download tool, they must be separated from the main options by "--". Without this, packages that use _DL_OPTS, such as the amd-catalyst package, cannot download their tarball anymore. Fixes: http://autobuild.buildroot.net/results/de818f6e4c8e63d5e8a49c445d10c34eccc40410/ Signed-off-by: Thomas Petazzoni Tested-by: "Yann E. MORIN" Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

core/download: fix when the BR2_DL_DIR does not accept hardlinks

2018-04-03T20:45:00+00:00

When the BR2_DL_DIR is a mountpoint (presumably shared between various machine, or mounted from the local host when running in a VM), it is possible that it does not support hardlinks (e.g. samba, or the VMWare VMFS, etc...). If the hardlink fails, fallback to copying the file. As a last resort, if that also fails, eventually fallback to doing the download. Note: this means that the dl-wrapper is no longer atomic-safe: the code suffers of a TOCTTOU condition: the file may be created in-between the check and the moment we try to ln/cp it. Fortunately, the dl-wrapper is now run under an flock, so we're still safe. If we eventually go for a more fine-grained implementation, we'll have to be careful then. Reported-by: Arnout Vandecappelle Signed-off-by: "Yann E. MORIN" Cc: Arnout Vandecappelle Cc: Peter Korsgaard Cc: Maxime Hadjinlian Signed-off-by: Peter Korsgaard

core/download: look for archives in the global download dir first

2018-04-02T15:48:56+00:00

For existing setups, the global donload directory may have a lot of the required archives, so look into there before attempting a download. We simply hard-link them if found there and not in the new per-package loaction. Then we resume the existing procedure (which means the new hardlink will get removed if it happened to not match the hash). Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Peter Korsgaard Cc: Thomas Petazzoni Signed-off-by: Peter Korsgaard

download: add missing '-d' option

2018-04-02T15:48:56+00:00

The infrastructure needs to give the 'dl_dir' to the dl-wrapper which in its turn needs to give it to the helper. It will only be used by the 'git' helper as of now. Signed-off-by: Maxime Hadjinlian Signed-off-by: Peter Korsgaard