buildroot/support/download, branch 2019.02-op-build

Restrict tar transform to regular files

2019-04-18T06:14:18+00:00

Without this restriction, symlinks are rewritten and corrupted. Example without the restriction: Input tree (valid): package-githash/file1 package-githash/link -> ./file1 Output tree (broken): package-githash/file1 package-githash/link -> package-githash/file1

support/download/scp: fix download with scheme prefix 'scp://'

2019-02-05T19:32:53+00:00

The scp download helper is broken when the server URL starts with 'scp://'. Such prefix is used in two situations: 1. to let FOO_SITE point to an scp location without explicitly having to set 'FOO_SITE_METHOD = scp' 2. when BR2_PRIMARY_SITE or BR2_BACKUP_SITE points to an scp location. In this case, there is no equivalent of 'SITE_METHOD'. Strip out the scheme prefix, similarly to how the 'file' download helper does it. That helper has the same cases as above. Signed-off-by: Thomas De Schampheleire Signed-off-by: Peter Korsgaard

support/download: fix scp downloads

2019-01-03T21:05:43+00:00

scp download is broken, because scp is called without filename argument and only the server is specified. The call is: scp but should be: scp / Instead of assuming '-u' lists a full URL including filename (which it is not), align with the wget helper where -u is the server URL and -f gives the filename. With this commit, an scp download can work if FOO_SITE_METHOD is explicitly set to 'scp' and the server does not have a scheme prefix 'scp://'. The next commit will handle the case where a scheme prefix is present. Signed-off-by: Thomas De Schampheleire Reviewed-by: "Yann E. MORIN" [Thomas: s/URL/URI/, as noticed by Yann.] Signed-off-by: Thomas Petazzoni

support/download/dl-wrapper: fix urlencode option never being sent to backend

2018-12-09T16:16:04+00:00

Since commit 38de434123 ("download: fix file:// BR2_PRIMARY_SITE (download cache)"), the urlencode option is no longer passed to the download backend, because we use ${backend} instead of ${backend_urlencode}. We must get the urlencode information from backend_urlencode. Signed-off-by: Damien Thébault Reviewed-by: "Yann E. MORIN" [Thomas: rework commit log] Signed-off-by: Thomas Petazzoni

download/git: fix code-style

2018-09-10T20:44:26+00:00

This file uses leading spaces, not TABs. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Signed-off-by: Thomas Petazzoni

Merge branch 'next'

2018-09-07T11:13:17+00:00

Signed-off-by: Peter Korsgaard

download: fix file:// BR2_PRIMARY_SITE (download cache)

2018-08-20T22:24:29+00:00

wget is the only downloader currently usable with BR2_PRIMARY_SITE, and that doesn't work at all for file:// URLs. The symptoms are these: support/download/dl-wrapper -c '2.4.47' -d '/PATH/build/sw/source/attr' -D '/PATH/build/sw/source' -f 'attr-2.4.47.src.tar.gz' -H 'package/attr//attr.hash' -n 'attr-2.4.47' -N 'attr' -o '/PATH/build/sw/source/attr/attr-2.4.47.src.tar.gz' -u file\|urlencode+file:///NFS/buildroot_dl_cache/attr -u file\|urlencode+file:///NFS/buildroot_dl_cache -u http+http://download.savannah.gnu.org/releases/attr -u http\|urlencode+http://sources.buildroot.net/attr -u http\|urlencode+http://sources.buildroot.net -- file:///NFS/buildroot_dl_cache/attr/attr-2.4.47.src.tar.gz: Unsupported scheme `file'. ERROR: attr-2.4.47.src.tar.gz has wrong sha256 hash: ERROR: expected: 25772f653ac5b2e3ceeb89df50e4688891e21f723c460636548971652af0a859 ERROR: got : e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ERROR: Incomplete download, or man-in-the-middle (MITM) attack In the case of custom Linux kernel versions, this is fatal, because there isn't necessarily a hash file to indicate that wget's empty tarball is wrong. This seems to have been broken by commit c8ef0c03b0b, because: 1. BR2_PRIMARY_SITE always appends "urlencode" (package/pkg-download.mk) 2. Anything with the "|urlencode" suffix in $uri will end up using wget due to the backend case wildcarding. 3. The wget backend rejects file:/// URLs ("unsupported scheme"), and we end up with an empty .tar.gz file in the downloads directory. Fix that by shell-extracting the backend name from the left of "|". I'm not positive if all URLs will have a "|", so this code only looks for a "|" left of the "+". Signed-off-by: Hollis Blanchard Reviewed-by: "Yann E. MORIN" Tested-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

support/download: remove help from wrapper

2018-08-09T21:47:54+00:00

The download wrapper is a purely internal helper, and is not supposed to be callable manually. No need to offer some help. Besides, the help text was way out-dated. Signed-off-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

dl-wrapper: Fix support for URIs containing '+'

2018-06-04T19:59:55+00:00

'+' is a valid character in a url. The current dl-wrapper gets the URI scheme by dropping everything after the last '+' character, with the intension of finding 'git' from e.g. 'git+https://uri'. If a uri has a '+' anywhere in it, it ends up using too much of the string as a scheme, and fails to match the handler properly. An example of where this form of URI is used is when using deploy tokens in gitlab. It uses a form like https://:@gitlab.com// where username for deploy token is of the form 'gitlab+deploy-token-'. Use the %% operator to search backwards until the last '+' character when dropping the rest of the string as we know that the first '+' in the string should be the scheme. Signed-off-by: Robert Beckett Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

download/cvs: add a 10 minute timeout

2018-05-31T17:32:45+00:00

Apparently, CVS servers can be deadlocked and in that case clients will retry connecting to them indefinitely. Cfr. http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78/build-end.log Apparently, the sf.net CVS server got in such a deadlock on 2018-05-18, and almost 2 weeks later it is still not fixed. Instead of just hanging, we should fall back on BR2_SECONDARY_SITE. To achieve this, it's sufficient to add a timeout to the CVS command. The timeout value is of course arbitrary. However, we can assume that nobody will be putting large projects under CVS any more. So if the download takes more than 5 minutes, it's probably broken. Let's put the timeout at 10 minutes then. Fixes: http://autobuild.buildroot.net/results/db3/db33d4fa507fb3b4132423cd0a7e25a1fe6e4105 http://autobuild.buildroot.net/results/b6d/b6d927dcc73ac8d754422577dacefff4ff918a5c http://autobuild.buildroot.net/results/23d/23d1034b33d0354de15de2ec4a8ccd0603e8db78 http://autobuild.buildroot.net/results/127/1272a3aa3077e434c9805ec3034f35e6fcc330d4 Signed-off-by: Arnout Vandecappelle (Essensium/Mind) Signed-off-by: Peter Korsgaard