buildroot/support/download, branch 2016.02

support/download: really, really make git archives reproducible

2016-02-27T22:13:55+00:00

The way we use it, gzip will store the current time in the header, which leads to unreproducible archives. Fix that by telling gzip to not store the name and date of the file it compresses, with the -n option. Since it compresses its stdin, there was already no filename stored; now there's even no date stored. Note: gzip has had -n since at least 1.2.4, released in 1993, so virtually every gzip out there nowadays has it. Signed-off-by: "Yann E. MORIN" Signed-off-by: Peter Korsgaard

support/download: alternative access methods to CVS

2016-01-20T22:26:03+00:00

Allows user to specify other access methods than :pserver:anonymous@ on CVS repositories. This shall be defined in the _SITE variable. [Thomas: - as suggested by Yann, quote the variable expansion - as suggested by Yann, use a regexp match - tweak commit log] Signed-off-by: Joao Mano Signed-off-by: Thomas Petazzoni

support/download: support older bazaar versions

2016-01-18T19:46:04+00:00

In efe7f68 (support/download: generate reproducible Bazaar archives), bzr was instructed to store files with the timestamp set to the date they were last modified in the repository, instead of the current date, using the --per-file-timestamp option. However, this option has been added only in bzr-2.2 (August 2010) which is not available on older distros. We fix that by not using --per-file-timestamp when the bzr version is older than 2.2, in which case we just generate the archive with the current date set on files. This means the archive is thus non-reproducible, and if a hash is available for that archive, the hash will not match, and Buildroot will try to download from the mirror (if any) or fail (if no mirror). Fixes: http://autobuild.buildroot.org/results/51f/51f4ff5462c15a85937d411f457096224d00fdcd http://autobuild.buildroot.org/results/b88/b8828b5fbc16128408c2f44169ac23de7e34d770 http://autobuild.buildroot.org/results/fb4/fb4b0fb2131b40c18273dbe5e51b393cb6df18ec ... [Peter: simplify sed invocation] Reported-by: Thomas Petazzoni Signed-off-by: "Yann E. MORIN" Signed-off-by: Peter Korsgaard

support/download: generate reproducible Bazaar archives

2016-01-03T20:51:50+00:00

Similarly to what has previously been done for the Hg download backend, instruct bzr to generate the archive on stdout, so that we can generate reproducible archives. When instructing bzr to generate the output file by itself, it uses a temporary file that is then fed to gzip, which in turn stores the timestamp of that file in the generated archive, whereas when the output is generated on stdout, there is no timestamp, so the archive is then reproducible. Bizarely enough, we can tell 'bazaar' not to generate a bazaar in the archive. Cool, uh? ;-] Signed-off-by: "Yann E. MORIN" Signed-off-by: Peter Korsgaard

support/download: generate reproducible Hg archives

2016-01-03T20:50:38+00:00

When hg directly creates the output file, the hash for that file changes everytime. However, if we just tell hg to output the archive on stdout and we do the redirect to the file, then the archive is reproducible. (The reason is that in the first case, a temporary file is created and then compressed, and gzip is adding the filename and its timestamp in the gzip header, while in the second case, there is no temporary file, and thus no timestamp and thus it is reproducible.) Signed-off-by: "Yann E. MORIN" Cc: Yegor Yefremov Tested-by: Yegor Yefremov Signed-off-by: Peter Korsgaard

support/download: protect from custom commands with spaces in args

2015-12-12T16:00:46+00:00

Some users may provide custom download commands with spaces in their arguments, like so: BR2_HG="hg --config foo.bar='some space-separated value'" However, the way we currently call those commands does not account for the extra quotes, and each space-separated part of the command is interpreted as separate arguments. Fix that by calling 'eval' on the commands. Because of the eval, we must further quote our own arguments, to avoid the eval further splitting them in case there are spaces (even though we do not support paths with spaces, better be clean from the onset to avoid breakage in the future). We change all the wrappers to use a wrapper-function, even those with a single call, so they all look alike. Note that we do not single-quote some of the variables, like ${verbose} because it can be empty and we really do not want to generate an empty-string argument. That's not a problem, as ${verbose} would not normally contain space-separated values (it could get set to something like '-q -v' but in that case we'd still want two arguments, so that's fine). Reported-by: Thomas De Schampheleire Signed-off-by: "Yann E. MORIN" Cc: Thomas De Schampheleire Reviewed-by: Thomas De Schampheleire Tested-by: Thomas De Schampheleire Signed-off-by: Thomas Petazzoni

support/download: fetch all refs on full git clone

2015-11-29T17:52:44+00:00

When specifying BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION, a user may want to specify the SHA of a reference different than a branch or tag. For instance, Gerrit stores the patchsets under refs/changes/xx/xxx, and Github stores the pull requests under refs/pull/xxx/head. When cloning a repository with --bare, you don't fetch these references. This patch uses --mirror for a full clone, in order to give the user access to all references of the Git repository. Signed-off-by: Vivien Didelot Reviewed-by: "Maxime Hadjinlian" Signed-off-by: Thomas Petazzoni

support/download: fix the Hg backend for tags

2015-08-29T09:38:00+00:00

When the version of a package is a Mercurial tag, the download fails, with: abort: unknown revision 'X.Y.Z'! This is because, in Mercurial, tags are commits like the others, and when we clone, we actively request a tag. But then, the server "dereferences" that tag and sends us the revision pointed to by that tag. Of course, since the tag is a commit after the revision we got, we do not have the revision adding the tag. So, we just have to download the full repository to be sure we have the tags in our local clone. Reported-by: Thomas Petazzoni Signed-off-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

core/download: don't be needlessly verbose in backends

2015-07-26T13:18:39+00:00

In 50c8b7e (support/download: support -q in all download backends), the backend were made to respect the quietness of the main Makefile, when -s is poassed on the 'make' command line. In doing so, they were all made to be verbose by default. However, the verbosity of some of the tools, like scp, is very high, and is in fact intended for debug purposes. Drop being verbose by default, just use whatever each tool deems normal output. Only respect the quietness requested by the user. Reported-by: Thomas De Schampheleire Signed-off-by: "Yann E. MORIN" Reviewed-by: Thomas De Schampheleire Signed-off-by: Thomas Petazzoni

support/download/cvs: add support to use a date instead of a tag

2015-07-10T09:09:20+00:00

This is useful when a tag is not avaiable. Also fix support for Fedora where the command "cvs -r :" doesn't work. Signed-off-by: Fabio Porcedda Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni