buildroot/support/scripts/graph-depends, branch 2016.02

graph-depends: add support for excluding host packages

2016-02-08T20:29:06+00:00

Just like the --stop-on and --exclude options allow to stop on or exclude virtual packages from the list by passing the "virtual" magic value, this commit extends the graph-depends logic to support a "host" magic value for --stop-on and --exclude. This will allow to draw the graph by stopping on host packages, or by excluding host packages. Signed-off-by: Thomas Petazzoni Tested-by: "Yann E. MORIN" Reviewed-by: "Yann E. MORIN" [Thomas: minor code beautification suggested by Yann E. Morin.] Signed-off-by: Thomas Petazzoni

graph-depends: fix handling of "virtual" in exclude_list

2016-02-08T20:28:19+00:00

The condition to determine if a virtual package should be excluded from the list due to "virtual" being passed in --exclude is under a loop iterating over each entry of the exclude_list, but it doesn't use the iterator of this list. Indeed, the condition contains: "virtual" in exclude_list which checks automatically if "virtual" was passed in the list. Due to this, there is no need for this check to be within the "for p in exclude_list" iteration. This commit fixes that by moving the check outside of the loop. Signed-off-by: Thomas Petazzoni Tested-by: "Yann E. MORIN" Acked-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni

support/graph-depends: teach it to only check dependencies

2016-02-07T22:21:40+00:00

Add an option to graph-depends to only do the dependency checks and not generate the dot program. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Samuel Martin Signed-off-by: Thomas Petazzoni

support/graph-depends: detect circular dependencies

2016-02-07T22:20:38+00:00

Currently, if there is a circular dependency in the packages, the graph-depends script just errors out with a Python RuntimeError which is not caught, resulting in a very-long backtrace which does not provide any hint as what the real issue is (even if "RuntimeError: maximum recursion depth exceeded" is a pretty good hint at it). We fix that by recursing the dependency chain of each package, until we either end up with a package with no dependency, or with a package already seen along the current dependency chain. We need to introduce a new function, check_circular_deps(), because we can't re-use the existing ones: - remove_mandatory_deps() does not iterate, - remove_transitive_deps() does iterate, but we do not call it for the top-level package if it is not 'all' - it does not make sense to use those functions anyway, as they were not designed to _check_ but to _act_ on the dependency chain. Since we've had time-related issues in the past, we do not want to introduce yet another time-hog, so here are timings with the circular dependency check: $ time python -m cProfile -s cumtime support/scripts/graph-depends [...] 28352654 function calls (20323050 primitive calls) in 87.292 seconds Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) 1 0.012 0.012 87.292 87.292 graph-depends:24() 21 0.000 0.000 73.685 3.509 subprocess.py:473(_eintr_retry_call) 7 0.000 0.000 73.655 10.522 subprocess.py:768(communicate) 7 73.653 10.522 73.653 10.522 {method 'read' of 'file' objects} 5/1 0.027 0.005 43.488 43.488 graph-depends:164(get_all_depends) 5 0.003 0.001 43.458 8.692 graph-depends:135(get_depends) 1 0.001 0.001 25.712 25.712 graph-depends:98(get_version) 1 0.001 0.001 13.457 13.457 graph-depends:337(remove_extra_deps) 1717 1.672 0.001 13.050 0.008 graph-depends:290(remove_transitive_deps) 9784086/2672326 5.079 0.000 11.363 0.000 graph-depends:274(is_dep) 2883343/1980154 2.650 0.000 6.942 0.000 graph-depends:262(is_dep_uncached) 1 0.000 0.000 4.529 4.529 graph-depends:121(get_targets) 2883343 1.123 0.000 1.851 0.000 graph-depends:246(is_dep_cache_insert) 9784086 1.783 0.000 1.783 0.000 graph-depends:255(is_dep_cache_lookup) 2881580 0.728 0.000 0.728 0.000 {method 'update' of 'dict' objects} 1 0.001 0.001 0.405 0.405 graph-depends:311(check_circular_deps) 12264/1717 0.290 0.000 0.404 0.000 graph-depends:312(recurse) [...] real 1m27.371s user 1m15.075s sys 0m12.673s The cumulative time spent in check_circular_deps is just below 0.5s, which is largely less than 1% of the total run time. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Samuel Martin Signed-off-by: Thomas Petazzoni

support/graph-depends: add option to specify output file

2016-02-07T22:15:30+00:00

Currently, graph-depends outputs the dotfile program to stdout, and uses stderr to trace the dependencies it is currently looking for. Redirection was done because the output was directly piped into the dot program to generate the final PDF/SVG/... dependency graph, but that meant that an error in the graph-depends script was never caught (because shell pipes only return the final command exit status, and an empty dot program is perfectly valid so dot would not complain). Add an option to tell graph-depends where to store the generated dot program, and keep stdout as the default if not specified. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Samuel Martin [Thomas: rename metavar from DOT_FILE to OUT_FILE for consistency with the rest of the new option naming.] Signed-off-by: Thomas Petazzoni

graph-depends: correct is_dep() comment

2015-12-29T22:46:10+00:00

The uncached variant is called is_dep_uncached(), not is_dep_full(). Signed-off-by: Peter Korsgaard

graph-depends: optimize remove_transitive_deps()

2015-12-29T22:26:22+00:00

For large configurations, the execution time of remove_transitive_deps() becomes really high, due to the number of nested loops + the is_dep() function being recursive. For an allyespackageconfig, the remove_extra_deps() function takes 334 seconds to execute, and the overall time to generate the .dot file is 6 minutes and 39 seconds. Here is a timing of the different graph-depends steps and the overall execution time: Getting dependencies: 42.5735 seconds Turn deps into a dict: 0.0023 seconds Remove extra deps: 334.1542 seconds Get version: 22.4919 seconds Generate .dot: 0.0197 seconds real 6m39.289s user 6m16.644s sys 0m8.792s By adding a very simple cache for the results of is_dep(), we bring down the execution time of the "Remove extra deps" step from 334 seconds to just 4 seconds, reducing the overall execution time to 1 minutes and 10 seconds: Getting dependencies: 42.9546 seconds Turn deps into a dict: 0.0025 seconds Remove extra deps: 4.9643 seconds Get version: 22.1865 seconds Generate .dot: 0.0207 seconds real 1m10.201s user 0m47.716s sys 0m7.948s Signed-off-by: Thomas Petazzoni Tested-by: Gustavo Zacarias [yann.morin.1998@free.fr: - rename is_dep() to is_dep_uncached(), keep existig code as-is - add is_dep() as a cached-version of is_dep_uncached() - use constructs more conform with 2to3 - use exceptions (EAFP) rather than check-before-use (LBYL) to be more pythonist; that even decreases the duration yet a little bit more! ] Signed-off-by: Yann E. MORIN

graph-depends: Strip skeleton from dependency

2015-07-14T16:10:58+00:00

skeleton being a mandatory dependency, we don't want all our packages to have a link back to that node, the graph would be awful. Signed-off-by: Maxime Hadjinlian Signed-off-by: Thomas Petazzoni

Makefile: rename TARGETS to PACKAGES

2015-04-14T08:16:51+00:00

For clarity, this commit renames the TARGETS variable to the more meaningful PACKAGES variable. Indeed, only packages (handled by one of the package infrastructures) should be listed in this variable, and not other random non-package targets. Signed-off-by: Thomas Petazzoni Reviewed-by: "Yann E. MORIN"

support/graph-depends: allow excluding virtual packages

2015-04-01T18:02:54+00:00

Like for --stop-on, make --exclude recognise the keyword 'virtual', to stop on virtual packages (as explained in the help...). Reported-by: Thomas Petazzoni Signed-off-by: "Yann E. MORIN" Signed-off-by: Thomas Petazzoni