| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
|
| |
This is especially useful when lit is invoked indirectly by the build
system, and additional arguments can not be easily specified.
Differential Revision: https://reviews.llvm.org/D35091
llvm-svn: 307339
|
|
|
|
|
|
|
| |
This is a very light refactoring aimed at improving readability. There
is definitely still room for improvement here.
llvm-svn: 307310
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix by Andrew Ng!
The Visual Studio build can contain output for multiple configuration types (
e.g. Debug, Release & RelWithDebInfo) within the same build output
directory. Therefore when discovering unit tests, the "build mode" sub directory
containing the appropriate configuration is included in the search. This sub
directory may not always be present, so a test for its existence is required.
Reviewers: zturner, modocache, dlj
Reviewed By: zturner, dlj
Subscribers: grimar, bd1976llvm, gbreynoo, edd, jhenderson, llvm-commits
Differential Revision: https://reviews.llvm.org/D34976
llvm-svn: 307235
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
The lit test formats use largely the same logic for discovering tests. There are
some superficial differences in the logic, which seem reasonable enough to
handle in a single routine.
At a high level, the common goal is "look for files that end with one of these
suffixes, and skip anything starting with a dot." The balance of the logic
specific to ShTest and GoogleTest collapses quite a bit, so that
getTestsInDirectory is only a couple of lines around a call to the new function.
Reviewers: zturner, MatzeB, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34855
llvm-svn: 306895
|
|
|
|
|
|
|
|
|
| |
This reverts commit da6318a92fba793e4f2447ec478b001392d57d43.
This is causing failures on some build bots due to what appears
to be some kind of lit ordering dependency.
llvm-svn: 306833
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Presently lit leaks files in the tests' output directories.
Specifically, if a test creates output files, lit makes no
effort to remove them prior to the next test run. This is
problematic because it leads to false positives whenever a
test passes because stale files were present. In general
it is a source of flakiness that should be removed.
This patch addresses this by building the list of all test
directories that are part of the current run set, and then
deleting those directories and recreating them anew. This
gives each test a clean baseline to start from.
Differential Revision: https://reviews.llvm.org/D34732
llvm-svn: 306832
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
de-duplicate across modules that used it.
(Take 2: this patch re-applies r306625, which was reverted in r306629. This
patch includes only trivial fixes.)
In Python2 and Python3, the various (non-)?Unicode string types are sort of
spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which
is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere"
approach, with 'str' representing a Unicode string.
Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes.
However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting
problems when an interface requires a precise type, but has to run under both
Python2 and Python3.
The previous logic appeared to be correct in all cases, but went through more
layers of indirection than necessary. This change does the necessary conversions
in one shot, with documentation about which paths might be taken in Python2 or
Python3.
Changes from r306625: some tests just print binary outputs, so in those cases,
fall back to str() in Python3. For googletests, add one missing call to
to_string().
(Tested by verifying the visible breakage with Python3. Verified that everything
works in py2 and py3.)
llvm-svn: 306643
|
|
|
|
|
|
|
|
| |
de-duplicate across modules that used it."
This reverts r306625.
llvm-svn: 306629
|
|
|
|
|
|
|
| |
Remember kids: there is no 'I' in str or bytes, but there is ALWAYS an
'I' in unicode.
llvm-svn: 306626
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
across modules that used it.
Summary:
In Python2 and Python3, the various (non-)?Unicode string types are sort of
spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which
is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere"
approach, with 'str' representing a Unicode string.
Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes.
However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting
problems when an interface requires a precise type, but has to run under both
Python2 and Python3.
The previous logic appeared to be correct in all cases, but went through more
layers of indirection than necessary. This change does the necessary conversions
in one shot, with documentation about which paths might be taken in Python2 or
Python3.
Reviewers: zturner, modocache
Subscribers: llvm-commits, sanjoy
Differential Revision: https://reviews.llvm.org/D34793
llvm-svn: 306625
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This change removes the intermediate 'FileBasedTest' format from lit. This
format is only ever used by the ShTest format, so the logic can be moved into
ShTest directly.
In order to better clarify what the TestFormat subclasses do, I fleshed out the
TestFormat base class with Python's notion of abstract methods, using
@abc.abstractmethod. This gives a convenient way to document the expected
interface, without the risk of instantiating an abstract class (that's what
ABCMeta does -- it raises an exception if you try to instantiate a class which
has abstract methods, but not if you instantiate a subclass that implements
them).
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34792
llvm-svn: 306623
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
names.
Summary:
The dead code seems to be unreferenced, according to textual search across the
LLVM SVN repo.
The clarification part of this change alters the name of a module-level function
so that it is different from the name of the class-methods that call it.
Currently, there are no erroneous references, but stylistically (c.f. PEP-8),
internal "helper" functions should generally be named accordingly by prepending
an underscore. (I also chose to add '_impl', which isn't necessary, but helps me
at least to mentally disambiguate the interface and implementation functions.)
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34775
llvm-svn: 306600
|
|
|
|
|
|
|
|
|
| |
on macOS
This function will be used to tie Clang's Integeration tests to a particular
SDK version. See https://reviews.llvm.org/D32178 for more context.
llvm-svn: 304541
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
For various clang analyzer tests, which were unsupported, I got lit
exceptions, similar to the following:
Exception during script execution:
Traceback (most recent call last):
File "utils/lit/lit/run.py", line 190, in execute_test
result = test.config.test_format.execute(test, lit_config)
File "tools/clang/test/Analysis/analyzer_test.py", line 11, in execute
if result.code == lit.Test.FAIL:
AttributeError: 'tuple' object has no attribute 'code'
This is because executeShTest() in utils/lit/lit/TestRunner.py is
supposed to return a lit.Test.Result object, but in case of unsupported
tests, it returns a plain tuple.
Fix this by returning a properly initialized lit.Test.Result object
instead.
Reviewers: rnk, rafael, modocache
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D33579
llvm-svn: 303943
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This seems to have been present since the beginning of time,
which is quite surprising. The symptom was this: Suppose you
have a test with a run line that looks like this:
RUN: foo | FileCheck %s
foo prints some output and then due to a bug in the program it
asserts. On Windows this results in the program returning a
negative exit code. But if enough output had been printed
already by the tool so that the FileCheck match would succeed
then FileCheck would return 0, and because of bad logic in
lit this 0 return value would overwrite the failed return
value from previous items in the pipeline. This only happened
with negative exit codes.
The most sensible behavior is to just take whatever the first
exit code is. There is no logical ordering defined on exit
codes, so comparing with < and > does not make a lot of sense.
Instead, as soon as we find the first non-successful return
value, that should be the result of the entire expression.
This fixes the issue, as now tests which fail on non-Windows
platforms also fail for me on Windows as well.
llvm-svn: 303440
|
|
|
|
|
|
| |
Some bots are using it.
llvm-svn: 303282
|
|
|
|
|
|
| |
This seems to work on freebsd and openbsd these days.
llvm-svn: 303280
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If all jobs complete successfully, use pool.close() instead of
pool.terminate() before waiting for the workers. Zach Turner reported
that he was getting "access denied" exceptions from pool.terminate().
Make the workers abort immediately without printing to stderr when they
are interrupted.
Finally, catch exceptions when attempting to remove our temporary
testing directory. On abnormal exit, there can often be open handles
that haven't been cleaned up yet.
llvm-svn: 301941
|
|
|
|
|
|
|
|
|
|
|
| |
Both pickling errors encountered on clang bots and Darwin compiler-rt
should now be fixed.
This has no impact on testing time on Linux, and on Windows goes from
88s to 63s for 'check'. The tests pass on Mac, but I haven't compared
execution time.
llvm-svn: 299775
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is necessary to pass the lit test suite at llvm/utils/lit/tests.
There are some pre-existing failures here, but now switching to pools
doesn't regress any tests.
I had to change test-data/lit.cfg to import DummyConfig from a module to
fix pickling problems, but I think it'll be OK if we require test
formats to be written in real .py modules outside lit.cfg files.
I also discovered that in some circumstances AsyncResult.wait() will not
raise KeyboardInterrupt in a timely manner, but you can pass a non-zero
timeout to work around this. This makes threading.Condition.wait use a
polling loop that runs through the interpreter, so it's capable of
asynchronously raising KeyboardInterrupt.
llvm-svn: 299605
|
|
|
|
| |
llvm-svn: 299565
|
|
|
|
| |
llvm-svn: 299564
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This drastically reduces lit test execution startup time on Windows. Our
previous strategy was to manually create one Process per job and manage
the worker pool ourselves. Instead, let's use the worker pool provided
by multiprocessing. multiprocessing.Pool(jobs) returns almost
immediately, and initializes the appropriate number of workers, so they
can all start executing tests immediately. This avoids the ramp-up
period that the old implementation suffers from. This appears to speed
up small test runs.
Here are some timings of the llvm-readobj tests on Windows using the
various execution strategies:
# multiprocessing.Pool:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-process-pool |& grep real: ; done
real: 0m1.156s
real: 0m1.078s
real: 0m1.094s
# multiprocessing.Process:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-processes |& grep real: ; done
real: 0m6.062s
real: 0m5.860s
real: 0m5.984s
# threading.Thread:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-threads |& grep real: ; done
real: 0m9.438s
real: 0m10.765s
real: 0m11.079s
I kept the old code to launch processes in case this change doesn't work
on all platforms that LLVM supports, but at some point I would like to
remove both the threading and old multiprocessing execution strategies.
Reviewers: modocache, rafael
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31677
llvm-svn: 299560
|
|
|
|
| |
llvm-svn: 299475
|
|
|
|
|
|
| |
Requested on post commit code review.
llvm-svn: 299232
|
|
|
|
|
|
| |
This will be used to avoid various call to basename in the asan tests.
llvm-svn: 299216
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes tests that do things like
mkdir <dir>
cd <dir>
..
<cmd> *.foo
llvm-svn: 299209
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is needed by TestCases/Posix/coverage-direct.cc
The problem is that the test does:
mkdir <dir>
cd <dir>
cd ..
rm -rf <dir>
<more commands>
the current directory currently looks like "/.../<dir>/../" which
doesn't exist when dir is deleted.
at some point we should probably switch to using the os current
directory (specially if we want to add subshell), but this is a small
incremental improvement.
llvm-svn: 299113
|
|
|
|
|
|
|
|
|
|
|
| |
This adds support for commands like
FileCheck < foobar*
which is used by some asan tests because the file they want to read
has a pid in the name.
llvm-svn: 299111
|
|
|
|
| |
llvm-svn: 298994
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
`assert.assertItemEqual` went away in Python 3. Seeing how lists
are ordered, comparing a list against each other should work just
as well.
Patch by @jbergstroem (Johan Bergström).
Reviewers: modocache, gparker42
Reviewed By: modocache
Differential Revision: https://reviews.llvm.org/D31229
llvm-svn: 298479
|
|
|
|
|
|
|
|
|
|
|
| |
This will enable removing hacks throughout the codebase
in clang and compiler-rt that feed multiple inputs to a
testing utility by globbing, all of which are either disabled
on Windows currently or using xargs / find hacks.
Differential Revision: https://reviews.llvm.org/D30380
llvm-svn: 296904
|
|
|
|
|
|
|
|
| |
and UNSUPPORTED"
This reverts the revert in r292942.
llvm-svn: 293007
|
|
|
|
|
|
|
|
|
|
|
| |
and UNSUPPORTED"
After r292904 llvm-lit fails to emit the test results in the XML format for
Apple's internal buildbots.
rdar://30164800
llvm-svn: 292942
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292904
|
|
|
|
|
|
| |
This change needs to be better-coordinated with libc++.
llvm-svn: 292900
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292896
|
|
|
|
|
|
|
|
|
|
| |
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292548
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This change equips lit.py with two new options, --num-shards=M and
--run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD).
The options must be used together, and N must be in 1..M.
Together these options effect only test selection: they partition the testsuite
into M equal-sized "shards", then select only the Nth shard. They can be used
in a cluster of test machines to achieve a very crude (static) form of
parallelism, with minimal configuration work.
Reviewers: modocache, ddunbar
Reviewed By: ddunbar
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28789
llvm-svn: 292417
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary: The parameter `input` to `subprocess.Popen.communicate(...)` must be an object of type `bytes` . This is strictly enforced in python3. This patch (1) allows `to_bytes` to be safely called redundantly. (2) Explicitly convert `input` within `executeCommand`. This allows for usages like `executeCommand(['clang++', '-'], input='int main() {}\n')`.
Reviewers: ddunbar, BinaryKhaos, modocache, dim, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28736
llvm-svn: 292308
|
|
|
|
| |
llvm-svn: 292237
|
|
|
|
|
|
|
|
|
|
| |
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292231
|
|
|
|
|
|
|
|
|
| |
This required re-working the streaming support and lit's support for
'--gtest_list_tests' but otherwise seems to be a clean upgrade.
Differential Revision: https://reviews.llvm.org/D28154
llvm-svn: 291029
|
|
|
|
|
|
| |
This allows us to use `lit` to run on-target execution tests.
llvm-svn: 289769
|
|
|
|
|
|
|
| |
This reverts commit r289648, as it's an execution test and relies on the
emulator/dispatcher being available on all builders.
llvm-svn: 289651
|
|
|
|
|
|
| |
This test runs on actual AVR hardware.
llvm-svn: 289648
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
llvm-svn: 288694
|
|
|
|
|
|
|
|
| |
Darwin, use '--sdk macosx' to find the right SDK path.
This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'.
llvm-svn: 288385
|
|
|
|
|
|
|
|
|
| |
right SDK path."
This reverts commit r287403. It breaks an internal asan bot. According
to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929
llvm-svn: 287804
|
|
|
|
|
|
|
|
| |
SDK path.
This will make sure that we find an actual path in case you have Command Line Tools installed.
llvm-svn: 287403
|