| Commit message (Collapse) | Author | Age | Files | Lines |
... | |
|
|
|
|
|
|
|
|
|
|
|
| |
Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot.
Reviewers: asmith, delcypher, zturner
Subscribers: stella.stamenova, llvm-commits
Differential Revision: https://reviews.llvm.org/D51871
llvm-svn: 341856
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
diagnose buildbot failures
Summary:
Right now this test is failing on the builtbots on Windows but we have a very similar setup where the test passes. The test is meant to test that specifying a timeout works correctly by running an infnite loop and having it timeout - on the buildbot, the infinite loop doesn't actually execute. This change runs all of the tests in the set using an internal shell rather than an external shell. I expect this will make the test pass which means that either the way the external shell is invoked or the external shell setup on the buildbots is not correct. Regardless of whether the test passes with this change, we'll need to undo this change and have a real fix.
@gkistanova was able to get logs from the buildbot to rule out a number of theories as to why this test is failing, but they didn't have enough information to confirm exactly what the issue is. The purpose of this change is to narrow it down, but if someone has a local repro and can aid in debugging, that would make it much speedier (and less prone to making the bots fail).
Reviewers: gkistanova, asmith, zturner, modocache, rnk, delcypher
Reviewed By: rnk
Subscribers: delcypher, llvm-commits, gkistanova
Differential Revision: https://reviews.llvm.org/D51326
llvm-svn: 340840
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
other test fixes
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3.
This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests.
Reviewers: asmith, zturner
Subscribers: stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50397
llvm-svn: 339179
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
The problem here is that on windows double quotes are used for paths (usually) while single quotes are not. This is not generally a problem for the tests because the lit infrastructure tends to treat both the same. One (and possibly only) exception is when some tests are run in an external shell such as some of the shtest-format tests. In this case on windows the path to python was not created correctly because it had single quotes and the test failed.
This same test is already failing with python 3 which is why our testing missed the new failure. This patch will take care of the immediate failure with python 2 and I'll send a follow up for the python 3 failure.
Reviewers: asmith, zturner
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50373
llvm-svn: 339091
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.
This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests.
Reviewers: asmith, zturner, alexshap, jakehehrlich
Reviewed By: zturner, alexshap, jakehehrlich
Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50206
llvm-svn: 339073
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(Relands r333584, reverted in 333592.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333614
|
|
|
|
|
|
| |
It breaks test-suite.
llvm-svn: 333592
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
(Relands r330755 (reverted in r330848) with fix for PR37239.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333584
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.
rdar://problem/40221572
llvm-svn: 332987
|
|
|
|
|
|
|
|
|
|
|
| |
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.
rdar://problem/40221572
llvm-svn: 332986
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>".
Reviewers: cmatthews
Reviewed By: cmatthews
Differential Revision: https://reviews.llvm.org/D46886
llvm-svn: 332440
|
|
|
|
|
|
| |
Failures will increase coverage.
llvm-svn: 332056
|
|
|
|
|
|
| |
I forgot to commit this file.
llvm-svn: 331946
|
|
|
|
|
|
|
| |
It is causing many tests to fail on Windows buildbots:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211
llvm-svn: 330848
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
Reviewed By: asmith, delcypher
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 330755
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.
The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.
We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.
rdar://problem/38774530
llvm-svn: 328702
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This reverts commit r328596.
Checking if the arguments are strings before testing if they contain "/dev/null".
Reviewers: rnk
Reviewed By: rnk
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D44914
llvm-svn: 328603
|
|
|
|
|
|
| |
This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92.
llvm-svn: 328596
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Generalized /dev/null remapping on Windows, and added test.
Reviewers: rnk
Reviewed By: rnk
Subscribers: amccarth, zturner, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D44771
llvm-svn: 328589
|
|
|
|
|
|
|
|
|
|
| |
Fixes PR36449
Patch by Chamal de Silva
Differential Revision: https://reviews.llvm.org/D43501
llvm-svn: 328563
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
support for microbenchmarks.
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.
This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.
This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.
Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).
These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.
Reviewers: MatzeB, hfinkel, rengolin, delcypher
Differential Revision: https://reviews.llvm.org/D43314
llvm-svn: 327422
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.
Reviewers: zturner, morehouse, vsk
Reviewed By: zturner
Subscribers: kcc, llvm-commits
Differential Revision: https://reviews.llvm.org/D41776
llvm-svn: 322102
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands.
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’
commands as the internal shell builtins.
Reviewed by: Zachary Turner, Reid Kleckner
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D39567
llvm-svn: 319528
|
|
|
|
| |
llvm-svn: 318914
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
internally
Summary:
The internal shell already supports 'cd', ‘export’ and ‘echo’ commands.
This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’
commands as the internal shell builtins.
Reviewers: Zachary Turner, Reid Kleckner
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D39567
llvm-svn: 318911
|
|
|
|
|
|
|
| |
This reverts r315697 and my ill-fated attempts to fix it on Windows.
I'll try again when I get access to a Windows machine.
llvm-svn: 315793
|
|
|
|
|
|
|
|
|
|
|
|
| |
This refers to a temporary path that can be shared across all tests,
identified by a particular label. This can be used for things like
caches.
At the moment, the character set for the LABEL is limited to C
identifier characters, plus '-', '+', '=', and '.'. This is the same
set of characters currently allowed in REQUIRES clause identifiers.
llvm-svn: 315697
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This has gone back and forth, but it seems this is necessary
after all. realpath is not sufficient because if you have a
file named 'C:\foo.txt', then both realpath('c:\foo.txt') and
realpath(C:\foo.txt') return the string that was passed to them
exactly as is, meaning the case of the drive-letter won't match.
The problem before was not that we were normalizing the case of
items going into the config map, but rather that we were
normalizing the case of something we needed to print. The value
that is used to key on the config map should never be printed.
llvm-svn: 313918
|
|
|
|
|
|
|
|
| |
This makes all paths lowercase on Windows, which seemed like a
good idea at the time, but it means that tests can't properly
use FileCheck to match expected path names.
llvm-svn: 313889
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Config map is not exposed through the command line, so testing this
is somewhat tricky. But basically we need a test that if a custom
driver builds a config map and passes it to main, it gets respected.
A config map allows config files in the source tree to be mapped
to alternate config files in the build tree. This particular test
works by having two config files in separate directories, and
setting up a config map to have that redirects A/lit.site.cfg
to B/altconfig. Then, we print a message in A/lit.site.cfg
and B/altconfig and check that we do see the output from B
but don't see the output from A. Additionally we test that
the test suite specified by A's config map is properly discovered.
Differential Revision: https://reviews.llvm.org/D38105
llvm-svn: 313887
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Many editors and Python-related diagnostics tools such as
debuggers break or fail in mysterious ways when python files
don't end in .py. This is especially true on Windows, but
still exists on other platforms. I don't want to be too heavy
handed in changing everything across the board, but I do want
to at least *allow* lit configs to have .py extensions. This
patch makes the discovery process first look for a config file
with a .py extension, and if one is not found, then looks for
a config file using the old method. So for existing users, there
should be no functional change.
Differential Revision: https://reviews.llvm.org/D37838
llvm-svn: 313849
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton.
The patch was accepted but not landed.
This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour.
Differential Revision: https://reviews.llvm.org/D36403
llvm-svn: 311180
|
|
|
|
|
|
|
| |
This reverts r309602, check-lit still leaves Output directories in the
source directory.
llvm-svn: 309833
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
This is an alternative solution to running the lit test suite on bots
without polluting the source directory. Each input test suite gets an
auto-generated site config in the build directory that points back to
the test input source directory.
This adds some cmake comlexity, but now we don't need to remove and
re-copy the test input directory before every test.
Reviewers: delcypher, modocache
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D36026
llvm-svn: 309602
|
|
|
|
|
|
|
|
|
|
| |
Traceback (most recent call last):
File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module>
sys.stdout.write(b"a line with bad encoding: \xc2.")
sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts.
llvm-svn: 309473
|
|
|
|
| |
llvm-svn: 309434
|
|
|
|
|
|
|
|
|
|
| |
When using win32 cmd.exe, turn off command echoing at the beginning of
the script (@echo off).
Replace a bash shell script with a python script for the
fail_with_bad_encoding test.
llvm-svn: 309399
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a valid win32 executable.
Instead, rename all the python scripts masquerading as gtest executables
to have .py extensions, so we can easily detect then and call the python
executable for them. Do this on Linux as well as Windows for
consistency.
The test suite directory names also come out in lower-case on Windows.
We can consider removing that in a later patch. This change just updates
the FileCheck lines to match on Windows.
Fixes PR33933
Reviewers: modocache, mgorny
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35909
llvm-svn: 309347
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt`
expects that first a string printed to stdout is seen, and then a
string printed to stderr. Add `flush()` calls to ensure that stdout is
printed before stderr, as expected.
Reviewers: rnk, mgorny, jroelofs
Reviewed By: rnk
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35947
llvm-svn: 309292
|
|
|
|
|
|
|
| |
Previously these tests would use the same Output directory leading to
flaky non-deterministic failures.
llvm-svn: 309227
|
|
|
|
|
|
|
|
|
| |
Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell
scripts as python scripts and call python on them.
Fixes PR33940
llvm-svn: 309200
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is necessary to pass the lit test suite at llvm/utils/lit/tests.
There are some pre-existing failures here, but now switching to pools
doesn't regress any tests.
I had to change test-data/lit.cfg to import DummyConfig from a module to
fix pickling problems, but I think it'll be OK if we require test
formats to be written in real .py modules outside lit.cfg files.
I also discovered that in some circumstances AsyncResult.wait() will not
raise KeyboardInterrupt in a timely manner, but you can pass a non-zero
timeout to work around this. This makes threading.Condition.wait use a
polling loop that runs through the interpreter, so it's capable of
asynchronously raising KeyboardInterrupt.
llvm-svn: 299605
|
|
|
|
|
|
|
|
| |
and UNSUPPORTED"
This reverts the revert in r292942.
llvm-svn: 293007
|
|
|
|
|
|
|
|
|
|
|
| |
and UNSUPPORTED"
After r292904 llvm-lit fails to emit the test results in the XML format for
Apple's internal buildbots.
rdar://30164800
llvm-svn: 292942
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292904
|
|
|
|
|
|
| |
This change needs to be better-coordinated with libc++.
llvm-svn: 292900
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292896
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
llvm-svn: 288694
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reviewers: ddunbar, echristo, beanz
Patch by Brian Gesiak!
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25169
llvm-svn: 283089
|
|
|
|
|
|
|
|
|
| |
- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
failed, but the main point of the test was, say, a `FileCheck` later on, then
the log wasn't helpful in showing why the command failed).
llvm-svn: 272021
|