summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* Support Unsupported Tests in xunit outputChris Matthews2018-05-114-12/+24
| | | | | | | | We were reporting "Unsupported" tests in xunit as passes, however since they are not run, it make more sense to mark them as skipped. The Junit xml standard has support for that, so lets use it. llvm-svn: 332065
* Refactor xunit test case builder to not use as much str additionChris Matthews2018-05-112-11/+12
| | | | | | | | String concatenation in python is slow. Refactor to not concatenate the possibly large strings of test output and instead write them directly to the output file. llvm-svn: 332064
* [LIT] Move xunit tests tests into their own location, and and add failuresChris Matthews2018-05-105-9/+59
| | | | | | Failures will increase coverage. llvm-svn: 332056
* [LIT] Add the missing fileChris Matthews2018-05-101-0/+7
| | | | | | I forgot to commit this file. llvm-svn: 331946
* Refactor test incase results are backwardsChris Matthews2018-05-101-14/+3
| | | | | | Looks like results can come in either way in this file. Loosen the ordering constraints. llvm-svn: 331945
* [LIT] Handle xml characters in test namesChris Matthews2018-05-093-2/+14
| | | | | | | | | | Lit creates malformed xml when the test case has an & in the name. Escape those correctly. This also adds a test case which I will add other nasty encoding issues to in some followup commits. llvm-svn: 331942
* Remove 'abi-breaking-checks' lit feature.Nico Weber2018-05-091-4/+0
| | | | | | | | | | Its only two uses were removed in r311730. Effectively reverts r304851 (but that code has removed around a bit since then). https://reviews.llvm.org/D46619 clang side done in r331871. llvm-svn: 331872
* lit: flesh out `SubsituteCaptures` furtherSaleem Abdulrasool2018-05-031-0/+6
| | | | | | | | Add overloads for `__len__` and `__getitem__` to allow use of this class on Linux as well as Windows. With these overloads, lit can be used on both hosts for the swift testsuite. llvm-svn: 331431
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-2515-153/+16
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-2415-16/+153
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* [lit] Remove spurious `-` in invocation of lit inDan Liew2018-04-241-1/+1
| | | | | | | | | | `shtest-xunit-output.py` test. Although there is no `-` file Jeremy Morse has reported to me that it causes problems in their setup because lit tries to find it and ends up loading an out of tree lit configuration file. llvm-svn: 330728
* Mostly revert r330672.Nico Weber2018-04-241-0/+4
| | | | | | | | | The test is apparently needed e.g. for check-cfi on Windows where we get 'C:/b/slave/sanitizer-windows/build/./bin/clang.exe': command not found without it. Try to fix the problem that was fixed by r330672 by also checking for isabs() instead. llvm-svn: 330673
* Remove code that's almost always dead, and harmful if not.Nico Weber2018-04-241-4/+0
| | | | | | | | | | | | | | lit's util.which() would check if the passed-in path existed directly, and if so return it as-is. This is never the case when running llvm's, clang's, or lld's tests normally. But when running `./llvm-lit path/to/clang/test` with a cwd of llvm-build/bin, this if would detect that clang exists at path 'clang' and return 'clang' as the discovered clang binary -- and then lit would use the " clang " -> "*** Do not use 'clang' in tests, use '%clang'. ***" substitution to replace that with a broken test. By removing this early return, lit ends up with the usual absolute path and everything works even in this uncommon case. llvm-svn: 330672
* [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnitDan Liew2018-04-202-1/+80
| | | | | | | | | | | XML printer. A test has been added that tries to comprehensively test emitting XUnit XML output for shell tests. Differential Revision: https://reviews.llvm.org/D45567 llvm-svn: 330409
* [lit] Remove duplicate to_string methodAaron Smith2018-04-121-5/+0
| | | | | | | | | | | | | | | There are two versions of to_string used by TestRunner.py. The one defined in TestRunner.py and the one defined in utils/lit/lit/util.py. The util.py version is superior to the TestRunner.py version. This change removes the duplicate to_string in TestRunner.py in favor of always using the version from util.py. Beside removing duplicate code, this makes it easier to debug TestRunner.py since only one version of to_string is used. Patch by Stella Stamenova! llvm-svn: 329972
* [lit] Fix several Python 2/3 compatibility issues and testsAaron Smith2018-04-071-1/+7
| | | | | | | | | | | | | | | | | | | - In Python 2.x, basestring is the base string type, but in Python 3.x basestring is not defined and instead str includes unicode strings. - When Python is in a path that includes spaces, it needs to be specified with quotes in the test files for it to run. - The cache.ll test relies on files of a specific size being created by Python, but on some versions of Windows the files that are created by the current code are one byte larger than expected. To fix the test, update file creation to always make files of the expected size. Patch by Stella Stamenova! llvm-svn: 329466
* 'cat' command for internal shell - Support Python 3Reid Kleckner2018-04-031-4/+12
| | | | | | | | | | | | | | | LLVM Bug Id : 36449 Revision 328563 caused tests to fail under python 3. This patch modified cat.py file to support both python 2 and 3. This patch also fixes CRLF issues on Windows. Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D45077 llvm-svn: 329123
* [lit] Prefer opening files with open (Python 2) rather than io.open which ↵Aaron Smith2018-04-031-7/+9
| | | | | | | | | | | requires io. Only rely on Python 3 (io.open) when necessary. This puts TestRunnyer.py closer to how it behaved before the changes introduced in D43165 and silences a few Windows build bot failures. Thanks to Stella Stamenova for the patch! llvm-svn: 329037
* [lit] One more try at fixing TestRunner.py for D43165Aaron Smith2018-04-021-0/+3
| | | | llvm-svn: 329026
* [lit] Attempt to fix builtin diff code for Python 2Reid Kleckner2018-04-021-0/+3
| | | | llvm-svn: 329024
* [lit] Fix problem in how Python versions open files with different encodingsAaron Smith2018-04-021-1/+41
| | | | | | | | | Reapply D43165 which was reverted because of different versions of python failing. The one line fix for the different python versions was commited at the same time that D43165 was reverted. If this change is giving you issues then get in touch with your python version and we will fix it. llvm-svn: 329022
* [lit] Use io.open to compare two files since it supports different encodings ↵Aaron Smith2018-04-021-1/+1
| | | | | | while older versions of open do not llvm-svn: 329020
* Revert r329012 "[lit] Fix problem in how Python versions open files with ↵Reid Kleckner2018-04-021-44/+1
| | | | | | | | | different encodings" This doesn't work with Python 2. See this build-in-progress: http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/11105 llvm-svn: 329017
* [lit] Fix problem in how Python versions open files with different encodingsAaron Smith2018-04-021-1/+44
| | | | | | | | | | | | | | | | | | | Summary: This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified. This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison. Patch by Stella Stamenova! Reviewers: zturner, llvm-commits, rnk, MaggieYi Reviewed By: zturner Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D43165 llvm-svn: 329012
* Revert "[lit] Temporarily disable shtest-timeout.py on darwin"Dan Liew2018-03-281-4/+1
| | | | | | | | | | This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598) Hopefully the test will now pass on the bots. rdar://problem/38774530 llvm-svn: 328703
* [lit] Remove a timing senstive part of `shtest-timeout.py`Dan Liew2018-03-282-52/+2
| | | | | | | | | | | | | | | | | | | | | | | The `shtest-timeout.py` test was failing intermittently. It looks like the issue is that on a resource constrained system lit is unable to run `quick_then_slow.py` twice and print out the messages the tests expects within the one second timeout. The underlying issue is that the test is dependent on the performance of the host machine is a rather fragile way. This is due to hardcoding timeout values and having assumptions that the host machine is able to perform a certain amount of work within the hardcoded timeout values. We could increase the timeout values but that doesn't really fix the underlying issue. Instead this patch removes one of fragile assumptions in the hope that this will be enough to fix the bots. There are other fragile assumptions in this test (e.g. `quick.py` can be executed in less than 1 second). If the bots continue to fail we'll have to revisit this. rdar://problem/38774530 llvm-svn: 328702
* Revert "Revert "[lit] Generalized /dev/null support on Windows.""Mircea Trofin2018-03-273-3/+32
| | | | | | | | | | | | | | | | | Summary: This reverts commit r328596. Checking if the arguments are strings before testing if they contain "/dev/null". Reviewers: rnk Reviewed By: rnk Subscribers: delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44914 llvm-svn: 328603
* [lit] Temporarily disable shtest-timeout.py on darwinJan Korous2018-03-271-1/+4
| | | | | | | | Disabled until fixed in order to avoid random failures on green dragon. rdar://problem/38774530 llvm-svn: 328598
* Revert "[lit] Generalized /dev/null support on Windows."Mircea Trofin2018-03-263-32/+3
| | | | | | This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92. llvm-svn: 328596
* [lit] Generalized /dev/null support on Windows.Mircea Trofin2018-03-263-3/+32
| | | | | | | | | | | | | | Generalized /dev/null remapping on Windows, and added test. Reviewers: rnk Reviewed By: rnk Subscribers: amccarth, zturner, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44771 llvm-svn: 328589
* [lit] Implement 'cat' command for internal shellReid Kleckner2018-03-268-12/+183
| | | | | | | | | | Fixes PR36449 Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D43501 llvm-svn: 328563
* [lit] - Allow 1 test to report multiple micro-test results to provide ↵Brian Homerding2018-03-137-0/+201
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | support for microbenchmarks. Summary: These changes are to allow to a Result object to have nested Result objects in order to support microbenchmarks. Currently lit is restricted to reporting one result object for one test, this change provides support tests that want to report individual timings for individual kernels. This revision is the result of the discussions in https://reviews.llvm.org/D32272#794759, https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496. It is a separation of the changes purposed in https://reviews.llvm.org/D40077. This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite) collection of loop kernels to the llvm test suite using the google benchmark library (https://reviews.llvm.org/D43319) with tracking of individual kernel timings. Previously microbenchmarks had been handled by using macros to section groups of microbenchmarks together and build many executables while still getting a grouped timing (MultiSource/TSVC). Recently the google benchmark library was added to the test suite and utilized with a litsupport plugin. However the limitation of 1 test 1 result limited its use to passing a runtime option to run only 1 microbenchmark with several hand written tests (MicroBenchmarks/XRay). This runs the same executable many times with different hand-written tests. I will update the litsupport plugin to utilize the new functionality (https://reviews.llvm.org/D43316). These changes allow lit to report micro test results if desired in order to get many precise timing results from 1 run of 1 test executable. Reviewers: MatzeB, hfinkel, rengolin, delcypher Differential Revision: https://reviews.llvm.org/D43314 llvm-svn: 327422
* utils: add a helper class to lit for captured substitutionsSaleem Abdulrasool2018-03-091-0/+19
| | | | | | | | | On Windows, if the substitution contains a back reference, it would removed due to the replacement of the escape character in lit. Create a helper class to avoid this which will simply ignore the replacement and mark the substitution as having capture groups being referenced. llvm-svn: 327082
* Losen time contraint to accommodate system loadsLei Huang2018-02-281-1/+1
| | | | llvm-svn: 326359
* Remove "--full-shutdown" and instead use an environment variable LLD_IN_TEST.Rui Ueyama2018-02-161-3/+0
| | | | | | | | | | | | | | | | | We are running lld tests with "--full-shutdown" option because we don't want to call _exit() in lld if it is running tests. Regular shutdown is needed for leak sanitizer. This patch changes the way how we tell lld that it is running tests. Now "--full-shutdown" is removed, and LLD_IN_TEST environment variable is used instead. This patch enables full shutdown on all ports, e.g. ELF, COFF and wasm. Previously, we enabled it only for ELF. Differential Revision: https://reviews.llvm.org/D43410 llvm-svn: 325413
* Make sure we invoke ld64.lld and ld-wasm in the build directory.Rui Ueyama2018-02-161-1/+1
| | | | llvm-svn: 325397
* [lit] Pass CLANG env var to testing configurationVictor Leschuk2018-02-091-1/+1
| | | | | | | | Allow CLANG environment variable be copied into the testing configuration and proper support testing with a custom path to the clang executable. Signed-off-by: Vladimir Vereschaka <vvereschaka@accesssoftek.com> llvm-svn: 324706
* [lit] Implement "-r" option for builtin "diff" command + a test using that.Max Moroz2018-01-0911-12/+245
| | | | | | | | | | | | | | | | Summary: That would allow to recursively compare directories in tests using "diff -r" on Windows in a similar way as it can be done on Linux or Mac. Reviewers: zturner, morehouse, vsk Reviewed By: zturner Subscribers: kcc, llvm-commits Differential Revision: https://reviews.llvm.org/D41776 llvm-svn: 322102
* [lit] Bump version numberTom Stellard2018-01-031-1/+1
| | | | llvm-svn: 321735
* Disable detect_leaks in the ASanified build of LLVM when using Apple LLVM. ↵Kuba Mracek2017-12-051-1/+2
| | | | | | The released Apple LLVM versions don't support LSan. llvm-svn: 319738
* [NFC][lit] Use proper semantic versioning names for variablesJonas Devlieghere2017-12-041-2/+3
| | | | | | | | | The variable named `minor` was actually pointing to the patch part of the version. While I was changing this I also made the check for Apple clang more robust by checking both patch and minor rather than just minor. llvm-svn: 319656
* [lit] Don't enable LSan on Darwin for Apple clang 9.0.0Jonas Devlieghere2017-12-011-4/+5
| | | | | | | | | The latest clang that ships with Xcode (clang 900 or 9.0.0) does not support LSan. This fixes the lit configuration to reflect that. Differential revision: https://reviews.llvm.org/D40672 llvm-svn: 319530
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-12-0119-2/+456
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewed by: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 319528
* Make check-lit tests respect LLVM_LIT_TOOLS_DIRGreg Bedwell2017-11-293-5/+9
| | | | | | Differential Revision: https://reviews.llvm.org/D40520 llvm-svn: 319329
* lit: Bring back -Dtool=xxx feature lost in r313928Matthias Braun2017-11-281-3/+8
| | | | llvm-svn: 319139
* Reverted rL318911 since it broke the sanitizer-windows.Ying Yi2017-11-2319-437/+2
| | | | llvm-svn: 318914
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-11-2319-2/+437
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewers: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 318911
* Re-revert "Refactor debuginfo-tests."Zachary Turner2017-11-211-6/+3
| | | | | | | | | | This is still breaking greendragon. At this point I give up until someone can fix the greendragon bots, and I will probably abandon this effort in favor of using a private github repository. llvm-svn: 318722
* Resubmit "Refactor debuginfo-tests" again.Zachary Turner2017-11-201-3/+6
| | | | | | | | | | | | This was reverted due to the tests being run twice on some build bots. Each run had a slightly different configuration due to the way in which it was being invoked. This fixes the problem (albeit in a somewhat hacky way). Hopefully in the future we can get rid of the workflow of running debuginfo-tests as part of clang, and then this hack can go away. llvm-svn: 318697
* [LIT] Fix testing out-of-tree Clang buildsEric Fiselier2017-11-191-4/+4
| | | | | | | | | | | | | | | | | Summary: Currently, LIT configures the LLVM binary path before the Clang binary path. However this breaks testing out-of-tree Clang builds (where the LLVM binary path includes a copy of Clang). This patch reverses the order of the paths when looking for Clang, putting the Clang binary directory first. Reviewers: zturner, beanz, chapuni, modocache, EricWF Reviewed By: EricWF Subscribers: mgorny, cfe-commits, llvm-commits Differential Revision: https://reviews.llvm.org/D40217 llvm-svn: 318607
OpenPOWER on IntegriCloud