summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* Revert r333584: [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3016-186/+17
| | | | | | It breaks test-suite. llvm-svn: 333592
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3016-17/+186
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r330755 (reverted in r330848) with fix for PR37239.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333584
* Use quoteattr to ensure we make well formed attributesChris Matthews2018-05-254-38/+40
| | | | | | | | We were making malformed XML on tests with ' in the name. Switch to using saxutils to set all of our attributes, so it can handle quotes etc correctly. llvm-svn: 333249
* [lit] Try to make `shtest-timeout.py` test more reliable by using aDan Liew2018-05-223-6/+13
| | | | | | | | | | | | | larger timeout value. This really isn't very good because it will still be susceptible to machine performance. While we are here also fix a bug in validation of `maxIndividualTestTime` where previously it wasn't checked if the type was an int. rdar://problem/40221572 llvm-svn: 332987
* [lit] Don't run `slow.py` in `shtest-timeout.py` test.Dan Liew2018-05-222-27/+2
| | | | | | | | | | | The program used to be used in `quick_then_slow.py` but that was removed in r328702. The tests always run `slow.py` on its own but this doesn't really test additional code so we'll just drop running `slow.py` so the tests run faster. rdar://problem/40221572 llvm-svn: 332986
* [lit] Don't check output of commands used in `shtest-timeout.py` test.Dan Liew2018-05-221-6/+0
| | | | | | | | | | If the system is under heavy load 1 second might not be long enough for it to produce output which could lead to spurious test failures. What matters is that the right test cases reach a timeout. rdar://problem/40221572 llvm-svn: 332985
* Mark test with "REQUIRES: shell" since it directly invokes "sh" and was ↵Douglas Yung2018-05-171-0/+2
| | | | | | failing on Windows. llvm-svn: 332563
* Escape ]]> in xunit xml outputAlexander Richardson2018-05-163-4/+8
| | | | | | | | | | | | | | Summary: This sequence ends the CDATA block so any characters after that are no longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>". Reviewers: cmatthews Reviewed By: cmatthews Differential Revision: https://reviews.llvm.org/D46886 llvm-svn: 332440
* remove output xml incase it is leftover from another runChris Matthews2018-05-161-0/+1
| | | | llvm-svn: 332424
* Use not to catch unexpected pass as well as remove old test resultsChris Matthews2018-05-161-1/+2
| | | | | | | As per review feedback, make sure we rm temp files, and make the return code checking for lit more specific. llvm-svn: 332423
* Requirements can have & in them!Chris Matthews2018-05-121-1/+1
| | | | | | Lets escape those so the XML is valid! llvm-svn: 332161
* Add the message attribute to skippedChris Matthews2018-05-112-6/+12
| | | | | | | JUnit xml allows for a message attribute to be displayed on skips. Lets populate that with an analysis of why we skipped the test. llvm-svn: 332156
* Overhaul unicode handling in xunit outputChris Matthews2018-05-111-6/+10
| | | | | | | I have seen a lot of errors where the xunit does not encode unicode test output correctly. Handle that explicitly now. llvm-svn: 332148
* [LIT] replace output escapes wit a cdata blockChris Matthews2018-05-112-6/+4
| | | | | | | CDATA blocks don't need to have XML stuff escaped. Makes sense to wrap output in them instead of escaping. llvm-svn: 332116
* Support Unsupported Tests in xunit outputChris Matthews2018-05-114-12/+24
| | | | | | | | We were reporting "Unsupported" tests in xunit as passes, however since they are not run, it make more sense to mark them as skipped. The Junit xml standard has support for that, so lets use it. llvm-svn: 332065
* Refactor xunit test case builder to not use as much str additionChris Matthews2018-05-112-11/+12
| | | | | | | | String concatenation in python is slow. Refactor to not concatenate the possibly large strings of test output and instead write them directly to the output file. llvm-svn: 332064
* [LIT] Move xunit tests tests into their own location, and and add failuresChris Matthews2018-05-105-9/+59
| | | | | | Failures will increase coverage. llvm-svn: 332056
* [LIT] Add the missing fileChris Matthews2018-05-101-0/+7
| | | | | | I forgot to commit this file. llvm-svn: 331946
* Refactor test incase results are backwardsChris Matthews2018-05-101-14/+3
| | | | | | Looks like results can come in either way in this file. Loosen the ordering constraints. llvm-svn: 331945
* [LIT] Handle xml characters in test namesChris Matthews2018-05-093-2/+14
| | | | | | | | | | Lit creates malformed xml when the test case has an & in the name. Escape those correctly. This also adds a test case which I will add other nasty encoding issues to in some followup commits. llvm-svn: 331942
* Remove 'abi-breaking-checks' lit feature.Nico Weber2018-05-091-4/+0
| | | | | | | | | | Its only two uses were removed in r311730. Effectively reverts r304851 (but that code has removed around a bit since then). https://reviews.llvm.org/D46619 clang side done in r331871. llvm-svn: 331872
* lit: flesh out `SubsituteCaptures` furtherSaleem Abdulrasool2018-05-031-0/+6
| | | | | | | | Add overloads for `__len__` and `__getitem__` to allow use of this class on Linux as well as Windows. With these overloads, lit can be used on both hosts for the swift testsuite. llvm-svn: 331431
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-2515-153/+16
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-2415-16/+153
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* [lit] Remove spurious `-` in invocation of lit inDan Liew2018-04-241-1/+1
| | | | | | | | | | `shtest-xunit-output.py` test. Although there is no `-` file Jeremy Morse has reported to me that it causes problems in their setup because lit tries to find it and ends up loading an out of tree lit configuration file. llvm-svn: 330728
* Mostly revert r330672.Nico Weber2018-04-241-0/+4
| | | | | | | | | The test is apparently needed e.g. for check-cfi on Windows where we get 'C:/b/slave/sanitizer-windows/build/./bin/clang.exe': command not found without it. Try to fix the problem that was fixed by r330672 by also checking for isabs() instead. llvm-svn: 330673
* Remove code that's almost always dead, and harmful if not.Nico Weber2018-04-241-4/+0
| | | | | | | | | | | | | | lit's util.which() would check if the passed-in path existed directly, and if so return it as-is. This is never the case when running llvm's, clang's, or lld's tests normally. But when running `./llvm-lit path/to/clang/test` with a cwd of llvm-build/bin, this if would detect that clang exists at path 'clang' and return 'clang' as the discovered clang binary -- and then lit would use the " clang " -> "*** Do not use 'clang' in tests, use '%clang'. ***" substitution to replace that with a broken test. By removing this early return, lit ends up with the usual absolute path and everything works even in this uncommon case. llvm-svn: 330672
* [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnitDan Liew2018-04-202-1/+80
| | | | | | | | | | | XML printer. A test has been added that tries to comprehensively test emitting XUnit XML output for shell tests. Differential Revision: https://reviews.llvm.org/D45567 llvm-svn: 330409
* [lit] Remove duplicate to_string methodAaron Smith2018-04-121-5/+0
| | | | | | | | | | | | | | | There are two versions of to_string used by TestRunner.py. The one defined in TestRunner.py and the one defined in utils/lit/lit/util.py. The util.py version is superior to the TestRunner.py version. This change removes the duplicate to_string in TestRunner.py in favor of always using the version from util.py. Beside removing duplicate code, this makes it easier to debug TestRunner.py since only one version of to_string is used. Patch by Stella Stamenova! llvm-svn: 329972
* [lit] Fix several Python 2/3 compatibility issues and testsAaron Smith2018-04-071-1/+7
| | | | | | | | | | | | | | | | | | | - In Python 2.x, basestring is the base string type, but in Python 3.x basestring is not defined and instead str includes unicode strings. - When Python is in a path that includes spaces, it needs to be specified with quotes in the test files for it to run. - The cache.ll test relies on files of a specific size being created by Python, but on some versions of Windows the files that are created by the current code are one byte larger than expected. To fix the test, update file creation to always make files of the expected size. Patch by Stella Stamenova! llvm-svn: 329466
* 'cat' command for internal shell - Support Python 3Reid Kleckner2018-04-031-4/+12
| | | | | | | | | | | | | | | LLVM Bug Id : 36449 Revision 328563 caused tests to fail under python 3. This patch modified cat.py file to support both python 2 and 3. This patch also fixes CRLF issues on Windows. Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D45077 llvm-svn: 329123
* [lit] Prefer opening files with open (Python 2) rather than io.open which ↵Aaron Smith2018-04-031-7/+9
| | | | | | | | | | | requires io. Only rely on Python 3 (io.open) when necessary. This puts TestRunnyer.py closer to how it behaved before the changes introduced in D43165 and silences a few Windows build bot failures. Thanks to Stella Stamenova for the patch! llvm-svn: 329037
* [lit] One more try at fixing TestRunner.py for D43165Aaron Smith2018-04-021-0/+3
| | | | llvm-svn: 329026
* [lit] Attempt to fix builtin diff code for Python 2Reid Kleckner2018-04-021-0/+3
| | | | llvm-svn: 329024
* [lit] Fix problem in how Python versions open files with different encodingsAaron Smith2018-04-021-1/+41
| | | | | | | | | Reapply D43165 which was reverted because of different versions of python failing. The one line fix for the different python versions was commited at the same time that D43165 was reverted. If this change is giving you issues then get in touch with your python version and we will fix it. llvm-svn: 329022
* [lit] Use io.open to compare two files since it supports different encodings ↵Aaron Smith2018-04-021-1/+1
| | | | | | while older versions of open do not llvm-svn: 329020
* Revert r329012 "[lit] Fix problem in how Python versions open files with ↵Reid Kleckner2018-04-021-44/+1
| | | | | | | | | different encodings" This doesn't work with Python 2. See this build-in-progress: http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/11105 llvm-svn: 329017
* [lit] Fix problem in how Python versions open files with different encodingsAaron Smith2018-04-021-1/+44
| | | | | | | | | | | | | | | | | | | Summary: This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified. This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison. Patch by Stella Stamenova! Reviewers: zturner, llvm-commits, rnk, MaggieYi Reviewed By: zturner Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D43165 llvm-svn: 329012
* Revert "[lit] Temporarily disable shtest-timeout.py on darwin"Dan Liew2018-03-281-4/+1
| | | | | | | | | | This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598) Hopefully the test will now pass on the bots. rdar://problem/38774530 llvm-svn: 328703
* [lit] Remove a timing senstive part of `shtest-timeout.py`Dan Liew2018-03-282-52/+2
| | | | | | | | | | | | | | | | | | | | | | | The `shtest-timeout.py` test was failing intermittently. It looks like the issue is that on a resource constrained system lit is unable to run `quick_then_slow.py` twice and print out the messages the tests expects within the one second timeout. The underlying issue is that the test is dependent on the performance of the host machine is a rather fragile way. This is due to hardcoding timeout values and having assumptions that the host machine is able to perform a certain amount of work within the hardcoded timeout values. We could increase the timeout values but that doesn't really fix the underlying issue. Instead this patch removes one of fragile assumptions in the hope that this will be enough to fix the bots. There are other fragile assumptions in this test (e.g. `quick.py` can be executed in less than 1 second). If the bots continue to fail we'll have to revisit this. rdar://problem/38774530 llvm-svn: 328702
* Revert "Revert "[lit] Generalized /dev/null support on Windows.""Mircea Trofin2018-03-273-3/+32
| | | | | | | | | | | | | | | | | Summary: This reverts commit r328596. Checking if the arguments are strings before testing if they contain "/dev/null". Reviewers: rnk Reviewed By: rnk Subscribers: delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44914 llvm-svn: 328603
* [lit] Temporarily disable shtest-timeout.py on darwinJan Korous2018-03-271-1/+4
| | | | | | | | Disabled until fixed in order to avoid random failures on green dragon. rdar://problem/38774530 llvm-svn: 328598
* Revert "[lit] Generalized /dev/null support on Windows."Mircea Trofin2018-03-263-32/+3
| | | | | | This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92. llvm-svn: 328596
* [lit] Generalized /dev/null support on Windows.Mircea Trofin2018-03-263-3/+32
| | | | | | | | | | | | | | Generalized /dev/null remapping on Windows, and added test. Reviewers: rnk Reviewed By: rnk Subscribers: amccarth, zturner, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44771 llvm-svn: 328589
* [lit] Implement 'cat' command for internal shellReid Kleckner2018-03-268-12/+183
| | | | | | | | | | Fixes PR36449 Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D43501 llvm-svn: 328563
* [lit] - Allow 1 test to report multiple micro-test results to provide ↵Brian Homerding2018-03-137-0/+201
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | support for microbenchmarks. Summary: These changes are to allow to a Result object to have nested Result objects in order to support microbenchmarks. Currently lit is restricted to reporting one result object for one test, this change provides support tests that want to report individual timings for individual kernels. This revision is the result of the discussions in https://reviews.llvm.org/D32272#794759, https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496. It is a separation of the changes purposed in https://reviews.llvm.org/D40077. This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite) collection of loop kernels to the llvm test suite using the google benchmark library (https://reviews.llvm.org/D43319) with tracking of individual kernel timings. Previously microbenchmarks had been handled by using macros to section groups of microbenchmarks together and build many executables while still getting a grouped timing (MultiSource/TSVC). Recently the google benchmark library was added to the test suite and utilized with a litsupport plugin. However the limitation of 1 test 1 result limited its use to passing a runtime option to run only 1 microbenchmark with several hand written tests (MicroBenchmarks/XRay). This runs the same executable many times with different hand-written tests. I will update the litsupport plugin to utilize the new functionality (https://reviews.llvm.org/D43316). These changes allow lit to report micro test results if desired in order to get many precise timing results from 1 run of 1 test executable. Reviewers: MatzeB, hfinkel, rengolin, delcypher Differential Revision: https://reviews.llvm.org/D43314 llvm-svn: 327422
* utils: add a helper class to lit for captured substitutionsSaleem Abdulrasool2018-03-091-0/+19
| | | | | | | | | On Windows, if the substitution contains a back reference, it would removed due to the replacement of the escape character in lit. Create a helper class to avoid this which will simply ignore the replacement and mark the substitution as having capture groups being referenced. llvm-svn: 327082
* Losen time contraint to accommodate system loadsLei Huang2018-02-281-1/+1
| | | | llvm-svn: 326359
* Remove "--full-shutdown" and instead use an environment variable LLD_IN_TEST.Rui Ueyama2018-02-161-3/+0
| | | | | | | | | | | | | | | | | We are running lld tests with "--full-shutdown" option because we don't want to call _exit() in lld if it is running tests. Regular shutdown is needed for leak sanitizer. This patch changes the way how we tell lld that it is running tests. Now "--full-shutdown" is removed, and LLD_IN_TEST environment variable is used instead. This patch enables full shutdown on all ports, e.g. ELF, COFF and wasm. Previously, we enabled it only for ELF. Differential Revision: https://reviews.llvm.org/D43410 llvm-svn: 325413
* Make sure we invoke ld64.lld and ld-wasm in the build directory.Rui Ueyama2018-02-161-1/+1
| | | | llvm-svn: 325397
OpenPOWER on IntegriCloud