summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit/tests
Commit message (Collapse)AuthorAgeFilesLines
* [lit] Terminate ": RUN at line N" with ";" not "&&"Joel E. Denny2018-05-313-16/+16
| | | | | | | | This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was broken by r333614. The trouble was that "&&" changes the command for which "$!" gives the pid. llvm-svn: 333620
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3115-6/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r333584, reverted in 333592.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333614
* Revert r333584: [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3015-140/+6
| | | | | | It breaks test-suite. llvm-svn: 333592
* [lit] Report line number for failed RUN commandJoel E. Denny2018-05-3015-6/+140
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Relands r330755 (reverted in r330848) with fix for PR37239.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333584
* Use quoteattr to ensure we make well formed attributesChris Matthews2018-05-252-26/+26
| | | | | | | | We were making malformed XML on tests with ' in the name. Switch to using saxutils to set all of our attributes, so it can handle quotes etc correctly. llvm-svn: 333249
* [lit] Try to make `shtest-timeout.py` test more reliable by using aDan Liew2018-05-222-6/+11
| | | | | | | | | | | | | larger timeout value. This really isn't very good because it will still be susceptible to machine performance. While we are here also fix a bug in validation of `maxIndividualTestTime` where previously it wasn't checked if the type was an int. rdar://problem/40221572 llvm-svn: 332987
* [lit] Don't run `slow.py` in `shtest-timeout.py` test.Dan Liew2018-05-222-27/+2
| | | | | | | | | | | The program used to be used in `quick_then_slow.py` but that was removed in r328702. The tests always run `slow.py` on its own but this doesn't really test additional code so we'll just drop running `slow.py` so the tests run faster. rdar://problem/40221572 llvm-svn: 332986
* [lit] Don't check output of commands used in `shtest-timeout.py` test.Dan Liew2018-05-221-6/+0
| | | | | | | | | | If the system is under heavy load 1 second might not be long enough for it to produce output which could lead to spurious test failures. What matters is that the right test cases reach a timeout. rdar://problem/40221572 llvm-svn: 332985
* Mark test with "REQUIRES: shell" since it directly invokes "sh" and was ↵Douglas Yung2018-05-171-0/+2
| | | | | | failing on Windows. llvm-svn: 332563
* Escape ]]> in xunit xml outputAlexander Richardson2018-05-162-3/+5
| | | | | | | | | | | | | | Summary: This sequence ends the CDATA block so any characters after that are no longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>". Reviewers: cmatthews Reviewed By: cmatthews Differential Revision: https://reviews.llvm.org/D46886 llvm-svn: 332440
* remove output xml incase it is leftover from another runChris Matthews2018-05-161-0/+1
| | | | llvm-svn: 332424
* Use not to catch unexpected pass as well as remove old test resultsChris Matthews2018-05-161-1/+2
| | | | | | | As per review feedback, make sure we rm temp files, and make the return code checking for lit more specific. llvm-svn: 332423
* Add the message attribute to skippedChris Matthews2018-05-111-5/+5
| | | | | | | JUnit xml allows for a message attribute to be displayed on skips. Lets populate that with an analysis of why we skipped the test. llvm-svn: 332156
* [LIT] replace output escapes wit a cdata blockChris Matthews2018-05-111-3/+1
| | | | | | | CDATA blocks don't need to have XML stuff escaped. Makes sense to wrap output in them instead of escaping. llvm-svn: 332116
* Support Unsupported Tests in xunit outputChris Matthews2018-05-112-7/+12
| | | | | | | | We were reporting "Unsupported" tests in xunit as passes, however since they are not run, it make more sense to mark them as skipped. The Junit xml standard has support for that, so lets use it. llvm-svn: 332065
* [LIT] Move xunit tests tests into their own location, and and add failuresChris Matthews2018-05-105-9/+59
| | | | | | Failures will increase coverage. llvm-svn: 332056
* [LIT] Add the missing fileChris Matthews2018-05-101-0/+7
| | | | | | I forgot to commit this file. llvm-svn: 331946
* Refactor test incase results are backwardsChris Matthews2018-05-101-14/+3
| | | | | | Looks like results can come in either way in this file. Loosen the ordering constraints. llvm-svn: 331945
* [LIT] Handle xml characters in test namesChris Matthews2018-05-092-1/+13
| | | | | | | | | | Lit creates malformed xml when the test case has an & in the name. Escape those correctly. This also adds a test case which I will add other nasty encoding issues to in some followup commits. llvm-svn: 331942
* Revert r330755 "[lit] Report line number for failed RUN command"Reid Kleckner2018-04-2514-131/+7
| | | | | | | It is causing many tests to fail on Windows buildbots: http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/builds/10211 llvm-svn: 330848
* [lit] Report line number for failed RUN commandJoel E. Denny2018-04-2414-7/+131
| | | | | | | | | | | | | | | | | | | | | When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. Reviewed By: asmith, delcypher Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 330755
* [lit] Remove spurious `-` in invocation of lit inDan Liew2018-04-241-1/+1
| | | | | | | | | | `shtest-xunit-output.py` test. Although there is no `-` file Jeremy Morse has reported to me that it causes problems in their setup because lit tries to find it and ends up loading an out of tree lit configuration file. llvm-svn: 330728
* [lit] Fix a bug where UNRESOLVED tests were not handled in the XUnitDan Liew2018-04-201-0/+78
| | | | | | | | | | | XML printer. A test has been added that tries to comprehensively test emitting XUnit XML output for shell tests. Differential Revision: https://reviews.llvm.org/D45567 llvm-svn: 330409
* Revert "[lit] Temporarily disable shtest-timeout.py on darwin"Dan Liew2018-03-281-4/+1
| | | | | | | | | | This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598) Hopefully the test will now pass on the bots. rdar://problem/38774530 llvm-svn: 328703
* [lit] Remove a timing senstive part of `shtest-timeout.py`Dan Liew2018-03-282-52/+2
| | | | | | | | | | | | | | | | | | | | | | | The `shtest-timeout.py` test was failing intermittently. It looks like the issue is that on a resource constrained system lit is unable to run `quick_then_slow.py` twice and print out the messages the tests expects within the one second timeout. The underlying issue is that the test is dependent on the performance of the host machine is a rather fragile way. This is due to hardcoding timeout values and having assumptions that the host machine is able to perform a certain amount of work within the hardcoded timeout values. We could increase the timeout values but that doesn't really fix the underlying issue. Instead this patch removes one of fragile assumptions in the hope that this will be enough to fix the bots. There are other fragile assumptions in this test (e.g. `quick.py` can be executed in less than 1 second). If the bots continue to fail we'll have to revisit this. rdar://problem/38774530 llvm-svn: 328702
* Revert "Revert "[lit] Generalized /dev/null support on Windows.""Mircea Trofin2018-03-272-0/+28
| | | | | | | | | | | | | | | | | Summary: This reverts commit r328596. Checking if the arguments are strings before testing if they contain "/dev/null". Reviewers: rnk Reviewed By: rnk Subscribers: delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44914 llvm-svn: 328603
* [lit] Temporarily disable shtest-timeout.py on darwinJan Korous2018-03-271-1/+4
| | | | | | | | Disabled until fixed in order to avoid random failures on green dragon. rdar://problem/38774530 llvm-svn: 328598
* Revert "[lit] Generalized /dev/null support on Windows."Mircea Trofin2018-03-262-28/+0
| | | | | | This reverts commit ca7fdbb974384ce5a05528b22a41d46b1cc13e92. llvm-svn: 328596
* [lit] Generalized /dev/null support on Windows.Mircea Trofin2018-03-262-0/+28
| | | | | | | | | | | | | | Generalized /dev/null remapping on Windows, and added test. Reviewers: rnk Reviewed By: rnk Subscribers: amccarth, zturner, delcypher, llvm-commits Differential Revision: https://reviews.llvm.org/D44771 llvm-svn: 328589
* [lit] Implement 'cat' command for internal shellReid Kleckner2018-03-266-2/+107
| | | | | | | | | | Fixes PR36449 Patch by Chamal de Silva Differential Revision: https://reviews.llvm.org/D43501 llvm-svn: 328563
* [lit] - Allow 1 test to report multiple micro-test results to provide ↵Brian Homerding2018-03-135-0/+150
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | support for microbenchmarks. Summary: These changes are to allow to a Result object to have nested Result objects in order to support microbenchmarks. Currently lit is restricted to reporting one result object for one test, this change provides support tests that want to report individual timings for individual kernels. This revision is the result of the discussions in https://reviews.llvm.org/D32272#794759, https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496. It is a separation of the changes purposed in https://reviews.llvm.org/D40077. This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite) collection of loop kernels to the llvm test suite using the google benchmark library (https://reviews.llvm.org/D43319) with tracking of individual kernel timings. Previously microbenchmarks had been handled by using macros to section groups of microbenchmarks together and build many executables while still getting a grouped timing (MultiSource/TSVC). Recently the google benchmark library was added to the test suite and utilized with a litsupport plugin. However the limitation of 1 test 1 result limited its use to passing a runtime option to run only 1 microbenchmark with several hand written tests (MicroBenchmarks/XRay). This runs the same executable many times with different hand-written tests. I will update the litsupport plugin to utilize the new functionality (https://reviews.llvm.org/D43316). These changes allow lit to report micro test results if desired in order to get many precise timing results from 1 run of 1 test executable. Reviewers: MatzeB, hfinkel, rengolin, delcypher Differential Revision: https://reviews.llvm.org/D43314 llvm-svn: 327422
* Losen time contraint to accommodate system loadsLei Huang2018-02-281-1/+1
| | | | llvm-svn: 326359
* [lit] Implement "-r" option for builtin "diff" command + a test using that.Max Moroz2018-01-0910-2/+132
| | | | | | | | | | | | | | | | Summary: That would allow to recursively compare directories in tests using "diff -r" on Windows in a similar way as it can be done on Linux or Mac. Reviewers: zturner, morehouse, vsk Reviewed By: zturner Subscribers: kcc, llvm-commits Differential Revision: https://reviews.llvm.org/D41776 llvm-svn: 322102
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-12-0118-2/+279
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewed by: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 319528
* Make check-lit tests respect LLVM_LIT_TOOLS_DIRGreg Bedwell2017-11-293-5/+9
| | | | | | Differential Revision: https://reviews.llvm.org/D40520 llvm-svn: 319329
* Reverted rL318911 since it broke the sanitizer-windows.Ying Yi2017-11-2318-273/+2
| | | | llvm-svn: 318914
* [lit] Implement non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands ↵Ying Yi2017-11-2318-2/+273
| | | | | | | | | | | | | | | | | internally Summary: The internal shell already supports 'cd', ‘export’ and ‘echo’ commands. This patch adds implementation of non-pipelined ‘mkdir’, ‘diff’ and ‘rm’ commands as the internal shell builtins. Reviewers: Zachary Turner, Reid Kleckner Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D39567 llvm-svn: 318911
* Revert "lit.py: Add new %{shared_output(LABEL)} substitution"Jordan Rose2017-10-145-25/+0
| | | | | | | This reverts r315697 and my ill-fated attempts to fix it on Windows. I'll try again when I get access to a Windows machine. llvm-svn: 315793
* lit.py: Give up and disable the new shared-output.py test on WindowsJordan Rose2017-10-141-0/+3
| | | | | | | | | | "No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp" And yet other forward-slashes don't seem to be causing the same problem. I'll see if I can get ahold of a Windows machine to poke at this directly later. llvm-svn: 315792
* lit.py: Previous test fix was a red herring; backslashes are escapesJordan Rose2017-10-141-1/+1
| | | | | | | I don't have access to a Windows machine at the moment, so if this doesn't fix it I'll just revert for now. llvm-svn: 315782
* lit.py: Fix new test harder for systems that don't use / as os.path.sepJordan Rose2017-10-141-1/+1
| | | | | | | I didn't think about '%{inputs}' having the same problem. This one should be a fully Windows path name. llvm-svn: 315779
* lit.py: Fix new test for systems that don't use / as os.path.sepJordan Rose2017-10-131-1/+1
| | | | llvm-svn: 315773
* lit.py: Add new %{shared_output(LABEL)} substitutionJordan Rose2017-10-135-0/+22
| | | | | | | | | | | | This refers to a temporary path that can be shared across all tests, identified by a particular label. This can be used for things like caches. At the moment, the character set for the LABEL is limited to C identifier characters, plus '-', '+', '=', and '.'. This is the same set of characters currently allowed in REQUIRES clause identifiers. llvm-svn: 315697
* [lit] Fix running lit tests in unconfigured source dirMichal Gorny2017-10-011-1/+1
| | | | | | | | | | | | | Fix llvm_tools_dir attribute access not to fail when the variable is not present. This directory is not really necessary to run lit tests, and the code already accounts for it being None. The reference was added in r313407, and it breaks the stand-alone lit package in Gentoo. Differential Revision: https://reviews.llvm.org/D38442 llvm-svn: 314620
* [lit] Actually do normalize the case of files in the config map.Zachary Turner2017-09-212-2/+4
| | | | | | | | | | | | | | | This has gone back and forth, but it seems this is necessary after all. realpath is not sufficient because if you have a file named 'C:\foo.txt', then both realpath('c:\foo.txt') and realpath(C:\foo.txt') return the string that was passed to them exactly as is, meaning the case of the drive-letter won't match. The problem before was not that we were normalizing the case of items going into the config map, but rather that we were normalizing the case of something we needed to print. The value that is used to key on the config map should never be printed. llvm-svn: 313918
* [lit] Don't norm case when inserting into the config map.Zachary Turner2017-09-213-3/+3
| | | | | | | | This makes all paths lowercase on Windows, which seemed like a good idea at the time, but it means that tests can't properly use FileCheck to match expected path names. llvm-svn: 313889
* [lit] Add a test for the builtin config map.Zachary Turner2017-09-217-0/+46
| | | | | | | | | | | | | | | | | | | Config map is not exposed through the command line, so testing this is somewhat tricky. But basically we need a test that if a custom driver builds a config map and passes it to main, it gets respected. A config map allows config files in the source tree to be mapped to alternate config files in the build tree. This particular test works by having two config files in separate directories, and setting up a config map to have that redirects A/lit.site.cfg to B/altconfig. Then, we print a message in A/lit.site.cfg and B/altconfig and check that we do see the output from B but don't see the output from A. Additionally we test that the test suite specified by A's config map is properly discovered. Differential Revision: https://reviews.llvm.org/D38105 llvm-svn: 313887
* [lit] Make lit support config files with .py extension.Zachary Turner2017-09-212-0/+33
| | | | | | | | | | | | | | | | | Many editors and Python-related diagnostics tools such as debuggers break or fail in mysterious ways when python files don't end in .py. This is especially true on Windows, but still exists on other platforms. I don't want to be too heavy handed in changing everything across the board, but I do want to at least *allow* lit configs to have .py extensions. This patch makes the discovery process first look for a config file with a .py extension, and if one is not found, then looks for a config file using the old method. So for existing users, there should be no functional change. Differential Revision: https://reviews.llvm.org/D37838 llvm-svn: 313849
* [lit] Fix the lit unit tests.Zachary Turner2017-09-161-0/+1
| | | | | | | | A few tests were manually constructing a LitConfig object, since I added a new argument to it this was triggering some failures I didn't detect. `ninja check-lit` passes now. llvm-svn: 313461
* Resubmit "[lit] Force site configs to run before source-tree configs"Zachary Turner2017-09-151-3/+2
| | | | | | | | | | | | | | | | | | | | This is a resubmission of r313270. It broke standalone builds of compiler-rt because we were not correctly generating the llvm-lit script in the standalone build directory. The fixes incorporated here attempt to find llvm/utils/llvm-lit from the source tree returned by llvm-config. If present, it will generate llvm-lit into the output directory. Regardless, the user can specify -DLLVM_EXTERNAL_LIT to point to a specific lit.py on their file system. This supports the use case of someone installing lit via a package manager. If it cannot find a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or invalid, then we print a warning that tests will not be able to run. Differential Revision: https://reviews.llvm.org/D37756 llvm-svn: 313407
OpenPOWER on IntegriCloud