summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* The canonical way to XFAIL a test for all targets is XFAIL: *, not XFAIL:Benjamin Kramer2016-02-041-1/+1
| | | | | | | | Fix the lit bug that enabled this "feature" (empty triple is substring of all possible target triples) and change the two outliers to use the documented * syntax. llvm-svn: 259799
* Add "/dev/tty" as a special file name for lit tests.Yunzhong Gao2016-01-271-0/+4
| | | | | | | | | | | | If a lit test has a RUN line that includes a redirection to "/dev/tty", the redirection goes to the special device file corresponding to the console. It is /dev/tty on UNIX-like systems and "CON" on Windows. This patch is needed to implement a test like PR25717 (caused by the size limit of the Windows system call WriteConsole() prior to Windows 8) where the test only breaks when outputing to the console and won't fail if using a pipe. llvm-svn: 258898
* [lit] Fix handling of per test timeout when the installed psutil versionDan Liew2016-01-131-1/+8
| | | | | | | | | | is < ``2.0``. Older versions of psutil (e.g. ``1.2.1`` which is the version shipped with Ubuntu 14.04) use a different API for retrieving the child processes. To handle this try the new API first and if that fails try the old API. llvm-svn: 257616
* Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew2016-01-093-50/+2
| | | | | | | | | | | | | | This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
* Teach the CMake build system to run lit's test suite. These can be runDan Liew2016-01-083-2/+50
| | | | | | | | | | | | | | | directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
* [lit] Implement support of per test timeout in lit.Dan Liew2015-12-2717-32/+585
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
* lit: Limit number of processes on Windows to 32.Nico Weber2015-12-221-1/+3
| | | | llvm-svn: 256291
* [lit] Fix bug when using Python3 where a failing test would not showDan Liew2015-11-191-1/+3
| | | | | | | | | | | | | | | | the script when running a ShTest with an external or internal shell. This bug is caused by use of the ``map`` function in Python 3 which returns an iterable (rather than a list in Python 2). After the iterable is exhausted it won't return any more output and consequently when ``_runShTest()`` tries to access the ``script`` which has already been iterated over it is empty. Converting to a list immediatley after calling ``map()`` fixes this. This fixes the ``tests/shtest-format.py`` test when running under Python3 which was previously failing. llvm-svn: 253556
* [lit] Improve error message when lit fails to executable a command byDan Liew2015-11-131-1/+1
| | | | | | showing the executable it tried to use. llvm-svn: 253032
* [lit] Fix bug where ``lit.util.which()`` would return a directoryDan Liew2015-11-131-1/+1
| | | | | | | | | | | | | | | | | instead of executable if the argument was found inside a directory contained in PATH. An example where this could cause a problem is if there was a RUN line that ran the ``test`` command and if the user had a directory in their PATH that contained a directory called ``test/`` (that occured before ``/usr/bin/``). Lit would try to use the directory as the executable which would fail with the rather cryptic message. ``` Could not create process due to [Errno 13] Permission denied ``` llvm-svn: 253031
* lit: Show all output with --show-all, even in combination with --succinctMatthias Braun2015-11-111-0/+1
| | | | | | | I missed an earlier exit for the --succinct case when I introduced the -a option. llvm-svn: 252698
* lit: Add '-a' option to display commands+output of all testsMatthias Braun2015-11-021-4/+9
| | | | | | | The existing -v option only displays commands and outputs for failed tests, the newly introduced -a displays it for all executed tests. llvm-svn: 251806
* lit/TestRunner.py: Factor variable subsitution into an own function; NFCIMatthias Braun2015-10-281-25/+26
| | | | | | | This is a clearer separation of concerns and makes it easier to reuse the function. llvm-svn: 251481
* lit/TestRunner.py: Factor out Substitution construction; NFCMatthias Braun2015-10-281-24/+27
| | | | | | | This is a clearer separation of concerns and makes it easier to reuse the functions. llvm-svn: 251480
* lit/TestRunner.py: Get execdir from test.getExecPath() instead of passing it ↵Matthias Braun2015-10-281-5/+5
| | | | | | around; NFC llvm-svn: 251479
* lit/TestRunner.py: Make parseIntegratedTestScriptCommands() keyword list a ↵Matthias Braun2015-10-281-3/+3
| | | | | | | | | parameter; NFC This allows the function to be easily reused and also simplifies the code as the keyword list is next to the keyword handling now. llvm-svn: 251478
* Lit: Rework r249161; Move RLIMIT_NPROC to main.py.NAKAMURA Takumi2015-10-152-22/+22
| | | | | | The message "raised the process limit..." prevented the progress bar. llvm-svn: 250420
* [lit] Raise the default soft process limit when possibleHal Finkel2015-10-021-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | It is common to have a default soft process limit, at least on some families of Linux distributions, of 1024. This is normally more than enough, but if you have many cores, and you're running tests that create many threads, this can become a problem. My POWER7 development machine has 48 cores, and when running the lld regression tests, which often want to create up to 48 threads, I run into problems. lit, by default, will want to run 48 tests in parallel, and 48*48 < 1024, and so many tests fail like this: terminate called after throwing an instance of 'std::system_error' what(): Resource temporarily unavailable or lit fails like this when launching a test: OSError: [Errno 11] Resource temporarily unavailable lit can easily detect this situation and attempt to repair it before launching tests (by raising the soft process limit to something that will allow ncpus^2 threads to be created), and should do so to prevent spurious test failures. This is the follow-up to this thread: http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html llvm-svn: 249161
* Fix passed env var name in lit for Android tests.Evgeniy Stepanov2015-09-041-1/+1
| | | | | | | | The variable is actually called ANDROID_SERIAL. This was not exercised on the bots until today. Should fix the sanitizer-x86_64-linux failures. llvm-svn: 246898
* [lit] Add basic flaky test retry functionalityReid Kleckner2015-09-023-2/+16
| | | | | | | | | | The plan is to use this for the sanitizer test suite on Windows. See PR24554 for more details on why we need this. Tested manually by injecting rand() into a sanitizer test and watching what it does. llvm-svn: 246704
* Revert "[lit] Speculatively fix PR24554 by manually closing the process handle"Reid Kleckner2015-08-311-3/+0
| | | | | | | | This reverts commit r245946. It didn't help the problem: http://lab.llvm.org:8011/builders/sanitizer-windows/builds/9179/steps/run%20tests/logs/stdio LINK : fatal error LNK1104: cannot open file llvm-svn: 246491
* lit: Match UNSUPPORTED against target triple as we do for XFAIL.Peter Collingbourne2015-08-281-0/+7
| | | | llvm-svn: 246343
* [lit] Speculatively fix PR24554 by manually closing the process handleReid Kleckner2015-08-251-0/+3
| | | | | | | | | | | My theory is that somehow Python's refcounting and GC strategy isn't closing the subprocess handle in a timely fashion. This accesses the private '_handle' field of the Popen object, but I see no other way to do this. If this doesn't address the problem on the sanitizer-windows buildbot, we can revert this change. If it does, then let's keep the hack. llvm-svn: 245946
* [lit] Fix launching executables relative to the cwd after 'cd'Reid Kleckner2015-07-221-1/+8
| | | | | | | | | | | | | | This was affecting test/asan/TestCases/Windows/coverage-basic.cc in compiler-rt. It does something like: cd %T/mydir %clang %s -o t.exe ./t.exe Previously, we'd end up looking for t.exe relative to the cwd of the lit process, not the cwd of the test. llvm-svn: 242941
* [LIT] Fix discovery.py test once againEric Fiselier2015-07-211-4/+4
| | | | llvm-svn: 242758
* [lit] Implement 'env' in the internal shellReid Kleckner2015-07-201-7/+28
| | | | | | | | | | | | | | | | | | | | The MSys 2 version of 'env' cannot be used to set 'TZ' in the environment due to some portability hacks in the process spawning compatibility layer[1]. This affects test/Object/archive-toc.test, which tries to set TZ in the environment. Other than that, this saves a subprocess invocation of a small unix utility, which is makes the tests faster. The internal shell does not support shell variable expansion, so this idiom in the ASan tests isn't supported yet: RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:opt=1 ... [1] https://github.com/Alexpux/MSYS2-packages/issues/294 Differential Revision: http://reviews.llvm.org/D11350 llvm-svn: 242696
* [LIT] Allow for executeCommand to take the stdin input.Eric Fiselier2015-07-191-3/+3
| | | | | | | | | | | | Summary: This patch allows executeCommand to pass a string to the processes stdin. Reviewers: ddunbar, jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D11332 llvm-svn: 242631
* Pass 2 more variables to lit tests.Evgeniy Stepanov2015-06-231-1/+1
| | | | | | | | Pass ADB and ADB_SERIAL environment variables to lit tests. This would allow running Android tests in compiler-rt when there is more than one device attached to the host. llvm-svn: 240459
* [LIT] Fix failing LIT testsEric Fiselier2015-06-134-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | Summary: I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine. I made the following changes for the following reasons. 1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes. 2. discovery.py: The output appears in a different order on my machine than it did in the test. 3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test. 4. The classname is now formed differently in `getJUnitXML(...)`. I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts? Reviewers: ddunbar, danalbert, jroelofs Reviewed By: jroelofs Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9864 llvm-svn: 239663
* lit: Allow configurations to restrict the set of tests to runArnold Schwaighofer2015-06-012-1/+14
| | | | | | | | | | By setting limit_to_features to a non empty list of features a configuration can restrict the set of tests to run to only include tests that require a feature in this list. rdar://21082253 llvm-svn: 238766
* Lit: Allow overriding llvm tool paths+arguments, make -D an alias for --paramMatthias Braun2015-05-041-1/+1
| | | | | | | | | | | These changes allow usages where you want to pass an additional commandline option to all invocations of a specific llvm tool. Example: > llvm-lit -Dllc=llc -enable-misched -verify-machineinstrs Differential Revision: http://reviews.llvm.org/D9487 llvm-svn: 236461
* [lit] Allow disabling an entire gtest suite, as is done in tsanReid Kleckner2015-04-071-1/+2
| | | | llvm-svn: 234336
* [lit] Skip gtest names starting with DISABLED_Reid Kleckner2015-04-061-0/+4
| | | | | | | | | | | The sanitizer test suite uses this idiom to disable a test. Now that we actually check if a test ran after invoking it, we see that zero tests ran, and complain. Instead, ignore tests starting with DISABLED_ completely. Fixes the sanitizer test suite failures on Windows. llvm-svn: 234247
* [lit] Fix running gtest type-parameterized tests on WindowsReid Kleckner2015-04-061-4/+11
| | | | | | | | | | | | | | The '/' character in the test name of a type-parameterized test is not a path separator, and should not be '\' on Windows. We were passing a test name to --gtest_filter which found no tests, so the exit code was zero, indicating a passed test. This bug has been here since r84387 in 2009, when Jeff Yasskin added the original lit support for type-paratermized tests. Somewhere along the line some of the ValueMapTests started failing, but we can fix those separately. llvm-svn: 234242
* lit: Add 'cd' support to the internal shell and port some testsReid Kleckner2015-03-021-13/+39
| | | | | | | | | | | The internal shell was already threading around a 'cwd' parameter. We just have to make it mutable so that we can update it as the test script executes. If the shell ever grows support for environment variable substitution, we could also implement support for export. llvm-svn: 231017
* Don't warn or note if bash is missingReid Kleckner2015-02-041-2/+0
| | | | | | | | | We haven't needed bash on Windows to run the test suite for a long time now. Patch by Michael Edwards! llvm-svn: 228221
* Fix some bashims. More information on https://wiki.ubuntu.com/DashAsBinSh. ↵Sylvestre Ledru2015-02-011-1/+1
| | | | | | Reported initially on https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772302 & https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772301 llvm-svn: 227744
* [lit] Format JSONMetricValue strings better.Eric Fiselier2015-01-211-1/+2
| | | | llvm-svn: 226672
* Don't set LD_PRELOAD to ''. It doesn't work on OpenBSD.Rafael Espindola2015-01-141-1/+5
| | | | | | Patch by Brad Smith. llvm-svn: 225890
* [LIT] Remove string decoding in gtest discovery code. lit.util.capture now ↵Eric Fiselier2015-01-121-1/+0
| | | | | | does decoding. llvm-svn: 225693
* [LIT] Decode string result in lit.util.captureEric Fiselier2015-01-121-9/+9
| | | | | | | | | | | | | | Summary: I think this is probably a bug, but I'm putting this up for review just to be sure. I think that `lit.util.capture` should decode the resulting string in the same way `lit.util.executeCommand` does. Reviewers: ddunbar, EricWF Reviewed By: EricWF Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D6769 llvm-svn: 225681
* Pass LSAN_OPTIONS down so that it is possible to add suppressions.Rafael Espindola2014-12-231-1/+2
| | | | llvm-svn: 224777
* Split executeShTest into two parts so that it can be better leveraged by libc++Eric Fiselier2014-12-201-13/+18
| | | | llvm-svn: 224672
* [LIT] Add JSONMetricValue type to wrap types supported by the json encoder.Eric Fiselier2014-12-191-0/+36
| | | | | | | | | | | | | | | | | | | | | | Summary: The following types can be encoded and decoded by the json library: `dict`, `list`, `tuple`, `str`, `unicode`, `int`, `long`, `float`, `bool`, `NoneType`. `JSONMetricValue` can be constructed with any of these types, and used as part of Test.Result. This patch also adds a toMetricValue function that converts a value into a MetricValue. Reviewers: ddunbar, EricWF Reviewed By: EricWF Subscribers: cfe-commits, llvm-commits Differential Revision: http://reviews.llvm.org/D6576 llvm-svn: 224628
* Now that we require a newer python, use the new exception syntax.Rafael Espindola2014-12-121-1/+1
| | | | llvm-svn: 224130
* Require python 2.7.Rafael Espindola2014-12-121-2/+0
| | | | | | | | | | | | | We were already requiring 2.5, which meant that people on old linux distros had to upgrade anyway. Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X exception syntax. According to the discussion on llvmdev, there is not much value is requiring just 2.6, we may as well just require 2.7. llvm-svn: 224129
* [LIT] Add support for `UNSUPPORTED` tag to ↵Eric Fiselier2014-12-102-19/+29
| | | | | | | | | | | | | | | | | | | | | | | `TestRunner.parseIntegratedTestScript` Summary: This patch gives me just enough to leverage the existing functionality in `TestRunner` for use in `libc++` and `libc++abi` . It does the following: * Adds the `UNSUPPORTED` tag to `TestRunner.parseIntegratedTestScript`. * Allows `parseIntegratedTestScript` to return an empty script if a script is not required by the caller. Reviewers: ddunbar, EricWF Reviewed By: EricWF Subscribers: cfe-commits, llvm-commits Differential Revision: http://reviews.llvm.org/D6589 llvm-svn: 223915
* lit: Don't use python 2.6 featuresDavid Majnemer2014-12-081-1/+1
| | | | | | LLVM supports python 2.5, this fixes a (somewhat) recent regression. llvm-svn: 223626
* Fix for xunit output to work around issue in Jenkins when tests are at the ↵Chris Matthews2014-12-061-1/+1
| | | | | | root level llvm-svn: 223562
* Fix corner cases in lit xunit for paths with dotsChris Matthews2014-12-061-2/+9
| | | | llvm-svn: 223549
OpenPOWER on IntegriCloud