summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
...
* [lit] Throw in unimplemented method (NFC)Chris Bieneman2016-10-031-1/+1
| | | | | | | | | | | | | | | | | | Summary: lit's `OneCommandFileTest` class implements an abstract method that raises if called. However, it raises by referencing an undefined symbol. Instead, raise explicitly by throwing a `NotImplementedError`. This is clearer, and appeases Python linters. Patch By Brian Gesiak! Reviewers: ddunbar, echristo, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25170 llvm-svn: 283090
* [lit] Remove unused imports (NFC)Chris Bieneman2016-10-034-5/+6
| | | | | | | | | | | | Reviewers: ddunbar, echristo, beanz Patch by Brian Gesiak! Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25169 llvm-svn: 283089
* [lit] Compare to None using identity, not equalityChris Bieneman2016-10-032-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | Summary: In Python, `None` is a singleton, so checking whether a variable is `None` may be done with `is` or `is not`. This has a slight advantage over equiality comparisons `== None` and `!= None`, since `__eq__` may be overridden in Python to produce sometimes unexpected results. Using `is None` and `is not None` is also recommended practice in https://www.python.org/dev/peps/pep-0008: > Comparisons to singletons like `None` should always be done with `is` or > `is not`, never the equality operators. Patch by Brian Gesiak! Reviewers: ddunbar, echristo, beanz Subscribers: llvm-commits, mehdi_amini Differential Revision: https://reviews.llvm.org/D25168 llvm-svn: 283088
* [lit] Add instructions to run lit's test suiteDaniel Dunbar2016-09-271-0/+16
| | | | | | | | - Patch by Brian Gesiak. - https://reviews.llvm.org/D24968 llvm-svn: 282525
* [lit] Fix refacto introduced by rL282479.Daniel Dunbar2016-09-271-1/+3
| | | | llvm-svn: 282501
* Trying to fix lldb build breakage probably caused by rL282452Dimitar Vlahovski2016-09-271-11/+11
| | | | llvm-svn: 282479
* [lit] Add a --max-failures option.Daniel Dunbar2016-09-264-3/+41
| | | | | | | | | - This is primarily useful as a "fail fast" mode for lit, where it will stop running tests after the first failure. - Patch by Max Moiseev. llvm-svn: 282452
* [lit] Downgrade error to warning on gtest crashes during discovery.Ahmed Bougacha2016-09-051-2/+2
| | | | | | | | | | Lots of unittests started failing under asan after r280455. It seems they've been failing for a long time, but lit silently ignored them. Downgrade the error so we can figure out what is going on. Filed http://llvm.org/PR30285. llvm-svn: 280674
* lit/util.py: Another fix for py3.NAKAMURA Takumi2016-09-051-0/+2
| | | | | | 'str' object has no attribute 'decode'. llvm-svn: 280641
* Make lit/util.py py3-compatible.NAKAMURA Takumi2016-09-031-1/+1
| | | | llvm-svn: 280579
* lit: print process output, if getting the list of google-tests failed.Ivan Krasin2016-09-022-6/+11
| | | | | | | | | | | | | Summary: This is a follow up to r280455, where a check for the process exit code was introduced. Some ASAN bots throw this error now, but it's impossible to understand what's wrong with them, and the issue is not reproducible. Reviewers: vitalybuka Differential Revision: https://reviews.llvm.org/D24210 llvm-svn: 280550
* Quick fix to make LIT_PRESERVES_TMP work againReid Kleckner2016-09-021-1/+2
| | | | llvm-svn: 280502
* [lit] Clean up temporary files created by testsReid Kleckner2016-09-022-11/+28
| | | | | | | | | | | | Do this by creating a temp directory in the normal system temp directory, and cleaning it up on exit. It is still possible for this temp directory to leak if Python exits abnormally, but this is probably good enough for now. Fixes PR18335 llvm-svn: 280501
* [lit] Fail testing if a googletest executable crashes during test discoveryGreg Parker2016-09-022-2/+6
| | | | | | | | | googletest formatted tests are discovered by running the test executable. Previously testing would silently succeed if the test executable crashed during the discovery process. Now testing fails with "error: unable to discover google-tests ..." if the test executable exits with a non-zero status. llvm-svn: 280455
* [lit] Use multiprocessing by default on WindowsReid Kleckner2016-09-011-6/+3
| | | | | | | | | | | | | | | | Apparently nobody evaluated multiprocessing on Windows since Daniel enabled multiprocessing on Unix in r193279. It works so far as I can tell. Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24 core Windows machine. Hopefully this will improve Windows buildbot cycle time, where currently it takes more time to run check-all than it does to self-host with assertions enabled: http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20 build stage 2 ninja all ( 28 mins, 22 secs ) ninja check 2 stage 2 ( 37 mins, 38 secs ) llvm-svn: 280382
* [lit] Use full config path in diagnostics.Daniel Dunbar2016-07-211-1/+1
| | | | | | | | | - This allows tools like emacs to automatically find the config file path when you step through errors. - Patch by Dave Abrahams. llvm-svn: 276357
* [lit] Bump version number.Daniel Dunbar2016-07-211-2/+2
| | | | llvm-svn: 276353
* [lit] Add SANITIZER_IGNORE_CVE_2016_2143 to pass_vars.Marcin Koscielnicki2016-06-261-1/+2
| | | | | | | | | | | | | | | This variable is used by ASan (and other sanitizers in the future) on s390x-linux to override a check for CVE-2016-2143 in the running kernel (see revision 267747 on compiler-rt). Since the check simply checks if the kernel version is in a whitelist of known-good versions, it may miss distribution kernels, or manually-patched kernels - hence the need for this variable. To enable running the ASan testsuite on such kernels, this variable should be passed from the environment down to the testcases. Differential Revision: http://reviews.llvm.org/D19888 llvm-svn: 273825
* [lit] Only gather redirected files for command failures.Daniel Dunbar2016-06-101-10/+11
| | | | | | | | | | | - The intended use of this was just in diagnostics, so we shouldn't pay the cost of reading these all the time. - This will avoid including the full output of each command in tests which fail, but the most important use case for this was to gather the output of the specific command which failed. llvm-svn: 272365
* Revert "[lit] Use os.devnull instead of named temp files"Vedant Kumar2016-06-091-5/+19
| | | | | | | | | This reverts commit r272290. It breaks a test that depends on being able to seek the /dev/null equivalent on Windows: http://bb.pgr.jp/builders/ninja-clang-x64-mingw64-RA/builds/11360 llvm-svn: 272293
* [lit] Use os.devnull instead of named temp filesVedant Kumar2016-06-091-19/+5
| | | | | | | | | | | Use os.devnull instead of tempfiles when substituting '/dev/null' on Windows machines. This should make the bots just a bit speedier. Thanks to Yunzhong Gao for testing this patch on Windows! Differential Revision: http://reviews.llvm.org/D20549 llvm-svn: 272290
* [lit] Ensure we get bytes when reading redirected output files.Daniel Dunbar2016-06-081-1/+1
| | | | llvm-svn: 272147
* [lit] Ignore errors when decoding redirected output.Daniel Dunbar2016-06-071-1/+1
| | | | llvm-svn: 272066
* [lit] Fix an uninitialized var on Windows.Daniel Dunbar2016-06-071-1/+2
| | | | llvm-svn: 272052
* [utils/lit] Show available_features with --show-suites.Daniel Dunbar2016-06-071-0/+3
| | | | llvm-svn: 272022
* [lit] Improve logging with file redirection.Daniel Dunbar2016-06-074-9/+39
| | | | | | | | | - This will cause lit to automatically include the first 1K of data in redirected output files when a command fails (previously if the command failed, but the main point of the test was, say, a `FileCheck` later on, then the log wasn't helpful in showing why the command failed). llvm-svn: 272021
* [lit] Improve readability of failing scripts.Daniel Dunbar2016-06-026-14/+69
| | | | | | | | | | | | | | | | | | | | | | | - This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
* [lit] Factor out a helper for shell command results.Daniel Dunbar2016-06-021-8/+22
| | | | llvm-svn: 271608
* Add "REQUIRES-ANY" feature testEric Fiselier2016-06-024-3/+19
| | | | | | | | | | | | | | | | Summary: This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available. Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed. Reviewers: rnk, ddunbar Subscribers: ddunbar, probinson, llvm-commits, cfe-commits Differential Revision: http://reviews.llvm.org/D20757 llvm-svn: 271468
* [lit] Add %:[STpst] to represent paths without colons on Windows.Rui Ueyama2016-04-301-0/+18
| | | | | | | | | | | | Summary: We need these variables to concatenate two absolute paths to construct a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo, which is not a valid path because ":" is not a valid path character on Windows. With this patch, %t will be expanded to C\foo. Differential Revision: http://reviews.llvm.org/D19757 llvm-svn: 268168
* lit: python3 compatibility fixMatthias Braun2016-03-311-1/+8
| | | | llvm-svn: 265070
* [lit][googletest] Handle upstream gtest outputDaniel Dunbar2016-03-314-0/+67
| | | | | | | | | | | | | | | | | | | Summary: Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior to running tests. LLVM removed that print statement in r61540. If a user were to use lit to run tests that use upstream googletest, however, lit reports "Running main()" as an invalid test name. To avoid such a failure, add an extra conditional to `formats/googletest.py`. Also add tests to demonstrate the modified behavior. Reviewers: abdulras, ddunbar Subscribers: ddunbar, llvm-commits, kastiglione Differential Revision: http://reviews.llvm.org/D18606 llvm-svn: 265034
* [lit] Enqueue tests on a separate thread to not hit limits on parallel queuesFilipe Cabecinhas2016-03-171-2/+14
| | | | | | | | | | | | | | | | | Summary: The multiprocessing.Queue.put() call can hang if we try queueing all the tests before starting to take them out of the queue. The current implementation hangs if tests exceed 2^^15, on Mac OS X. This might happen with a ninja check-all if one has a bunch of llvm projects. Reviewers: delcypher, bkramer Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D17609 llvm-svn: 263731
* [lit] Hack lit to allow a test suite to request that it is run "early".Chandler Carruth2016-03-123-2/+14
| | | | | | | | | | | | | This lets us for example start running the unit test suite early. For 'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!! It's pretty ugly. I barely know how to write Python, so feel free to just tell me how I should write it instead. =D Thanks to Filipe and others for help. Differential Revision: http://reviews.llvm.org/D18089 llvm-svn: 263329
* The canonical way to XFAIL a test for all targets is XFAIL: *, not XFAIL:Benjamin Kramer2016-02-041-1/+1
| | | | | | | | Fix the lit bug that enabled this "feature" (empty triple is substring of all possible target triples) and change the two outliers to use the documented * syntax. llvm-svn: 259799
* Add "/dev/tty" as a special file name for lit tests.Yunzhong Gao2016-01-271-0/+4
| | | | | | | | | | | | If a lit test has a RUN line that includes a redirection to "/dev/tty", the redirection goes to the special device file corresponding to the console. It is /dev/tty on UNIX-like systems and "CON" on Windows. This patch is needed to implement a test like PR25717 (caused by the size limit of the Windows system call WriteConsole() prior to Windows 8) where the test only breaks when outputing to the console and won't fail if using a pipe. llvm-svn: 258898
* [lit] Fix handling of per test timeout when the installed psutil versionDan Liew2016-01-131-1/+8
| | | | | | | | | | is < ``2.0``. Older versions of psutil (e.g. ``1.2.1`` which is the version shipped with Ubuntu 14.04) use a different API for retrieving the child processes. To handle this try the new API first and if that fails try the old API. llvm-svn: 257616
* Revert "Teach the CMake build system to run lit's test suite. These can be run"Dan Liew2016-01-093-50/+2
| | | | | | | | | | | | | | This reverts r257221. This caused several build bot failures * It looks like some of the tests don't work correctly under Windows * It looks like the lit per test timeout tests fail So I'm reverting for now. Once the above failures are fixed running lit's tests can be enabled again. llvm-svn: 257268
* Teach the CMake build system to run lit's test suite. These can be runDan Liew2016-01-083-2/+50
| | | | | | | | | | | | | | | directy with ``make check-lit`` and are run as part of ``make check-all``. In principle we should run lit's testsuite before testing LLVM using lit so that any problems with lit get discovered before testing LLVM so we can bail out early. However this implementation (``check-all`` runs all tests together) seemed simpler and will still report failing lit tests. Note that the tests and the configured ``lit.site.cfg`` have to be copied into the build directory to avoid polluting the source tree. llvm-svn: 257221
* [lit] Implement support of per test timeout in lit.Dan Liew2015-12-2717-32/+585
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This should work with ShTest (executed externally or internally) and GTest test formats. To set the timeout a new option ``--timeout=`` has been added which specifies the maximum run time of an individual test in seconds. By default this 0 which causes no timeout to be enforced. The timeout can also be set from a lit configuration file by modifying the ``lit_config.maxIndividualTestTime`` property. To implement a timeout we now require the psutil Python module if a timeout is requested. This dependency is confined to the newly added ``lit.util.killProcessAndChildren()``. A note has been added into the TODO document describing how we can remove the dependency on the ``pustil`` module in the future. It would be nice to remove this immediately but that is a lot more work and Daniel Dunbar believes it is better that we get a working implementation first and then improve it. To avoid breaking the existing behaviour the psutil module will not be imported if no timeout is requested. The included testcases are derived from test cases provided by Jonathan Roelofs which were in an previous attempt to add a per test timeout to lit (http://reviews.llvm.org/D6584). Thanks Jonathan! Reviewers: ddunbar, jroelofs, cmatthews, MatzeB Subscribers: cmatthews, llvm-commits Differential Revision: http://reviews.llvm.org/D14706 llvm-svn: 256471
* lit: Limit number of processes on Windows to 32.Nico Weber2015-12-221-1/+3
| | | | llvm-svn: 256291
* [lit] Fix bug when using Python3 where a failing test would not showDan Liew2015-11-191-1/+3
| | | | | | | | | | | | | | | | the script when running a ShTest with an external or internal shell. This bug is caused by use of the ``map`` function in Python 3 which returns an iterable (rather than a list in Python 2). After the iterable is exhausted it won't return any more output and consequently when ``_runShTest()`` tries to access the ``script`` which has already been iterated over it is empty. Converting to a list immediatley after calling ``map()`` fixes this. This fixes the ``tests/shtest-format.py`` test when running under Python3 which was previously failing. llvm-svn: 253556
* [lit] Improve error message when lit fails to executable a command byDan Liew2015-11-131-1/+1
| | | | | | showing the executable it tried to use. llvm-svn: 253032
* [lit] Fix bug where ``lit.util.which()`` would return a directoryDan Liew2015-11-131-1/+1
| | | | | | | | | | | | | | | | | instead of executable if the argument was found inside a directory contained in PATH. An example where this could cause a problem is if there was a RUN line that ran the ``test`` command and if the user had a directory in their PATH that contained a directory called ``test/`` (that occured before ``/usr/bin/``). Lit would try to use the directory as the executable which would fail with the rather cryptic message. ``` Could not create process due to [Errno 13] Permission denied ``` llvm-svn: 253031
* lit: Show all output with --show-all, even in combination with --succinctMatthias Braun2015-11-111-0/+1
| | | | | | | I missed an earlier exit for the --succinct case when I introduced the -a option. llvm-svn: 252698
* lit: Add '-a' option to display commands+output of all testsMatthias Braun2015-11-021-4/+9
| | | | | | | The existing -v option only displays commands and outputs for failed tests, the newly introduced -a displays it for all executed tests. llvm-svn: 251806
* lit/TestRunner.py: Factor variable subsitution into an own function; NFCIMatthias Braun2015-10-281-25/+26
| | | | | | | This is a clearer separation of concerns and makes it easier to reuse the function. llvm-svn: 251481
* lit/TestRunner.py: Factor out Substitution construction; NFCMatthias Braun2015-10-281-24/+27
| | | | | | | This is a clearer separation of concerns and makes it easier to reuse the functions. llvm-svn: 251480
* lit/TestRunner.py: Get execdir from test.getExecPath() instead of passing it ↵Matthias Braun2015-10-281-5/+5
| | | | | | around; NFC llvm-svn: 251479
* lit/TestRunner.py: Make parseIntegratedTestScriptCommands() keyword list a ↵Matthias Braun2015-10-281-3/+3
| | | | | | | | | parameter; NFC This allows the function to be easily reused and also simplifies the code as the keyword list is next to the keyword handling now. llvm-svn: 251478
OpenPOWER on IntegriCloud