summaryrefslogtreecommitdiffstats
path: root/llvm/utils/lit
Commit message (Collapse)AuthorAgeFilesLines
* lit::shtest-format.py: Make write-bad-encoding.py py3-aware.NAKAMURA Takumi2017-07-291-1/+1
| | | | | | | | | | Traceback (most recent call last): File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module> sys.stdout.write(b"a line with bad encoding: \xc2.") sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts. llvm-svn: 309473
* [lit] Use a %{python} substitution to avoid relying on python being on PATHReid Kleckner2017-07-285-9/+7
| | | | llvm-svn: 309434
* [lit] Remove stale test inputs before running check-litReid Kleckner2017-07-281-0/+1
| | | | | | | | This should fix googletest-format test failures on the clang modules buildbots, which have a stale copy of the OneTest script in the build directory. llvm-svn: 309432
* [lit] Dump some FileCheck inputs to try to debug some failing testsReid Kleckner2017-07-282-0/+6
| | | | llvm-svn: 309400
* [lit] Fix shtest-format external_shell failuresReid Kleckner2017-07-284-4/+8
| | | | | | | | | | When using win32 cmd.exe, turn off command echoing at the beginning of the script (@echo off). Replace a bash shell script with a python script for the fail_with_bad_encoding test. llvm-svn: 309399
* [lit] Port googletest lit tests to WindowsReid Kleckner2017-07-287-26/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: The technique of directly calling subprocess.Popen on a python script doesn't work on Windows. The executable path of the command must refer to a valid win32 executable. Instead, rename all the python scripts masquerading as gtest executables to have .py extensions, so we can easily detect then and call the python executable for them. Do this on Linux as well as Windows for consistency. The test suite directory names also come out in lower-case on Windows. We can consider removing that in a later patch. This change just updates the FileCheck lines to match on Windows. Fixes PR33933 Reviewers: modocache, mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35909 llvm-svn: 309347
* [lit] Fix TestRunner unit test on WindowsBrian Gesiak2017-07-271-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Normally Python converts all newline characters, Windows or Unix, to Unix newlines when opening a file. However, lit opens files in binary mode, which does not perform this conversion. As a result, trailing Windows newlines are not stripped from test input, which caused a failure in the TestRunner unit test: ``` FAIL: test_custom (__main__.TestIntegratedTestKeywordParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\bgesiak\src\llvm\llvm\utils\lit\tests\unit\TestRunner.py", line 109, in test_custom self.assertItemsEqual(value, ['a', 'b', 'c']) AssertionError: Element counts were not equal: First has 1, Second has 0: 'c\r' First has 0, Second has 1: 'c' ``` Fix the discrepancy in behavior across the two platforms by manually stripping Windows newlines before yielding each line in the test file. Reviewers: echristo, beanz, ddunbar, delcypher, rnk Reviewed By: rnk Subscribers: mehdi_amini, llvm-commits Differential Revision: https://reviews.llvm.org/D27746 llvm-svn: 309312
* Un-revert "Teach the CMake build system to run lit's test suite. These can ↵Brian Gesiak2017-07-273-2/+50
| | | | | | | | | | | | | | | | | | | | | | be run" Summary: Depends on https://reviews.llvm.org/D35879. This reverts rL257268, which in turn was a revert of rL257221. https://reviews.llvm.org/D35879 marks the tests in the lit test suite that fail on Windows as XFAIL, which should allow these tests to pass on Windows-based buildbots. Reviewers: delcypher, beanz, mgorny, jroelofs, rnk Reviewed By: mgorny Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits Differential Revision: https://reviews.llvm.org/D35880 llvm-svn: 309310
* [lit] Fix order of checks in shtest-shell.py testBrian Gesiak2017-07-272-0/+10
| | | | | | | | | | | | | | | | | | Summary: An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt` expects that first a string printed to stdout is seen, and then a string printed to stderr. Add `flush()` calls to ensure that stdout is printed before stderr, as expected. Reviewers: rnk, mgorny, jroelofs Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35947 llvm-svn: 309292
* [lit] Fix race between shtest-shell and max-failures testsReid Kleckner2017-07-262-4/+10
| | | | | | | Previously these tests would use the same Output directory leading to flaky non-deterministic failures. llvm-svn: 309227
* [lit] Fix shtest-shell and max-failures lit tests on WindowsReid Kleckner2017-07-267-16/+10
| | | | | | | | | Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell scripts as python scripts and call python on them. Fixes PR33940 llvm-svn: 309200
* [lit] Fix shtest-output-printing.py on Windows by matching either / or \\Reid Kleckner2017-07-261-4/+1
| | | | | | Fixes PR33938 llvm-svn: 309198
* [lit] Fix discovery.py on Windows by matching backslashes when necessaryReid Kleckner2017-07-261-22/+19
| | | | | | Fixes PR33932 llvm-svn: 309194
* [lit] Un-XFAIL selecting.py test on WindowsReid Kleckner2017-07-261-3/+0
| | | | | | | | This passes locally for me, which fails the overall lit test suite. I can't debug a passing test, but I will try to help debug the test when we get some failing logs. llvm-svn: 309190
* Un-XFAIL some internal lit tests on Windows, they pass for me locallyReid Kleckner2017-07-263-9/+0
| | | | llvm-svn: 309144
* Fix LIT test breakageGeorge Karpenkov2017-07-261-1/+1
| | | | | | Differential Revision: https://reviews.llvm.org/D35867 llvm-svn: 309140
* [lit] Mark several of lit's tests XFAIL on WindowsBrian Gesiak2017-07-2613-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: rL257221 attempted to run lit's own test suite continuously, but that commit was reverted because lit's test suite does not pass on Windows. Because lit's tests do not run continuously, they often regress. In order to un-revert rL257221, mark lit tests that fail as XFAIL for Windows platforms. Test Plan: On a Windows development environment, follow the instructions in utils/lit/README.txt to run lit's test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test suite is run and a successful exit code is returned. Reviewers: mgorny, rnk, delcypher, beanz Reviewed By: rnk Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35879 llvm-svn: 309123
* [lit] Fix type error for parallelism groupsBrian Gesiak2017-07-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Whereas rL299560 and rL309071 call `parallelism_groups.items()`, under the assumption that `parallelism_groups` is a `dict` type, the default parameter for that attribute is a `list`. Change the default to a `dict` for type correctness. This regression in the unit tests would have been caught if the unit tests were being run continously. It also would have been caught if the lit project used a Python type checker such as `mypy`. Test Plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the test `lit :: unit/TestRunner.py` fails before applying this patch, but passes once this patch is applied. Reviewers: mgorny, rnk, rafael Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35878 llvm-svn: 309122
* Revert "[lit] Remove dead code not referenced in the LLVM SVN repo."Brian Gesiak2017-07-263-74/+123
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This reverts rL306623, which removed `FileBasedTest`, an abstract base class, but did not also remove the usages of that class in the lit unit tests. The revert fixes four test failures in the lit unit test suite. Test plan: As per the instructions in `utils/lit/README.txt`, run the lit unit test suite: ``` utils/lit/lit.py \ --path /path/to/your/llvm/build/bin \ utils/lit/tests ``` Verify that the following tests fail before applying this patch, and pass once the patch is applied: ``` lit :: test-data.py lit :: test-output.py lit :: xunit-output.py ``` In addition, run `check-llvm` to make sure the existing LLVM test suite executes normally. Reviewers: george.karpenkov, mgorny, dlj Reviewed By: mgorny Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35877 llvm-svn: 309120
* [lit] Attempt to fix Python unittest adaptor logicReid Kleckner2017-07-261-7/+6
| | | | llvm-svn: 309071
* [lit] Fix UnboundLocalError for invalid shtest redirectsMichal Gorny2017-07-251-1/+1
| | | | | | | | | | | | | | | | | Replace the incorrect variable reference when invalid redirect is used. This fixes the following issue: File "/usr/src/llvm/utils/lit/lit/TestRunner.py", line 316, in processRedirects raise InternalShellError(cmd, "Unsupported redirect: %r" % (r,)) UnboundLocalError: local variable 'r' referenced before assignment which in turn broke shtest-shell.py and max-failures.py lit tests. The breakage was introduced during refactoring in rL307310. Differential Revision: https://reviews.llvm.org/D35857 llvm-svn: 309044
* [lit] add a -vv option to echo all executed commands.George Karpenkov2017-07-133-2/+16
| | | | | | | | | | | | | | | | | | | Debugging LIT scripts can be rather painful, as LIT directly does not specify which line has failed. Rather, FileCheck is expected to report the failing location, but it can be often ambiguous if multiple commands are tested against the same prefix. This change adds a -vv option, which echoes all output. Then detecting the error becomes straightforward: last printed line is the failing one. Of course, it could be desired to try to get failing line number directly from bash, but it involves excessive hacks on older bash versions (cf. https://stackoverflow.com/questions/24398691/how-to-get-the-real-line-number-of-a-failing-bash-command) Differential Revision: https://reviews.llvm.org/D35330 llvm-svn: 307938
* [lit] Fix import StringIO errors in Python 3Reid Kleckner2017-07-111-3/+3
| | | | | | | Remove the cStringIO micro-optimization, as it isn't portable to Python 3. llvm-svn: 307669
* [lit] Implement non-pipelined echo commands internallyReid Kleckner2017-07-111-0/+74
| | | | | | | | | | | | | | | Summary: This speeds up the LLD test suite on Windows by 3x. Most of the time is spent on lld/test/ELF/linkerscript/diagnostics.s, which repeatedly constructs linker scripts with appending echo commands. Reviewers: dlj, zturner, modocache Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D35093 llvm-svn: 307668
* [lit] Modify LIT to accept environment variable LIT_FILTER to select tests.George Karpenkov2017-07-072-1/+7
| | | | | | | | | This is especially useful when lit is invoked indirectly by the build system, and additional arguments can not be easily specified. Differential Revision: https://reviews.llvm.org/D35091 llvm-svn: 307339
* [lit] Factor out some shell input/output redirection logic, NFCReid Kleckner2017-07-061-70/+97
| | | | | | | This is a very light refactoring aimed at improving readability. There is definitely still room for improvement here. llvm-svn: 307310
* [lit] Fix unit test discovery for Visual Studio builds.David L. Jones2017-07-061-1/+4
| | | | | | | | | | | | | | | | | | | | Fix by Andrew Ng! The Visual Studio build can contain output for multiple configuration types ( e.g. Debug, Release & RelWithDebInfo) within the same build output directory. Therefore when discovering unit tests, the "build mode" sub directory containing the appropriate configuration is included in the search. This sub directory may not always be present, so a test for its existence is required. Reviewers: zturner, modocache, dlj Reviewed By: zturner, dlj Subscribers: grimar, bd1976llvm, gbreynoo, edd, jhenderson, llvm-commits Differential Revision: https://reviews.llvm.org/D34976 llvm-svn: 307235
* [lit] Factor out listdir logic shared by different test formats.David L. Jones2017-06-303-44/+58
| | | | | | | | | | | | | | | | | | | | Summary: The lit test formats use largely the same logic for discovering tests. There are some superficial differences in the logic, which seem reasonable enough to handle in a single routine. At a high level, the common goal is "look for files that end with one of these suffixes, and skip anything starting with a dot." The balance of the logic specific to ShTest and GoogleTest collapses quite a bit, so that getTestsInDirectory is only a couple of lines around a call to the new function. Reviewers: zturner, MatzeB, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34855 llvm-svn: 306895
* Revert "[lit] Clean output directories before running tests."Zachary Turner2017-06-303-51/+39
| | | | | | | | | This reverts commit da6318a92fba793e4f2447ec478b001392d57d43. This is causing failures on some build bots due to what appears to be some kind of lit ordering dependency. llvm-svn: 306833
* [lit] Clean output directories before running tests.Zachary Turner2017-06-303-39/+51
| | | | | | | | | | | | | | | | | | Presently lit leaks files in the tests' output directories. Specifically, if a test creates output files, lit makes no effort to remove them prior to the next test run. This is problematic because it leads to false positives whenever a test passes because stale files were present. In general it is a source of flakiness that should be removed. This patch addresses this by building the list of all test directories that are part of the current run set, and then deleting those directories and recreating them anew. This gives each test a clean baseline to start from. Differential Revision: https://reviews.llvm.org/D34732 llvm-svn: 306832
* [lit] Re-apply: Fix some convoluted logic around Unicode encoding, and ↵David L. Jones2017-06-292-51/+69
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | de-duplicate across modules that used it. (Take 2: this patch re-applies r306625, which was reverted in r306629. This patch includes only trivial fixes.) In Python2 and Python3, the various (non-)?Unicode string types are sort of spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere" approach, with 'str' representing a Unicode string. Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes. However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting problems when an interface requires a precise type, but has to run under both Python2 and Python3. The previous logic appeared to be correct in all cases, but went through more layers of indirection than necessary. This change does the necessary conversions in one shot, with documentation about which paths might be taken in Python2 or Python3. Changes from r306625: some tests just print binary outputs, so in those cases, fall back to str() in Python3. For googletests, add one missing call to to_string(). (Tested by verifying the visible breakage with Python3. Verified that everything works in py2 and py3.) llvm-svn: 306643
* Revert "[lit] Fix some convoluted logic around Unicode encoding, and ↵David L. Jones2017-06-292-62/+51
| | | | | | | | de-duplicate across modules that used it." This reverts r306625. llvm-svn: 306629
* Fix spelling: uncode -> unicode.David L. Jones2017-06-291-1/+1
| | | | | | | Remember kids: there is no 'I' in str or bytes, but there is ALWAYS an 'I' in unicode. llvm-svn: 306626
* [lit] Fix some convoluted logic around Unicode encoding, and de-duplicate ↵David L. Jones2017-06-292-51/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | across modules that used it. Summary: In Python2 and Python3, the various (non-)?Unicode string types are sort of spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere" approach, with 'str' representing a Unicode string. Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes. However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting problems when an interface requires a precise type, but has to run under both Python2 and Python3. The previous logic appeared to be correct in all cases, but went through more layers of indirection than necessary. This change does the necessary conversions in one shot, with documentation about which paths might be taken in Python2 or Python3. Reviewers: zturner, modocache Subscribers: llvm-commits, sanjoy Differential Revision: https://reviews.llvm.org/D34793 llvm-svn: 306625
* [lit] Remove dead code not referenced in the LLVM SVN repo.David L. Jones2017-06-293-120/+90
| | | | | | | | | | | | | | | | | | | | | | | Summary: This change removes the intermediate 'FileBasedTest' format from lit. This format is only ever used by the ShTest format, so the logic can be moved into ShTest directly. In order to better clarify what the TestFormat subclasses do, I fleshed out the TestFormat base class with Python's notion of abstract methods, using @abc.abstractmethod. This gives a convenient way to document the expected interface, without the risk of instantiating an abstract class (that's what ABCMeta does -- it raises an exception if you try to instantiate a class which has abstract methods, but not if you instantiate a subclass that implements them). Reviewers: zturner, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34792 llvm-svn: 306623
* [lit] Remove dead code (not referenced anywhere), and clarify some function ↵David L. Jones2017-06-281-177/+41
| | | | | | | | | | | | | | | | | | | | | | | names. Summary: The dead code seems to be unreferenced, according to textual search across the LLVM SVN repo. The clarification part of this change alters the name of a module-level function so that it is different from the name of the class-methods that call it. Currently, there are no erroneous references, but stylistically (c.f. PEP-8), internal "helper" functions should generally be named accordingly by prepending an underscore. (I also chose to add '_impl', which isn't necessary, but helps me at least to mentally disambiguate the interface and implementation functions.) Reviewers: zturner, modocache Subscribers: sanjoy, llvm-commits Differential Revision: https://reviews.llvm.org/D34775 llvm-svn: 306600
* [lit][macOS] Add a utility function to find the platform SDK versionAlex Lorenz2017-06-021-0/+14
| | | | | | | | | on macOS This function will be used to tie Clang's Integeration tests to a particular SDK version. See https://reviews.llvm.org/D32178 for more context. llvm-svn: 304541
* Return a lit.Test.Result object from TestRunner's executeShTest()Dimitry Andric2017-05-251-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: For various clang analyzer tests, which were unsupported, I got lit exceptions, similar to the following: Exception during script execution: Traceback (most recent call last): File "utils/lit/lit/run.py", line 190, in execute_test result = test.config.test_format.execute(test, lit_config) File "tools/clang/test/Analysis/analyzer_test.py", line 11, in execute if result.code == lit.Test.FAIL: AttributeError: 'tuple' object has no attribute 'code' This is because executeShTest() in utils/lit/lit/TestRunner.py is supposed to return a lit.Test.Result object, but in case of unsupported tests, it returns a plain tuple. Fix this by returning a properly initialized lit.Test.Result object instead. Reviewers: rnk, rafael, modocache Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D33579 llvm-svn: 303943
* [lit] Take the last error when executing pipelines.Zachary Turner2017-05-191-6/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This seems to have been present since the beginning of time, which is quite surprising. The symptom was this: Suppose you have a test with a run line that looks like this: RUN: foo | FileCheck %s foo prints some output and then due to a bug in the program it asserts. On Windows this results in the program returning a negative exit code. But if enough output had been printed already by the tool so that the FileCheck match would succeed then FileCheck would return 0, and because of bad logic in lit this 0 return value would overwrite the failed return value from previous items in the pipeline. This only happened with negative exit codes. The most sensible behavior is to just take whatever the first exit code is. There is no logical ordering defined on exit codes, so comparing with < and > does not make a lot of sense. Instead, as soon as we find the first non-successful return value, that should be the result of the entire expression. This fixes the issue, as now tests which fail on non-Windows platforms also fail for me on Windows as well. llvm-svn: 303440
* Add back a dummy --use-processes.Rafael Espindola2017-05-171-0/+3
| | | | | | Some bots are using it. llvm-svn: 303282
* Always use the multiprocess module.Rafael Espindola2017-05-172-115/+3
| | | | | | This seems to work on freebsd and openbsd these days. llvm-svn: 303280
* [lit] Try to exit more cleanlyReid Kleckner2017-05-022-11/+25
| | | | | | | | | | | | | | | If all jobs complete successfully, use pool.close() instead of pool.terminate() before waiting for the workers. Zach Turner reported that he was getting "access denied" exceptions from pool.terminate(). Make the workers abort immediately without printing to stderr when they are interrupted. Finally, catch exceptions when attempting to remove our temporary testing directory. On abnormal exit, there can often be open handles that haven't been cleaned up yet. llvm-svn: 301941
* [lit] Try using process pools by default againReid Kleckner2017-04-071-1/+1
| | | | | | | | | | | Both pickling errors encountered on clang bots and Darwin compiler-rt should now be fixed. This has no impact on testing time on Linux, and on Windows goes from 88s to 63s for 'check'. The tests pass on Mac, but I haven't compared execution time. llvm-svn: 299775
* [lit] Implement timeouts and max_time for process pool testingReid Kleckner2017-04-063-50/+83
| | | | | | | | | | | | | | | | | | | This is necessary to pass the lit test suite at llvm/utils/lit/tests. There are some pre-existing failures here, but now switching to pools doesn't regress any tests. I had to change test-data/lit.cfg to import DummyConfig from a module to fix pickling problems, but I think it'll be OK if we require test formats to be written in real .py modules outside lit.cfg files. I also discovered that in some circumstances AsyncResult.wait() will not raise KeyboardInterrupt in a timely manner, but you can pass a non-zero timeout to work around this. This makes threading.Condition.wait use a polling loop that runs through the interpreter, so it's capable of asynchronously raising KeyboardInterrupt. llvm-svn: 299605
* [lit] Revert to old execution strategy while I debug these pickling errorsReid Kleckner2017-04-051-1/+1
| | | | llvm-svn: 299565
* [lit] Use Python 3 style print to satisfy some botsReid Kleckner2017-04-051-1/+1
| | | | llvm-svn: 299564
* [lit] Use process pools for test execution by defaultReid Kleckner2017-04-052-42/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This drastically reduces lit test execution startup time on Windows. Our previous strategy was to manually create one Process per job and manage the worker pool ourselves. Instead, let's use the worker pool provided by multiprocessing. multiprocessing.Pool(jobs) returns almost immediately, and initializes the appropriate number of workers, so they can all start executing tests immediately. This avoids the ramp-up period that the old implementation suffers from. This appears to speed up small test runs. Here are some timings of the llvm-readobj tests on Windows using the various execution strategies: # multiprocessing.Pool: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-process-pool |& grep real: ; done real: 0m1.156s real: 0m1.078s real: 0m1.094s # multiprocessing.Process: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-processes |& grep real: ; done real: 0m6.062s real: 0m5.860s real: 0m5.984s # threading.Thread: $ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-threads |& grep real: ; done real: 0m9.438s real: 0m10.765s real: 0m11.079s I kept the old code to launch processes in case this change doesn't work on all platforms that LLVM supports, but at some point I would like to remove both the threading and old multiprocessing execution strategies. Reviewers: modocache, rafael Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31677 llvm-svn: 299560
* [lit] Add a minimum export implementation.Rafael Espindola2017-04-041-9/+21
| | | | llvm-svn: 299475
* Rename variable.Rafael Espindola2017-03-311-1/+1
| | | | | | Requested on post commit code review. llvm-svn: 299232
* Add a %basename substitution.Rafael Espindola2017-03-311-1/+4
| | | | | | This will be used to avoid various call to basename in the asan tests. llvm-svn: 299216
OpenPOWER on IntegriCloud