summaryrefslogtreecommitdiffstats
path: root/lldb/packages/Python/lldbsuite/test/dotest.py
Commit message (Collapse)AuthorAgeFilesLines
* test infra: catch bad decorators and import-time errorsTodd Fiala2016-05-131-54/+79
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: This change enhances the LLDB test infrastructure to convert load-time exceptions in a given Python test module into errors. Before this change, specifying a non-existent test decorator, or otherwise having some load-time error in a python test module, would not get flagged as an error. With this change, typos and other load-time errors in a python test file get converted to errors and reported by the test runner. This change also includes test infrastructure tests that include covering the new work here. I'm going to wait until we have these infrastructure tests runnable on the main platforms before I try to work that into all the normal testing workflows. The test infrastructure tests can be run by using the standard python module testing practice of doing the following: cd packages/Python/lldbsuite/test_event python -m unittest discover -s test/src -p 'Test*.py' Those tests run the dotest inferior with a known broken test and verify that the errors are caught. These tests did not pass until I modified dotest.py to capture them properly. @zturner, if you have the chance, if you could try those steps above (the python -m unittest ... line) on Windows, that would be great if we can address any python2/3/Windows bits there. I don't think there's anything fancy, but I didn't want to hook it into test flow until I know it works there. I'll be slowly adding more tests that cover some of the other breakage I've occasionally seen that didn't get collected as part of the summarization. This is the biggest one I'm aware of. Reviewers: zturner, labath Subscribers: zturner, lldb-commits Differential Revision: http://reviews.llvm.org/D20193 llvm-svn: 269489
* fix argument usage for '-#' command line optionTodd Fiala2016-05-051-1/+1
| | | | | | | This was broken in the grand configuration change. Now using -# works again. llvm-svn: 268638
* Store absolute path for lldb executable in dotest.pyFrancis Ricci2016-04-251-1/+1
| | | | | | | | | | | | | | | | | | | | Summary: lldb-server tests are currently being skipped on the check-lldb target. This is because we get the path of lldb-server by modifying the path to the lldb executable. However, by this point, we've changed directories, and a relative path to the build/bin directory will no longer point to the location of lldb-server. Storing an absolute path solves this issue. Reviewers: vharron, zturner, tfiala, labath Subscribers: labath, lldb-commits, sas Differential Revision: http://reviews.llvm.org/D19082 llvm-svn: 267463
* test infra: move test event-related handling into its own packageTodd Fiala2016-04-201-10/+8
| | | | | | | | | | | | | | | | | | | | | | | This change moves all the test event handling and its related ResultsFormatter classes out of the packages/Python/lldbsuite/test dir into a packages/Python/lldbsuite/test_event package. Formatters are moved into a sub-package under that. I am limiting the scope of this change to just the motion and a few minor issues caught by a static Python checker (e.g. removing unused import statements). This is a pre-step for adding package-level tests to the test event system. I also intend to simplify test event results formatter selection after I make sure this doesn't break anybody. See: http://reviews.llvm.org/D19288 Reviewed by: Pavel Labath llvm-svn: 266885
* Unicode support on Win32.Zachary Turner2016-03-221-1/+1
| | | | | | | | | | | | | Win32 API calls that are Unicode aware require wide character strings, but LLDB uses UTF8 everywhere. This patch does conversions wherever necessary when passing strings into and out of Win32 API calls. Patch by Cameron Differential Revision: http://reviews.llvm.org/D17107 Reviewed By: zturner, amccarth llvm-svn: 264074
* Eliminate the TestStarted-XXX and TestFinished-XXX files from check-lldb runs.Adrian McCarthy2016-03-101-11/+0
| | | | | | | | Nobody seems to know what purpose these files serve, yet they were accumulating by the thousands in the test traces directory. I'm proposing we delete them. Creating these files accounted for about 2.5% of the time to run ninja check-lldb on my machine, which isn't a lot, but it's something. llvm-svn: 263122
* Add a symbolic link from the test directory to the actual - elsewhere ↵Enrico Granata2015-12-161-1/+1
| | | | | | | | | | located - path that contains the test cases - and teach the test suite driver to resolve paths that contain symbolic links to test cases This is meant to reduce the typing that one needs to do to get from the test subdirectory to actual test cases. Now one can just do $ ./dotest.py ./testcases/<yaddayaddayadda> llvm-svn: 255741
* test-infra: refactored new summary results into base ResultsFormatter classTodd Fiala2015-12-151-1/+1
| | | | | | | This allows more specialized formatters to still reuse the results summarization display from the base class. llvm-svn: 255676
* test infra: enable single-worker rerun phase for flakey tests.Todd Fiala2015-12-141-0/+1
| | | | | | | | | | | | | | | | Use of --rerun-all-issues will enable any test method failure, not just test methods marked with the flakey decorator, to rerun. Currently this does not change the flakey logic's immediate rerun attempt. I want to make sure this doesn't cause any significant issues before changing that part. The rerun reporting is only known to work properly with the default (new) BasicResultsFormatter reporting. Once we work out any issues, I'll go back and make sure the curses output handles it properly as well. llvm-svn: 255543
* Make debug info specification use categories system.Zachary Turner2015-12-141-17/+0
| | | | | | | Reviewed By: Tamas Berghammer, Pavel Labath Differential Revision: http://reviews.llvm.org/D15428 llvm-svn: 255525
* test infra: adds book-keeping for rerunnable testsTodd Fiala2015-12-121-0/+3
| | | | | | | | | | | | | | Also adds full path info for exceptional exits and timeouts when no test method is currently running. Adds --rerun-all-issues command line arg. If specified, all test issues are eligible for rerun. If not specified, only tests marked flakey are eligible for rerun. The actual rerunning will occur in an upcoming change. This change just handles tha accounting of what should be rerun. llvm-svn: 255438
* Revert "Revert "Turn on new test summary results by default.""Todd Fiala2015-12-111-0/+6
| | | | | | This reverts commit f994b46a2028c8a8b9b55fe010a95122bca07540. llvm-svn: 255381
* Remove -S option from dotest.py.Zachary Turner2015-12-111-3/+0
| | | | llvm-svn: 255361
* Remove -r and -R options from dotest.py.Zachary Turner2015-12-111-100/+1
| | | | llvm-svn: 255360
* Revert "Turn on new test summary results by default."Pavel Labath2015-12-111-6/+0
| | | | | | | The new test summary formatter does not honor the "expected timeout" markings, which makes our buildbots all red. I'm switching it off by default until we figure out a way to make this work. llvm-svn: 255335
* Remove the -y option from dotest.py.Zachary Turner2015-12-101-4/+0
| | | | llvm-svn: 255280
* Remove the -x option from dotest.py.Zachary Turner2015-12-101-5/+0
| | | | llvm-svn: 255279
* Remove deprecated command line options from dotest.pyZachary Turner2015-12-101-10/+0
| | | | llvm-svn: 255278
* Remove the --output-on-success command line argument from dotest.Zachary Turner2015-12-101-7/+2
| | | | llvm-svn: 255277
* Remove the -T option from dotest.py.Zachary Turner2015-12-101-13/+0
| | | | llvm-svn: 255276
* Remove -w option from dotest.py.Zachary Turner2015-12-101-3/+0
| | | | llvm-svn: 255275
* Remove the -P option from dotest.pyZachary Turner2015-12-091-7/+0
| | | | | | | | This was an option to display a graphical progress bar. Nobody is using this, and it doesn't work correctly anyway with the new result formatter. llvm-svn: 255153
* Delete the -n command line option from dotest.py.Zachary Turner2015-12-091-14/+7
| | | | | | | This removes the option to not print some one time version and diagnostic information when running the test suite. llvm-svn: 255152
* Remove the -i command line option from dotest.py.Zachary Turner2015-12-091-7/+0
| | | | | | | This is part of a larger effort to remove unused command line options from dotest.py. llvm-svn: 255151
* Delete the -F command line option from dotest.py.Zachary Turner2015-12-091-5/+0
| | | | | | | | This removes the failfast command line option as part of an effort to simplify dotest and remove unused command line options. You can still Ctrl+C any time you want to exit early. llvm-svn: 255150
* Remove -k command line option from dotest.py.Zachary Turner2015-12-091-3/+0
| | | | | | This is part of an effort to remove unused command line options. llvm-svn: 255143
* Remove -e option from dotest.py.Zachary Turner2015-12-091-7/+0
| | | | | | | This is part of an effort to clean up dotest command line options that are no longer used. llvm-svn: 255142
* Remove -libcxx option from dotest.pyZachary Turner2015-12-091-3/+0
| | | | | | | Nobody was using this, and plus it can be achieved just as well by using -E to set an environment variable. llvm-svn: 255141
* Turn on new test summary results by default.Todd Fiala2015-12-091-0/+6
| | | | llvm-svn: 255130
* wire timeouts and exceptional inferior process exits through the test event ↵Todd Fiala2015-12-091-2/+11
| | | | | | | | | | | | | | | | | | | | | | | system The results formatter system is now fed timeouts and exceptional process exits (i.e. inferior dotest.py process that exited by signal on POSIX systems). If a timeout or exceptional exit happens while a test method is running on the worker queue, the timeout or exceptional exit is charged and reported against that test method. Otherwise, if no test method was running at the time of the timeout or exceptional exit, only the test filename will be reported as the TIMEOUT or ERROR. Implements: https://llvm.org/bugs/show_bug.cgi?id=24830 https://llvm.org/bugs/show_bug.cgi?id=25703 In support of: https://llvm.org/bugs/show_bug.cgi?id=25450 llvm-svn: 255097
* Remove the -c option from dotest.py.Zachary Turner2015-12-081-78/+0
| | | | | | | | | | This seems to be a legacy relic from days gone by where the remote test suite runner operated completely differently than it does today. git blames and comments traced this functionality back to about 2012, and nobody seems to know anything about it now. llvm-svn: 255060
* Remove the -X option from dotest.pyZachary Turner2015-12-081-6/+3
| | | | | | | | This removes the option to exclude a single directory. This is part of an effort to remove unused options and cleanup the interface to the test suite. llvm-svn: 255048
* Remove the -g option from dotest.pyZachary Turner2015-12-081-5/+1
| | | | | | | | This removes the non-exclusive filterspec option as part of an effort to remove unused / deprecated command line options from dotest. llvm-svn: 255041
* Remove the -b option from dotest.pyZachary Turner2015-12-081-11/+0
| | | | | | | This removes the blacklist option as part of an effort to remove unused / unmaintained command line options from the test suite. llvm-svn: 255040
* Remove +b option from dotest.pyZachary Turner2015-12-081-3/+0
| | | | llvm-svn: 255037
* Remove the -D option from dotest.py.Zachary Turner2015-12-081-5/+0
| | | | | | | This removes the option to dump Python sys.path variable as part of an effort to remove unused options. llvm-svn: 255035
* Fixup dotest.py on mac for the configuration packagePavel Labath2015-12-081-0/+1
| | | | llvm-svn: 255013
* Fixup dotest.py after the configuration package introductionPavel Labath2015-12-081-1/+1
| | | | llvm-svn: 255009
* Move LLDBTestResult class to its own module.Zachary Turner2015-12-081-231/+5
| | | | llvm-svn: 254983
* Get rid of global variables in dotest.pyZachary Turner2015-12-081-623/+232
| | | | | | | | | | | | | | | | | | | | | | | | | | This moves all the global variables into a separate module called `configuration`. This has a number of advantages: 1. Configuration data is centrally maintained so it's easy to get a high level overview of what configuration data the test suite makes use of. 2. The method of sharing configuration data among different parts of the test suite becomes standardized. Previously we would put some things into the `lldb` module, some things into the `lldbtest_config` module, and some things would not get shared. Now everything is shared through one module and is available to the entire test suite. 3. It opens the door to moving some of the initialization code into the `configuration` module, simplifying the implementation of `dotest.py`. There are a few stragglers that didn't get converted over to using the `configuration` module in this patch, because it would have grown the size of the patch unnecessarily. This includes everything currently in the `lldbtest_config` module, as well as the `lldb.remote_platform` variable. We can address these in the future. llvm-svn: 254982
* Refactor ResultsFormatter creation into result_formatter.Todd Fiala2015-12-081-73/+15
| | | | | | | | | This cleans up dotest.py and is a pre-step for getting the test inferior runner to send post-inferior run events to the events collector, as this code needs to be accessed from within dosep.py. llvm-svn: 254979
* Rename test_results.py to result_formatter.py.Zachary Turner2015-12-071-5/+5
| | | | | | | | There is already a class called LLDBTestResults which I would like to move into a separate file, but the most appropriate filename was taken. llvm-svn: 254946
* Make --results-file stdout implied if unspecified when using a results ↵Todd Fiala2015-12-021-3/+13
| | | | | | | | formatter. Also cleans up pylint warnings (stock settings) in the modified function. llvm-svn: 254546
* Add --curses shortcut for specifying the curses-based test results formatter.Todd Fiala2015-11-091-1/+5
| | | | | | | This commit closes the following review: http://reviews.llvm.org/D14488 llvm-svn: 252498
* Fix Linux tests after r252348.Chaoren Lin2015-11-061-0/+1
| | | | llvm-svn: 252353
* Python 3 - Port use of string.maketrans and don't use sets.Set.Zachary Turner2015-11-061-2/+4
| | | | | | | | | | `sets.Set` has been deprecated in favor of `set` since 2.6, and `string.maketrans` has to be special cased. In Python 3 there is `str.maketrans`, `bytes.maketrans`, and `bytearray.maketrans` and you have to choose the correct one. So we need to introduce a runtime version check at this site. llvm-svn: 252348
* Another import fix for OS X.Chaoren Lin2015-11-051-1/+1
| | | | llvm-svn: 252230
* Fix OS X tests.Chaoren Lin2015-11-051-1/+1
| | | | llvm-svn: 252218
* Python 3 - Turn on absolute imports, and fix existing imports.Zachary Turner2015-11-051-11/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Absolute imports were introduced in Python 2.5 as a feature (e.g. from __future__ import absolute_import), and made default in Python 3. When absolute imports are enabled, the import system changes in a couple of ways: 1) The `import foo` syntax will *only* search sys.path. If `foo` isn't in sys.path, it won't be found. Period. Without absolute imports, the import system will also search the same directory that the importing file resides in, so that you can easily import from the same folder. 2) From inside a package, you can use a dot syntax to refer to higher levels of the current package. For example, if you are in the package lldbsuite.test.utility, then ..foo refers to lldbsuite.test.foo. You can use this notation with the `from X import Y` syntax to write intra-package references. For example, using the previous locationa s a starting point, writing `from ..support import seven` would import lldbsuite.support.seven Since this is now the default behavior in Python 3, this means that importing from the same directory with `import foo` *no longer works*. As a result, the only way to have portable code is to force absolute imports for all versions of Python. See PEP 0328 [https://www.python.org/dev/peps/pep-0328/] for more information about absolute and relative imports. Differential Revision: http://reviews.llvm.org/D14342 Reviewed By: Todd Fiala llvm-svn: 252191
* Python 3 - Don't use `os.path.walk`, it's removed in Py3.Zachary Turner2015-11-051-4/+2
| | | | | | | | It was deprecated even in 2.7, but not removed until 3.x. os.walk provides all of the same functionality and is the correct way to do this now. llvm-svn: 252127
OpenPOWER on IntegriCloud