The primary motivation for this is to add another dimension to the
Swift LLDB test matrix, but this seems generally useful.
Differential Revision: https://reviews.llvm.org/D72662
There already are decorators and "--excluded" option to mark test-cases/files
as expected to fail. However, when a new test file is added and it which relates
to a feature that a target doesn't support, this requires either adding decorators
to that file or modifying the file provided as "--excluded" option value.
The purpose of this patch is to avoid any modifications in such cases.
E.g. if a target doesn't support "watchpoints" and passes "--xfail-category watchpoint"
to dotest, a testing job will not fail after a new watchpoint-related test file is added.
Differential Revision: https://reviews.llvm.org/D71906
Summary:
A lot of tests do this trick but the vast majority of them don't even call `print()`.
Most of this patch was generated by a script that just looks at all the files and deletes the line if there is no `print (` or `print(` anywhere else in the file.
I checked the remaining tests manually and deleted the import if we never call print (but instead do stuff like `expr print(...)` and similar false-positives).
I also corrected the additional empty lines after the import in the files that I manually edited.
Reviewers: JDevlieghere, labath, jfb
Reviewed By: labath
Subscribers: dexonsmith, wuzish, nemanjai, kbarton, christof, arphaman, abidh, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D71452
Before the test reorganization, everything was part of a single test
suite with a single module cache. Now that things are properly separated
this is no longer the case. Only the shell tests inherited the logic to
properly configure and wipe the module caches. This patch adds that
logic back for the API tests. While doing so, I noticed that we were
configuring a Clang module cache in CMake, but weren't actually using it
from dotest.py. I included a fix for that in this patch as well.
Differential revision: https://reviews.llvm.org/D68755
llvm-svn: 374386
Summary:
dotest.py currently requires a filterspec to be of the
form `TestCase.test_method`. This patch makes it more
flexible, so you can pass `TestModule.TestCase.test_method`
or `TestModule.TestCase` or `TestCase.test_method` or just
`test_method`.
This makes it more convenient to just copy a test name
out of the terminal after running a bunch of tests and use
it as a filterspec.
Reviewers: JDevlieghere, jasonmolenda, labath
Reviewed By: JDevlieghere
Subscribers: jingham, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D68545
llvm-svn: 373997
Currently, lit tests don't set neither the module cache for building
inferiors nor the module cache used by lldb when running tests.
Furthermore, we have several places where we rely on the path to the
module cache being always the same, rather than passing the correct
value around. This makes it hard to specify a different module cache
path when debugging a a test.
This patch reworks how we determine and pass around the module cache
paths and fixes the omission on the lit side. It also adds a sanity
check to the lit and dotest suites.
Differential revision: https://reviews.llvm.org/D66966
llvm-svn: 370394
This removes the curses result formatter which appears to be broken.
Passing --curses to dotest.py screws up my terminal and doesn't run any
tests. It even crashes Python on occasion.
Differential revision: https://reviews.llvm.org/D66917
llvm-svn: 370386
This removes support for reading the LLDB_TEST_ARGUMENTS environment
variable and instead requires all arguments to be specified as part of
the invocation. This ensures that dotest.py invocations are easily
repeatable.
Differential revision: https://reviews.llvm.org/D66912
llvm-svn: 370278
This argument was used by dosep.py to pass information around from the
workers. With dosep.py gone, I'm fairly sure we don't need this any
longer.
llvm-svn: 370266
This patch removes the -q (quiet) flag and changing the default
behavior. Currently the flag serves two purposes that are somewhat
contradictory, as illustrated by the difference between the argument
name (quiet) and the configuration flag (parsable). On the one hand it
reduces output, but on the other hand it prints more output, like the
result of individual tests. My proposal is to guard the extra output
behind the verbose flag and always print the individual test results.
Differential revision: https://reviews.llvm.org/D66837
llvm-svn: 370226
The results port was used by dosep.py to deal with test results coming
form different processes. With dosep.py gone, I don't think we need this
any longer.
Differential revision: https://reviews.llvm.org/D66811
llvm-svn: 370090
Today I discovered the skipLongRunningTest decorator and to my surprise
all the tests were passing without the decorator. They don't seem to be
that expensive either, they take a few seconds but we have tests that
take much longer than that. As such I propose to remove the decorator
and enable them by default.
Differential revision: https://reviews.llvm.org/D66774
llvm-svn: 369995
On the matrix bot on GreenDragon [1] we want to run the test suite
against different DWARF versions. The idea here is not to replace
targeted tests for certain DWARF features, but rather to provide an easy
way to support this configuration.
[1] http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake-matrix/
Differential revision: https://reviews.llvm.org/D66370
llvm-svn: 369272
Now that the Xcode project is removed, I want to focus on dotest as a
test framework, and remove its driver capabilities for which we already
rely on llvm's lit. Removing multiprocessing is the first step in that
direction.
Differential revision: https://reviews.llvm.org/D65311
llvm-svn: 367331
This patch adds a dotest flag for setting environment variables for the
inferior. This is different from the current --env flag, which sets
variables in the debugger's environment. This allows us to set things
like LD_LIBRARY_PATH for testing.
Differential revision: https://reviews.llvm.org/D63790
llvm-svn: 364443
Add a "filecheck" method to the LLDB test base. This allows test authors
to pattern match command output using FileCheck, making it possible to
write stricter tests than what `self.expect` allows.
For context (motivation, examples of stricter checking, etc), see the
lldb-dev thread: "Using FileCheck in lldb inline tests".
Differential Revision: https://reviews.llvm.org/D50751
llvm-svn: 342508
Summary:
With the upstream implementation of dsymutil containing almost all
functionality from the one shipped with Xcode, we want to use the
in-tree version for running the test suite.
This will also allow us to re-enable TestUnicodeSymbols which was
failing because of the discrepancy in how Unicode symbols were hashed in
lldb and older versions of dsymutil.
Reviewers: aprantl, davide, jingham, labath
Subscribers: mgorny, llvm-commits, lldb-commits
Differential Revision: https://reviews.llvm.org/D45518
llvm-svn: 329889
The test "test_fp_special_purpose_register_read" in TestRegisters.py
fails on Darwin machines configured to use an out-of-tree debugserver.
The error message is: 'register read ftag' returns expected result, got
'ftag = 0x80'. This indicates that the debugserver in use is too old.
This commit introduces a decorator which can be used to skip tests which
rely on having a just-built debugserver. This resolves the issue:
$ ./bin/llvm-dotest -p TestRegisters.py -v
1 out of 617 test suites processed - TestRegisters.py
Test Methods: 7
Success: 6
Skip: 1
...
llvm-svn: 327052
This patch is the result of a discussion on lldb-dev, see
http://lists.llvm.org/pipermail/lldb-dev/2018-January/013111.html for
background.
For each test (should be eventually: each test configuration) a
separate build directory is created and we execute
make VPATH=$srcdir/path/to/test -C $builddir/path/to/test -f $srcdir/path/to/test/Makefile -I $srcdir/path/to/test
In order to make this work all LLDB tests need to be updated to find
the executable in the test build directory, since CWD still points at
the test's source directory, which is a requirement for unittest2.
Although we have done extensive testing, I'm expecting that this first
attempt will break a few bots. Please DO NOT HESITATE TO REVERT this
patch in order to get the bots green again. We will likely have to
iterate on this some more.
Differential Revision: https://reviews.llvm.org/D42281
llvm-svn: 323803
Summary:
This has been broken at least since the new test result framework was
added, which was over a year ago. It looks like nobody has missed it
since.
Removing this makes the gmodules handling code saner, as it already did
not know how to handle the multiple-compilers case.
My motivation for this is libc++ data formatters support on android -- I
am trying make a central way of determining whether libc++ tests can be
run, and without this, I would have to resort to similar hacks as the
gmodules code.
Reviewers: jingham, zturner
Subscribers: danalbert, tfiala, lldb-commits
Differential Revision: https://reviews.llvm.org/D30779
llvm-svn: 297811
This patch adds support to the test suite for overriding the path to debugserver, and uses the override to point to the build tree's debugserver on Darwin.
llvm-svn: 297776
Summary: Not everyone names their code sign identity "lldb_codesign", so it is nice to allow this to be overridden.
Reviewers: zturner, tfiala
Subscribers: labath, mgorny, lldb-commits
Differential Revision: https://reviews.llvm.org/D25714
llvm-svn: 284893
Summary: Default installations of OS X do not have system headers installed at /usr/include. This patch allows the LLDB test executables to properly compile when built on a system without headers at /usr/include by specifying a default value for the apple-sdk flag as "macosx".
Reviewers: tfiala, zturner
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D25487
llvm-svn: 284042
Summary:
This patch is necessary because individual test cases are not required
to have unique names. Therefore, test cases must now
be specified explicitly in the form <TestCase>.<TestMethod>.
Because it works by regex matching, passing just <TestCase> will
still disable an entire file.
This also allows for multiple exclusion files to be specified.
Reviewers: zturner, labath, jingham, tfiala
Subscribers: lldb-commits, sas
Differential Revision: https://reviews.llvm.org/D24988
llvm-svn: 283238
Summary:
The current implementation of the test suite allows the user to run
a certain subset of tests using '-p', but does not allow the inverse,
where a user wants to run all but some number of known failing tests.
Implement this functionality.
Reviewers: labath, zturner, tfiala
Subscribers: jingham, sas, lldb-commits
Differential Revision: https://reviews.llvm.org/D24629
llvm-svn: 282298
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:
Firstly, merging this particular commit into a downstream fork may be a huge
effort. Alternatively, it may be worth merging all changes up to this commit,
performing the same reformatting operation locally, and then discarding the
merge for this particular commit. The commands used to accomplish this
reformatting were as follows (with current working directory as the root of
the repository):
find . \( -iname "*.c" -or -iname "*.cpp" -or -iname "*.h" -or -iname "*.mm" \) -exec clang-format -i {} +
find . -iname "*.py" -exec autopep8 --in-place --aggressive --aggressive {} + ;
The version of clang-format used was 3.9.0, and autopep8 was 1.2.4.
Secondly, “blame” style tools will generally point to this commit instead of
a meaningful prior commit. There are alternatives available that will attempt
to look through this change and find the appropriate prior commit. YMMV.
llvm-svn: 280751
Use of --rerun-all-issues will enable any test method failure, not just
test methods marked with the flakey decorator, to rerun.
Currently this does not change the flakey logic's immediate rerun
attempt. I want to make sure this doesn't cause any significant issues
before changing that part.
The rerun reporting is only known to work properly with the
default (new) BasicResultsFormatter reporting. Once we work out
any issues, I'll go back and make sure the curses output handles
it properly as well.
llvm-svn: 255543
Also adds full path info for exceptional exits and timeouts when
no test method is currently running.
Adds --rerun-all-issues command line arg. If specified, all
test issues are eligible for rerun. If not specified, only tests
marked flakey are eligible for rerun.
The actual rerunning will occur in an upcoming change. This
change just handles tha accounting of what should be rerun.
llvm-svn: 255438
This was an option to display a graphical progress bar. Nobody
is using this, and it doesn't work correctly anyway with the new
result formatter.
llvm-svn: 255153
This removes the failfast command line option as part of an effort
to simplify dotest and remove unused command line options. You can
still Ctrl+C any time you want to exit early.
llvm-svn: 255150
This seems to be a legacy relic from days gone by where the
remote test suite runner operated completely differently than it
does today. git blames and comments traced this functionality
back to about 2012, and nobody seems to know anything about it
now.
llvm-svn: 255060
This removes the option to exclude a single directory. This is
part of an effort to remove unused options and cleanup the interface
to the test suite.
llvm-svn: 255048
