Also adds full path info for exceptional exits and timeouts when
no test method is currently running.
Adds --rerun-all-issues command line arg. If specified, all
test issues are eligible for rerun. If not specified, only tests
marked flakey are eligible for rerun.
The actual rerunning will occur in an upcoming change. This
change just handles tha accounting of what should be rerun.
llvm-svn: 255438
The new test summary formatter does not honor the "expected timeout" markings, which makes our
buildbots all red. I'm switching it off by default until we figure out a way to make this work.
llvm-svn: 255335
This was an option to display a graphical progress bar. Nobody
is using this, and it doesn't work correctly anyway with the new
result formatter.
llvm-svn: 255153
This removes the failfast command line option as part of an effort
to simplify dotest and remove unused command line options. You can
still Ctrl+C any time you want to exit early.
llvm-svn: 255150
The results formatter system is now fed timeouts and exceptional process
exits (i.e. inferior dotest.py process that exited by signal on POSIX
systems).
If a timeout or exceptional exit happens while a test method is running
on the worker queue, the timeout or exceptional exit is charged and
reported against that test method. Otherwise, if no test method was
running at the time of the timeout or exceptional exit, only the test
filename will be reported as the TIMEOUT or ERROR.
Implements:
https://llvm.org/bugs/show_bug.cgi?id=24830https://llvm.org/bugs/show_bug.cgi?id=25703
In support of:
https://llvm.org/bugs/show_bug.cgi?id=25450
llvm-svn: 255097
This seems to be a legacy relic from days gone by where the
remote test suite runner operated completely differently than it
does today. git blames and comments traced this functionality
back to about 2012, and nobody seems to know anything about it
now.
llvm-svn: 255060
This removes the option to exclude a single directory. This is
part of an effort to remove unused options and cleanup the interface
to the test suite.
llvm-svn: 255048
This moves all the global variables into a separate module called
`configuration`. This has a number of advantages:
1. Configuration data is centrally maintained so it's easy to get
a high level overview of what configuration data the test suite
makes use of.
2. The method of sharing configuration data among different parts
of the test suite becomes standardized. Previously we would
put some things into the `lldb` module, some things into the
`lldbtest_config` module, and some things would not get shared.
Now everything is shared through one module and is available to
the entire test suite.
3. It opens the door to moving some of the initialization code into
the `configuration` module, simplifying the implementation of
`dotest.py`.
There are a few stragglers that didn't get converted over to using
the `configuration` module in this patch, because it would have grown
the size of the patch unnecessarily. This includes everything
currently in the `lldbtest_config` module, as well as the
`lldb.remote_platform` variable. We can address these in the future.
llvm-svn: 254982
This cleans up dotest.py and is a pre-step for getting
the test inferior runner to send post-inferior run events
to the events collector, as this code needs to be accessed
from within dosep.py.
llvm-svn: 254979
There is already a class called LLDBTestResults which I would like
to move into a separate file, but the most appropriate filename
was taken.
llvm-svn: 254946
`sets.Set` has been deprecated in favor of `set` since 2.6, and
`string.maketrans` has to be special cased. In Python 3 there
is `str.maketrans`, `bytes.maketrans`, and `bytearray.maketrans`
and you have to choose the correct one. So we need to introduce
a runtime version check at this site.
llvm-svn: 252348
Absolute imports were introduced in Python 2.5 as a feature
(e.g. from __future__ import absolute_import), and made default
in Python 3.
When absolute imports are enabled, the import system changes in
a couple of ways:
1) The `import foo` syntax will *only* search sys.path. If `foo`
isn't in sys.path, it won't be found. Period. Without absolute
imports, the import system will also search the same directory
that the importing file resides in, so that you can easily
import from the same folder.
2) From inside a package, you can use a dot syntax to refer to higher
levels of the current package. For example, if you are in the
package lldbsuite.test.utility, then ..foo refers to
lldbsuite.test.foo. You can use this notation with the
`from X import Y` syntax to write intra-package references. For
example, using the previous locationa s a starting point, writing
`from ..support import seven` would import lldbsuite.support.seven
Since this is now the default behavior in Python 3, this means that
importing from the same directory with `import foo` *no longer works*.
As a result, the only way to have portable code is to force absolute
imports for all versions of Python.
See PEP 0328 [https://www.python.org/dev/peps/pep-0328/] for more
information about absolute and relative imports.
Differential Revision: http://reviews.llvm.org/D14342
Reviewed By: Todd Fiala
llvm-svn: 252191
It was deprecated even in 2.7, but not removed until 3.x. os.walk
provides all of the same functionality and is the correct way to
do this now.
llvm-svn: 252127
This allows for command-line debugging of iOS simulator binaries (as long as UI is not required, or a full UI simulator has previously been otherwise launched), as well as execution of the LLDB test suite on the iOS simulator
This is known to compile on OSX 10.11 GM - feedback from people on other platforms and/or older versions of OSX as to the buildability of this code is greatly appreciated
llvm-svn: 252112
By default in Python 3, check_output() returns a program's output as
an encoded byte sequence. This means it returns a Py3 `bytes` object,
which cannot be compared to a string since it's a different fundamental
type.
Although it might not be correct from a purist standpoint, from a
practical one we can assume that all output is encoded in the default
locale, in which case using universal_newlines=True will decode it
according to the current locale. Anyway, universal_newlines also
has the nice behavior that it converts \r\n to \n on Windows platforms
so this makes parsing code easier, should we need that. So it seems
like a win/win.
llvm-svn: 252025
This module was originally intended to be imported by top-level
scripts to be able to find the LLDB packages and third party
libraries. Packages themselves shouldn't need to import it,
because by the time it gets into the package, the top-level
script should have already done this. Indeed, it was just
adding the same values to sys.path multiple times, so this
patch is essentially no functional change.
To make sure it doesn't get re-introduced, we also delete the
`use_lldb_suite` module from `lldbsuite/test`, although the
original copy still remains in `lldb/test`
llvm-svn: 251963
The `commands` module was deprecated in 2.7 and removed in 3.x.
As a workaround, we introduce a new module `seven` in
lldbsuite.support, and write helper functions in there that delegate
to the commands module if it is available, and re-implement their
functionality for cases where it is not available.
llvm-svn: 251959
For convenience, we had added the folder that dotest.py was in
to sys.path, so that we could easily write things like
`import lldbutil` from anywhere and any test. This introduces
a subtle problem when using Python's package system, because when
unittest2 imports a particular test suite, the test suite is detached
from the package. Thus, writing "import lldbutil" from dotest imports
it as part of the package, and writing the same line from a test
does a fresh import since the importing module was not part of
the same package.
The real way to fix this is to use absolute imports everywhere. Instead
of writing "import lldbutil", we need to write "import
lldbsuite.test.util". This patch fixes up that and all other similar
cases, and additionally removes the script directory from sys.path
to ensure that this can't happen again.
llvm-svn: 251886
I think the underlying problem was fixed by r251819, but I can't
reproduce the problem. So this is to check whether it does in
fact fix the problem.
llvm-svn: 251822
