mirrors/llvm-project
0
0
Fork 0
mirror of https://github.com/llvm/llvm-project.git synced 2025-05-17 18:56:04 +00:00
Code Issues Projects Releases Wiki Activity
llvm-project/llvm/utils/lit/tests/shtest-format.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

171 lines
6.7 KiB
Python
Raw Normal View History

[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# Check the various features of the ShTest format.
#
[lit] Move the shtest-xunit-output check lines into shtest-format These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
2018-07-23 18:08:36 +00:00
# RUN: rm -f %t.xml
# RUN: not %{lit} -j 1 -v %{inputs}/shtest-format --xunit-xml-output %t.xml > %t.out
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# RUN: FileCheck < %t.out %s
[lit] Move the shtest-xunit-output check lines into shtest-format These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
2018-07-23 18:08:36 +00:00
# RUN: FileCheck --check-prefix=XUNIT < %t.xml %s
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# END.
# CHECK: -- Testing:
[lit] Fix internal shell's argv[0] handling. - At least on OS X, it is important for correct behavior of /bin/[ that argv[0] is passed as written, and not as the full executable path. llvm-svn: 189559
2013-08-29 02:52:10 +00:00
# CHECK: PASS: shtest-format :: argv0.txt
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: FAIL: shtest-format :: external_shell/fail.txt
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail.txt' FAILED ***
# CHECK: Command Output (stdout):
# CHECK-NEXT: --
# CHECK-NEXT: line 1: failed test output on stdout
# CHECK-NEXT: line 2: failed test output on stdout
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: Command Output (stderr):
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK-NEXT: --
[lit][tests][AIX] Update expected form of diagnostic messages; use `not` to normalize non-zero exit values Summary: Various tests in the `lit` testing suite expect specific return codes and forms of diagnostic message from utility programs. As per POSIX.1-2017 XCU Section 1.4, Utility Description Defaults, "[the] format of diagnostic messages for most utilities is unspecified". The STDERR subsections of the `cat` and `wc` utilities merely indicate that "[the] standard error shall be used only for diagnostic messages". The corresponding EXIT STATUS subsections merely indicate, with regard to errors, an exit value of >0. The affected tests are updated to accept the applicable diagnostic message as produced by the utilities on AIX. The exit value is normalized using `not` as necessary. Reviewers: xingxue, sfertile, jasonliu Reviewed By: xingxue Subscribers: delcypher, jsji, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D60553 llvm-svn: 359690
2019-05-01 15:47:16 +00:00
# CHECK-NEXT: cat{{(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK: --
# CHECK: FAIL: shtest-format :: external_shell/fail_with_bad_encoding.txt
# CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail_with_bad_encoding.txt' FAILED ***
# CHECK: Command Output (stdout):
# CHECK-NEXT: --
# CHECK-NEXT: a line with bad encoding:
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: --
# CHECK: PASS: shtest-format :: external_shell/pass.txt
# CHECK: FAIL: shtest-format :: fail.txt
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK-NEXT: *** TEST 'shtest-format :: fail.txt' FAILED ***
# CHECK-NEXT: Script:
# CHECK-NEXT: --
# CHECK-NEXT: printf "line 1
# CHECK-NEXT: false
# CHECK-NEXT: --
# CHECK-NEXT: Exit Code: 1
#
# CHECK: Command Output (stdout):
# CHECK-NEXT: --
[lit] Report line number for failed RUN command (Relands r333584, reverted in 333592.) When debugging test failures with -vv (or -v in the case of the internal shell), this makes it easier to locate the RUN line that failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines after concatenation for line continuations. When reading the generated shell script, this also makes it easier to locate the RUN line that produced each command. To support reporting RUN line numbers in the case of the internal shell, this patch extends the internal shell to support the null command, ":", except pipelines are not supported. To support reporting RUN line numbers in the case of windows cmd.exe as the external shell, this patch extends -vv to set "echo on" instead of "echo off" in bat files. (Support for windows cmd.exe as a lit external shell will likely be dropped later, but I found out too late.) Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk Differential Revision: https://reviews.llvm.org/D44598 llvm-svn: 333614
2018-05-31 00:55:32 +00:00
# CHECK-NEXT: $ ":" "RUN: at line 1"
[lit] Improve readability of failing scripts. - This only applies to scripts executed by the _internal_ shell script interpreter. - This patch reworks the log to look more like a shell transcript, and be less verbose (but in the interest of calling attention to the important parts). Here is an example of the new format, for commands with/without failures and with/without output: ``` $ true $ echo hi hi $ false note: command had no output on stdout or stderr error: command failed with exit status 1 ``` llvm-svn: 271610
2016-06-02 23:49:42 +00:00
# CHECK-NEXT: $ "printf"
# CHECK-NEXT: # command output:
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK-NEXT: line 1: failed test output on stdout
# CHECK-NEXT: line 2: failed test output on stdout
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
# CHECK: PASS: shtest-format :: pass.txt
Revert "[lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED" This change needs to be better-coordinated with libc++. llvm-svn: 292900
2017-01-24 08:58:20 +00:00
# CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt
# CHECK: PASS: shtest-format :: requires-any-present.txt
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
# CHECK: PASS: shtest-format :: requires-present.txt
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
2017-01-25 02:26:03 +00:00
# CHECK: UNRESOLVED: shtest-format :: requires-star.txt
# CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
# CHECK: PASS: shtest-format :: unsupported-expr-false.txt
# CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
# CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
2017-01-25 02:26:03 +00:00
# CHECK: PASS: shtest-format :: xfail-expr-false.txt
# CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: XFAIL: shtest-format :: xfail-feature.txt
# CHECK: XFAIL: shtest-format :: xfail-target.txt
# CHECK: XFAIL: shtest-format :: xfail.txt
# CHECK: XPASS: shtest-format :: xpass.txt
[lit] Lift XFAIL handling to core infrastructure. llvm-svn: 188949
2013-08-21 22:26:42 +00:00
# CHECK-NEXT: *** TEST 'shtest-format :: xpass.txt' FAILED ***
# CHECK-NEXT: Script
# CHECK-NEXT: --
# CHECK-NEXT: true
# CHECK-NEXT: --
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: Testing Time
# CHECK: Unexpected Passing Tests (1)
# CHECK: shtest-format :: xpass.txt
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK: Failing Tests (3)
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: shtest-format :: external_shell/fail.txt
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: shtest-format :: fail.txt
[lit] Fix UnicodeEncodeError when test commands contain non-ASCII chars Ensure that the bash script written by lit TestRunner is open with UTF-8 encoding when using Python 3. Otherwise, attempt to write non-ASCII characters causes UnicodeEncodeError. This happened e.g. with the following LLD test: UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119) ******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ******************** Exception during script execution: Traceback (most recent call last): File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test result = test.config.test_format.execute(test, lit_config) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute self.execute_external) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest res = _runShTest(test, litConfig, useExternalSh, script, tmpBase) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest res = executeScript(test, litConfig, tmpBase, script, execdir) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript f.write('{ ' + '; } &&\n{ '.join(commands) + '; }') UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128) Differential Revision: https://reviews.llvm.org/D63254 llvm-svn: 363388
2019-06-14 13:31:48 +00:00
# CHECK: Expected Passes : 8
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED" This reverts the revert in r292942. llvm-svn: 293007
2017-01-25 02:26:03 +00:00
# CHECK: Expected Failures : 4
# CHECK: Unsupported Tests : 5
# CHECK: Unresolved Tests : 3
[lit] Add a test for the various ShTest format features. llvm-svn: 174072
2013-01-31 18:05:01 +00:00
# CHECK: Unexpected Passes : 1
[lit] Ensure test output is converted to strings where possible. - This cleans up the text output of failing tests when run under PY3. llvm-svn: 188416
2013-08-14 22:21:01 +00:00
# CHECK: Unexpected Failures: 3
[lit] Move the shtest-xunit-output check lines into shtest-format These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
2018-07-23 18:08:36 +00:00
# XUNIT: <?xml version="1.0" encoding="UTF-8" ?>
# XUNIT-NEXT: <testsuites>
[lit] Fix UnicodeEncodeError when test commands contain non-ASCII chars Ensure that the bash script written by lit TestRunner is open with UTF-8 encoding when using Python 3. Otherwise, attempt to write non-ASCII characters causes UnicodeEncodeError. This happened e.g. with the following LLD test: UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119) ******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ******************** Exception during script execution: Traceback (most recent call last): File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test result = test.config.test_format.execute(test, lit_config) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute self.execute_external) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest res = _runShTest(test, litConfig, useExternalSh, script, tmpBase) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest res = executeScript(test, litConfig, tmpBase, script, execdir) File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript f.write('{ ' + '; } &&\n{ '.join(commands) + '; }') UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128) Differential Revision: https://reviews.llvm.org/D63254 llvm-svn: 363388
2019-06-14 13:31:48 +00:00
# XUNIT-NEXT: <testsuite name="shtest-format" tests="24" failures="7" skipped="5">
[lit] Move the shtest-xunit-output check lines into shtest-format These two tests are operating on the same test suite, which causes them to be racy about writing temporary files and can cause spurious failures. Merge them into one test to avoid the issue. llvm-svn: 337718
2018-07-23 18:08:36 +00:00
# XUNIT: <testcase classname="shtest-format.shtest-format" name="argv0.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.external_shell" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.external_shell" name="fail_with_bad_encoding.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.external_shell" name="pass.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="fail.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="no-test-line.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="pass.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-any-missing.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT:<skipped message="Skipping because of: a-missing-feature || a-missing-feature-2" />
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-any-present.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-missing.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT:<skipped message="Skipping because of: a-missing-feature" />
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-present.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-star.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="requires-triple.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT:<skipped message="Skipping because of: x86_64" />
# XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-false.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-expr-true.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT:<skipped message="Skipping because of configuration." />
# XUNIT: <testcase classname="shtest-format.shtest-format" name="unsupported-star.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: <testcase classname="shtest-format.unsupported_dir" name="some-test.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT:<skipped message="Skipping because of configuration." />
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-expr-false.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-expr-true.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-feature.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail-target.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xfail.txt" time="{{[0-9]+\.[0-9]+}}"/>
# XUNIT: <testcase classname="shtest-format.shtest-format" name="xpass.txt" time="{{[0-9]+\.[0-9]+}}">
# XUNIT-NEXT: <failure{{[ ]*}}>
# XUNIT: </failure>
# XUNIT-NEXT: </testcase>
# XUNIT: </testsuite>
# XUNIT-NEXT: </testsuites>
Reference in New Issue Copy Permalink