So far CSA was relying on the LLVM Statistic package that allowed us to
gather some data about analysis of an entire translation unit. However,
the translation unit consists of a collection of loosely related entry
points. Aggregating data across multiple such entry points is often
counter productive.
This change introduces a new lightweight always-on facility to collect
Boolean or numerical statistics for each entry point and dump them in a
CSV format. Such format makes it easy to aggregate data across multiple
translation units and analyze it with common data-processing tools.
We break down the existing statistics that were collected on the per-TU
basis into values per entry point.
Additionally, we enable the statistics unconditionally (STATISTIC ->
ALWAYS_ENABLED_STATISTIC) to facilitate their use (you can gather the
data with a simple run-time flag rather than having to recompile the
analyzer). These statistics are very light and add virtually no
overhead.
Co-authored-by: Balazs Benics <benicsbalazs@gmail.com>
CPP-6160
We need to be able to read in and parse files using the ClangIR dialect
in order to test this part of the functionality.
This change adds the minimum cir-opt tool needed to read and parse cir
files and write them back to text. This tool will later be extended to
add features for lowering from CIR to other MLIR dialects and to run CIR
passes as they are upstreamed.
This PR is one of the many PRs in the SYCL upstreaming effort focusing
on device code linking during the SYCL offload compilation process. RFC:
https://discourse.llvm.org/t/rfc-offloading-design-for-sycl-offload-kind-and-spir-targets/74088
In this PR, we introduce a new tool that will be used to perform device
code linking for SYCL offload kind. It accepts SYCL device objects in
LLVM IR bitcode format and will generate a fully linked device object
that can then be wrapped and linked into the host object.
A primary use case for this tool is to perform device code linking for
objects with SYCL offload kind inside the clang-linker-wrapper. It can
also be invoked via clang driver as follows:
`clang --target=spirv64 --sycl-link input.bc`
Device code linking for SYCL offloading kind has a number of known
quirks that makes it difficult to use in a unified offloading setting.
Two of the primary issues are:
1. Several finalization steps are required to be run on the fully-linked
LLVM IR bitcode to gaurantee conformance to SYCL standards. This step is
unique to SYCL offloading compilation flow.
2. SPIR-V LLVM Translator tool is an extenal tool and hence SPIR-V IR
code generation cannot be done as part of LTO. This limitation will be
lifted once SPIR-V backend is available as a viable LLVM backend.
Hence, we introduce this new tool to provide a clean wrapper to perform
SYCL device linking.
Co-Author: Michael Toguchi
Thanks
---------
Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>
This adds a flag to lit for detecting and updating failing tests when
possible to do so automatically. The flag uses a plugin architecture
where config files can add additional auto-updaters for the types of
tests in the test suite. When a test fails with `--update-tests` enabled
lit passes the test RUN invocation and output to each registered test
updater until one of them signals that it updated the test (or all test
updaters have been run). As such it is the responsibility of the test
updater to only update tests where it is reasonably certain that it will
actually fix the test, or come close to doing so.
Initially adds support for UpdateVerifyTests and UpdateTestChecks. The
flag is currently only implemented for lit's internal shell, so
`--update-tests` implies `LIT_USE_INTERNAL_SHELL=1`.
Builds on work in #97369Fixes#81320
Summary:
The `clang-nvlink-wrapper` is a utility that I removed awhile back
during the transition to the new driver. This patch adds back in a new,
upgraded version that does LTO + archive linking. It's not an easy
choice to reintroduce something I happily deleted, but this is the only
way to move forward with improving GPU support in LLVM.
While NVIDIA provides a linker called 'nvlink', its main interface is
very difficult to work with. It does not provide LTO, or static linking,
requires all files to be named a non-standard `.cubin`, and rejects link
jobs that other linkers would be fine with (i.e empty). I have spent a
great deal of time hacking around this in the GPU `libc` implementation,
where I deliberately avoid LTO and static linking and have about 100
lines of hacky CMake dedicated to storing these files in a format that
the clang-linker-wrapper accepts to avoid this limitation.
The main reason I want to re-intorudce this tool is because I am
planning on creating a more standard C/C++ toolchain for GPUs to use.
This will install files like the following.
```
<install>/lib/nvptx64-nvidia-cuda/libc.a
<install>/lib/nvptx64-nvidia-cuda/libc++.a
<install>/lib/nvptx64-nvidia-cuda/libomp.a
<install>/lib/clang/19/lib/nvptx64-nvidia-cuda/libclang_rt.builtins.a
```
Linking in these libraries will then simply require passing `-lc` like
is already done for non-GPU toolchains. However, this doesn't work with
the currently deficient `nvlink` linker, so I consider this a blocking
issue to massively improving the state of building GPU libraries.
In the future we may be able to convince NVIDIA to port their linker to
`ld.lld`, but for now this is the only workable solution that allows us
to hack around the weird behavior of their closed-source software.
This also copies some amount of logic from the clang-linker-wrapper,
but not enough for it to be worthwhile to merge them I feel. In the
future it may be possible to delete that handling from there entirely.
The goal of this patch is to enable utilizing LLVM plugin passes and
remarks for GPU offload code at link time. Specifically, this patch
extends clang-linker-wrapper's `--offload-opt` (and consequently
`-mllvm`) to accept the various LLVM pass options that tools like opt
usually accept. Those options include `--passes`, `--load-pass-plugin`,
and various remarks options.
Unlike many other LLVM options that are inherited from linked code by
clang-linker-wrapper (e.g., `-pass-remarks` is already implemented in
`llvm/lib/IR/DiagnosticHandler.cpp`), these options are implemented
separately as needed by each tool (e.g., opt, llc). Fortunately, this
patch is able to handle most of the implementation by passing the option
values to `lto::Config`.
For testing plugin support, this patch uses the simple `Bye` plugin from
LLVM core, but that requires several small Clang test suite config
extensions.
Installapi has important distinctions when compared to the clang driver,
so much that, it doesn't make much sense to try to integrate into it.
This patch partially reverts the CC1 action & driver support to replace
with its own driver as a clang tool.
For distribution, we could use `LLVM_TOOL_LLVM_DRIVER_BUILD` mechanism
for integrating the functionality into clang such that the toolchain
size is less impacted.
This introduces a basic outline of installapi as a clang driver option.
It captures relevant information as cc1 args, which are common arguments
already passed to the linker to encode into TBD file outputs. This is
effectively an upstream for what already exists as `tapi installapi` in
Xcode toolchains, but directly in Clang. This patch does not handle any
AST traversing on input yet.
InstallAPI is broadly an operation that takes a series of header files
that represent a single dynamic library and generates a TBD file out of
it which represents all the linkable symbols and necessary attributes
for statically linking in clients. It is the linkable object in all
Apple SDKs and when building dylibs in Xcode. `clang -installapi` also
will support verification where it compares all the information recorded
for the TBD files against the already built binary, to catch possible
mismatches like when a declaration is missing a definition for an
exported symbol.
We now have 64-bit XCOFF object file support, so these tests can be
enabled again. However, some tests still fail due to unsupported debug
sections, so I cleaned up their comments.
Summary:
llvm-ar is symlinked as llvm-ranlib and will act as ranlib when invoked in that mode. llvm-ar since [[ 4f2cfbe531 | compiler/llvm-project@4f2cfbe ]] supports the -X options, but doesn't seem to accept them when running as llvm-ranlib.
In AIX OS , according to https://www.ibm.com/docs/en/aix/7.2?topic=r-ranlib-command
-X mode Specifies the type of object file ranlib should examine. The mode must be one of the following:
32
Processes only 32-bit object files
64
Processes only 64-bit object files
32_64, any
Processes both 32-bit and 64-bit object files
The default is to process 32-bit object files (ignore 64-bit objects). The mode can also be set with the OBJECT_MODE environment variable. For example, OBJECT_MODE=64 causes ranlib to process any 64-bit objects and ignore 32-bit objects. The -X flag overrides the OBJECT_MODE variable.
Reviewers: James Henderson, MaskRay, Stephen Peckham
Differential Revision: https://reviews.llvm.org/D142660
CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used with all
__device__ functions.
Differential Revision: https://reviews.llvm.org/D146389
This is an ongoing series of commits that are reformatting our
Python code.
Reformatting is done with `black`.
If you end up having problems merging this commit because you
have made changes to a python file, the best way to handle that
is to run git checkout --ours <yourfile> and then reformat it
with black.
If you run into any problems, post to discourse about it and
we will try to help.
RFC Thread below:
https://discourse.llvm.org/t/rfc-document-and-standardize-python-code-style
Reviewed By: MatzeB
Differential Revision: https://reviews.llvm.org/D150761
CUDA support can be enabled in clang-repl with --cuda flag.
Device code linking is not yet supported. inline must be used with all
__device__ functions.
Differential Revision: https://reviews.llvm.org/D146389
This reverts commit 029212617ca8bdedd949d17bf2d33750605ea607.
Reverting because of failures when setting
`CLANG_CRASH_DIAGNOSTICS_DIR` before invoking `ninja check-clang`.
This is useful for retriving crash reports of LIT runs when the
temporary folder is not accessible.
Reviewed By: michaelplatings
Differential Revision: https://reviews.llvm.org/D147209
Summary:
In patch https://reviews.llvm.org/D127864. we add a new option -X for AIX OS, and default value is -X32. In order not effect the test cases in clang/test. we need to change "llvm-ar" to "env OBJECT_MODE=any llvm-ar" in clang/test for AIX OS
Differential Revision: https://reviews.llvm.org/D145600
Similar to issue fixed in D107155, some lit tests invoke `llvm-lto`,
`llvm-lto2` and `llvm-profdata` without going through the substitution
system. While the test runner correctly picks up these binaries from
the build directory, it doesn't print its absolute path. When copying
the invocations when reproducing test failures, this can result in
`command not found: llvm-lto` or other errors (caused by using wrong
version of the tool).
This patch adds these tools to the `tools` variable in
`clang/test/lit.cfg.py` which will then call add_tool_substitutions.
Reviewed By: dblaikie
Differential Revision: https://reviews.llvm.org/D143806
This patch adds AArch64 to the LP64 feature to avoid running LLP64
incompatible test on AArch64/Windows platform.
Differential Revision: https://reviews.llvm.org/D136070
Remove the ability to disable opaque pointers by default in clang.
It is still possible to explicitly disable them via cc1
-no-opaque-pointers.
Differential Revision: https://reviews.llvm.org/D135259
Add support for a CLANG_NO_DEFAULT_CONFIG envvar that works like
the --no-default-config option when set to a non-empty value. Use it
to disable loading system configuration files during the test suite
runs.
Configuration files can change the driver behavior in extensive ways,
and it is neither really possible nor feasible to account for or undo
the effects of even the most common configuration uses. Therefore,
the most reasonable option seems to be to ignore configuration files
while running the majority of tests (with the notable exception of tests
for configuration file support).
Due to the diversity of ways that %clang is used in the test suite,
including using it to copy or symlink the clang executable, as well to
call -cc1 and -cc1as modes, it is not feasible to pass the explicit
options to disable config loading either. Using an environment variable
has the advantage of being easily applied across the test suite
and easily unset for default configuration file loading tests.
Differential Revision: https://reviews.llvm.org/D134905
Move the `%clang_dxc` substitution from local definition in clang/test
to lit's `llvm/config.py` module where all other driver definitions
are found. This improves consistency and makes it easier to control
global clang options.
Differential Revision: https://reviews.llvm.org/D134871
Summary:
since default object mode of llvm-nm is change to -X32 (from default -Xany) in patch https://reviews.llvm.org/D132494.In order not effect the test cases in clang/test we need to change "llvm-nm" to "env OBJECT_MODE=any llvm-nm" in clang/test for AIX OS
Reviewers: James Henderson,David Tenty, Hubert Tong
Differential Revision: https://reviews.llvm.org/D134284
Newer SDKs don't even provide libstdc++ headers, so it's effectively
never valid to build for libstdc++ unless the user explicitly asks
for it (in which case they will need to provide include paths and more).
This is a re-application of c5ccb78ade81 which had been reverted in
33171df9cc7f because it broke the Fuchsia CI bots. The issue was that
the test was XPASSing because it didn't fail anymore when the
CLANG_DEFAULT_CXX_LIB was set to libc++, which seems to be done for
Fuchsia. Instead, the test only fails if CLANG_DEFAULT_CXX_LIB is
set to libstdc++.
As a fly-by fix, also adjust the triple used by various tests to
something that is supported. Those tests were shown to fail on
internal bots.
Differential Revision: https://reviews.llvm.org/D131274
We have seen random symbol not found "__cxa_throw" error in fuschia build bots and out-of-tree users. The understanding have been that they are built without exception support, but it turned out that these platforms have LLVM_STATIC_LINK_CXX_STDLIB ON so that they link libstdc++ to llvm statically. The reason why this is problematic for clang-repl is that by default clang-repl tries to find symbols from symbol table of executable and dynamic libraries loaded by current process. It needs to load another libstdc++, but the platform that had LLVM_STATIC_LINK_CXX_STDLIB turned on is usally those with missing or obsolate shared libstdc++ in the first place -- trying to load it again would be destined to fail eventually with a risk to introuduce mixed libstdc++ versions.
A proper solution that doesn't take a workaround is statically link the same libstdc++ by clang-repl side, but this is not possible with old JIT linker runtimedyld. New just-in-time linker JITLink handles this relatively well, but it's not availalbe in majority of platforms. For now, this patch just disables the building of clang-repl when LLVM_STATIC_LINK_CXX_STDLIB is ON and removes the "__cxa_throw" check in exception unittest as well as reverting previous exception check flag patch.
Reviewed By: v.g.vassilev
Differential Revision: https://reviews.llvm.org/D130788
Add host exception support check utility flag. This is needed to not run tests that require exception support in few buildbots that lacks related symbols for some reason.
Reviewed By: lhames
Differential Revision: https://reviews.llvm.org/D129242
Some tests perform deep recursion, which requires a larger pthread stack
size than the relatively low default of 192 KiB for 64-bit processes on
AIX. The `AIXTHREAD_STK` environment variable provides a non-intrusive
way to request a larger pthread stack size for the tests. The required
pthread stack size depends on the build configuration.
A 4 MiB default is generous compared to the 512 KiB of macOS; however,
it is known that some compilers on AIX produce code that uses
comparatively more stack space.
This patch expands the solution from D65688 to apply to all Clang LIT
tests.
This also reverts commit c3c75d805c2174388417080f762230961b3433d6,
"[clang][test] Mark test arm-float-abi-lto.c unsupported on AIX".
The problem was caused by the test running up against the per-thread
stack limit on AIX. This is resolved by having the tests run with
`AIXTHREAD_STK` set for 4 MiB.
Reviewed By: xingxue
Differential Revision: https://reviews.llvm.org/D129165
In order to do offloading compilation we need to embed files into the
host and create fatbainaries. Clang uses a special binary format to
bundle several files along with their metadata into a single binary
image. This is currently performed using the `-fembed-offload-binary`
option. However this is not very extensibile since it requires changing
the command flag every time we want to add something and makes optional
arguments difficult. This patch introduces a new tool called
`clang-offload-packager` that behaves similarly to CUDA's `fatbinary`.
This tool takes several input files with metadata and embeds it into a
single image that can then be embedded in the host.
Reviewed By: tra
Differential Revision: https://reviews.llvm.org/D125165
The linker wrapper is used to perform linking and wrapping of embedded
device object files. Currently its internals are not able to be tested
easily. This patch adds the `--dry-run` and `--print-wrapped-module`
options to investigate the link jobs that will be run along with the
wrapped code that will be created to register the binaries.
Reviewed By: JonChesterfield
Differential Revision: https://reviews.llvm.org/D124039
The target profile option(/T) decide the shader model when compile hlsl.
The format is shaderKind_major_minor like ps_6_1.
The shader model is saved as llvm::Triple is clang/llvm like
dxil-unknown-shadermodel6.1-hull.
The main job to support the option is translating ps_6_1 into
shadermodel6.1-pixel.
That is done inside tryParseProfile at HLSL.cpp.
To integrate the option into clang Driver, a new DriverMode DxcMode is
created. When DxcMode is enabled, OSType for TargetTriple will be
forced into Triple::ShaderModel. And new ToolChain HLSLToolChain will
be created when OSType is Triple::ShaderModel.
In HLSLToolChain, ComputeEffectiveClangTriple is overridden to call
tryParseProfile when targetProfile option is set.
To make test work, Fo option is added and .hlsl is added for active
-xhlsl.
Reviewed By: beanz
Differential Revision: https://reviews.llvm.org/D122865
Patch by: Xiang Li <python3kgae@outlook.com>
The current cc1 CLANG_ENABLE_OPAQUE_POINTERS=on default difference is not ideal
in that people contribute %clang_cc1 tests may assume the default ON behavior,
which will cause failures on systems set to OFF.
cc1 option default dependent on CMake options should be used prudently
(generally avoided). We prefer to limit target differences to Driver.
Change the CLANG_ENABLE_OPAQUE_POINTERS_INTERNAL mechanism introduced in D123122
to use a driver default instead. This is similar to the mechanism used for the
-flegacy-pass-manager transition to new PM transition.
Reviewed By: #opaque-pointers, rsmith, aeubanks
Differential Revision: https://reviews.llvm.org/D123744
Or rather, error out if it is set to something other than ON. This
removes the ability to enable the legacy pass manager by default,
but does not remove the ability to explicitly enable it through
various flags like -flegacy-pass-manager or -enable-new-pm=0.
I checked, and our test suite definitely doesn't pass with
LLVM_ENABLE_NEW_PASS_MANAGER=OFF anymore.
Differential Revision: https://reviews.llvm.org/D123126
Bringing in HLSL as a language as well as language options for each of
the HLSL language standards.
While the HLSL language is unimplemented, this patch adds the
HLSL-specific preprocessor defines which enables testing of the command
line options through the driver.
Reviewed By: pete, rnk
Differential Revision: https://reviews.llvm.org/D122087
This patch gets rid of the ridiculous relative path we use to invoke the `module-deps-to-rsp.py` script and creates proper lit substitution, cleaning up the tests.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D121525
Exactly what it says on the tin! We had a nasty crash with the following incovation:
$ clang --analyze -Xclang -analyzer-constraints=z3 test.c
fatal error: error in backend: LLVM was not compiled with Z3 support, rebuild with -DLLVM_ENABLE_Z3_SOLVER=ON
... <stack trace> ...
Differential Revision: https://reviews.llvm.org/D120325
The modified tests fail because 64-bit XCOFF object files are not currently supported on AIX. This patch disables these tests on 64-bit AIX for now.
This patch is similar to D111887 except the failures on this patch are on a 64-bit build.
Reviewed By: shchenz, #powerpc
Differential Revision: https://reviews.llvm.org/D113049
