Summary:
The standard GNU atomic operations are a very common way to target
hardware atomics on the device. With more heterogenous devices being
introduced, the concept of memory scopes has been in the LLVM language
for awhile via the `syncscope` modifier. For targets, such as the GPU,
this can change code generation depending on whether or not we only need
to be consistent with the memory ordering with the entire system, the
single GPU device, or lower.
Previously these scopes were only exported via the `opencl` and `hip`
variants of these functions. However, this made it difficult to use
outside of those languages and the semantics were different from the
standard GNU versions. This patch introduces a `__scoped_atomic` variant
for the common functions. There was some discussion over whether or not
these should be overloads of the existing ones, or simply new variants.
I leant towards new variants to be less disruptive.
The scope here can be one of the following
```
__MEMORY_SCOPE_SYSTEM // All devices and systems
__MEMORY_SCOPE_DEVICE // Just this device
__MEMORY_SCOPE_WRKGRP // A 'work-group' AKA CUDA block
__MEMORY_SCOPE_WVFRNT // A 'wavefront' AKA CUDA warp
__MEMORY_SCOPE_SINGLE // A single thread.
```
Naming consistency was attempted, but it is difficult to capture to full
spectrum with no many names. Suggestions appreciated.
We used to assume that the CXXRecordDecl passed to the 1st argument
always had a definition. This is not true since a pointer to an
incomplete type was not excluded.
Fixes https://github.com/llvm/llvm-project/issues/63506
Just follow along with the reassociate pragma. This allows locally
setting the arcp fast math flag. Previously you could only access this
through the global -freciprocal-math.
Fixes#64798
The data size is required for implementing the `memmove` optimization
for `std::copy`, `std::move` etc. correctly as well as replacing
`__compressed_pair` with `[[no_unique_address]]` in libc++. Since the
compiler already knows the data size, we can avoid some complexity by
exposing that information.
Adds a new `__builtin_vectorelements()` function which returns the
number of elements for a given vector either at compile-time for
fixed-sized vectors, e.g., created via `__attribute__((vector_size(N)))`
or at runtime via a call to `@llvm.vscale.i32()` for scalable vectors,
e.g., SVE or RISCV V.
The new builtin follows a similar path as `sizeof()`, as it essentially
does the same thing but for the number of elements in vector instead of
the number of bytes. This allows us to re-use a lot of the existing
logic to handle types etc.
A small side addition is `Type::isSizelessVectorType()`, which we need
to distinguish between sizeless vectors (SVE, RISCV V) and sizeless
types (WASM).
This is the [corresponding
discussion](https://discourse.llvm.org/t/new-builtin-function-to-get-number-of-lanes-in-simd-vectors/73911).
This is information that the compiler already has, and should be exposed
so that the library doesn't need to reimplement the exact same
functionality.
Differential Revision: https://reviews.llvm.org/D135341
I'm reverting this on principle, since it didn't get the Phabricator
approval I thought it had (only an informal LGTM). Will re-apply once
it has been properly approved.
This reverts commit e1bfeb6bcc627a94c5ab3a5417d290c7dc516d54.
This is information that the compiler already has, and should be exposed
so that the library doesn't need to reimplement the exact same
functionality.
Differential Revision: https://reviews.llvm.org/D135341
Builtin function `__builtin_isfpclass` now can be called for a vector
of floating-point values. In this case it is applied to the vector
elementwise and produces vector of integer values.
Differential Revision: https://reviews.llvm.org/D153339
As discussed on #46593 - this enables us to use __lzcnt / __popcnt intrinsics inside constexpr code.
Differential Revision: https://reviews.llvm.org/D157420
This changes some public references of C2x to be C23, corrects standard
citations to use the final paragraph numbers, and adds some information
about differences between C17 and C23 modes.
Add codegen for llvm bitreverse elementwise builtin
The bitreverse elementwise builtin is necessary for HLSL codegen.
Tests were added to make sure that the expected errors are encountered when these functions are given inputs of incompatible types, or too many inputs.
The new builtin is restricted to integer types only.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D156357
Add codegen for llvm pow elementwise builtin
The pow elementwise builtin is necessary for HLSL codegen.
Tests were added to make sure that the expected errors are encountered when these functions are given inputs of incompatible types, or too many inputs.
The new builtin is restricted to floating point types only.
Reviewed By: arsenm
Differential Revision: https://reviews.llvm.org/D153310
These are basically the same thing and only differ for strictfp,
so add both for future proofing. Note all the elementwise functions are
currently broken for strictfp, and use non-constrained ops. Add a test
that demonstrates this, but doesn't attempt to fix it.
A new builtin function __builtin_isfpclass is added. It is called as:
__builtin_isfpclass(<floating point value>, <test>)
and returns an integer value, which is non-zero if the floating point
argument falls into one of the classes specified by the second argument,
and zero otherwise. The set of classes is an integer value, where each
value class is represented by a bit. There are ten data classes, as
defined by the IEEE-754 standard, they are represented by bits:
0x0001 (__FPCLASS_SNAN) - Signaling NaN
0x0002 (__FPCLASS_QNAN) - Quiet NaN
0x0004 (__FPCLASS_NEGINF) - Negative infinity
0x0008 (__FPCLASS_NEGNORMAL) - Negative normal
0x0010 (__FPCLASS_NEGSUBNORMAL) - Negative subnormal
0x0020 (__FPCLASS_NEGZERO) - Negative zero
0x0040 (__FPCLASS_POSZERO) - Positive zero
0x0080 (__FPCLASS_POSSUBNORMAL) - Positive subnormal
0x0100 (__FPCLASS_POSNORMAL) - Positive normal
0x0200 (__FPCLASS_POSINF) - Positive infinity
They have corresponding builtin macros to facilitate using the builtin
function:
if (__builtin_isfpclass(x, __FPCLASS_NEGZERO | __FPCLASS_POSZERO) {
// x is any zero.
}
The data class encoding is identical to that used in llvm.is.fpclass
function.
Differential Revision: https://reviews.llvm.org/D152351
This commit implements support for WebAssembly table types and
respective builtins. Table tables are WebAssembly objects to store
reference types. They have a large amount of semantic restrictions
including, but not limited to, only being allowed to be declared
at the top-level as static arrays of zero-length. Not being arguments
or result of functions, not being stored ot memory, etc.
This commit introduces the __attribute__((wasm_table)) to attach to
arrays of WebAssembly reference types. And the following builtins to
manage tables:
* ref __builtin_wasm_table_get(table, idx)
* void __builtin_wasm_table_set(table, idx, ref)
* uint __builtin_wasm_table_size(table)
* uint __builtin_wasm_table_grow(table, ref, uint)
* void __builtin_wasm_table_fill(table, idx, ref, uint)
* void __builtin_wasm_table_copy(table, table, uint, uint, uint)
This commit also enables reference-types feature at bleeding-edge.
This is joint work with Alex Bradbury (@asb).
Reviewed By: aaron.ballman
Differential Revision: https://reviews.llvm.org/D139010
_Generic accepts an expression operand whose type is matched against a
list of associations. The expression operand is unevaluated, but the
type matched is the type after lvalue conversion. This conversion loses
type information, which makes it more difficult to match against
qualified or incomplete types.
This extension allows _Generic to accept a type operand instead of an
expression operand. The type operand form does not undergo any
conversions and is matched directly against the association list.
This extension is also supported in C++ as we already supported
_Generic selection expressions there.
The RFC for this extension can be found at:
https://discourse.llvm.org/t/rfc-generic-selection-expression-with-a-type-operand/70388
Differential Revision: https://reviews.llvm.org/D149904
Pursuant to discussions at
https://discourse.llvm.org/t/rfc-c-23-p1467r9-extended-floating-point-types-and-standard-names/70033/22,
this commit enhances the handling of the __bf16 type in Clang.
- Firstly, it upgrades __bf16 from a storage-only type to an arithmetic
type.
- Secondly, it changes the mangling of __bf16 to DF16b on all
architectures except ARM. This change has been made in
accordance with the finalization of the mangling for the
std::bfloat16_t type, as discussed at
https://github.com/itanium-cxx-abi/cxx-abi/pull/147.
- Finally, this commit extends the existing excess precision support to
the __bf16 type. This applies to hardware architectures that do not
natively support bfloat16 arithmetic.
Appropriate tests have been added to verify the effects of these
changes and ensure no regressions in other areas of the compiler.
Reviewed By: rjmccall, pengfei, zahiraam
Differential Revision: https://reviews.llvm.org/D150913
Arrays and their element types are identically qualified as of C2x,
and we support that behavior as far back as C89.
As a drive-by, this adds the paper number for designated initializers
so that all the WG14 features can be uniquely identified despite the
lack of a feature testing macro.
This patch adds a new trait to allow standard libraries to forward `std::equal` calls to `memcmp` in more cases.
Reviewed By: aaron.ballman
Spies: Mordante, shafik, xbolva00, libcxx-commits, cfe-commits, ldionne
Differential Revision: https://reviews.llvm.org/D147175
Extend `#pragma clang __debug dump` to support not only single identifier, but an expression as well. This makes it possible to test ADL and overload resolution directly, without being creative to make them observable via diagnostics (e.g. when [[ http://eel.is/c++draft/over.match.best | over.match.best ]] is involved). This implementation has a known limitation of not supporting dependent expressions properly, but it's quite useful even without such support.
Differential Revision: https://reviews.llvm.org/D144115
Plumbing from the language level to the assume intrinsics with
separate_storage operand bundles.
Patch by David Goldblatt (davidtgoldblatt)
Differential Revision: https://reviews.llvm.org/D136515
Add '__builtin_FILE_NAME()', which expands to the filename because the
full path is not always needed. It corresponds to the '__FILE_NAME__'
predefined macro and is consistent with the other '__builin' functions
added for predefined macros.
Differential Revision: https://reviews.llvm.org/D144878
Add codegen for llvm exp/exp2 elementwise builtin
The exp/exp2 elementwise builtins are necessary for HLSL codegen.
Tests were added to make sure that the expected errors are encountered when these functions are given inputs of incompatible types.
The new builtins are restricted to floating point types only.
Reviewed By: fhahn
Differential Revision: https://reviews.llvm.org/D145270
I didn't understand why the other builtins have promotion logic,
or how it would apply for a ternary operation. Implicit conversions
are evil to begin with, and even more so when the purpose is to get
an exact IR intrinsic. This checks all the arguments have the same type.
Also move the line about __has_extension(gnu_asm_goto_with_outputs) so
that it is more generally about asm goto, not the paragraph on symbolic
references.
Reviewed By: efriedma, void
Differential Revision: https://reviews.llvm.org/D143205
Initial support for asm goto w/ outputs (D69876) only supported outputs
along the "default" (aka "fallthrough") edge.
We can support outputs along all edges by repeating the same pattern of
stores along the indirect edges that we allready do for the default
edge. One complication is that these indirect edges may be critical
edges which would need to be split. Another issue is that mid-codgen of
LLVM IR, the control flow graph might not reflect the control flow of
the final function.
To avoid this "chicken and the egg" problem assume that any given
indirect edge may become a critical edge, and pro-actively split it.
This is unnecessary if the edge does not become critical, but LLVM will
optimize such cases via tail duplication.
Fixes: https://github.com/llvm/llvm-project/issues/53562
Reviewed By: void
Differential Revision: https://reviews.llvm.org/D136497
