This commit introduces basic annotations for `std::basic_string`,
mirroring the approach used in `std::vector` and `std::deque`.
Initially, only long strings with the default allocator will be
annotated. Short strings (_SSO - short string optimization_) and strings
with non-default allocators will be annotated in the near future, with
separate commits dedicated to enabling them. The process will be similar
to the workflow employed for enabling annotations in `std::deque`.
**Please note**: these annotations function effectively only when libc++
and libc++abi dylibs are instrumented (with ASan). This aligns with the
prevailing behavior of Memory Sanitizer.
To avoid breaking everything, this commit also appends
`_LIBCPP_INSTRUMENTED_WITH_ASAN` to `__config_site` whenever libc++ is
compiled with ASan. If this macro is not defined, string annotations are
not enabled. However, linking a binary that does **not** annotate
strings with a dynamic library that annotates strings, is not permitted.
Originally proposed here: https://reviews.llvm.org/D132769
Related patches on Phabricator:
- Turning on annotations for short strings:
https://reviews.llvm.org/D147680
- Turning on annotations for all allocators:
https://reviews.llvm.org/D146214
This PR is a part of a series of patches extending AddressSanitizer C++
container overflow detection capabilities by adding annotations, similar
to those existing in `std::vector` and `std::deque` collections. These
enhancements empower ASan to effectively detect instances where the
instrumented program attempts to access memory within a collection's
internal allocation that remains unused. This includes cases where
access occurs before or after the stored elements in `std::deque`, or
between the `std::basic_string`'s size (including the null terminator)
and capacity bounds.
The introduction of these annotations was spurred by a real-world
software bug discovered by Trail of Bits, involving an out-of-bounds
memory access during the comparison of two strings using the
`std::equals` function. This function was taking iterators
(`iter1_begin`, `iter1_end`, `iter2_begin`) to perform the comparison,
using a custom comparison function. When the `iter1` object exceeded the
length of `iter2`, an out-of-bounds read could occur on the `iter2`
object. Container sanitization, upon enabling these annotations, would
effectively identify and flag this potential vulnerability.
This Pull Request introduces basic annotations for `std::basic_string`.
Long strings exhibit structural similarities to `std::vector` and will
be annotated accordingly. Short strings are already implemented, but
will be turned on separately in a forthcoming commit. Look at [a
comment](https://github.com/llvm/llvm-project/pull/72677#issuecomment-1850554465)
below to read about SSO issues at current moment.
Due to the functionality introduced in
[D132522](dd1b7b797a),
the `__sanitizer_annotate_contiguous_container` function now offers
compatibility with all allocators. However, enabling this support will
be done in a subsequent commit. For the time being, only strings with
the default allocator will be annotated.
If you have any questions, please email:
- advenam.tacet@trailofbits.com
- disconnect3d@trailofbits.com
There are a few drive-by fixes:
- Since the combination RTTI disabled and exceptions enabled do not
work, this combination is prohibited.
- A small NFC in any fixing clang-tidy.
The code in the Buildkite configuration is prepared for using the std
module. There are more fixes needed for that configuration which will be
done in a separate commit.
Patch 4b1fe097f9a3882f437bc3b829ef02331e28a8d6 introduced a bug when
building libc++ for Fuchsia, it disabled the libc++.so linker script by
default. This patch restores its original behavior.
We were detecting which sanitizer flags to use when building
libc++.dylib but we were never actually adding those flags to the
targets, which means that our sanitized builds would basically build the
dylib without any sanitizers enabled.
We really shouldn't be depending on far away configuration options like
LLVM_HAVE_LINK_VERSION_SCRIPT here. This patch simplifies the enablement
of the linker scripts and as a result gets rid of an undesirable
dependency on HandleLLVMOptions.cmake.
As a drive-by, the patch also stops taking into account whether Python3
is available. This should have no bearing on whether we generate a
linker script or not, which is required for correctness. If someone
tries to build libc++ and generate a linker script but Python3 is not
available, they should get an error instead of silently getting an
incorrect installation of the library.
1. Instead of using individual "boolean" macros, have an "enum" macro
`_LIBCPP_HARDENING_MODE`. This avoids issues with macros being
mutually exclusive and makes overriding the hardening mode within a TU
more straightforward.
2. Rename the safe mode to debug-lite.
This brings the code in line with the RFC:
https://discourse.llvm.org/t/rfc-hardening-in-libc/73925Fixes#65101
This is necessary in order to implement some papers like P2467R1, which
require using C++23 declarations in the dylib. It is a good habit to
keep building the dylib with a recent standard version regardless.
With this patch, we also stop strictly enforcing that the targets are
built with C++23. Concretely, C++23 will soon be required in order to
build the dylib, but not enforcing it strictly works around some issues
like the documentation bots using an old and unsupported compiler. Since
these bots do not actually build the library, not strictly enforcing the
C++ Standard makes our CMake build more resilient to these kinds of
situation. This is just a workaround though, the better way of going
about would be to update the compiler on the documentation bot but we
don't seem to have control over that.
We support GCC 13, which supports the flag. This allows simplifying the
CMake logic around the use of -nostdlib++. Note that there are other
places where we don't assume -nostdlib++ yet in the build, but this
patch is intentionally trying to be small because this part of our CMake
is pretty tricky.
To allow for a smoother transition, keep the safe mode working as is in
the LLVM 18 release (the first release that aims to make hardening
available), then deprecate it in LLVM 19.
On Apple platforms, we always support the -nostdlib++ flag. Hence, it is
not necessary to manually link against system libraries. In fact, doing
so causes us to link against libSystem explicitly, which messes up with
the order of libraries we should use. Indeed:
Before patch, using the system unwinder (LIBCXXABI_USE_LLVM_UNWINDER = OFF)
===========================================================================
$ otool -L lib/{libc++.1.dylib,libc++abi.1.dylib,libunwind.1.dylib}
lib/libc++.1.dylib:
@rpath/libc++.1.dylib
/usr/lib/libSystem.B.dylib
@rpath/libc++abi.1.dylib
lib/libc++abi.1.dylib:
@rpath/libc++abi.1.dylib
/usr/lib/libSystem.B.dylib
lib/libunwind.1.dylib:
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
After patch, using the system unwinder (LIBCXXABI_USE_LLVM_UNWINDER = OFF)
===========================================================================
$ otool -L lib/{libc++.1.dylib,libc++abi.1.dylib,libunwind.1.dylib}
lib/libc++.1.dylib:
@rpath/libc++.1.dylib
@rpath/libc++abi.1.dylib
/usr/lib/libSystem.B.dylib
lib/libc++abi.1.dylib:
@rpath/libc++abi.1.dylib
/usr/lib/libSystem.B.dylib
lib/libunwind.1.dylib:
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
Before patch, with the LLVM unwinder (LIBCXXABI_USE_LLVM_UNWINDER = ON)
=======================================================================
$ otool -L lib/{libc++.1.dylib,libc++abi.1.dylib,libunwind.1.dylib}
lib/libc++.1.dylib:
@rpath/libc++.1.dylib
/usr/lib/libSystem.B.dylib
@rpath/libc++abi.1.dylib
@rpath/libunwind.1.dylib
lib/libc++abi.1.dylib:
@rpath/libc++abi.1.dylib
/usr/lib/libSystem.B.dylib
@rpath/libunwind.1.dylib
lib/libunwind.1.dylib:
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
After patch, with the LLVM unwinder (LIBCXXABI_USE_LLVM_UNWINDER = ON)
======================================================================
$ otool -L lib/{libc++.1.dylib,libc++abi.1.dylib,libunwind.1.dylib}
lib/libc++.1.dylib:
@rpath/libc++.1.dylib
@rpath/libc++abi.1.dylib
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
lib/libc++abi.1.dylib:
@rpath/libc++abi.1.dylib
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
lib/libunwind.1.dylib:
@rpath/libunwind.1.dylib
/usr/lib/libSystem.B.dylib
As we can see, libSystem appears before the just-built libraries before
the patch, which causes the libunwind.dylib bundled in libSystem.dylib
to be used instead of the just-built libunwind.dylib.
We didn't notice the issue until recently when I tried to update the
macOS CI builders to macOS 13.5, where it is necessary to use the right
libunwind library (the exact reason still needs to be investigated).
The safe mode is in-between the hardened and the debug modes, extending
the checks contained in the hardened mode with certain checks that are
relatively cheap and prevent common sources of errors but aren't
security-critical. Thus, the safe mode trades off some performance for
a wider set of checks, but unlike the debug mode, it can still be used
in production.
Differential Revision: https://reviews.llvm.org/D158823
This is the first step to implement time zone support in libc++. This
adds the complete tzdb_list class and a minimal tzdb class. The tzdb
class only contains the version, which is used by reload_tzdb.
Next to these classes it contains documentation and build system support
needed for time zone support. The code depends on the IANA Time Zone
Database, which should be available on the platform used or provided by
the libc++ vendors.
The code is labeled as experimental since there will be ABI breaks
during development; the tzdb class needs to have the standard headers.
Implements parts of:
- P0355 Extending <chrono> to Calendars and Time Zones
Addresses:
- LWG3319 Properly reference specification of IANA time zone database
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D154282
The building of the std module has been moved from `params.py` and
`dsl.py` to a `lit.local.cfg` for the entire test suite. In theory this
change allows testing modules in most configurations, except:
- combined with clang modules
- C++ versions that don't support the std module
Currently only C++23 with all parts enabled works.
C++26 is expected to work properly with CMake 3.27. That versions of CMake
knows how to invoke clang using C++26.
The parts disabled modi of libc++ have not been modularized yet.
It still is the goal that in the future CMake will be able to do the work
done in `lit.local.cfg`. Doing this in CMake would require a more mature
libc++ implementation.
Thanks a lot to @ldionne for giving hints how to enable modules in a
`lit.local.cfg`.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D157625
This patch fixes a few CMake options that were set using incorrect
mechanisms.
CMake's man page for the -D <var>=<value> option states: If a command in
the project sets the type to PATH or FILEPATH, then the <value> will be
converted to an absolute path. That's not what we want for most of the
paths we have as configuration options. Otherwise, using -D to set the
configuration option results in an absolute path being used, which
breaks things.
option() denotes a boolean variable, but what was desired was a
string/list variable. Fix this to prevent cmake from changing any
non-empty user provided values to 'ON'.
Differential Revision: https://reviews.llvm.org/D157926
If CMAKE_MSVC_RUNTIME_LIBRARY isn't set, then CMake defaults to
the debug CRT, if CMAKE_BUILD_TYPE is set to Debug. If
CMAKE_MSVC_RUNTIME_LIBRARY is set though, that overrides any
implicit defaults.
Match this in libcxx's own manual linking logic. This allows
decoupling the debug CRT from the CMAKE_BUILD_TYPE and allows users
to configure their builds exactly how they want.
Differential Revision: https://reviews.llvm.org/D155561
This respects the CMAKE_MSVC_RUNTIME_LIBRARY option for selecting
the right CRT to use.
Add a CI configuration that tests building this way.
Based on a patch by Andrew Ng.
The test config files end up accumulating and duplicating a fair
bit of cmake-specific logic here; if preferred, we could also add
that in `libcxx/test/CMakeLists.txt` and export a few more variables
to `cmake-bridge.cfg.in` instead.
Differential Revision: https://reviews.llvm.org/D155560
The library msvcrt.lib pulls in ucrt.lib and vcruntime.lib anyway,
there's no need to manually link against the individual dependencies.
This matches how the tests link against libraries - they only link
against msvcrt and msvcprt, not directly against ucrt and vcruntime.
Differential Revision: https://reviews.llvm.org/D155555
Our threading support layer is currently a huge mess. There are too many
configurations with too many confusing names, and none of them are tested
in the usual CI. Here's a list of names related to these configurations:
LIBCXX_BUILD_EXTERNAL_THREAD_LIBRARY
_LIBCPP_BUILDING_THREAD_LIBRARY_EXTERNAL
LIBCXXABI_BUILD_EXTERNAL_THREAD_LIBRARY
_LIBCPP_HAS_THREAD_LIBRARY_EXTERNAL
LIBCXX_HAS_EXTERNAL_THREAD_API
_LIBCPP_HAS_THREAD_API_EXTERNAL
This patch cleans this up by removing the ability to build libc++ with
an "external" threading library for testing purposes, removing 4 out of
6 "names" above. That setting was meant to be used by libc++ developers,
but we don't use it in-tree and it's not part of our CI.
I know the ability to use an external threading API is used by some folks
out-of-tree, and this patch doesn't change that. This only changes the
way they will have to test their external threading support. After this
patch, the intent would be for them to set `-DLIBCXX_HAS_EXTERNAL_THREAD_API=ON`
when building the library, and to provide their usual `<__external_threading>`
header when they are testing the library. This can be done easily now
that we support custom lit configuration files in test suites.
The motivation for this patch is that our threading support layer is
basically unmaintainable -- anything beyond adding a new "backend" in
the slot designed for it requires incredible attention. The complexity
added by this setting just doesn't pull its weigh considering the
available alternatives.
Concretely, this will also allow future patches to clean up
`<__threading_support>` significantly.
Differential Revision: https://reviews.llvm.org/D154466
`_LIBCPP_ENABLE_ASSERTIONS` was used to enable the "safe" mode in
libc++. Libc++ now provides the hardened mode and the debug mode that
replace the safe mode.
For backward compatibility, enabling `_LIBCPP_ENABLE_ASSERTIONS` now
enables the hardened mode. Note that the hardened mode provides
a narrower set of checks than the previous "safe" mode (only
security-critical checks that are performant enough to be used in
production).
Differential Revision: https://reviews.llvm.org/D154997
It's no longer necessary to define `_DEBUG` because we use our own
internal assertion mechanism in libc++.
Differential Revision: https://reviews.llvm.org/D155038
This patch only adds new configuration knobs -- the actual assertions
will be added in follow-up patches.
Differential Revision: https://reviews.llvm.org/D153902
MSVC is set for actual msvc and clang-cl. LIBXX_TARGETTING_CLANG_CL is
still used over in runtimes to configure warning flags.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D154698
Previously we added both `-GR-` and `-fno-rtti` if RTTI was disabled.
When building with clang 16.x, that caused this error in part of the build:
```
clang-16: error: argument unused during compilation: '-G R-' [-Werror,-Wunused-command-line-argument]
```
I think the strange message is because clang is seeing `R-` as the argument
to `-G`, which is a valid clang option.
`-GR-` is an alternate syntax for the `/GR-` option for MSVC
(the dash means disable RTTI):
https://learn.microsoft.com/en-us/cpp/build/reference/gr-enable-run-time-type-information?view=msvc-170
This error is sort of fixed by cd18efb61d759405956dbd30e4b5f2720d8e1783
but not intentionally. Also, we'd have to wait for 17.x to benefit from that.
The proper fix here is to only add `-GR-` if we are building with MSVC
or the MSVC-like clang-cl, and add `-fno-rtti` if not.
Reviewed By: #libc, simon_tatham, michaelplatings, ldionne
Differential Revision: https://reviews.llvm.org/D154582
Whether we include operator new and delete into libc++ has always
been a build time setting, and piggy-backing on a macro like
_LIBCPP_DISABLE_NEW_DELETE_DEFINITIONS is inconsistent with how
we handle similar cases for e.g. LIBCXX_ENABLE_RANDOM_DEVICE. Instead,
simply avoid including new.cpp in the sources of the library when we
do not wish to include these operators in the build.
This also makes us much closer to being able to share the definitions
between libc++ and libc++abi, since we could technically build those
definitions into a standalone static library and decide whether we link
it into libc++abi.dylib or libc++.dylib.
Differential Revision: https://reviews.llvm.org/D153272
The patch is based on D144994.
D151030 added the module definitions for the module std.
This patch wires in the module and enables the basic testing.
Some notable features are missing:
- There is no test that libc++ can be fully imported as a module.
- This lacks the parts for the std.compat module.
- The module is not shipped with libc++.
Implements parts of
- P2465R3 Standard Library Modules std and std.compat
Reviewed By: ldionne, aaronmondal, #libc
Differential Revision: https://reviews.llvm.org/D151814
LIBCXX_ENABLE_FILESYSTEM should represent whether the platform has
support for a filesystem, not just whether we support <filesystem>.
This patch slightly generalizes the setting to also encompass whether
we provide <fstream>, since that only makes sense when a filesystem is
supported.
Differential Revision: https://reviews.llvm.org/D152168
Back in 2020 [1], we went very close to enabling Filesystem on MSVC
by disabling int128_t, but decided to wait because MSVC support
for int128_t was supposed to come shortly after. Since it's not
there yet, I propose turning off int128_t support by default on MSVC.
This will make <filesystem> available by default on MSVC, and most
importantly will open the possibility for changing
LIBCXX_ENABLE_FILESYSTEM to mean "the system doesn't have support
for a filesystem" instead of simply "don't build the std::filesystem
library", which is what I'm really after with this change.
In a way, this is a resurection of D91139.
[1]: https://reviews.llvm.org/D91139#2429595
Differential Revision: https://reviews.llvm.org/D134912
This reverts commit d763c6e5e2d0a6b34097aa7dabca31e9aff9b0b6.
Adds the patch by @hans from
https://github.com/llvm/llvm-project/issues/62719
This patch fixes the Windows build.
d763c6e5e2d0a6b34097aa7dabca31e9aff9b0b6 reverted the reviews
D144509 [CMake] Bumps minimum version to 3.20.0.
This partly undoes D137724.
This change has been discussed on discourse
https://discourse.llvm.org/t/rfc-upgrading-llvms-minimum-required-cmake-version/66193
Note this does not remove work-arounds for older CMake versions, that
will be done in followup patches.
D150532 [OpenMP] Compile assembly files as ASM, not C
Since CMake 3.20, CMake explicitly passes "-x c" (or equivalent)
when compiling a file which has been set as having the language
C. This behaviour change only takes place if "cmake_minimum_required"
is set to 3.20 or newer, or if the policy CMP0119 is set to new.
Attempting to compile assembly files with "-x c" fails, however
this is workarounded in many cases, as OpenMP overrides this with
"-x assembler-with-cpp", however this is only added for non-Windows
targets.
Thus, after increasing cmake_minimum_required to 3.20, this breaks
compiling the GNU assembly for Windows targets; the GNU assembly is
used for ARM and AArch64 Windows targets when building with Clang.
This patch unbreaks that.
D150688 [cmake] Set CMP0091 to fix Windows builds after the cmake_minimum_required bump
The build uses other mechanism to select the runtime.
Fixes#62719
Reviewed By: #libc, Mordante
Differential Revision: https://reviews.llvm.org/D151344
This is just to test that the PSTL works with parallelization. This is not supposed to be a production-ready backend.
Reviewed By: ldionne, #libc
Spies: EricWF, arichardson, libcxx-commits
Differential Revision: https://reviews.llvm.org/D150284
This reverts commit 65429b9af6a2c99d340ab2dcddd41dab201f399c.
Broke several projects, see https://reviews.llvm.org/D144509#4347562 onwards.
Also reverts follow-up commit "[OpenMP] Compile assembly files as ASM, not C"
This reverts commit 4072c8aee4c89c4457f4f30d01dc9bb4dfa52559.
Also reverts fix attempt "[cmake] Set CMP0091 to fix Windows builds after the cmake_minimum_required bump"
This reverts commit 7d47dac5f828efd1d378ba44a97559114f00fb64.
The source file is used to anchor the destructor of format_error. When
format is moved from experimental to stable this code would move to the
dylib. One issue with code in the dylib is that it can't be used in
constexpr context. There is a proposal to make format work during
constant evaluation
P2758 Emitting messages at compile time
This paper has initially been received favourable by EWG. Therefore move
the code to the header. This also avoids possible availability issues on
Mac back deployment targets.
Note it is expected that format will no longer be experimental with the
next LLVM release.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D150073
This removes the need for a custom libc++ build to have a basic set of PSTL algorithms.
Reviewed By: ldionne, #libc
Spies: miyuki, libcxx-commits, arichardson
Differential Revision: https://reviews.llvm.org/D149624
We decided to go a different route. To make the switch easier, rip out the old integration first and build on a clean base.
Reviewed By: ldionne, #libc, #libc_abi
Spies: arichardson, libcxx-commits
Differential Revision: https://reviews.llvm.org/D148480
This mostly keeps the same warning flags. The most important exceptions are `-Wpedantic` and `-Wconversion`, which are now removed from libc++abi and libunwind.
Reviewed By: ldionne, #libunwind, #libc, #libc_abi
Spies: mikhail.ramalho, phosek, libcxx-commits
Differential Revision: https://reviews.llvm.org/D144252
Clang wants to enable this flag by default, but libc++ isn't working with it yet.
Reviewed By: Mordante, #libc, #libc_abi, EricWF
Spies: libcxx-commits, arichardson
Differential Revision: https://reviews.llvm.org/D144667
