Since this standalone build configuration uses the runtime libraries that
are being built just now, we need to ensure that e.g. the TSan unit tests
depend on the tsan runtime library. Also fix TSAN_DEPS being overridden
to not include the tsan runtime (commit .....).
This change fixes a build race seen in the CI checks for
TsanRtlTest-x86_64-Test in https://github.com/llvm/llvm-project/pull/83088.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83650
When building optimized versions of the runtime libraries the compiler
is generally able to elide these references, but when building them
for maximum debug info (with -O0), these references remain which causes
the test suite to fail for tests that do not pull in the C++ standard
library.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/84613
Some buildbots (e.g.,
https://lab.llvm.org/buildbot/#/builders/18/builds/16061/steps/10/logs/stdio)
have recently started complaining about
```
cast from 'SignalHandlerType' (aka 'void (*)(int, void *, void *)') to 'sa_sigaction_t' (aka 'void (*)(int, siginfo_t *, void *)') converts to incompatible function type [-Werror,-Wcast-function-type-strict]
219 | sigact.sa_sigaction = (sa_sigaction_t)handler;
```
This patch does an intermediate cast to `'(void (*) (void))'` to suppress the warning.
N.B. SignalHandlerType has `'void*'` instead of `'siginfo_t*'` because it is typedef'ed in sanitizer_common/sanitizer_common.h, which does not have access to the header (signal.h) that defines siginfo_t; we therefore cannot fix SignalHandlerType.
DFSan's shadow mappings are incompatible with 32 bits of ASLR entropy
('sudo sysctl vm.mmap_rnd_bits=32; ninja check-dfsan') and it is
difficult to fix this via increasing the size of the shadow mappings,
due to the overhead of shadow memory. This patch works around the issue
by detecting if the memory layout is incompatible, and if so,
re-exec'ing without ASLR.
DFSan and MSan share copy-pasted shadow memory code, hence this
workaround is ported from MSan:
- "[msan] Re-exec with no ASLR if memory layout is incompatible on
Linux"
(58f7251820)
- "[msan] Add 'MappingDesc::ALLOCATOR' type and check it is available"
(af2bf86a37)
(which in turn are inspired by TSan: "Re-exec TSan with no ASLR if
memory layout is incompatible on Linux"
(0784b1eefa
))
aeubanks had remarked in
https://github.com/llvm/llvm-project/pull/85142#issuecomment-2004442883
that this issue occurs in Chromium:
https://ci.chromium.org/ui/p/chromium/builders/try/linux_upload_clang/5066/overview
memprof often fails when ASLR entropy is too high ('sudo sysctl
vm.mmap_rnd_bits=32; ninja check-memprof'), which is the default setting
for newer versions of Ubuntu
(https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=hwe-6.5-next--2024.03.04-1--auto&id=6b522637c6a7dabd8530026ae933fb5ff17e877f).
This patch fixes the issue by moving the allocator base, analogously to
ASan (https://reviews.llvm.org/D148280).
Explanation from the ASan patch: when CONFIG_ARCH_MMAP_RND_BITS == 32,
it will frequently conflict with memprof's allocator, because the PIE
program segment base address of 0x555555555554 plus an ASLR shift of up
to ((2**32) * 4K == 0x100000000000) will sometimes exceed memprof's
hardcoded base address of 0x600000000000. We fix this by simply moving
the allocator base to 0x500000000000, which is below the PIE program
segment base address. This is cleaner than trying to move it to another
location that is sandwiched between the PIE program and library
segments, because if either of those grow too large, it will collide
with the allocator region.
Note that we will never need to change this base address again (unless
we want to increase the size of the allocator), because ASLR cannot be
set above 32-bits for x86-64 Linux (the PIE program segment and library
segments would collide with each other; see also ARCH_MMAP_RND_BITS_MAX
in https://github.com/torvalds/linux/blob/master/arch/x86/Kconfig).
With https://github.com/llvm/llvm-project/pull/83088, we now need the
runtimes to be built before running test if
COMPILER_RT_TEST_STANDALONE_BUILD_LIBS is true, since otherwise we
get failures running `ninja check-all` such as the following:
```
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.fuzzer-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-basic-x86_64.a: No such file or directory
/usr/bin/ld: cannot find .../compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.xray-fdr-x86_64.a: No such file or directory
```
This is a follow-up to 058e9b03 which started removing these checks
and it should make it easier to stop forcing COMPILER_RT_STANDALONE_BUILD
for runtimes builds in the future.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83651
this DCHECK was not valid for hwasan_load1, and was not necessary for
the. the function is written without any assumptions of alignment of the
pointer.
Updated the corelimit.cpp test to handle buildbots with RLIMIT_CORE
rlim_max already set to zero.
Original commit message:
I was trying to debug why `ninja check-compiler-rt` was taking so long
to run on my system and after some debugging it turned out that most of
the time was being spent generating core dumps.
On many current Linux systems, coredumps are no longer dumped in the CWD
but instead piped to a utility such as systemd-coredumpd that stores
them in a deterministic location. This can be done by setting the
kernel.core_pattern sysctl to start with a '|'. However, when using such
a setup the kernel ignores a coredump limit of 0 (since there is no file
being written) and we can end up piping many gigabytes of data to
systemd-coredumpd which causes the test suite to freeze for a long time.
While most piped coredump handlers do respect the crashing processes'
RLIMIT_CORE, this is notable not the case for Debian's systemd-coredump
due to a local patch that changes sysctl.d/50-coredump.conf to ignore
the specified limit and instead use RLIM_INFINITY
(https://salsa.debian.org/systemd-team/systemd/-/commit/64599ffe44f0d).
Fortunately there is a workaround: the kernel recognizes the magic value
of 1 for RLIMIT_CORE to disable coredumps when piping. One byte is also
too small to generate any coredump, so it effectively behaves as if we
had set the value to zero.
The alternative to using RLIMIT_CORE=1 would be to use prctl() with the
PR_SET_DUMPABLE flag, however that also prevents ptrace(), so makes it
impossible to attach a debugger.
Fixes: https://github.com/llvm/llvm-project/issues/45797
This reverts commit 0b9f19a9880eb786871194af116f223d2ad30c52.
Contains test changes from
https://github.com/llvm/llvm-project/pull/81677 that are simple test
changes. For the most part just makes the tests allow more flexibility
in the callstacks produced by asan.
Note: this PR has the exact changes from
https://github.com/llvm/llvm-project/pull/81677 in addition to a revert
commit to remove the ones that require other things from that PR. This
will be squashed, ofc. I just left things unsquashed for reviewing
pleasure :).
This is a non-functional-change
This ports the change from TSan
(0784b1eefa).
Testing notes: run 'sudo sysctl vm.mmap_rnd_bits=32; ninja check-msan'
before and after this patch.
N.B. aggressive ASLR may also cause the app to overlap with the
allocator region; for MSan, this was fixed in
af2bf86a37
Blocking that signal causes inter-blocking for profilers that monitor
threads through that signal.
Update tests accordingly to use an uncaught signal.
This is a recommit of 6f3f659ce9ab91002b4a490b0ce4b085981383cd with the
tests fixed.
Fix#83844 and #83561
MSan divides the virtual address space into APP, INVALID, SHADOW and
ORIGIN memory. The allocator usually just steals a bit of the APP
address space: typically the bottom portion of the PIE binaries section,
which works because the Linux kernel maps from the top of the PIE
binaries section. However, if ASLR is very aggressive, the binary may
end up mapped in the same location where the allocator wants to live;
this results in a segfault.
This patch adds in a MappingDesc::ALLOCATOR type and enforces that the
memory range for the allocator is not occupied by anything else.
Since the allocator range information is not readily available in
msan.h, we duplicate the information from msan_allocator.cpp.
Note: aggressive ASLR can also lead to a different type of failure,
where the PIE binaries/libraries are mapped entirely outside of the
APP/ALLOCATOR sections; that will be addressed in a separate patch
(https://github.com/llvm/llvm-project/pull/85142).
Reapply "[sanitizer][asan][win] Intercept _strdup on Windows instead of
strdup
This includes test changes and interface changes that are duplicated in
https://github.com/llvm/llvm-project/pull/81677
This reverts commit 03dc87e93937bf25a48bc77d0006c59f37b1855d.
Instead of explicitly disabling a feature by declaring the variable and
set it to false, this change supports the optional flags. I.e., you can
skip certain flags if you are not using it.
This optional feature supports both forms,
1. Value: A parameter for a feature. E.g., EnableRandomOffset
2. Type: A C++ type implementing a feature. E.g., ConditionVariableT
On the other hand, to access the flags will be through one of the
wrappers, BaseConfig/PrimaryConfig/SecondaryConfig/CacheConfig
(CacheConfig is embedded in SecondaryConfig). These wrappers have the
getters to access the value and the type. When adding a new feature, we
need to add it to `allocator_config.def` and mark the new variable with
either *_REQUIRED_* or *_OPTIONAL_* macro so that the accessor will be
generated properly.
In addition, also remove the need of `UseConditionVariable` to flip
on/off of condition variable. Now we only need to define the type of
condition variable.
I was trying to debug why `ninja check-compiler-rt` was taking so long
to run on my system and after some debugging it turned out that most of
the time was being spent generating core dumps.
On many current Linux systems, coredumps are no longer dumped in the CWD
but instead piped to a utility such as systemd-coredumpd that stores
them in a deterministic location. This can be done by setting the
kernel.core_pattern sysctl to start with a '|'. However, when using such
a setup the kernel ignores a coredump limit of 0 (since there is no file
being written) and we can end up piping many gigabytes of data to
systemd-coredumpd which causes the test suite to freeze for a long time.
While most piped coredump handlers do respect the crashing processes'
RLIMIT_CORE, this is notable not the case for Debian's systemd-coredump
due to a local patch that changes sysctl.d/50-coredump.conf to ignore
the specified limit and instead use RLIM_INFINITY
(https://salsa.debian.org/systemd-team/systemd/-/commit/64599ffe44f0d).
Fortunately there is a workaround: the kernel recognizes the magic value
of 1 for RLIMIT_CORE to disable coredumps when piping. One byte is also
too small to generate any coredump, so it effectively behaves as if we
had set the value to zero.
The alternative to using RLIMIT_CORE=1 would be to use prctl() with the
PR_SET_DUMPABLE flag, however that also prevents ptrace(), so makes it
impossible to attach a debugger.
Fixes: https://github.com/llvm/llvm-project/issues/45797
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83701
All these unit tests already include ${COMPILER_RT_GTEST_SOURCE} as an
input source file and the target llvm_gtest does not exist for
standalone builds. Currently the DEPS argument is ignored for standalone
builds so the missing target is not a problem, but as part of fixing a
build race for standalone builds I am planning to include those
dependencies in COMPILER_RT_TEST_STANDALONE_BUILD_LIBS configurations.
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83649
On AArch64 with BTI, we have to start functions with the appropriate
BTI hint to indicate that the function is a valid call target.
To support interceptors with AArch64 BTI, add "BTI c".
Most sanitizers don't support static linking. One primary reason is the
incompatibility with interceptors. `GetTlsSize` is another reason.
asan/memprof use `__interception::DoesNotSupportStaticLinking`
(`_DYNAMIC` reference) to reject -static at link time. Port this
detector to other sanitizers. dfsan actually supports -static for
certain cases. Don't touch dfsan.
`__tsan::Acquire()`, which is called by `__tsan_acquire()`, has a
performance optimization which attempts to avoid acquiring the atomic
variable's mutex if the variable has no associated memory model state.
However, if the atomic variable was recently written to by a
`compare_exchange_weak/strong` on another thread, the memory model state
may be created *after* the atomic variable is updated. This is a data
race, and can cause the thread calling `Acquire()` to not realize that
the atomic variable was previously written to by another thread.
Specifically, if you have code that writes to an atomic variable using
`compare_exchange_weak/strong`, and then in another thread you read the
value using a relaxed load, followed by an
`atomic_thread_fence(memory_order_acquire)`, followed by a call to
`__tsan_acquire()`, TSAN may not realize that the store happened before
the fence, and so it will complain about any other variables you access
from both threads if the thread-safety of those accesses depended on the
happens-before relationship between the store and the fence.
This change eliminates the unsafe optimization in `Acquire()`. Now,
`Acquire()` acquires the mutex before checking for the existence of the
memory model state.
If we're in a blocking call, we need to run the signal immediately, as
the call may not return for a very long time (if ever). Not running the
handler can cause deadlocks if the rest of the program waits (in one way
or another) for the signal handler to execute.
I've gone through the list of functions in
sanitizer_common_interceptors and marked as blocking those that I know
can block, but I don't claim the list to be exhaustive. In particular, I
did not mark libc FILE* functions as blocking, because these can end up
calling user functions. To do that correctly, /I think/ it would be
necessary to clear the "is in blocking call" flag inside the fopencookie
wrappers.
The test for the bug (deadlock) uses the read call (which is the one
that I ran into originally), but the same kind of test could be written
for any other blocking syscall.
This can be useful because dlsym() may call malloc on failure which could
result in other interposed functions being called that could eventually
make use of TLS. While the crash that I experienced originally has been
fixed differently (by not using global-dynamic TLS accesses in the mutex
deadlock detector, see https://github.com/llvm/llvm-project/pull/83890),
moving this interception earlier is still a good since it makes the code
a bit more robust against initialization order problems.
Reviewed By: MaskRay, vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83886
TestCases/Misc/Linux/sigaction.cpp fails because dlsym() may call malloc
on failure. And then the wrapped malloc appears to access thread local
storage using global dynamic accesses, thus calling
___interceptor___tls_get_addr, before REAL(__tls_get_addr) has
been set, so we get a crash inside ___interceptor___tls_get_addr. For
example, this can happen when looking up __isoc23_scanf which might not
exist in some libcs.
Fix this by marking the thread local variable accessed inside the
debug checks as "initial-exec", which does not require __tls_get_addr.
This is probably a better alternative to https://github.com/llvm/llvm-project/pull/83886.
This fixes a different crash but is related to https://github.com/llvm/llvm-project/issues/46204.
Backtrace:
```
#0 0x0000000000000000 in ?? ()
#1 0x00007ffff6a9d89e in ___interceptor___tls_get_addr (arg=0x7ffff6b27be8) at /path/to/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2759
#2 0x00007ffff6a46bc6 in __sanitizer::CheckedMutex::LockImpl (this=0x7ffff6b27be8, pc=140737331846066) at /path/to/llvm/compiler-rt/lib/sanitizer_common/sanitizer_mutex.cpp:218
#3 0x00007ffff6a448b2 in __sanitizer::CheckedMutex::Lock (this=0x7ffff6b27be8, this@entry=0x730000000580) at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_mutex.h:129
#4 __sanitizer::Mutex::Lock (this=0x7ffff6b27be8, this@entry=0x730000000580) at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_mutex.h:167
#5 0x00007ffff6abdbb2 in __sanitizer::GenericScopedLock<__sanitizer::Mutex>::GenericScopedLock (mu=0x730000000580, this=<optimized out>) at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_mutex.h:383
#6 __sanitizer::SizeClassAllocator64<__tsan::AP64>::GetFromAllocator (this=0x7ffff7487dc0 <__tsan::allocator_placeholder>, stat=stat@entry=0x7ffff570db68, class_id=11, chunks=chunks@entry=0x7ffff5702cc8, n_chunks=n_chunks@entry=128) at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_primary64.h:207
#7 0x00007ffff6abdaa0 in __sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >::Refill (this=<optimized out>, c=c@entry=0x7ffff5702cb8, allocator=<optimized out>, class_id=<optimized out>)
at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_local_cache.h:103
#8 0x00007ffff6abd731 in __sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >::Allocate (this=0x7ffff6b27be8, allocator=0x7ffff5702cc8, class_id=140737311157448)
at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_local_cache.h:39
#9 0x00007ffff6abc397 in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__tsan::AP64>, __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Allocate (this=0x7ffff5702cc8, cache=0x7ffff6b27be8, size=<optimized out>, size@entry=175, alignment=alignment@entry=16)
at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_combined.h:69
#10 0x00007ffff6abaa6a in __tsan::user_alloc_internal (thr=0x7ffff7ebd980, pc=140737331499943, sz=sz@entry=175, align=align@entry=16, signal=true) at /path/to/llvm/compiler-rt/lib/tsan/rtl/tsan_mman.cpp:198
#11 0x00007ffff6abb0d1 in __tsan::user_alloc (thr=0x7ffff6b27be8, pc=140737331846066, sz=11, sz@entry=175) at /path/to/llvm/compiler-rt/lib/tsan/rtl/tsan_mman.cpp:223
#12 0x00007ffff6a693b5 in ___interceptor_malloc (size=175) at /path/to/llvm/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:666
#13 0x00007ffff7fce7f2 in malloc (size=175) at ../include/rtld-malloc.h:56
#14 __GI__dl_exception_create_format (exception=exception@entry=0x7fffffffd0d0, objname=0x7ffff7fc3550 "/path/to/llvm/compiler-rt/cmake-build-all-sanitizers/lib/linux/libclang_rt.tsan-x86_64.so",
fmt=fmt@entry=0x7ffff7ff2db9 "undefined symbol: %s%s%s") at ./elf/dl-exception.c:157
#15 0x00007ffff7fd50e8 in _dl_lookup_symbol_x (undef_name=0x7ffff6af868b "__isoc23_scanf", undef_map=<optimized out>, ref=0x7fffffffd148, symbol_scope=<optimized out>, version=<optimized out>, type_class=0, flags=2, skip_map=0x7ffff7fc35e0) at ./elf/dl-lookup.c:793
--Type <RET> for more, q to quit, c to continue without paging--
#16 0x00007ffff656d6ed in do_sym (handle=<optimized out>, name=0x7ffff6af868b "__isoc23_scanf", who=0x7ffff6a3bb84 <__interception::InterceptFunction(char const*, unsigned long*, unsigned long, unsigned long)+36>, vers=vers@entry=0x0, flags=flags@entry=2) at ./elf/dl-sym.c:146
#17 0x00007ffff656d9dd in _dl_sym (handle=<optimized out>, name=<optimized out>, who=<optimized out>) at ./elf/dl-sym.c:195
#18 0x00007ffff64a2854 in dlsym_doit (a=a@entry=0x7fffffffd3b0) at ./dlfcn/dlsym.c:40
#19 0x00007ffff7fcc489 in __GI__dl_catch_exception (exception=exception@entry=0x7fffffffd310, operate=0x7ffff64a2840 <dlsym_doit>, args=0x7fffffffd3b0) at ./elf/dl-catch.c:237
#20 0x00007ffff7fcc5af in _dl_catch_error (objname=0x7fffffffd368, errstring=0x7fffffffd370, mallocedp=0x7fffffffd367, operate=<optimized out>, args=<optimized out>) at ./elf/dl-catch.c:256
#21 0x00007ffff64a2257 in _dlerror_run (operate=operate@entry=0x7ffff64a2840 <dlsym_doit>, args=args@entry=0x7fffffffd3b0) at ./dlfcn/dlerror.c:138
#22 0x00007ffff64a28e5 in dlsym_implementation (dl_caller=<optimized out>, name=<optimized out>, handle=<optimized out>) at ./dlfcn/dlsym.c:54
#23 ___dlsym (handle=<optimized out>, name=<optimized out>) at ./dlfcn/dlsym.c:68
#24 0x00007ffff6a3bb84 in __interception::GetFuncAddr (name=0x7ffff6af868b "__isoc23_scanf", trampoline=140737311157448) at /path/to/llvm/compiler-rt/lib/interception/interception_linux.cpp:42
#25 __interception::InterceptFunction (name=0x7ffff6af868b "__isoc23_scanf", ptr_to_real=0x7ffff74850e8 <__interception::real___isoc23_scanf>, func=11, trampoline=140737311157448)
at /path/to/llvm/compiler-rt/lib/interception/interception_linux.cpp:61
#26 0x00007ffff6a9f2d9 in InitializeCommonInterceptors () at /path/to/llvm/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:10315
```
Reviewed By: vitalybuka, MaskRay
Pull Request: https://github.com/llvm/llvm-project/pull/83890
We can rely on the compiler-provided macro __SIZE_TYPE__ for all
non-MSVC compilers and fall back to `uptr` otherwise.
I verified via https://godbolt.org/z/MW9KMjv5f that this works for MSVC
as well as GCC 4.5 Clang 3.0, so that should cover supported compilers.
While touching this also rename operator_new_size_type to usize which
makes it more obvious that this is the equivalent to size_t within
the sanitizers runtime (which I plan to use in follow-up changes).
Reviewed By: vitalybuka
Pull Request: https://github.com/llvm/llvm-project/pull/83912
