Instead of building the benchmarks separately via CMake and running them
separately from the test suite, this patch merges the benchmarks into
the test suite and handles both uniformly.
As a result:
- It is now possible to run individual benchmarks like we run tests
(e.g. using libcxx-lit), which is a huge quality-of-life improvement.
- The benchmarks will be run under exactly the same configuration as
the rest of the tests, which is a nice simplification. This does
mean that one has to be careful to enable the desired optimization
flags when running benchmarks, but that is easy with e.g.
`libcxx-lit <...> --param optimization=speed`.
- Benchmarks can use the same annotations as the rest of the test
suite, such as `// UNSUPPORTED` & friends.
When running the tests via `check-cxx`, we only compile the benchmarks
because running them would be too time consuming. This introduces a bit
of complexity in the testing setup, and instead it would be better to
allow passing a --dry-run flag to GoogleBenchmark executables, which is
the topic of https://github.com/google/benchmark/issues/1827.
I am not really satisfied with the layering violation of adding the
%{benchmark_flags} substitution to cmake-bridge, however I believe
this can be improved in the future.
- Landing page: add link to the libc++ Discord channel
- Landing page: reorder "Getting Involved" above "Design documents"
- Landing page: remove "Notes and Known Issues" which was completely outdated
- Rename "Using Libc++" to "User Documentation" and update contents
- Rename "Building Libc++" to "Vendor Documentation" and update contents
The "BuildingLibcxx" and "UsingLibcxx" pages have basically been used for
vendor and user documentation respectively. However, they were named in
a way that doesn't really make that clear. Renaming the pages now gives
us a location to clearly document what we target at vendors and what we
target at users, and to do that separately.
This is an intermediate and fairly mechanical step towards unifying the
benchmarks with the rest of the test suite. Moving this around requires
a few changes, notably making sure we don't throw a wrench into the
discovery process of the normal test suite. This won't be a problem
anymore once benchmarks are taken into account by the test setup out of
the box.
There were some ad-hoc settings that allowed running the benchmarks
against the native C++ Standard Library. While this ability is very
useful, it was done before the test suite was quite independent of
libc++ itself. Instead, it is better to streamline running the
benchmarks on the native standard library by using a custom Lit
configuration like we do with the test suite.
A follow-up patch will rework the integration of benchmarks with the Lit
configuration used for the test suite so that we can reuse the same
mechanism for both, making it easy to benchmark the native standard
library.
It will also make benchmarks way more user-friendly to run since we will
be able to run them like we run individual tests, which is a pain point
right now.
Revert "Revert #76246 and #76083"
This reverts commit 5c150e7eeba9db13cc65b329b3c3537b613ae61d.
Adds a small fix that should properly disable the tests on Windows.
Unfortunately the original poster has not provided feedback and the
original patch did not fail in the LLVM CI infrastructure.
Modules are known to fail on Windows due to non compliance of the
C library. Currently not having this patch prevents testing on other
platforms.
These cause test build failures on Windows.
This reverts the following commits:
57ca74843586c9a93c425036c5538aae0a2cfa60
d06ae33ec32122bb526fb35025c1f0cf979f1090
This adds a new module test infrastructure. This requires tagging tests
using modules. The test runner uses this information to determine the
compiler flags needed to build and use the module.
Currently modules are build per test, which allows testing them for
tests with ADDITIONAL_COMPILE_FLAGS. At the moment only 4 tests use
modules. Therefore the performance penalty is not measurable. If in the
future more tests use modules it would be good to measure the overhead
and determine whether it's acceptable.
As a new contributor, I found it hard to find the documentation for the
meaning of the names of different tests and how those names translate to
Lit. This patch moves the documentation to the RST documentation we
publish on the website instead of leaving it in the source code only.
These macros are intended to replace the macros in rapid-cxx-test.h.
Reviewed By: #libc, ldionne
Differential Revision: https://reviews.llvm.org/D142808
The debug mode has been broken pretty much ever since it was shipped
because it was possible to enable the debug mode in user code without
actually enabling it in the dylib, leading to ODR violations that
caused various kinds of failures.
This commit makes the debug mode a knob that is configured when
building the library and which can't be changed afterwards. This is
less flexible for users, however it will actually work as intended
and it will allow us, in the future, to add various kinds of checks
that do not assume the same ABI as the normal library. Furthermore,
this will make the debug mode more robust, which means that vendors
might be more tempted to support it properly, which hasn't been the
case with the current debug mode.
This patch shouldn't break any user code, except folks who are building
against a library that doesn't have the debug mode enabled and who try
to enable the debug mode in their code. Such users will get a compile-time
error explaining that this configuration isn't supported anymore.
In the future, we should further increase the granularity of the debug
mode checks so that we can cherry-pick which checks to enable, like we
do for unspecified behavior randomization.
Differential Revision: https://reviews.llvm.org/D122941
Seems this section is not updated since we have transited to llvm-project monorepo.
At the start, we build libcxx under monorepo configuration but later try to make the separate configuration for libcxx build
and running benchmark.
Reviewed By: ldionne, #libc
Differential Revision: https://reviews.llvm.org/D110722
This fixes a long standing issue where the triple is not always set
consistently in all configurations. This change also moves the
back-deployment Lit features to using the proper target triple
instead of using something ad-hoc.
This will be necessary for using from scratch Lit configuration files
in both normal testing and back-deployment testing.
Differential Revision: https://reviews.llvm.org/D102012
After this patch, we can use `--param std=c++20` even if the compiler only
supports -std=c++2a. The test suite will handle that for us. The only Lit
feature that isn't fully baked will always be the "in development" one,
since we don't know exactly what year the standard will be ratified in.
This is another take on https://reviews.llvm.org/D99789.
Differential Revision: https://reviews.llvm.org/D100210
Prior to this patch, we would generate a fancy <__config> header by
concatenating <__config_site> and <__config>. This complexifies the
build system and also increases the difference between what's tested
and what's actually installed.
This patch removes that complexity and instead simply installs <__config_site>
alongside the libc++ headers. <__config_site> is then included by <__config>,
which is much simpler. Doing this also opens the door to having different
<__config_site> headers depending on the target, which was impossible before.
It does change the workflow for testing header-only changes to libc++.
Previously, we would run `lit` against the headers in libcxx/include.
After this patch, we run it against a fake installation root of the
headers (containing a proper <__config_site> header). This makes use
closer to testing what we actually install, which is good, however it
does mean that we have to update that root before testing header changes.
Thus, we now need to run `ninja check-cxx-deps` before running `lit` by
hand.
Differential Revision: https://reviews.llvm.org/D97572
Several subprojects have targets that do the same thing, and they all
follow the same naming convention: llvm-test-depends, clang-test-depends,
lld-test-depends, etc.
This makes libc++ consistent with other LLVM projects.
Thanks to Duncan Exon Smith for noticing and suggesting the change.
Differential Revision: https://reviews.llvm.org/D94499
This makes us closer to running the test suite on platforms where the
legacy test suite configuration doesn't work.
One notable change after this commit is that the tests will be run with
warnings enabled on GCC too, which wasn't the case before. However,
previous commits should have tweaked the test suite to make sure it
passes with warnings enabled on GCC.
Note that warnings can still be disabled with `--param enable_warnings=False`,
as before.
Differential Revision: https://reviews.llvm.org/D90432
This is a massive revert of the following commits (from most revent to oldest):
2b9b7b5775a1d8fcd7aa5abaa8fc0bc303434f1a.
529ac33197f6408952ae995075ac5e2dc5287e81
28270234f1478047e35879f4ba8838b47edfcc14
69c2087283cf7b17ca75f69daebf4ffc158b754a
b5aa67446e01bd277727b05710a42e69ac41e74b
5d796645d6c8cadeb003715c33e231a8ba05b6de
After checking-in the __config_site change, a lot of things started breaking
due to widespread reliance on various aspects of libc++'s build, notably the
fact that we can include the headers from the source tree, but also reliance
on various "internal" CMake variables used by the runtimes build and compiler-rt.
These were unintended consequences of the change, and after two days, we
still haven't restored all the bots to being green. Instead, now that I
understand what specific areas this will blow up in, I should be able to
chop up the patch into smaller ones that are easier to digest.
See https://reviews.llvm.org/D89041 for more details on this adventure.
Prior to this patch, we would generate a fancy <__config> header by
concatenating <__config_site> and <__config>. This complexifies the
build system and also increases the difference between what's tested
and what's actually installed.
This patch removes that complexity and instead simply installs <__config_site>
alongside the libc++ headers. <__config_site> is then included by <__config>,
which is much simpler. Doing this also opens the door to having different
<__config_site> headers depending on the target, which was impossible before.
It does change the workflow for testing header-only changes to libc++.
Previously, we would run `lit` against the headers in libcxx/include.
After this patch, we run it against a fake installation root of the
headers (containing a proper <__config_site> header). This makes use
closer to testing what we actually install, which is good, however it
does mean that we have to update that root before testing header changes.
Thus, we now need to run `ninja check-cxx-deps` before running `lit` by
hand.
This commit was originally applied in 1e46d1aa3 and reverted in eb60c487
because it broke the libc++abi and libunwind test suites. This has now
been fixed.
Differential Revision: https://reviews.llvm.org/D89041
Prior to this patch, we would generate a fancy <__config> header by
concatenating <__config_site> and <__config>. This complexifies the
build system and also increases the difference between what's tested
and what's actually installed.
This patch removes that complexity and instead simply installs <__config_site>
alongside the libc++ headers. <__config_site> is then included by <__config>,
which is much simpler. Doing this also opens the door to having different
<__config_site> headers depending on the target, which was impossible before.
It does change the workflow for testing header-only changes to libc++.
Previously, we would run `lit` against the headers in libcxx/include.
After this patch, we run it against a fake installation root of the
headers (containing a proper <__config_site> header). This makes use
closer to testing what we actually install, which is good, however it
does mean that we have to update that root before testing header changes.
Thus, we now need to run `ninja check-cxx-deps` before running `lit` by
hand.
Differential Revision: https://reviews.llvm.org/D89041
Before this patch, the libc++ test suite first loads lit.site.cfg
(generated by CMake), and then lit.cfg. It's also possible to load
lit.cfg before lit.site.cfg and to point to a custom lit.site.cfg
file using '--param=libcxx_site_config'. However, in that case, lit.cfg
still relies on the site configuration filling up the 'config' object
like the default lit.site.cfg file does, which isn't flexible enough.
This commit simplifies the setup by having just a single Lit site config
file per CMake configuration, and always loading exactly that config file.
However, the config file to use can be selected when setting up CMake via
the LIBCXX_TEST_CONFIG setting. Furthermore, the site configs are entirely
standalone, which means that a new site config can be added that doesn't
need to conform what's expected by config.py.
Differential Revision: https://reviews.llvm.org/D81846
C++98 and C++03 are effectively aliases as far as Clang is concerned.
As such, allowing both std=c++98 and std=c++03 as Lit parameters is
just slightly confusing, but provides no value. It's similar to allowing
both std=c++17 and std=c++1z, which we don't do.
This was discovered because we had an internal bot that ran the test
suite under both c++98 AND c++03 -- one of which is redundant.
Differential Revision: https://reviews.llvm.org/D80926
This is already handled by setting cxx_runtime_root instead -- I don't
see a reason to have two ways of setting the runtime path of the library
we're running against.
This commit removes minor features of the test suite that I've never
seen used and that are basically just a maintenance burden:
- color_diagnostics: Diagnostics are colored by default when running
from a terminal, and not colored otherwise. This is the right behavior.
Being able to tweak this has minor value, and could be achieved by
modifying the %{compile_flags} instead if absolutely needed.
- ccache: This can be achieved by using a wrapper for the %{cxx}
substitution.
- _dump_macros_verbose is just a dead function now.
Summary:
When I read this on the website it looks like the `--` in the used font turns
into an em dash. I updated this with inline literal mark up so the `--` will
remain obvious.
Reviewers: EricWF, #libc!
Subscribers: libcxx-commits
Tags: #libc
Differential Revision: https://reviews.llvm.org/D78405
When running the tests through `lit` directly instead of through `check-cxx`,
it is required to manually build the `cxx` (and often `cxx_experimental`)
targets. Instead of having to do that manually, this commit adds a new
target `check-cxx-deps` that does that for you.
Our documentation for running LIT is basically wrong, since it doesn't
mention `llvm-lit`, and nothing works without it. Note that this
documentation improvement reflects the current reality outside of
the recent efforts on improving the test suite. My goal is to document
the current way of running the test suite, so that everybody agrees on
how things currently work. As the configuration system for libc++ gets
simplified, we can come back and keep this documentation up to date to
reflect those changes.
Differential Revision: https://reviews.llvm.org/D77673
We've been meaning to remove those targets for a while, and the fix is
simple enough cause they're all just aliases to other targets.
This is a re-application of f383fb40b17, wich was reverted in 04d48111b
because the build bots had not been updated yet. The build bot configurations
have now been updated not to use the deprecated targets, and I verified
that they were using the non-deprecated targets, so we should be good
unless I missed a bot.
Differential Revision: https://reviews.llvm.org/D76104
This reverts commit f383fb40b. It looks like several of our build bots
are still using the legacy target names, so we'll change those before
we commit this change again.
We've been meaning to remove those targets for a while, and the fix is
simple enough cause they're all just aliases to other targets.
There's no doubt this commit will break some CI systems, however the
fix is trivial.
Differential Revision: https://reviews.llvm.org/D76104
We're building tests with -nostdlib which means that we need to
explicitly include the builtins library. When using libgcc (default)
we can simply include -lgcc_s on the link line, but when using
compiler-rt builtins we need a complete path to the builtins library.
This path is already available in CMake as <PROJECT>_BUILTINS_LIBRARY,
so we just need to pass that path to lit and if config.compiler_rt is
true, link it to the test.
Prior to this patch, running tests when compiler-rt is being used as
the builtins library was broken as all tests would fail to link, but
with this change running tests when compiler-rt bultins library is
being used should be supported.
Differential Revision: https://reviews.llvm.org/D56701
llvm-svn: 353208
