This patch migrates the CI over to the new compute_projects.py script
for calculating what projects need to be tested based on a change to
LLVM.
Reviewers: lnihlen, ldionne, tstellar, Endilll, joker-eph, Keenuts
Reviewed By: Keenuts, tstellar
Pull Request: https://github.com/llvm/llvm-project/pull/132642
This reverts commit 95d28fe503cc3d2bc0bb980442d3defaf199ea5a.
I did not fully realize the implications of this change when reviewing.
With how it is set up currently, it causes clang and all of the runtimes
to be built and tested everytime a change to MLIR is made. This is a
large regression in build/test time, which seems to have been causing
large queueing delays.
Reverting for now. Once we rework the runtimes build for premerge (which
I hope to do soon, ideally in the next week), I will make sure flang-rt
gets added in.
Flang's runtime can now be built using LLVM's LLVM_ENABLE_RUNTIMES
mechanism, with the intent to remove the old mechanism in #124126.
Update the pre-merge builders to use the new mechanism.
In the current form, #124126 actually will add
LLVM_ENABLE_RUNTIMES=flang-rt implicitly, so no change is strictly
needed. I still think it is a good idea to do it explicitly and in
advance.
On Windows, flang-rt also requires compiler-rt, but which is not
building on Windows anyway.
This patch makes it so that the caller of monolithic-windows.sh can set
the maximum number of parallel compile/link jobs in an environment
variable rather than manually specifying it inside of the CMake.
Additionally, the env variable definitions for CC, CXX, and LD are sunk
into the shell script due to those config options being pretty inherent
to what the pipeline is testing.
This is intended to make things more flexible/useable for the new
premerge CI pipeline, particularly as we are looking at using larger
runners and want the increased flexibility to experiment.
This patch refactors some common functionality present in the CI scripts
to a separate shell script. This is mainly intended to make it easier to
reuse this functionality inside of a Github Actions pipeline as we make
the switch.
In this patch I'm using a new lit option so that the pipeline writes
many results files, one for each time lit is run:
```
--use-unique-output-file-name
When enabled, lit will add a unique element to the output file name, before the extension. For example "results.xml" will become "results.<something>.xml". The
"<something>" is not ordered in any way and is chosen so that existing files are not overwritten. [Default: Off]
```
(I added this to lit recently)
Alternatives were considered:
* mkfifo - does not work on bash for Windows.
* tail -f - does not print full content on file truncation
* lit wrapper script - more complication than using an option to lit
itself
* ninja/mv file/ninja/mv file etc - lots of changes needed to make the
scripts build each target separately
And after feedback I decided that using an option to lit itself is the
cleanest way to go. It can be removed when we no longer need it.
If I run the Linux build after this change:
```
$ bash ./.ci/monolithic-linux.sh "clang;lldb;lld" "check-lldb-shell check-lld" "libcxx;libcxxabi" "check-libcxx check-libcxxabi"
```
I get multiple test result files. In my case some tests fail so runtimes
aren't checked, but all projects are so there is 1 file for lldb and one
for lld:
```
$ ls build/*.xml
build/test-results.klc82utf.xml build/test-results.majylh73.xml
```
This change just collects the XML files as artifacts. Once I know that's
working, I can set up test reporting to make a summary of them.
Fixes#110265
Adding check-all causes us to run some tests twice if a project specific
target like check-clang is also added.
check-pstl is an alternative but as far as I can tell, check-all does
not include this so we have not been running the tests in CI anyway.
When I tried to run check-pstl locally I got a lot of compiler errors
but have not found any instructions on how to setup a correct build
environment. Even if such instructions exist, it's probably more than we
want to do in CI.
According to Louis Dionne, the project is probably not active. So if
it's ever revived it'll be up to the new contributors to enable testing.
Instead of "check-all" which leads to us running some tests twice if
there are other "check-..." targets. For example on one of my PRs this
script produced:
```
commands:
- './.ci/monolithic-linux.sh "clang;clang;lld;clang-tools-extra;compiler-rt;llvm" "check-all check-clang check-clang-tools" "libcxx;libcxxabi;libunwind" "check-cxx check-cxxabi check-unwind"'
commands:
- 'C:\BuildTools\Common7\Tools\VsDevCmd.bat -arch=amd64 -host_arch=amd64'
- 'bash .ci/monolithic-windows.sh "clang;clang-tools-extra;llvm" "check-clang check-clang-tools"'
```
Which meant that Linux ran the clang and clang-tools tests twice. These
extra tests were about 24% of the test run and increased testing time
(on my local machine) by 45%.
This problem can also happen with other projects but there isn't a
simple fix like this one at the moment.
* pstl has a check-pstl target but it is not part of check-all and when
I tried it locally I couldn't build it.
* libclc has no check- target.
I will deal with those projects later.
This is a temporary measure to alleviate Linux pre-commit CI waiting
times that started snowballing
[recently](https://discourse.llvm.org/t/long-wait-for-linux-presubmit-testing/79547/5).
My [initial
estimate](https://github.com/llvm/llvm-project/pull/94208#issuecomment-2155972973)
of 4 additional minutes spent per built seems to be in the right
ballpark, but looks like that was the last straw to break camel's back.
It seems that CI load got past the tipping point, and now it's not able
to burn through the queue over the night on workdays.
I don't intend to overthrow the consensus we reached in #94208, but it
shouldn't come at the expense of the whole LLVM community. I'll enable
this back as soon as we have news that we got more capacity for Linux
pre-commit CI.
This patch removes LLDB from a list of projects that are excluded from
building and testing on pre-merge CI on Linux.
Windows environment needs to be prepared in order to test LLDB
(https://github.com/llvm/llvm-project/pull/94208#issuecomment-2146256857),
but we don't have enough maintenance resources to do that at the moment.
Because LLDB has been in the list of projects that need to be tested on
Clang changes, this PR make this happen on Linux. This seems to be the
consensus in the discussion of this PR.
Flang triggers some OOM on Windows CI right now. This is disruptive to
MLIR and LLVM changes that don't touch Flang, as such we disable
building Flang on Windows only for these PR that don't touch flang. The
testing on Linux is unchanged, and the post-merge Windows testing is
still fully covering here.
Following the discussion in
https://github.com/llvm/llvm-project/pull/93233#issuecomment-2127920882,
this patch merges `clang-ci` pipeline into main `GitHub Pull Requests`
pipeline. `clang-ci` enables additional test coverage for Clang by
compiling it, and then using it to compile and test libc++, libc++abi,
and libunwind in C++03, C++26, and Clang Modules modes.
Additional work we skip and total time savings we should see:
1. Checking out the repo to generate the clang-ci pipeline (2 minutes)
2. Building Clang (3.5 minutes)
3. Uploading the artifacts once, then downloading them 3 times and
unpacking 3 times (0.5 minutes)
Note that because previously-split jobs for each mode are now under a
single Linux job, it now takes around 8 minutes more see the Linux CI
results despite total time savings.
The primary goal of this patch is to reduce the load of CI by removing
duplicated work. I consider this goal achieved. I could keep the job
parallelism we had (3 libc++ jobs depending on a main Linux job), but I
don't consider it worth the effort and opportunity cost, because
parallelism is not helping once the pool of builders is fully
subscribed.
Clang has a custom separate pipeline integrated with libc++ that only
runs in release mode. It means that changes which touches only clang
won't run the clang tests in the configuration used by LLVM premerge and
will break it unknowingly.
We should diff against the base branch, not always against `main`. This
allows the BuildKite pre-commit CI to work properly when we target other
branches, such as `release/18.x`.
we tried to generate a full diff against main in
ec9d80e but it resulted in wrong diffs.
It seems that the issue was that 'main' was not
updated after agent restart and diff main...HEAD kept growing.
Not enabling diff main...HEAD just yet and will check logs for new PRs
first.
Since we moved to Github PRs, the workflow has changed a bit and folks
often merge `main` back into their PR branch. This is fine, except the
previous way of determining modified files for pre-commit CI would use
the content modified just in the latest commit, whatever it is. This
means that in case someone merged main back into their PR branch, we'd
think that the files in the merge commit were modified by the PR, and
we'd spuriously trigger a CI run. This should fix this issue.
The downside is that the merge target is hardcoded to `main`, which
might not always be what we want. I still think this is an improvement
over the status quo.
- fixed build for linux (clang was missing)
- removed /monolithic-.. from build directory - it does not add anything
and makes path longer for windows which is not great;
- added env-based configuration to control cache and agent targeting;
- print (s)ccache stats to file not to pullute normal log.
This basically inlines the logic that was previously located in
https://github.com/google/llvm-premerge-checks so it is part of
the monorepo. This has the benefit of making it extremely easy
for individual projects to understand and modify this logic for
their own needs, unlike the current model where this logic lives
in a separate non-LLVM repository.
It also allows testing changes to the CI configuration using a simple
Phabricator review, since the code that defines the CI pipeline is taken
from the patch under review.
This (or something equivalent) is necessary if we want to retain the
current monolithic pre-commit CI throughout the GitHub PR transition.
Since triggering the monolithic CI is currently attached to the system
we use for triggering CI pipelines from Phabricator, we will lose that
part of the CI when we move to GitHub PRs if we don't do anything.
I've decided to rewrite the code as a shell script because the logic
was fairly simple and it seemed a lot easier than figuring out how to
pull only the relevant parts of llvm-premerge-checks into the monorepo.
Furthermore, I think we should strive to move away from the monolithic
CI altogether since sub-projects should understand, own and maintain the
tests that are relevant for them to run in the CI (with LLVM providing
the infrastructure). Hence, this is somewhat of a temporary solution
until monolithic CI is removed entirely.
Differential Revision: https://reviews.llvm.org/D158863
This simply moves existing CI-related scripts for Clang outside of libc++
to make the two projects independent with respect to CI.
The reason for making this change is that libc++'s CI pipeline history
was full of Clang runs, making it a lot harder to figure out information
about libc++'s own jobs. I created a pipeline for Clang and this patch
will run pre-commit CI for Clang inside that pipeline instead.
Differential Revision: https://reviews.llvm.org/D155078
