Previously, since response (.rsp) files weren't expanded at the very
beginning of clang-scan-deps, we only parsed the command-line as
provided in the Clang .cdb file. Unfortunately, when using Unreal
Engine, arguments are always generated in a .rsp file (ie.
`/path/to/clang-cl.exe @/path/to/filename_args.rsp`).
After this patch, `/Fo` can be parsed and added to the final
command-line. Without this option, the make targets that are emitted are
made up from the input file name alone. We have some cases where the
same input in the project generates several output files, so we end up
with duplicate make targets in the scan-deps emitted dependency file.
This makes it possible to pass "-o /dev/null" to `clang-scan-deps` and
skip some potentially expensive work, making timings less noisy. Also
removes the need for stream redirection.
The `clang-scan-deps` tool can be used for fast scanning of batches of
compilation commands passed in via the `-compilation-database` option.
This gets awkward in our tests where we have to resort to using
`.in`/`.template` JSON files and running them through `sed` in order to
embed LIT's `%t` variable into them. However, most of our tests only
need to pass single compilation command, so this dance is entirely
unnecessary.
This patch makes sure the existing "per-file" mode (where the
compilation command is passed in-line after the `--` argument) works for
all output formats, not only `P1689`.
I suddenly found that the clang scan deps may use all concurrent threads
to scan the files. It makes sense in the batch mode. But in P1689 per
file mode, it simply wastes times and resources.
This patch itself should be a NFC patch. It simply moves codes.
The base class llvm::ThreadPoolInterface will be renamed
llvm::ThreadPool in a subsequent commit.
This is a breaking change: clients who use to create a ThreadPool must
now create a DefaultThreadPool instead.
Canonicalize `-D` and `-U` flags by sorting them and only keeping the
last instance of a given name.
This optimization will only fire if all `-D` and `-U` flags start with a
simple identifier that we can guarantee a simple analysis of can
determine if two flags refer to the same identifier or not. See the
comment on `getSimpleMacroName()` for details of what the issues are.
Previous version of this had issues with sed differences between macOS,
Linux, and Windows. This test doesn't check paths, so just don't run
sed.
Other tests should use `sed -E 's:\\\\?:/:g'` to get portable behavior.
Windows has different command line parsing behavior than Linux for
compilation databases, so the test has been adjusted to ignore that
difference.
Canonicalize `-D` and `-U` flags by sorting them and only keeping the
last instance of a given name.
This optimization will only fire if all `-D` and `-U` flags start with a
simple identifier that we can guarantee a simple analysis of can
determine if two flags refer to the same identifier or not. See the
comment on `getSimpleMacroName()` for details of what the issues are.
`-ivfsoverlay` files are unused when building most modules. Enable
removing them by,
* adding a way to visit the filesystem tree with extensible RTTI to
access each `RedirectingFileSystem`.
* Adding tracking to `RedirectingFileSystem` to record when it
actually redirects a file access.
* Storing this information in each PCM.
Usage tracking is only enabled when iterating over the source manager
and affecting modulemaps. Here each path is stated to cause an access.
During scanning these stats all hit the cache.
Previously, some tools such as `clang` or `lld` which require strict
order for certain command-line options, such as `clang -cc1` or `lld
-flavor`, would not longer work on Windows, when these tools were linked
as part of `llvm-driver`. This was caused by `InitLLVM` which was part
of the `*_main()` function of these tools, which in turn calls
`windows::GetCommandLineArguments`. That function completly replaces
argc/argv by new UTF-8 contents, so any ajustements to argc/argv made by
`llvm-driver` prior to calling these tools was reset.
`InitLLVM` is now called by the `llvm-driver`. Any tool that
participates in (or is part of) the `llvm-driver` doesn't call
`InitLLVM` anymore.
This patch replaces uses of StringRef::{starts,ends}with with
StringRef::{starts,ends}_with for consistency with
std::{string,string_view}::{starts,ends}_with in C++20.
I'm planning to deprecate and eventually remove
StringRef::{starts,ends}with.
Make it easier to control which optimizations are enabled by making
OptimizeArgs a bit masked enum. There's currently only one such
optimization, but more will be added in followup commits.
This patch makes the generation of command lines for modular
dependencies lazy/on-demand. That operation is somewhat expensive and
prior to this patch used to be performed multiple times for the
identical `ModuleDeps` (i.e. when they were imported from multiple
different TUs).
D152495 makes clang warn on unused variables that are declared in conditions like `if (int var = init) {}`
This patch is an NFC fix to suppress the new warning in llvm,clang,lld builds to pass CI in the above patch.
Differential Revision: https://reviews.llvm.org/D158016
All command-line tools using `llvm::opt` create an enum of option IDs and a table of `OptTable::Info` object. Most of the tools use the same ID (`OPT_##ID`), kind (`Option::KIND##Class`), group ID (`OPT_##GROUP`) and alias ID (`OPT_##ALIAS`). This patch extracts that common code into canonical macros. This results in fewer changes when tweaking the `OPTION` macros emitted by the TableGen backend.
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D157028
Scanner's C++ API accepts a set of modular dependencies the client has already seen and for which it doesn't need the full details. This is currently a set of strings, which somewhat implies that it should contain the set of module names. However, scanner internally expects the values to be in the format "{hash}{name}". Besides not being documented, this is very unintuitive. This patch makes this expectation explicit by changing the type to set of `ModuleID`.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D156492
I once again stumbled across `IndexedModuleID` and was confused by the mutable `InputIndex` field.
This patch adds comments around that piece of code and unifies/simplifies related functions.
Reviewed By: Bigcheese
Differential Revision: https://reviews.llvm.org/D145197
This patch adds new `-print-timing` option to `clang-scan-deps`. It measures the wall and process time taken to scan dependencies for the compilation database. This provides more representative data compared to measuring the timing for the whole tool invocation, since that includes parsing and generating JSON files, which can be significant for larger inputs.
Reviewed By: akyrtzi
Differential Revision: https://reviews.llvm.org/D147815
This patch simplifies the loop inside each worker by extracting index retrieval into a lambda function.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D145101
This patch makes sure the ordering of TUs in the input CDB is preserved in the full output. Previously, the results were pushed into a global vector, potentially out-of-order and then sorted by the input file name. This is non-deterministic when the CDB contains multiple entries with the same input file. This patch fixes that by pre-allocating the output vector and writing directly to the position corresponding to the current input. This also eliminates one critical section.
Reviewed By: benlangmuir
Differential Revision: https://reviews.llvm.org/D145098
Required in https://reviews.llvm.org/D137534.
The build systems needs the information to know that "header X changed,
scanning may have changed, so please rerun scanning". Although it is
possible to get the information by running clang-scan-deps for the
second time with make format, it is not user friendly clearly.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D139168
In a private chat with @ben.boeckel , we get in consensus it would be
great for cmake if the invocation of clang-scan-deps can get rid of
compilation database. Due to the compilation database can't do very well
for the files which are not existed yet. @ben.boeckel may have more
context to add here.
This patch should be innocent for others usages.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D137534
The forwarding header is left in place because of its use in
`polly/lib/External/isl/interface/extract_interface.cc`, but I have
added a GCC warning about the fact it is deprecated, because it is used
in `isl` from where it is included by Polly.
This reverts commit e1354763b6e641e45cc2371270883fcd26edf705.
There is a build failure in m68k-linux testing bot
(https://lab.llvm.org/buildbot/#/builders/192/builds/267), which is
weird. Revert this for now and look at the reasons.
Required in https://reviews.llvm.org/D137534.
The build systems needs the information to know that "header X changed,
scanning may have changed, so please rerun scanning". Although it is
possible to get the information by running clang-scan-deps for the
second time with make format, it is not user friendly clearly.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D139168
In a private chat with @ben.boeckel , we get in consensus it would be
great for cmake if the invocation of clang-scan-deps can get rid of
compilation database. Due to the compilation database can't do very well
for the files which are not existed yet. @ben.boeckel may have more
context to add here.
This patch should be innocent for others usages.
Reviewed By: jansvoboda11
Differential Revision: https://reviews.llvm.org/D137534
When reseting modular options, propagate the values from certain options
that have ImpliedBy relations instead of setting to the default. Also,
verify in clang-scan-deps that the command line produced round trips
exactly.
Ideally we would automatically derive the set of options that need this
kind of propagation, but for now there aren't very many impacted.
rdar://105148590
Differential Revision: https://reviews.llvm.org/D143446
D140176 introduced new `FullDeps` API that's not thread-safe, breaking the class invariant. This was causing race condition when `clang-scan-deps` was run with multiple threads.
Reviewed By: steven_wu, akyrtzi
Differential Revision: https://reviews.llvm.org/D143428
For users of the C++ API, the return type of `getFullDependencies` doesn't make sense when asking for dependencies of a module. In the returned `FullDependenciesResult` instance, only `DiscoveredModules` is useful (the graph of modular dependecies). The `FullDeps` member is trying to describe a translation unit it was never given. Its command line also refers to a file in the in-memory VFS we create in the scanner, leaking the implementation detail.
This patch splits the API and improves layering and naming of the return types.
Depends on D140175.
Reviewed By: artemcm
Differential Revision: https://reviews.llvm.org/D140176
This API is no longer necessary, so let's remove it to simplify the internal APIs.
Reviewed By: benlangmuir, artemcm
Differential Revision: https://reviews.llvm.org/D140175
