2018-08-14 16:03:32 +00:00
|
|
|
//===--- Compiler.h ----------------------------------------------*- C++-*-===//
|
2017-12-04 13:49:59 +00:00
|
|
|
//
|
2019-01-19 08:50:56 +00:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
2017-12-04 13:49:59 +00:00
|
|
|
//
|
2018-08-14 16:03:32 +00:00
|
|
|
//===----------------------------------------------------------------------===//
|
2017-12-04 13:49:59 +00:00
|
|
|
//
|
|
|
|
// Shared utilities for invoking the clang compiler.
|
2019-09-04 07:35:00 +00:00
|
|
|
// Most callers will use this through Preamble/ParsedAST, but some features like
|
|
|
|
// CodeComplete run their own compile actions that share these low-level pieces.
|
2017-12-04 13:49:59 +00:00
|
|
|
//
|
2018-08-14 16:03:32 +00:00
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
2017-12-04 13:49:59 +00:00
|
|
|
#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H
|
|
|
|
#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H
|
2017-12-14 21:22:03 +00:00
|
|
|
|
2021-03-12 14:23:45 +01:00
|
|
|
#include "FeatureModule.h"
|
[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)
Alternatives to https://reviews.llvm.org/D153114.
Try to address https://github.com/clangd/clangd/issues/1293.
See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.
This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
to review and understand so that every one are in the same page:
> Don't attempt any cross-file or cross-version coordination: i.e. don't
> try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
> build the preamble.
This patch reflects the above opinions.
# Testing in real-world project
I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.
The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.
# What this patch does in detail
- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.
# What we need to do next
Let's split the TODOs into clang part and clangd part to make things
more clear.
The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.
TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.
And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).
2024-07-18 10:10:22 +08:00
|
|
|
#include "ModulesBuilder.h"
|
2020-11-25 18:35:34 +00:00
|
|
|
#include "TidyProvider.h"
|
2019-01-28 14:01:55 +00:00
|
|
|
#include "index/Index.h"
|
2020-06-17 11:53:32 +02:00
|
|
|
#include "support/ThreadsafeFS.h"
|
2017-12-04 13:49:59 +00:00
|
|
|
#include "clang/Frontend/CompilerInstance.h"
|
|
|
|
#include "clang/Frontend/PrecompiledPreamble.h"
|
2019-01-22 09:58:53 +00:00
|
|
|
#include "clang/Tooling/CompilationDatabase.h"
|
2021-03-12 14:23:45 +01:00
|
|
|
#include <memory>
|
|
|
|
#include <vector>
|
2017-12-04 13:49:59 +00:00
|
|
|
|
|
|
|
namespace clang {
|
|
|
|
namespace clangd {
|
|
|
|
|
|
|
|
class IgnoreDiagnostics : public DiagnosticConsumer {
|
|
|
|
public:
|
2018-02-12 12:48:51 +00:00
|
|
|
static void log(DiagnosticsEngine::Level DiagLevel,
|
|
|
|
const clang::Diagnostic &Info);
|
|
|
|
|
2017-12-04 13:49:59 +00:00
|
|
|
void HandleDiagnostic(DiagnosticsEngine::Level DiagLevel,
|
2018-02-12 12:48:51 +00:00
|
|
|
const clang::Diagnostic &Info) override;
|
2017-12-04 13:49:59 +00:00
|
|
|
};
|
|
|
|
|
2019-01-28 14:01:55 +00:00
|
|
|
// Options to run clang e.g. when parsing AST.
|
|
|
|
struct ParseOptions {
|
2025-02-18 00:59:45 -05:00
|
|
|
bool PreambleParseForwardingFunctions = true;
|
[clangd] Full support for #import insertions
These are still disabled by default, but will work in ObjC code if you
enable the `-import-insertions` flag.
Completion requires ASTSignals to be available; before ASTSignals are
available, we will always use #include. Once they are available, the
behavior varies as follows:
- For source files, use #import if the ObjC language flag is enabled
- For header files:
- If the ObjC language flag is disabled, use #include
- If the header file contains any #imports, use #import
- If the header file references any ObjC decls, use #import
- Otherwise, use #include
IncludeFixer support is similar, but it does not rely upon ASTSignals,
instead it does the above checks excluding the scan for ObjC symbols.
Differential Revision: https://reviews.llvm.org/D139458
2022-12-06 16:33:12 -05:00
|
|
|
|
|
|
|
bool ImportInsertions = false;
|
2019-01-28 14:01:55 +00:00
|
|
|
};
|
|
|
|
|
2019-01-22 09:58:53 +00:00
|
|
|
/// Information required to run clang, e.g. to parse AST or do code completion.
|
|
|
|
struct ParseInputs {
|
|
|
|
tooling::CompileCommand CompileCommand;
|
2020-06-17 18:09:54 +02:00
|
|
|
const ThreadsafeFS *TFS;
|
2019-01-22 09:58:53 +00:00
|
|
|
std::string Contents;
|
[clangd] Track document versions, include them with diags, enhance logs
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
2020-03-04 00:33:29 +01:00
|
|
|
// Version identifier for Contents, provided by the client and opaque to us.
|
|
|
|
std::string Version = "null";
|
2020-02-03 15:14:49 -05:00
|
|
|
// Prevent reuse of the cached preamble/AST. Slow! Useful to workaround
|
|
|
|
// clangd's assumption that missing header files will stay missing.
|
|
|
|
bool ForceRebuild = false;
|
2019-01-28 14:01:55 +00:00
|
|
|
// Used to recover from diagnostics (e.g. find missing includes for symbol).
|
|
|
|
const SymbolIndex *Index = nullptr;
|
2020-06-03 10:34:05 +02:00
|
|
|
ParseOptions Opts = ParseOptions();
|
2020-11-25 18:35:34 +00:00
|
|
|
TidyProviderRef ClangTidyProvider = {};
|
2021-03-12 14:23:45 +01:00
|
|
|
// Used to acquire ASTListeners when parsing files.
|
|
|
|
FeatureModuleSet *FeatureModules = nullptr;
|
[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (#66462)
Alternatives to https://reviews.llvm.org/D153114.
Try to address https://github.com/clangd/clangd/issues/1293.
See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.
This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
to review and understand so that every one are in the same page:
> Don't attempt any cross-file or cross-version coordination: i.e. don't
> try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
> build the preamble.
This patch reflects the above opinions.
# Testing in real-world project
I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.
The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.
# What this patch does in detail
- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.
# What we need to do next
Let's split the TODOs into clang part and clangd part to make things
more clear.
The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.
TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.
And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).
2024-07-18 10:10:22 +08:00
|
|
|
// Used to build and manage (C++) modules.
|
|
|
|
ModulesBuilder *ModulesManager = nullptr;
|
2019-01-22 09:58:53 +00:00
|
|
|
};
|
|
|
|
|
2021-07-23 16:57:33 +02:00
|
|
|
/// Clears \p CI from options that are not supported by clangd, like codegen or
|
|
|
|
/// plugins. This should be combined with CommandMangler::adjust, which provides
|
|
|
|
/// similar functionality for options that needs to be stripped from compile
|
|
|
|
/// flags.
|
|
|
|
void disableUnsupportedOptions(CompilerInvocation &CI);
|
|
|
|
|
2019-01-22 09:58:53 +00:00
|
|
|
/// Builds compiler invocation that could be used to build AST or preamble.
|
|
|
|
std::unique_ptr<CompilerInvocation>
|
2019-11-28 19:22:50 +01:00
|
|
|
buildCompilerInvocation(const ParseInputs &Inputs, clang::DiagnosticConsumer &D,
|
|
|
|
std::vector<std::string> *CC1Args = nullptr);
|
2019-01-22 09:58:53 +00:00
|
|
|
|
2018-01-18 15:17:00 +00:00
|
|
|
/// Creates a compiler instance, configured so that:
|
|
|
|
/// - Contents of the parsed file are remapped to \p MainFile.
|
|
|
|
/// - Preamble is overriden to use PCH passed to this function. It means the
|
|
|
|
/// changes to the preamble headers or files included in the preamble are
|
|
|
|
/// not visible to this compiler instance.
|
2018-10-10 13:27:25 +00:00
|
|
|
/// - llvm::vfs::FileSystem is used for all underlying file accesses. The
|
|
|
|
/// actual vfs used by the compiler may be an overlay over the passed vfs.
|
2018-01-18 15:17:00 +00:00
|
|
|
/// Returns null on errors. When non-null value is returned, it is expected to
|
|
|
|
/// be consumed by FrontendAction::BeginSourceFile to properly destroy \p
|
|
|
|
/// MainFile.
|
2017-12-04 13:49:59 +00:00
|
|
|
std::unique_ptr<CompilerInstance> prepareCompilerInstance(
|
|
|
|
std::unique_ptr<clang::CompilerInvocation>, const PrecompiledPreamble *,
|
|
|
|
std::unique_ptr<llvm::MemoryBuffer> MainFile,
|
2018-10-10 13:27:25 +00:00
|
|
|
IntrusiveRefCntPtr<llvm::vfs::FileSystem>, DiagnosticConsumer &);
|
2017-12-04 13:49:59 +00:00
|
|
|
|
2021-10-26 15:27:07 +02:00
|
|
|
/// Respect `#pragma clang __debug crash` etc, which are usually disabled.
|
|
|
|
/// This may only be called before threads are spawned.
|
|
|
|
void allowCrashPragmasForTest();
|
|
|
|
|
2017-12-04 13:49:59 +00:00
|
|
|
} // namespace clangd
|
|
|
|
} // namespace clang
|
|
|
|
|
2018-08-14 16:03:32 +00:00
|
|
|
#endif // LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H
|