mirror of
https://github.com/llvm/llvm-project.git
synced 2025-04-17 04:26:37 +00:00
e385e0d3e7
This is the following patch of https://github.com/llvm/llvm-project/pull/66462 to optimize its performance. # Motivation To avoid data races, we choose "per file owns its dependent modules" model. That said, every TU will own all the required module files so that we don't need to worry about thread safety. And it looks like we succeeded that we focus on the interfaces and structure of modules support in clangd. But after all, this model is not good for performance. Image we have 10000 TUs import std, we will have 10000 std.pcm in the memory. That is terrible both in time and space. Given the current modules support in clangd works pretty well (almost every issue report I received is more or less a clang's issue), I'd like to improve the performance. # High Level Changes After this patch, the built module files will be owned by the module builder and each TU will only have a reference to the built module files. The module builder have a map from module names to built module files. When a new TU ask for a module file, the module builder will check if the module file lives in the map and if the module file are up to date. If yes, the module file will be returned. If no, the module file entry would be erased in the module builder. We use `shared_ptr<>` to track module file here so that the other TU owning the out dated module file won't be affected. The out dated module file will be removed automatically if other TU gets update or closed. (I know the out dated module file may not exist due to the `CanReuse` mechanism. But the design here is natural and can be seen as a redundant design to make it more robust.) When we a build a module, we will use the mutex and the condition variable in the working thread to build it exclusively. All other threads that also want the module file would have to wait for that working thread. It might not sounds great but I think if we want to make it asynchronous, we have to refactor TUScheduler as far as I know. # Code Structure Changes Thanks for the previous hard working reviewing, the interfaces almost don't change in this patch. Almost all the work are isolated in ModulesBuilder.cpp. A outliner is that we convert `ModulesBuilder` to an abstract class since the implementation class needs to own the module files. And the core function to review is `ReusableModulesBuilder::getOrBuildModuleFile`. It implements the core logic to fetch the module file from the cache or build it if the module file is not in the cache or out of date. And other important entities are `BuildingModuleMutexes`, `BuildingModuleCVs`, `BuildingModules` and `ModulesBuildingMutex`. These are mutexes and condition variables to make sure the thread safety. # User experience I've implemented this in our downstream and ask our users to use it. I also sent it https://github.com/ChuanqiXu9/clangd-for-modules here as pre-version. The feedbacks are pretty good. And I didn't receive any bug reports (about the reusable modules builder) yet. # Other potential improvement The are other two potential improvements can be done: 1. Scanning cache and a mechanism to get the required module information more quickly. (Like the module maps in https://github.com/ChuanqiXu9/clangd-for-modules) 2. Persist the module files. So that after we close the vscode and reopen it, we can reuse the built module files since the last invocation.
109 lines
3.6 KiB
C++
109 lines
3.6 KiB
C++
//===----------------- ModulesBuilder.h --------------------------*- C++-*-===//
|
|
//
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
//
|
|
//===----------------------------------------------------------------------===//
|
|
//
|
|
// Experimental support for C++20 Modules.
|
|
//
|
|
// Currently we simplify the implementations by preventing reusing module files
|
|
// across different versions and different source files. But this is clearly a
|
|
// waste of time and space in the end of the day.
|
|
//
|
|
// TODO: Supporting reusing module files across different versions and
|
|
// different source files.
|
|
//
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_MODULES_BUILDER_H
|
|
#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_MODULES_BUILDER_H
|
|
|
|
#include "GlobalCompilationDatabase.h"
|
|
#include "ProjectModules.h"
|
|
#include "support/Path.h"
|
|
#include "support/ThreadsafeFS.h"
|
|
#include "clang/Frontend/CompilerInvocation.h"
|
|
#include "llvm/ADT/SmallString.h"
|
|
#include <memory>
|
|
|
|
namespace clang {
|
|
namespace clangd {
|
|
|
|
/// Store all the needed module files information to parse a single
|
|
/// source file. e.g.,
|
|
///
|
|
/// ```
|
|
/// // a.cppm
|
|
/// export module a;
|
|
///
|
|
/// // b.cppm
|
|
/// export module b;
|
|
/// import a;
|
|
///
|
|
/// // c.cppm
|
|
/// export module c;
|
|
/// import b;
|
|
/// ```
|
|
///
|
|
/// For the source file `c.cppm`, an instance of the class will store
|
|
/// the module files for `a.cppm` and `b.cppm`. But the module file for `c.cppm`
|
|
/// won't be stored. Since it is not needed to parse `c.cppm`.
|
|
///
|
|
/// Users should only get PrerequisiteModules from
|
|
/// `ModulesBuilder::buildPrerequisiteModulesFor(...)`.
|
|
///
|
|
/// Users can detect whether the PrerequisiteModules is still up to date by
|
|
/// calling the `canReuse()` member function.
|
|
///
|
|
/// The users should call `adjustHeaderSearchOptions(...)` to update the
|
|
/// compilation commands to select the built module files first. Before calling
|
|
/// `adjustHeaderSearchOptions()`, users should call `canReuse()` first to check
|
|
/// if all the stored module files are valid. In case they are not valid,
|
|
/// users should call `ModulesBuilder::buildPrerequisiteModulesFor(...)` again
|
|
/// to get the new PrerequisiteModules.
|
|
class PrerequisiteModules {
|
|
public:
|
|
/// Change commands to load the module files recorded in this
|
|
/// PrerequisiteModules first.
|
|
virtual void
|
|
adjustHeaderSearchOptions(HeaderSearchOptions &Options) const = 0;
|
|
|
|
/// Whether or not the built module files are up to date.
|
|
/// Note that this can only be used after building the module files.
|
|
virtual bool
|
|
canReuse(const CompilerInvocation &CI,
|
|
llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem>) const = 0;
|
|
|
|
virtual ~PrerequisiteModules() = default;
|
|
};
|
|
|
|
/// This class handles building module files for a given source file.
|
|
///
|
|
/// In the future, we want the class to manage the module files acorss
|
|
/// different versions and different source files.
|
|
class ModulesBuilder {
|
|
public:
|
|
ModulesBuilder(const GlobalCompilationDatabase &CDB);
|
|
~ModulesBuilder();
|
|
|
|
ModulesBuilder(const ModulesBuilder &) = delete;
|
|
ModulesBuilder(ModulesBuilder &&) = delete;
|
|
|
|
ModulesBuilder &operator=(const ModulesBuilder &) = delete;
|
|
ModulesBuilder &operator=(ModulesBuilder &&) = delete;
|
|
|
|
std::unique_ptr<PrerequisiteModules>
|
|
buildPrerequisiteModulesFor(PathRef File, const ThreadsafeFS &TFS);
|
|
|
|
private:
|
|
class ModulesBuilderImpl;
|
|
std::unique_ptr<ModulesBuilderImpl> Impl;
|
|
};
|
|
|
|
} // namespace clangd
|
|
} // namespace clang
|
|
|
|
#endif
|