570 Commits

Author SHA1 Message Date
Duncan P. N. Exon Smith
2f721476d1 Frontend: Simplify handling of non-seeking streams in CompilerInstance, NFC
Add a new `raw_pwrite_ostream` variant, `buffer_unique_ostream`, which
is like `buffer_ostream` but with unique ownership of the stream it's
wrapping. Use this in CompilerInstance to simplify the ownership of
non-seeking output streams, avoiding logic sprawled around to deal with
them specially.

This also simplifies future work to encapsulate output files in a
different class.

Differential Revision: https://reviews.llvm.org/D93260
2021-01-26 15:20:43 -08:00
Duncan P. N. Exon Smith
8afabff6b1 Frontend: Fix memory leak in CompilerInstance::setVerboseOutputStream
Found this memory leak in `CompilerInstance::setVerboseOutputStream` by
inspection; it looks like this wasn't previously exercised, since it was
never called twice.

Differential Revision: https://reviews.llvm.org/D93249
2021-01-26 14:57:23 -08:00
Argyrios Kyrtzidis
b0e89906f5 [ASTReader] Allow controlling separately whether validation should be disabled for a PCH vs a module file
This addresses an issue with how the PCH preable works, specifically:

1. When using a PCH/preamble the module hash changes and a different cache directory is used
2. When the preamble is used, PCH & PCM validation is disabled.

Due to combination of #1 and #2, reparsing with preamble enabled can end up loading a stale module file before a header change and using it without updating it because validation is disabled and it doesn’t check that the header has changed and the module file is out-of-date.

rdar://72611253

Differential Revision: https://reviews.llvm.org/D95159
2021-01-21 20:45:54 -08:00
Duncan P. N. Exon Smith
3ee43adfb2 Basic: Add native support for stdin to SourceManager and FileManager
Add support for stdin to SourceManager and FileManager. Adds
FileManager::getSTDIN, which adds a FileEntryRef for `<stdin>` and reads
the MemoryBuffer, which is stored as `FileEntry::Content`.

Eventually the other buffers in `ContentCache` will sink to here as well
-- we probably usually want to load/save a MemoryBuffer eagerly -- but
it's happening early for stdin to get rid of
CompilerInstance::InitializeSourceManager's final call to
`SourceManager::overrideFileContents`.

clang/test/CXX/modules-ts/dcl.dcl/dcl.module/dcl.module.export/p1.cpp
relies on building a module from stdin; supporting that requires setting
ContentCache::BufferOverridden.

Differential Revision: https://reviews.llvm.org/D93148
2020-12-23 15:18:50 -08:00
Duncan P. N. Exon Smith
245218bb35 Basic: Support named pipes natively in SourceManager and FileManager
Handle named pipes natively in SourceManager and FileManager, removing a
call to `SourceManager::overrideFileContents` in
`CompilerInstance::InitializeSourceManager` (removing a blocker for
sinking the content cache to FileManager (which will incidently sink
this new named pipe logic with it)).

SourceManager usually checks if the file entry's size matches the
eventually loaded buffer, but that's now skipped for named pipes since
the `stat` won't reflect the full size.  Since we can't trust
`ContentsEntry->getSize()`, we also need shift the check for files that
are too large until after the buffer is loaded... and load the buffer
immediately in `createFileID` so that no client gets a bad value from
`ContentCache::getSize`. `FileManager::getBufferForFile` also needs to
treat these files as volatile when loading the buffer.

Native support in SourceManager / FileManager means that named pipes can
also be `#include`d, and clang/test/Misc/dev-fd-fs.c was expanded to
check for that.

This is a new version of 3b18a594c7717a328c33b9c1eba675e9f4bd367c, which
was reverted in b34632201987eed369bb7ef4646f341b901c95b8 since it was
missing the `SourceManager` changes.

Differential Revision: https://reviews.llvm.org/D92531
2020-12-23 14:57:41 -08:00
Duncan P. N. Exon Smith
0eb4378290 Frontend: Fix confusing comment at call to clearOutputFiles, NFC
Fix the comment in front of `compileModuleImpl`'s call to
`CompilerInstance::clearOutputFiles`. The purpose of this call is to
delete any stray temporary files after the module generation thread
crashes.

The comment is from f545f67de3a1bfdbbfad88acde5b540ce3b82f4f, and
was associated with manually deleting a generated module map. Then
13afbf42d830dd43febbeb0855aa359ca9dbfbf9 added this `clearOutputFiles`
call between the comment and the code it referenced. Finally,
1f76c4e8101b9beaf8f1b10a57faa80256ab2b05 started sending the generated
module map directly to the SourceManager instead of putting it on disk,
deleting the call that the comment referenced.

No functionality change.
2020-12-15 14:21:37 -08:00
Duncan P. N. Exon Smith
347e1f6213 Frontend: Use a getVirtualFileRef for a named pipe main file, NFC 2020-12-09 16:17:14 -08:00
Duncan P. N. Exon Smith
a5c89bb021 Frontend: Migrate to FileEntryRef in CompilerInstance::InitializeSourceManager, NFC
Use `FileManager::getVirtualFileRef` to get the virtual file for stdin,
and add an overload of `SourceManager::overrideFileContents` that takes
a `FileEntryRef`, migrating `CompilerInstance::InitializeSourceManager`.

Differential Revision: https://reviews.llvm.org/D92680
2020-12-09 13:28:29 -08:00
Yuanfang Chen
1821265db6 [Time-report] Add a flag -ftime-report={per-pass,per-pass-run} to control the pass timing aggregation
Currently, -ftime-report + new pass manager emits one line of report for each
pass run. This potentially causes huge output text especially with regular LTO
or large single file (Obeserved in private tests and was reported in D51276).
The behaviour of -ftime-report + legacy pass manager is
emitting one line of report for each pass object which has relatively reasonable
text output size. This patch adds a flag `-ftime-report=` to control time report
aggregation for new pass manager.

The flag is for new pass manager only. Using it with legacy pass manager gives
an error. It is a driver and cc1 flag. `per-pass` is the new default so
`-ftime-report` is aliased to `-ftime-report=per-pass`. Before this patch,
functionality-wise `-ftime-report` is aliased to `-ftime-report=per-pass-run`.

* Adds an boolean variable TimePassesHandler::PerRun to control per-pass vs per-pass-run.
* Adds a new clang CodeGen flag CodeGenOptions::TimePassesPerRun to work with the existing CodeGenOptions::TimePasses.
* Remove FrontendOptions::ShowTimers, its uses are replaced by the existing CodeGenOptions::TimePasses.
* Remove FrontendTimesIsEnabled (It was introduced in D45619 which was largely reverted.)

Differential Revision: https://reviews.llvm.org/D92436
2020-12-08 10:13:19 -08:00
Duncan P. N. Exon Smith
b346322019 Revert "Frontend: Sink named pipe logic from CompilerInstance down to FileManager"
This reverts commit 3b18a594c7717a328c33b9c1eba675e9f4bd367c, since
apparently this doesn't work everywhere. E.g.,
clang-x86_64-debian-fast/3889
(http://lab.llvm.org:8011/#/builders/109/builds/3889) gives me:
```
+ : 'RUN: at line 8'
+ /b/1/clang-x86_64-debian-fast/llvm.obj/bin/clang -x c /dev/fd/0 -E
+ cat /b/1/clang-x86_64-debian-fast/llvm.src/clang/test/Misc/dev-fd-fs.c
fatal error: file '/dev/fd/0' modified since it was first processed
1 error generated.
```
2020-12-02 17:36:20 -08:00
Duncan P. N. Exon Smith
3b18a594c7 Frontend: Sink named pipe logic from CompilerInstance down to FileManager
Remove compilicated logic from CompilerInstance::InitializeSourceManager
to deal with named pipes, updating FileManager::getBufferForFile to
handle it in a more straightforward way. The existing test at
clang/test/Misc/dev-fd-fs.c covers the new behaviour (just like it did
the old behaviour).

Differential Revision: https://reviews.llvm.org/D90733
2020-12-02 17:14:27 -08:00
Ben Barham
5834996fef [Frontend] Add flag to allow PCM generation despite compiler errors
As with precompiled headers, it's useful for indexers to be able to
continue through compiler errors in dependent modules.

Resolves rdar://69816264

Reviewed By: akyrtzi

Differential Revision: https://reviews.llvm.org/D91580
2020-11-17 17:27:50 -08:00
Alexandre Rames
58c586e701 Allow searching for prebuilt implicit modules.
This reverts commit c67656b994c87224e0b33e2c4b09093986a5cfa6, and addresses the
build issue.
2020-11-10 10:14:13 -08:00
Stella Stamenova
c67656b994 Revert "Allow searching for prebuilt implicit modules."
This reverts commit 71e108cd86e70b06c5fa3a63689dcb3555c3d13f.

This change caused a build failure on Windows:
http://lab.llvm.org:8011/#/builders/83/builds/570
2020-11-05 17:16:14 -08:00
Alexandre Rames
71e108cd86 Allow searching for prebuilt implicit modules.
The behavior is controlled by the `-fprebuilt-implicit-modules` option, and
allows searching for implicit modules in the prebuilt module cache paths.

The current command-line options for prebuilt modules do not allow to easily
maintain and use multiple versions of modules. Both the producer and users of
prebuilt modules are required to know the relationships between compilation
options and module file paths. Using a particular version of a prebuilt module
requires passing a particular option on the command line (e.g.
`-fmodule-file=[<name>=]<file>` or `-fprebuilt-module-path=<directory>`).

However the compiler already knows how to distinguish and automatically locate
implicit modules. Hence this proposal to introduce the
`-fprebuilt-implicit-modules` option. When set, it enables searching for
implicit modules in the prebuilt module paths (specified via
`-fprebuilt-module-path`). To not modify existing behavior, this search takes
place after the standard search for prebuilt modules. If not

Here is a workflow illustrating how both the producer and consumer of prebuilt
modules would need to know what versions of prebuilt modules are available and
where they are located.

  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v1 <config 1 options>
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v2 <config 2 options>
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules_v3 <config 3 options>

  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules_v1 <config 1 options>
  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap <non-prebuilt config options>

With prebuilt implicit modules, the producer can generate prebuilt modules as
usual, all in the same output directory. The same mechanisms as for implicit
modules take care of incorporating hashes in the path to distinguish between
module versions.

Note that we do not specify the output module filename, so `-o` implicit modules are generated in the cache path `prebuilt_modules`.

  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 1 options>
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 2 options>
  clang -cc1 -x c modulemap -fmodules -emit-module -fmodule-name=foo -fmodules-cache-path=prebuilt_modules <config 3 options>

The user can now simply enable prebuilt implicit modules and point to the
prebuilt modules cache. No need to "parse" command-line options to decide
what prebuilt modules (paths) to use.

  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules <config 1 options>
  clang -cc1 -x c use.c -fmodules fmodule-map-file=modulemap -fprebuilt-module-path=prebuilt_modules -fprebuilt-implicit-modules <non-prebuilt config options>

This is for example particularly useful in a use-case where compilation is
expensive, and the configurations expected to be used are predictable, but not
controlled by the producer of prebuilt modules. Modules for the set of
predictable configurations can be prebuilt, and using them does not require
"parsing" the configuration (command-line options).

Reviewed By: Bigcheese

Differential Revision: https://reviews.llvm.org/D68997
2020-11-05 13:10:53 -08:00
Duncan P. N. Exon Smith
296314516d ContentCache: Simplify by always owning the MemoryBuffer
This changes `ContentCache::Buffer` to use
`std::unique_ptr<MemoryBuffer>` instead of the `PointerIntPair`. It
drops the (mostly unused) `DoNotFree` bit, instead creating a (new)
non-owning `MemoryBuffer` instance when passed a `MemoryBufferRef`.

Differential Revision: https://reviews.llvm.org/D67030
2020-10-20 21:03:53 -04:00
Duncan P. N. Exon Smith
51d1d585e5 clang/Frontend: Use MemoryBufferRef in FrontendInputFile (and remove SourceManager::getBuffer)
In order to drop the final callers to `SourceManager::getBuffer`, change
`FrontendInputFile` to use `Optional<MemoryBufferRef>`. Also updated
the "unowned" version of `SourceManager::createFileID` to take a
`MemoryBufferRef` (it now calls `MemoryBuffer::getMemBuffer`, which
creates a `MemoryBuffer` that does not own the buffer data).

Differential Revision: https://reviews.llvm.org/D89427
2020-10-20 13:35:46 -04:00
Volodymyr Sapsai
a28678e20a Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""
This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is failing on Windows.
2020-10-19 18:27:30 -07:00
Volodymyr Sapsai
4000c9ee18 Reland "[Modules] Add stats to measure performance of building and loading modules."
Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

On relanding drop stats in MemoryBuffer.cpp as their value is pretty low
but affects a lot of clients and many of those aren't interested in
modules and header search.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895
2020-10-19 15:44:11 -07:00
Volodymyr Sapsai
9eba6b20a0 Revert "[Modules] Add stats to measure performance of building and loading modules."
This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is failing. Reverting now
and will recommit after making the test not fail with the added stats.
2020-09-24 12:36:06 -07:00
Volodymyr Sapsai
c4bacc3c9b [Modules] Add stats to measure performance of building and loading modules.
Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895
2020-09-24 12:23:47 -07:00
Kevin P. Neal
d4ce862f2a Reland "[FPEnv][Clang][Driver] Disable constrained floating point on targets lacking support."
We currently have strict floating point/constrained floating point enabled
for all targets. Constrained SDAG nodes get converted to the regular ones
before reaching the target layer. In theory this should be fine.

However, the changes are exposed to users through multiple clang options
already in use in the field, and the changes are _completely_ _untested_
on almost all of our targets. Bugs have already been found, like
"https://bugs.llvm.org/show_bug.cgi?id=45274".

This patch disables constrained floating point options in clang everywhere
except X86 and SystemZ. A warning will be printed when this happens.

Use the new -fexperimental-strict-floating-point flag to force allowing
strict floating point on hosts that aren't already marked as supporting
it (X86 and SystemZ).

Differential Revision: https://reviews.llvm.org/D80952
2020-07-10 08:49:45 -04:00
Kevin P. Neal
916e2ca997 Revert "[FPEnv][Clang][Driver] Disable constrained floating point on targets lacking support."
My mistake, I had a blocking reviewer.

This reverts commit 39d2ae0afb2312a15e4d15a0855b35b4e1c49fc4.
This reverts commit bfdafa32a0fa4b2745627fe57dd253db10ac3fcf.
This reverts commit 2b35511350454dd22997f129ee529e3fdb129ac2.

Differential Revision: https://reviews.llvm.org/D80952
2020-07-06 14:57:45 -04:00
Kevin P. Neal
39d2ae0afb [FPEnv][Clang][Driver] Disable constrained floating point on targets lacking support.
We currently have strict floating point/constrained floating point enabled
for all targets. Constrained SDAG nodes get converted to the regular ones
before reaching the target layer. In theory this should be fine.

However, the changes are exposed to users through multiple clang options
already in use in the field, and the changes are _completely_ _untested_
on almost all of our targets. Bugs have already been found, like
"https://bugs.llvm.org/show_bug.cgi?id=45274".

This patch disables constrained floating point options in clang everywhere
except X86 and SystemZ. A warning will be printed when this happens.

Differential Revision: https://reviews.llvm.org/D80952
2020-07-06 13:32:49 -04:00
Andrzej Warzynski
ef875c228a [clang][NFC] Removed unused parameters in InitializeSourceManager 2020-07-03 16:16:20 +01:00
Richard Smith
fc76b4ad3d Rename IsMissingRequirement to IsUnimportable and set it for shadowed
modules too.

This more accurately reflects the semantics of this flag, as distinct
from "IsAvailable", which (in an explicit modules world) only describes
whether a module is buildable, not whether it's importable.
2020-04-17 22:48:56 -07:00
Reid Kleckner
e08464fb45 Avoid including FileManager.h from SourceManager.h
Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in FileManager.h and LLVM's FS headers.

The main code change here is to sink SM::createFileID into the cpp file.
I reason that this is not performance critical because it doesn't happen
on the diagnostic path, it happens along the paths of macro expansion
(could be hot) and new includes (less hot).

Saves some includes:
    309 -    /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileManager.h
    272 -    /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileSystemOptions.h
    271 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/VirtualFileSystem.h
    267 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/FileSystem.h
    266 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Chrono.h

Differential Revision: https://reviews.llvm.org/D75406
2020-03-11 13:53:12 -07:00
Volodymyr Sapsai
a6c8698924 clang/Modules: Finish renaming CompilerInstance::ModuleManager, NFC.
Follow-up to 20d51b2f14ac4488f684f8fc57cb0ba718a6b91d, rename the setter to
make it consistent with the getter. Also fixed a few comments along the
way, didn't try to find all references to a module manager.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D74939
2020-02-21 13:56:50 -08:00
Benjamin Kramer
ef83d46b6b Use heterogenous lookup for std;:map<std::string with a StringRef. NFCI. 2020-02-08 13:28:29 +01:00
Michael Liao
ccac6b2bf8 [hip] Properly populate macros based on host processor.
Summary:
- The device compilation needs to have a consistent source code compared
  to the corresponding host compilation. If macros based on the
  host-specific target processor is not properly populated, the device
  compilation may fail due to the inconsistent source after the
  preprocessor. So far, only the host triple is used to build the
  macros. If a detailed host CPU target or certain features are
  specified, macros derived from them won't be populated properly, e.g.
  `__SSE3__` won't be added unless `+sse3` feature is present. On
  Windows compilation compatible with MSVC, that missing macros result
  in that intrinsics are not included and cause device compilation
  failure on the host-side source.

- This patch addresses this issue by introducing two `cc1` options,
  i.e., `-aux-target-cpu` and `-aux-target-feature`. If a specific host
  CPU target or certain features are specified, the compiler driver will
  append them during the construction of the offline compilation
  actions. Then, the toolchain in `cc1` phase will populate macros
  accordingly.

- An internal option `--gpu-use-aux-triple-only` is added to fall back
  the original behavior to help diagnosing potential issues from the new
  behavior.

Reviewers: tra, yaxunl

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D73942
2020-02-04 15:36:14 -05:00
Benjamin Kramer
adcd026838 Make llvm::StringRef to std::string conversions explicit.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.
2020-01-28 23:25:25 +01:00
Hans Wennborg
dde7b6bcda Re-land "Add an -fno-temp-file flag for compilation"
This time making sure to initialize FrontendOptions::UseTemporary.

Patch by Zachary Henkel!

Differential revision: https://reviews.llvm.org/D70615
2019-12-19 13:34:52 +01:00
Mitch Phillips
b19d87b16f Revert "Add an -fno-temp-file flag for compilation"
This reverts commit d129aa1d5369781deff6c6b854cb612e160d3fb2.

This broke the MSan buildbots. More information available in the
original PR: https://reviews.llvm.org/D70615
2019-12-18 09:05:09 -08:00
Hans Wennborg
d129aa1d53 Add an -fno-temp-file flag for compilation
Our build system does not handle randomly named files created during
the build well. We'd prefer to write compilation output directly
without creating a temporary file. Function parameters already
existed to control this behavior but were not exposed all the way out
to the command line.

Patch by Zachary Henkel!

Differential revision: https://reviews.llvm.org/D70615
2019-12-18 15:07:43 +01:00
Alexey Bader
cb30ad728f [SYCL] Add support for auxiliary triple specification to Frontend
Summary:
Add host predefined macros to compilation for SYCL device, which is
required for pre-processing host specific includes (e.g. system
headers).

Reviewers: ABataev, jdoerfert

Subscribers: ebevhan, Anastasia, cfe-commits, keryell, Naghasan, Fznamznon

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71286

Signed-off-by: Alexey Bader <alexey.bader@intel.com>
2019-12-11 12:40:43 +03:00
Duncan P. N. Exon Smith
20d51b2f14 clang/Modules: Rename CompilerInstance::ModuleManager, NFC
Fix the confusing naming of `CompilerInstance::ModuleManager`.  This is
actually an instance of `ASTReader`, which contains an instance of
`ModuleManager`.  I have to assume there was a point in the past where
they were just one class, but it's been pretty confusing for a while.  I
think it's time to fix it.

The new name is `TheASTReader`; the annoying `The` prefix is so that we
don't shadow the `ASTReader` class.  I tried out `ASTRdr` but that
seemed less clear, and this choice matches `ThePCHContainerOperations`
just a couple of declarations below.

Also rename `CompilerInstance::getModuleManager` and
`CompilerInstance::createModuleManager` to `*ASTReader`, making some
cases of `getModuleManager().getModuleManager()` a little more clear.

https://reviews.llvm.org/D70583
2019-11-22 18:24:11 -08:00
Duncan P. N. Exon Smith
5cca622310 clang/Modules: Refactor CompilerInstance::loadModule, NFC
Refactor the logic on CompilerInstance::loadModule and a couple of
surrounding methods in order to clarify what's going on.

- Rename ModuleLoader::loadModuleFromSource to compileModuleFromSource
  and fix its documentation, since it never loads a module.  It just
  creates/compiles one.
- Rename one of the overloads of compileModuleImpl to compileModule,
  making it more obvious which one calls the other.
- Rename compileAndLoadModule to compileModuleAndReadAST.  This
  clarifies the relationship between this helper and its caller,
  CompilerInstance::loadModule (the old name implied the opposite
  relationship).  It also (correctly) indicates that more needs to be
  done to load the module than this function is responsible for.
- Split findOrCompileModuleAndReadAST out of loadModule.  Besides
  reducing nesting for this code thanks to early returns and the like,
  this refactor clarifies the logic in loadModule, particularly around
  calls to ModuleMap::cacheModuleLoad and
  ModuleMap::getCachedModuleLoad.  findOrCompileModuleAndReadAST also
  breaks early if the initial ReadAST call returns Missing or OutOfDate,
  allowing the last ditch call to compileModuleAndReadAST to come at the
  end of the function body.
    - Additionally split out selectModuleSource, clarifying the logic
      due to early returns.
    - Add ModuleLoadResult::isNormal and OtherUncachedFailure, so that
      loadModule knows whether to cache the result.
      OtherUncachedFailure was added to keep this patch NFC, but there's
      a chance that these cases were uncached by accident, through
      copy/paste/modify failures.  These should be audited as a
      follow-up (maybe we can eliminate this case).
    - Do *not* lift the setting of `ModuleLoadFailed = true` to
      loadModule because there isn't a clear pattern for when it's set.
      This should be reconsidered in a follow-up, in case it would be
      correct to set `ModuleLoadFailed` whenever no module is returned
      and the result is either Normal or OtherUncachedFailure.
- Add some header documentation where it was missing, and fix it where
  it was wrong.

This should have no functionality change.

https://reviews.llvm.org/D70556
2019-11-22 18:23:47 -08:00
Duncan P. N. Exon Smith
8700831734 clang/Modules: Early return in CompilerInstance::createModuleManager, NFC
Reduce nesting with an early `return`.
2019-11-19 18:16:23 -08:00
Duncan P. N. Exon Smith
8112a423a8 clang/Modules: Bring back optimization lost in 31e14f41a21f
31e14f41a21f9016050a20f07d5da03db2e8c13e accidentally dropped caching of
failed module loads.  This brings it back by making
ModuleMap::getCachedModuleLoad return an Optional.
2019-11-04 11:40:03 -08:00
Duncan P. N. Exon Smith
31e14f41a2 clang/Modules: Sink CompilerInstance::KnownModules into ModuleMap
Avoid use-after-frees when FrontendAction::BeginSourceFile is called
twice on the same CompilerInstance by sinking
CompilerInstance::KnownModules into ModuleMap.  On the way, rename the
map to CachedModuleLoads.  I considered (but rejected) merging this with
ModuleMap::Modules, since that only has top-level modules and this map
includes submodules.

This is an alternative to https://reviews.llvm.org/D58497.  Thanks to
nemanjai for the detailed analysis of the problem!
2019-11-03 19:57:33 -08:00
Scott Linder
87cb734c04 [Clang] Add VerboseOutputStream to CompilerInstance
Remove one instance of a hardcoded output stream in
CompilerInstance::ExecuteAction. There are still other cases of output
being hard-coded to standard streams in ExecuteCompilerInvocation, but
this patch covers the case when no flags like -version or -help are
passed, namely the "X warnings and Y errors generated." diagnostic.

Differential Revision: https://reviews.llvm.org/D53768

llvm-svn: 375442
2019-10-21 19:04:56 +00:00
Volodymyr Sapsai
930ada91ce Replace platform-dependent stat with llvm::sys::fs::status. NFC intended.
Reviewers: bruno, sammccall

Reviewed By: sammccall

Subscribers: jkorous, dexonsmith, arphaman, ributzka, cfe-commits

Differential Revision: https://reviews.llvm.org/D69011

llvm-svn: 375031
2019-10-16 19:12:34 +00:00
Bruno Cardoso Lopes
1731fc88d1 Reapply: [Modules][PCH] Hash input files content
Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:

- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.

This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.

I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.

Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".

rdar://problem/29320105

Reviewers: dexonsmith, arphaman, rsmith, aprantl

Subscribers: jkorous, cfe-commits, ributzka

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67249

> llvm-svn: 374841

llvm-svn: 374895
2019-10-15 14:23:55 +00:00
Eric Christopher
3be9169caa Temporarily Revert [Modules][PCH] Hash input files content
as it's breaking a few bots.

This reverts r374841 (git commit 2a1386c81de504b5bda44fbecf3f7b4cdfd748fc)

llvm-svn: 374842
2019-10-14 23:14:24 +00:00
Bruno Cardoso Lopes
2a1386c81d [Modules][PCH] Hash input files content
Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:

- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.

This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.

I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.

Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".

rdar://problem/29320105

Reviewers: dexonsmith, arphaman, rsmith, aprantl

Subscribers: jkorous, cfe-commits, ributzka

Tags: #clang

Differential Revision: https://reviews.llvm.org/D67249

llvm-svn: 374841
2019-10-14 23:02:03 +00:00
Duncan P. N. Exon Smith
9ef6c49baf FileManager: Use llvm::Expected in new getFileRef API
`FileManager::getFileRef` is a modern API which we expect to convert to
over time.  We should modernize the error handling as well, using
`llvm::Expected` instead of `llvm::ErrorOr`, to help clients that care
about errors to ensure nothing is missed.

However, not all clients care.  I've also added another path for those
that don't:

- `FileEntryRef` is now copy- and move-assignable (using a pointer
  instead of a reference).
- `FileManager::getOptionalFileRef` returns an `llvm::Optional` instead
  of `llvm::Expected`.
- Added an `llvm::expectedToOptional` utility in case this is useful
  elsewhere.

https://reviews.llvm.org/D66705

llvm-svn: 369943
2019-08-26 18:29:51 +00:00
Richard Smith
26a92d5852 Improve behavior in the case of stack exhaustion.
Summary:
Clang performs various recursive operations (such as template instantiation),
and may use non-trivial amounts of stack space in each recursive step (for
instance, due to recursive AST walks). While we try to keep the stack space
used by such steps to a minimum and we have explicit limits on the number of
such steps we perform, it's impractical to guarantee that we won't blow out the
stack on deeply recursive template instantiations on complex ASTs, even with
only a moderately high instantiation depth limit.

The user experience in these cases is generally terrible: we crash with
no hint of what went wrong. Under this patch, we attempt to do better:

 * Detect when the stack is nearly exhausted, and produce a warning with a
   nice template instantiation backtrace, telling the user that we might
   run slowly or crash.

 * For cases where we're forced to trigger recursive template
   instantiation in arbitrarily-deeply-nested contexts, check whether
   we're nearly out of stack space and allocate a new stack (by spawning
   a new thread) after producing the warning.

Reviewers: rnk, aaron.ballman

Subscribers: mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D66361

llvm-svn: 369940
2019-08-26 18:18:07 +00:00
Alex Lorenz
4dc5573acc Introduce FileEntryRef and use it when handling includes to report correct dependencies
when the FileManager is reused across invocations

This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns
a reference to the FileEntry, and the name that was used to access the file. In the case of
a VFS with 'use-external-names', the FileEntyRef contains the external name of the file,
not the filename that was used to access it.

The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the
accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using
the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file
is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations.

Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies
are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits.

Differential Revision: https://reviews.llvm.org/D65907

llvm-svn: 369680
2019-08-22 18:15:50 +00:00
Jonas Devlieghere
2b3d49b610 [Clang] Migrate llvm::make_unique to std::make_unique
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368942
2019-08-14 23:04:18 +00:00
Rainer Orth
09d890d728 Move LangStandard*, InputKind::Language to Basic
This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.

It moves LangStandard* and InputKind::Language to Basic.  It is mostly
mechanical, with only a few changes of note:

- enum Language has been changed into enum class Language : uint8_t to
  avoid a clash between OpenCL in enum Language and OpenCL in enum
  LangFeatures and not to increase the size of class InputKind.

- Now that getLangStandardForName, which is currently unused, also checks
  both canonical and alias names, I've introduced a helper getLangKind
  which factors out a code pattern already used 3 times.

The patch has been tested on x86_64-pc-solaris2.11, sparcv9-sun-solaris2.11,
and x86_64-pc-linux-gnu.

There's a companion patch for lldb which uses LangStandard.h
(https://reviews.llvm.org/D65717).

While polly includes isl which in turn uses InputKind::C, that part of the
code isn't even built inside the llvm tree.  I've posted a patch to allow
for both InputKind::C and Language::C upstream
(https://groups.google.com/forum/#!topic/isl-development/6oEvNWOSQFE).

Differential Revision: https://reviews.llvm.org/D65562

llvm-svn: 367864
2019-08-05 13:59:26 +00:00