165 Commits

Author SHA1 Message Date
Douglas Gregor
227c352bae We do parse hexfloats in C++11; make it actually work.
llvm-svn: 141798
2011-10-12 18:51:02 +00:00
Richard Smith
a9e33d44a6 Handle Perforce-style conflict markers like normal conflict markers. Perforce
swaps over the <<<< and >>>> markers, and uses shorter markers than traditional
tools.

llvm-svn: 141751
2011-10-12 00:37:51 +00:00
Abramo Bagnara
e398e60611 Fixed exapnsion range for # and ##.
llvm-svn: 141012
2011-10-03 18:39:03 +00:00
Argyrios Kyrtzidis
e6e67deeed Rename SourceLocation::getFileLocWithOffset -> getLocWithOffset.
It already works (and is useful with) macro locs as well.

llvm-svn: 140057
2011-09-19 20:40:19 +00:00
Francois Pichet
0706d203cf Rename LangOptions::Microsoft to LangOptions::MicrosoftExt to make it clear that this flag must be used only for Microsoft extensions and not emulation; to avoid confusion with the new LangOptions::MicrosoftMode flag.
Many of the code now under LangOptions::MicrosoftExt will eventually be moved under the LangOptions::MicrosoftMode flag.

llvm-svn: 139987
2011-09-17 17:15:52 +00:00
Benjamin Kramer
17ff23c708 Speed up BCPL comment lexing by looking aggressively for newlines and then scannig backwards to see if the newline is escaped.
3% speedup in preprocessing all of clang with -Eonly. Also includes a small testcase for coverage.

llvm-svn: 139116
2011-09-05 07:19:39 +00:00
Benjamin Kramer
dbfb18a0a9 Use the Lexer's definition of whitespace here.
llvm-svn: 139115
2011-09-05 07:19:35 +00:00
Argyrios Kyrtzidis
5cec2aea3f Support code-completion for C++ inline methods and ObjC buffering methods.
Previously we would cut off the source file buffer at the code-completion
point; this impeded code-completion inside C++ inline methods and,
recently, with buffering ObjC methods.

Have the code-completion inserted into the source buffer so that it can
be buffered along with a method body. When we actually hit the code-completion
point the cut-off lexing or parsing.

Fixes rdar://10056932&8319466

llvm-svn: 139086
2011-09-04 03:32:15 +00:00
Argyrios Kyrtzidis
a3deaeeb52 Fix Lexer::ComputePreamble when MaxLines parameter is non-zero.
The function was only counting lines that included tokens and not empty lines,
but MaxLines (mainly initiated to the line where the code-completion point resides)
is a count of overall lines (even empty ones).

llvm-svn: 139085
2011-09-04 03:32:04 +00:00
Douglas Gregor
081425343b Introduce support for a simple module import declaration, which
loads the named module. The syntax itself is intentionally hideous and
will be replaced at some later point with something more
palatable. For now, we're focusing on the semantics:
  - Module imports are handled first by the preprocessor (to get macro
  definitions) and then the same tokens are also handled by the parser
  (to get declarations). If both happen (as in normal compilation),
  the second one is redundant, because we currently have no way to
  hide macros or declarations when loading a module. Chris gets credit
  for this mad-but-workable scheme.
  - The Preprocessor now holds on to a reference to a module loader,
  which is responsible for loading named modules. CompilerInstance is
  the only important module loader: it now knows how to create and
  wire up an AST reader on demand to actually perform the module load.
  - We search for modules in the include path, using the module name
  with the suffix ".pcm" (precompiled module) for the file name. This
  is a temporary hack; we hope to improve the situation in the
  future.

llvm-svn: 138679
2011-08-26 23:56:07 +00:00
Argyrios Kyrtzidis
7aecbc7661 Make Lexer::ComputePreamble accept a LangOptions parameter, otherwise it may be
out-of-sync how a file is compiled. Patch by Matthias Kleine!

llvm-svn: 138580
2011-08-25 20:39:19 +00:00
Argyrios Kyrtzidis
f6a3b0ca4b In Lexer::isAtEndOfMacroExpansion use SourceManager::isInFileID and avoid
the extra SourceManager::getFileID call.

llvm-svn: 138376
2011-08-23 21:02:30 +00:00
Argyrios Kyrtzidis
161868db4c Make Lexer::GetBeginningOfToken able to handle macro arg expansion locations.
llvm-svn: 137795
2011-08-17 00:31:23 +00:00
Craig Topper
54edccafc5 Add support for C++0x raw string literals.
llvm-svn: 137298
2011-08-11 04:06:15 +00:00
Anna Zaks
59a3c80717 Add a utility function to the Lexer, which makes it easier to find a token after the given location. (It is a generalized version of trans::findLocationAfterSemi from ArcMigrate, which will be changed to use the Lexer utility).
llvm-svn: 136268
2011-07-27 21:43:43 +00:00
Douglas Gregor
fb65e592e0 Add support for C++0x unicode string and character literals, from Craig Topper!
llvm-svn: 136210
2011-07-27 05:40:30 +00:00
Chandler Carruth
ee4c1d1298 Migrate 'Instantiation' data and API bits of SLocEntry to 'Expansion'
etc. With this I think essentially all of the SourceManager APIs are
converted. Comments and random other bits of cleanup should be all thats
left.

llvm-svn: 136057
2011-07-26 04:56:51 +00:00
Chandler Carruth
73ee5d7fae Convert InstantiationInfo and much of the related code to ExpansionInfo
and various other 'expansion' based terms. I've tried to reformat where
appropriate and catch as many references in comments but I'm going to do
several more passes. Also I've tried to expand parameter names to be
more clear where appropriate.

llvm-svn: 136056
2011-07-26 04:41:47 +00:00
Chandler Carruth
115b077f30 Rename create(MacroArg)InstantiationLoc to create(MacroArg)ExpansionLoc.
llvm-svn: 136054
2011-07-26 03:03:05 +00:00
Chandler Carruth
ca757587a3 Rename SourceManager::getImmediateInstantiationRange to
getImmediateExpansionRange.

llvm-svn: 135960
2011-07-25 20:52:21 +00:00
Chandler Carruth
6d28d7f2a3 Rename SourceManager::getInstantiationRange to getExpansionRange.
llvm-svn: 135915
2011-07-25 16:56:02 +00:00
Chandler Carruth
35f5320d8e Mechanically rename SourceManager::getInstantiationLoc and
FullSourceLoc::getInstantiationLoc to ...::getExpansionLoc. This is part
of the API and documentation update from 'instantiation' as the term for
macros to 'expansion'.

llvm-svn: 135914
2011-07-25 16:49:02 +00:00
Chris Lattner
0e62c1cc0b remove unneeded llvm:: namespace qualifiers on some core types now that LLVM.h imports
them into the clang namespace.

llvm-svn: 135852
2011-07-23 10:55:15 +00:00
Joerg Sonnenberger
da5d2b761a Spelling
llvm-svn: 135545
2011-07-20 00:14:37 +00:00
Douglas Gregor
925296b4c2 Revamp the SourceManager to separate the representation of parsed
source locations from source locations loaded from an AST/PCH file.

Previously, loading an AST/PCH file involved carefully pre-allocating
space at the beginning of the source manager for the source locations
and FileIDs that correspond to the prefix, and then appending the
source locations/FileIDs used for parsing the remaining translation
unit. This design forced us into loading PCH files early, as a prefix,
whic has become a rather significant limitation.

This patch splits the SourceManager space into two parts: for source
location "addresses", the lower values (growing upward) are used to
describe parsed code, while upper values (growing downward) are used
for source locations loaded from AST/PCH files. Similarly, positive
FileIDs are used to describe parsed code while negative FileIDs are
used to file/macro locations loaded from AST/PCH files. As a result,
we can load PCH/AST files even during parsing, making various
improvemnts in the future possible, e.g., teaching #include <foo.h> to
look for and load <foo.h.gch> if it happens to be already available.

This patch was originally written by Sebastian Redl, then brought
forward to the modern age by Jonathan Turner, and finally
polished/finished by me to be committed.

llvm-svn: 135484
2011-07-19 16:10:42 +00:00
Chandler Carruth
e2c09ebcaa Convert terminology in the Lexer from 'instantiate' and variants to
'expand'. Also update the public API it provides to the new term, and
propagate that update to the various clients.

No functionality changed.

llvm-svn: 135138
2011-07-14 08:20:40 +00:00
Argyrios Kyrtzidis
61c58f7f43 Move SourceManager::isAt[Start/End]OfMacroInstantiation functions to the Lexer, since they depend on it now.
llvm-svn: 134644
2011-07-07 21:54:45 +00:00
Argyrios Kyrtzidis
41fb2d95a3 Make the Preprocessor more memory efficient and improve macro instantiation diagnostics.
When a macro instantiation occurs, reserve a SLocEntry chunk with length the
full length of the macro definition source. Set the spelling location of this chunk
to point to the start of the macro definition and any tokens that are lexed directly
from the macro definition will get a location from this chunk with the appropriate offset.

For any tokens that come from argument expansion, '##' paste operator, etc. have their
instantiation location point at the appropriate place in the instantiated macro definition
(the argument identifier and the '##' token respectively).
This improves macro instantiation diagnostics:

Before:

t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
        ^~~~
t.c:5:11: note: instantiated from:
int y = M(/);
          ^

After:

t.c:5:9: error: invalid operands to binary expression ('struct S' and 'int')
int y = M(/);
        ^~~~
t.c:3:20: note: instantiated from:
\#define M(op) (foo op 3);
                ~~~ ^  ~
t.c:5:11: note: instantiated from:
int y = M(/);
          ^

The memory savings for a candidate boost library that abuses the preprocessor are:

- 32% less SLocEntries (37M -> 25M)
- 30% reduction in PCH file size (900M -> 635M)
- 50% reduction in memory usage for the SLocEntry table (1.6G -> 800M)

llvm-svn: 134587
2011-07-07 03:40:34 +00:00
Argyrios Kyrtzidis
2cfce18645 Allow Lexer::getLocForEndOfToken to return the location just passed the macro instantiation
if the location given points at the last token of the macro instantiation.

Fixes rdar://9045701.

llvm-svn: 133804
2011-06-24 17:58:59 +00:00
Eli Friedman
86a5101c27 Don't strlen() every file before parsing it.
llvm-svn: 131132
2011-05-10 17:11:21 +00:00
Chris Lattner
57540c5be0 fix a bunch of comment typos found by codespell. Patch by
Luis Felipe Strano Moraes!

llvm-svn: 129559
2011-04-15 05:22:18 +00:00
Richard Smith
f7b6202e6c Implement C++0x [lex.pptoken]p3's handling of <::.
llvm-svn: 129525
2011-04-14 18:36:27 +00:00
Eric Christopher
7f36a79ee9 Eat the UTF-8 BOM at the beginning of a file since it's ignored anyhow.
Nom Nom Nom.

Patch by Anton Korobeynikov!

llvm-svn: 129174
2011-04-09 00:01:04 +00:00
John McCall
75ca6d72c2 Fix getLocForEndOfToken to not double-count spurious internal characters
within a token, like trigraphs and escaped newlines.               
Patch by Marcin Kowalczyk!

llvm-svn: 128978
2011-04-06 01:50:22 +00:00
Daniel Dunbar
1057f86d0e Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPL
comments.

llvm-svn: 127910
2011-03-18 21:23:38 +00:00
John McCall
462c055d85 Fix my earlier commit to work with escaped newlines and leave breadcrumbs
in case we want to make a world where we can check intermediate instantiations
for this kind of breadcrumb.

llvm-svn: 127221
2011-03-08 07:59:04 +00:00
Peter Collingbourne
2f1e36bfd0 Rename tok::eom to tok::eod.
The previous name was inaccurate as this token in fact appears at
the end of every preprocessing directive, not just macro definitions.
No functionality change, except for a diagnostic tweak.

llvm-svn: 126631
2011-02-28 02:37:51 +00:00
Argyrios Kyrtzidis
c541ade850 Warn for missing terminating " or ' instead of error for gcc compatibility. Fixed rdar://8914293.
llvm-svn: 125616
2011-02-15 23:45:31 +00:00
Peter Collingbourne
c1270f51fa Lexer: add CUDA kernel call tokens
llvm-svn: 125218
2011-02-09 21:08:21 +00:00
Douglas Gregor
86af98444f Harden Lexer::GetBeginningOfToken() against bogus source locations and
the disappearance/alteration of files.

llvm-svn: 124616
2011-01-31 22:42:36 +00:00
Abramo Bagnara
ea4f7c7761 Introduced raw_identifier token kind.
llvm-svn: 122394
2010-12-22 08:23:18 +00:00
Chris Lattner
39720111e0 move getSpelling from Preprocessor to Lexer, which it is more conceptually related to.
llvm-svn: 119479
2010-11-17 07:26:20 +00:00
Chris Lattner
2a6ee91619 move AdvanceToTokenCharacter and getLocForEndOfToken from
Preprocessor to Lexer where they make more sense.

llvm-svn: 119474
2010-11-17 07:05:50 +00:00
Chandler Carruth
c3ce5840af Update remaining attribute macros to new style.
llvm-svn: 117204
2010-10-23 08:44:57 +00:00
Sebastian Redl
517523014d In MeasureTokenLength, the FileLoc supplied to the lexer must point to the start of the buffer, or we risk overflow.
llvm-svn: 115117
2010-09-30 01:03:03 +00:00
Chris Lattner
0f0492e69c improve isHexaLiteral to work with escaped newlines and trigraphs,
patch by Francois Pichet!

llvm-svn: 112602
2010-08-31 16:42:00 +00:00
Chris Lattner
dec7334218 silence a warning
llvm-svn: 112549
2010-08-30 23:11:03 +00:00
Alexis Hunt
3b7918625c Revert my user-defined literal commits - r1124{58,60,67} pending
some issues being sorted out.

llvm-svn: 112493
2010-08-30 17:47:05 +00:00
Chris Lattner
5f183aa592 add a fixme.
llvm-svn: 112491
2010-08-30 17:11:14 +00:00
Chris Lattner
7a9e9e7d76 use 'features' instead of 'PP->getLangOptions'.
llvm-svn: 112490
2010-08-30 17:09:08 +00:00