summaryrefslogtreecommitdiffstats
path: root/clang/lib/Lex/Lexer.cpp
Commit message (Collapse)AuthorAgeFilesLines
...
* Fix getLocForEndOfToken to not double-count spurious internal characters John McCall2011-04-061-1/+1
| | | | | | | within a token, like trigraphs and escaped newlines. Patch by Marcin Kowalczyk! llvm-svn: 128978
* Lexer: Add extremely limited support for -traditional-cpp, ignoring BCPLDaniel Dunbar2011-03-181-3/+5
| | | | | | comments. llvm-svn: 127910
* Fix my earlier commit to work with escaped newlines and leave breadcrumbsJohn McCall2011-03-081-0/+48
| | | | | | | in case we want to make a world where we can check intermediate instantiations for this kind of breadcrumb. llvm-svn: 127221
* Rename tok::eom to tok::eod.Peter Collingbourne2011-02-281-7/+7
| | | | | | | | The previous name was inaccurate as this token in fact appears at the end of every preprocessing directive, not just macro definitions. No functionality change, except for a diagnostic tweak. llvm-svn: 126631
* Warn for missing terminating " or ' instead of error for gcc compatibility. ↵Argyrios Kyrtzidis2011-02-151-2/+2
| | | | | | Fixed rdar://8914293. llvm-svn: 125616
* Lexer: add CUDA kernel call tokensPeter Collingbourne2011-02-091-0/+8
| | | | llvm-svn: 125218
* Harden Lexer::GetBeginningOfToken() against bogus source locations andDouglas Gregor2011-01-311-0/+6
| | | | | | the disappearance/alteration of files. llvm-svn: 124616
* Introduced raw_identifier token kind.Abramo Bagnara2010-12-221-25/+24
| | | | llvm-svn: 122394
* move getSpelling from Preprocessor to Lexer, which it is more conceptually ↵Chris Lattner2010-11-171-0/+101
| | | | | | related to. llvm-svn: 119479
* move AdvanceToTokenCharacter and getLocForEndOfToken fromChris Lattner2010-11-171-0/+77
| | | | | | Preprocessor to Lexer where they make more sense. llvm-svn: 119474
* Update remaining attribute macros to new style.Chandler Carruth2010-10-231-4/+2
| | | | llvm-svn: 117204
* In MeasureTokenLength, the FileLoc supplied to the lexer must point to the ↵Sebastian Redl2010-09-301-1/+2
| | | | | | start of the buffer, or we risk overflow. llvm-svn: 115117
* improve isHexaLiteral to work with escaped newlines and trigraphs,Chris Lattner2010-08-311-7/+8
| | | | | | patch by Francois Pichet! llvm-svn: 112602
* silence a warningChris Lattner2010-08-301-1/+1
| | | | llvm-svn: 112549
* Revert my user-defined literal commits - r1124{58,60,67} pendingAlexis Hunt2010-08-301-29/+2
| | | | | | some issues being sorted out. llvm-svn: 112493
* add a fixme.Chris Lattner2010-08-301-1/+5
| | | | llvm-svn: 112491
* use 'features' instead of 'PP->getLangOptions'.Chris Lattner2010-08-301-8/+8
| | | | llvm-svn: 112490
* In Microsoft compatibility mode, don't parse the exponent as part ofDouglas Gregor2010-08-301-1/+10
| | | | | | | the pp-number in a hexadecimal floating point literal, from Francois Pichet! Fixes PR7968. llvm-svn: 112481
* Implement C++0x user-defined string literals.Alexis Hunt2010-08-291-2/+29
| | | | | | | | | | The extra data stored on user-defined literal Tokens is stored in extra allocated memory, which is managed by the PreprocessorLexer because there isn't a better place to put it that makes sure it gets deallocated, but only after it's used up. My testing has shown no significant slowdown as a result, but independent testing would be appreciated. llvm-svn: 112458
* Introduce a preprocessor code-completion hook for contexts where weDouglas Gregor2010-08-251-6/+22
| | | | | | | expect "natural" language and should not provide any completions, e.g., comments, string literals, #error. llvm-svn: 112054
* Introduce basic code-completion support for preprocessor directives,Douglas Gregor2010-08-241-18/+18
| | | | | | e.g., after a "#" we'll suggest #if, #ifdef, etc. llvm-svn: 111943
* Don't emit end-of-file diagnostics like "unterminated conditional" orDouglas Gregor2010-08-121-4/+7
| | | | | | "unterminated string" when we're performing code completion. llvm-svn: 110933
* Random temporary string cleanup.Benjamin Kramer2010-08-111-1/+1
| | | | llvm-svn: 110807
* Use precompiled preambles for in-process code completion.Douglas Gregor2010-08-091-2/+16
| | | | llvm-svn: 110596
* Introduce basic support for loading a precompiled preamble whileDouglas Gregor2010-07-261-2/+13
| | | | | | | | | | | | | | | | | | | | | | | reparsing an ASTUnit. When saving a preamble, create a buffer larger than the actual file we're working with but fill everything from the end of the preamble to the end of the file with spaces (so the lexer will quickly skip them). When we load the file, create a buffer of the same size, filling it with the file and then spaces. Then, instruct the lexer to start lexing after the preamble, therefore continuing the parse from the spot where the preamble left off. It's now possible to perform a simple preamble build + parse (+ reparse) with ASTUnit. However, one has to disable a bunch of checking in the PCH reader to do so. That part isn't committed; it will likely be handled with some other kind of flag (e.g., -fno-validate-pch). As part of this, fix some issues with null termination of the memory buffers created for the preamble; we were trying to explicitly NULL-terminate them, even though they were also getting implicitly NULL terminated, leading to excess warnings about NULL characters in source files. llvm-svn: 109445
* Improve performance during cursor traversal when a region of interestDouglas Gregor2010-07-221-0/+53
| | | | | | | | | | | | | | | | | | | | | is present. Rather than using clang_getCursorExtent(), which requires us to lex the token at the ending position to determine its length. Then, we'd be comparing [a, b) source ranges that cover the characters in the range rather than the normal behavior for Clang's source ranges, which covers the tokens in the range. However, relexing causes us to read the source file (which may come from a precompiled header), which is rather unfortunate and affects performance. In the new scheme, we only use Clang-style source ranges that cover the tokens in the range. At the entry points where this matters (clang_annotateTokens, clang_getCursor), we make sure to move source locations to the start of the token. Addresses most of <rdar://problem/8049381>. llvm-svn: 109134
* Introduce a new lexer function to compute the "preamble" of a file,Douglas Gregor2010-07-201-0/+125
| | | | | | | | | which is the part of the file that contains all of the initial comments, includes, and preprocessor directives that occur before any of the actual code. Added a new -print-preamble cc1 action that is only used for testing. llvm-svn: 108913
* fix PR4499, patch by Kyle Dean!Chris Lattner2010-07-071-24/+16
| | | | llvm-svn: 107836
* simpler fix for rdar://8044135 - escaped newlines have alreadyChris Lattner2010-05-301-10/+7
| | | | | | been processed, so they don't have to be tip-toed around. llvm-svn: 105182
* Improve our handling of NULL after an escaping '\' in a stringDouglas Gregor2010-05-301-2/+7
| | | | | | literal. Fixes <rdar://problem/8044135>. llvm-svn: 105181
* Improve code completion in failure cases in two ways:Douglas Gregor2010-05-251-0/+3
| | | | | | | | | | | 1) Suppress diagnostics as soon as we form the code-completion token, so we don't get any error/warning spew from the early end-of-file. 2) If we consume a code-completion token when we weren't expecting one, go into a code-completion recovery path that produces the best results it can based on the context that the parser is in. llvm-svn: 104585
* robustify the conflict marker stuff. Don't add 7 twice, which wouldChris Lattner2010-05-171-1/+2
| | | | | | | | | | | | | | | | | | | | | make it miss (invalid) things like: <<<<<<< >>>>>>> and crash if <<<<<<< was at the end of the line. When we find a >>>>>>> that is not at the end of the line, make sure to reset Pos so we don't crash on something like: <<<<<<< >>>>>>> This isn't worth making testcases for, since each would require a new file. rdar://7987078 - signal 11 compiling "<<<<<<<<<<" llvm-svn: 103968
* when code completing inside a C-style block comment, don't emit errors aboutChris Lattner2010-05-161-2/+3
| | | | | | | | a missing */ since we truncated the file. This fixes rdar://7948776 llvm-svn: 103913
* fix a minor bug I noticed while work with Jordy's patch for PR6101,Chris Lattner2010-04-121-2/+4
| | | | | | | | | | | | | | | | | in an input file like this: # 42 int x; we were emitting: # <something> int x; (with a space before the int) because we weren't clearing the leading whitespace flag properly after the \n from the directive was handled. llvm-svn: 101084
* Reinstate my CodeModificationHint -> FixItHint renaming patch, withoutDouglas Gregor2010-03-311-2/+1
| | | | | | the C-only "optimization". llvm-svn: 100022
* Revert r100008, which inexplicably breaks the clang-i686-darwin10 builderDouglas Gregor2010-03-311-1/+2
| | | | llvm-svn: 100018
* Rename CodeModificationHint to FixItHint, since we've been using theDouglas Gregor2010-03-311-2/+1
| | | | | | | term "fix-it" everywhere and even *I* get tired of long names sometimes. No functionality change. llvm-svn: 100008
* Remove unused variableDouglas Gregor2010-03-161-1/+0
| | | | llvm-svn: 98691
* Audit all Preprocessor::getSpelling() callers, improving failureDouglas Gregor2010-03-161-1/+5
| | | | | | recovery for those that need it. llvm-svn: 98689
* Audit all callers of SourceManager::getCharacterData(); update some ofDouglas Gregor2010-03-161-0/+1
| | | | | | them to recover more gracefully on failure. llvm-svn: 98672
* Let SourceManager::getBufferData return StringRef instead of a pair of two ↵Benjamin Kramer2010-03-161-5/+4
| | | | | | const char*. llvm-svn: 98630
* Give SourceManager a Diagnostic object with which to report errors,Douglas Gregor2010-03-161-5/+3
| | | | | | and start simplifying the interfaces in SourceManager that can fail. llvm-svn: 98594
* Introduce a new BufferResult class to act as the return type ofDouglas Gregor2010-03-151-1/+8
| | | | | | | | | | | | | | SourceManager's getBuffer() (and similar) operations. This abstract can be used to force callers to cope with errors in getBuffer(), such as missing files and changed files. Fix a bunch of callers to use the new interface. Add some very basic checks for file consistency (file size, modification time) into ContentCache::getBuffer(), although these checks don't help much until we've updated the main callers (e.g., SourceManager::getSpelling()). llvm-svn: 98585
* don't inform comment handlers about comments in #if 0 blocks,Chris Lattner2010-02-031-7/+9
| | | | | | | | | doing so invalidates the file guard optimization and is not in the spirit of "#if 0" because it is supposed to completely skip everything, even if it isn't lexically valid. Patch by Abramo Bagnara! llvm-svn: 95253
* Teach CIndex's cursor visitor to restrict its traversal to a specificDouglas Gregor2010-01-221-0/+4
| | | | | | | | | | | | | | region of interest (if provided). Implement clang_getCursor() in terms of this traversal rather than using the Index library; the unified cursor visitor is more complete, and will be The Way Forward. Minor other tweaks needed to make this work: - Extend Preprocessor::getLocForEndOfToken() to accept an offset from the end, making it easy to move to the last character in the token (rather than just past the end of the token). - In Lexer::MeasureTokenLength(), the length of whitespace is zero. llvm-svn: 94200
* allow the HandlerComment callback to push tokens into theChris Lattner2010-01-181-14/+24
| | | | | | | preprocessor. This could be used by an OpenMP implementation or something. Patch by Abramo Bagnara! llvm-svn: 93795
* add a TODO for a perf improvement in LexIdentifier.Chris Lattner2010-01-111-2/+5
| | | | llvm-svn: 93141
* Do not parse hexadecimal floating point literals in C++0x mode because they areAlexis Hunt2010-01-101-1/+2
| | | | | | | | | | | | | | | | | | incompatible with user-defined literals, specifically with the following form: 0x1p+1 The preprocessing-number token extends only as far as the 'p'; the '+' is not included. Previously we could get away with this extension as p was an invalid suffix, but now with user-defined literals, 'p' might well be a valid suffix and we are forced to consider it as such. This patch also adds a warning in non-0x C++ modes telling the user that this extension is incompatible with C++0x that is enabled by default (previously and with other languages, we warn only with a compliance option such as -pedantic). llvm-svn: 93135
* reimplement r90860, fixing a couple of problems:Chris Lattner2009-12-171-9/+21
| | | | | | | | | 1. Don't make a copy of LangOptions every time a lexer is created. 2. Don't make CharInfo global mutable state. 3. Fix the implementation to properly treat ^Z as EOF instead of as horizontal whitespace, which matches the semantic implemented by VC++. llvm-svn: 91586
* teach clang to recover gracefully from conflict markers left in sourceChris Lattner2009-12-141-14/+133
| | | | | | files: PR5238. llvm-svn: 91270
OpenPOWER on IntegriCloud