summaryrefslogtreecommitdiffstats
path: root/clang-tools-extra/clangd/index
Commit message (Collapse)AuthorAgeFilesLines
...
* [clangd] Fix buildbot failure.Haojian Wu2018-10-171-3/+5
| | | | llvm-svn: 344680
* [clangd] Print numbers of symbols and refs as well when loading theHaojian Wu2018-10-171-2/+7
| | | | | | index. llvm-svn: 344679
* [clangd] Collect refs from headers.Haojian Wu2018-10-174-17/+42
| | | | | | | | | | | | | | | | | Summary: Add a flag to SymbolCollector to collect refs fdrom headers. Note that we collect refs from headers in static index, and we don't do it for dynamic index because of the preamble (we skip function body in preamble, collecting it will result incomplete results). Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53322 llvm-svn: 344678
* [clangd] Fix threading bugs in (not-yet-used) BackgroundIndex, re-enable test.Sam McCall2018-10-162-3/+6
| | | | | | | | | | | | | | | | | | | Summary: One relatively boring bug: forgot to notify the CV after enqueue. One much more fun bug: the thread member could access instance variables before they were initialized. Although the thread was last in the init list, QueueCV etc were listed after Thread in the class, so their default constructors raced with the thread itself. We have to get very unlucky to lose this race, I saw it 0.02% of the time. Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, jfb, cfe-commits Differential Revision: https://reviews.llvm.org/D53313 llvm-svn: 344595
* [clangd] Optionally use dex for the preamble parts of the dynamic index.Sam McCall2018-10-164-16/+40
| | | | | | | | | | | | | | | Summary: Reuse the old -use-dex-index experiment flag for this. To avoid breaking the tests, make Dex deduplicate symbols, addressing an old FIXME. Reviewers: hokein Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53288 llvm-svn: 344594
* [clangd] Revert include path change in Dexp. NFCSam McCall2018-10-152-4/+2
| | | | llvm-svn: 344533
* [clangd] Add createIndex in dexpHaojian Wu2018-10-151-7/+9
| | | | | | | | | | | | | | | Summary: This would allow easily injecting our internal customization. Also updates the stale "symbol-collection-file" flag. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53292 llvm-svn: 344521
* [clangd] Use SyncAPI in more places in tests. NFCSam McCall2018-10-151-0/+3
| | | | llvm-svn: 344520
* [clangd] Minimal implementation of automatic static index (not enabled).Sam McCall2018-10-152-0/+270
| | | | | | | | | | | | | | | | | | | | | | | | | | Summary: See tinyurl.com/clangd-automatic-index for design and goals. Lots of limitations to keep this patch smallish, TODOs everywhere: - no serialization to disk - no changes to dynamic index, which now has a much simpler job - no partitioning of symbols by file to avoid duplication of header symbols - no reindexing of edited files - only a single worker thread - compilation database is slurped synchronously (doesn't scale) - uses memindex, rebuilds after every file (should be dex, periodically) It's not hooked up to ClangdServer/ClangdLSPServer yet: the layering isn't clear (it should really be in ClangdServer, but ClangdLSPServer has all the CDB interactions). Reviewers: ioeric Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, jfb, cfe-commits Differential Revision: https://reviews.llvm.org/D53032 llvm-svn: 344513
* [clangd] Remove an unused include header, NFC.Haojian Wu2018-10-151-1/+0
| | | | llvm-svn: 344510
* [clangd] dump xrefs information in dexp tool.Haojian Wu2018-10-152-9/+93
| | | | | | | | | | Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53019 llvm-svn: 344508
* [clangd] Fix some references missing in dynamic index.Haojian Wu2018-10-151-17/+19
| | | | | | | | | | | | | | | | | | | Summary: Previously, SymbolCollector postfilters all references at the end to find all references of interesting symbols. It was incorrect when indxing main AST where we don't see locations of symbol declarations and definitions in the main AST (as those are in preamble AST). The fix is to do earily check during collecting references. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53273 llvm-svn: 344507
* [clangd] NFC fix semicolon warningJonas Toth2018-10-121-1/+1
| | | | llvm-svn: 344384
* [clangd] Fix an accident change in r342999.Haojian Wu2018-10-091-1/+1
| | | | llvm-svn: 344054
* [clangd] fix miscompiling lower_bound callJonas Toth2018-10-091-2/+3
| | | | llvm-svn: 344044
* [clangd] NFC: Migrate to LLVM STLExtras API where possibleKirill Bobyrev2018-10-077-26/+20
| | | | | | | | | | | | | | This patch improves readability by migrating `std::function(ForwardIt start, ForwardIt end, ...)` to LLVM's STLExtras range-based equivalent `llvm::function(RangeT &&Range, ...)`. Similar change in Clang: D52576. Reviewed By: sammccall Differential Revision: https://reviews.llvm.org/D52650 llvm-svn: 343937
* [clangd] Remove last usage of ast matchers from SymbolCollector. NFCSam McCall2018-10-051-4/+2
| | | | llvm-svn: 343849
* [clangd] Simplify Dex query tree logic and fix missing-posting-list bugSam McCall2018-10-044-43/+28
| | | | | | | | | | | | | | | | | | | | | | | | Summary: The bug being fixed: when a posting list doesn't exist in the index, it was previously just dropped from the query rather than being treated as empty. Now that we have the FALSE iterator, we can use it instead. The query tree logic previously had a bunch of special cases to detect whether subtrees are empty. Now we just naively build the whole tree, and rely on the query optimizations to drop the trivial parts. Finally, there was a bug in trigram generation: the empty query would generate a single trigram "$$$" instead of no trigrams. This had no effect (there was no posting list, so the other bug cancelled it out). But we now have to fix this bug too. Reviewers: ilya-biryukov Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52796 llvm-svn: 343802
* [clangd] Dex: FALSE iterator, peephole optimizations, fix AND bugSam McCall2018-10-041-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | Summary: The FALSE iterator will be used in a followup patch to fix a logic bug in Dex (currently, tokens that don't have posting lists in the index are simply dropped from the query, changing semantics). It can usually be optimized away, so added the following opmitizations: - simplify booleans inside AND/OR - replace effectively-empty AND/OR with booleans - flatten nested AND/ORs While working on this, found a bug in the AND iterator: its constructor sync() assumes that ReachedEnd is set if applicable, but the constructor never sets it. This crashes if a non-first iterator is nonempty. Reviewers: ilya-biryukov Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52789 llvm-svn: 343801
* [clangd] expose MergedIndex classSam McCall2018-10-044-116/+109
| | | | | | | | | | | | | | Summary: This allows inheriting from it, so index() can ga away and allowing TestTU::index) to be fixed. Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52250 llvm-svn: 343780
* [clangd] clangd-indexer gathers refs and stores them in index files.Sam McCall2018-10-045-17/+176
| | | | | | | | | | Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52531 llvm-svn: 343778
* [clangd] Remove one-segment-skipping from Dex trigrams.Sam McCall2018-10-042-5/+3
| | | | | | | | | | | | | | | Summary: Currently queries like "ab" can match identifiers like a_yellow_bee. The value of allowing this for exactly one segment but no more seems dubious. It costs ~3% of overall ram (~9% of posting list ram) and some quality. Reviewers: ilya-biryukov, ioeric Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52885 llvm-svn: 343777
* [cland] Dex: fix/simplify short-trigram generationSam McCall2018-10-043-88/+43
| | | | | | | | | | | | | | | | Summary: 1) Instead of x$$ for a short-query trigram, just use x 2) Make rules more coherent: prefixes of length 1-2, and first char + next head 3) Fix Dex::fuzzyFind to mark results as incomplete, because short-trigram rules only yield a subset of results. Reviewers: ioeric Subscribers: ilya-biryukov, jkorous, mgrang, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52808 llvm-svn: 343775
* [clangd] Dex: FALSE iterator, peephole optimizations, fix AND bugSam McCall2018-10-042-13/+108
| | | | | | | | | | | | | | | | | | | | | | | | Summary: The FALSE iterator will be used in a followup patch to fix a logic bug in Dex (currently, tokens that don't have posting lists in the index are simply dropped from the query, changing semantics). It can usually be optimized away, so added the following opmitizations: - simplify booleans inside AND/OR - replace effectively-empty AND/OR with booleans - flatten nested AND/ORs While working on this, found a bug in the AND iterator: its constructor sync() assumes that ReachedEnd is set if applicable, but the constructor never sets it. This crashes if a non-first iterator is nonempty. Reviewers: ilya-biryukov Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52789 llvm-svn: 343774
* [clangd] Support refs() in dex. Largely cloned from MemIndex.Sam McCall2018-10-043-18/+32
| | | | | | | | | | Reviewers: hokein Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52726 llvm-svn: 343760
* [clangd] clangd-indexer: Drop support for MR-via-YAMLSam McCall2018-10-042-16/+1
| | | | | | | | | | | | | | Summary: It's slow, and the open-source reduce implementation doesn't scale properly. While here, tidy up some dead headers and comments. Reviewers: kadircet Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D52517 llvm-svn: 343759
* Reland r343589 "[clangd] Dex: add Corpus factory for iterators, rename, fold ↵Sam McCall2018-10-025-88/+94
| | | | | | | | constant. NFC"" This reverts commit r343610. llvm-svn: 343622
* Revert r343589 "[clangd] Dex: add Corpus factory for iterators, rename, fold ↵Reid Kleckner2018-10-025-94/+88
| | | | | | | | | | | | | | constant. NFC" Declaring a field with the same name as a type causes GCC to error out: Dex.h:104:10: error: declaration of 'clang::clangd::dex::Corpus clang::clangd::dex::Dex::Corpus' [-fpermissive] Corpus Corpus; ^ Iterator.h:127:7: error: changes meaning of 'Corpus' from 'class clang::clangd::dex::Corpus' [-fpermissive] class Corpus { llvm-svn: 343610
* [clangd] Zap TODONEsSam McCall2018-10-021-3/+0
| | | | llvm-svn: 343590
* [clangd] Dex: add Corpus factory for iterators, rename, fold constant. NFCSam McCall2018-10-025-88/+94
| | | | | | | | | | | | | | | | | | | Summary: - Corpus avoids having to pass size to the true iterator, and (soon) any iterator that might optimize down to true. - Shorten names of factory functions now they're scoped to the Corpus. intersect() and unionOf() rather than createAnd() or createOr() as this seems to read better to me, and fits with other short names. Opinion wanted! - DEFAULT_BOOST_SCORE --> 1. This is a multiplier, don't obfuscate identity. - Simplify variadic templates in Iterator.h Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52711 llvm-svn: 343589
* [clangd] Dex iterator printer shows query structure, not iterator state.Sam McCall2018-10-025-30/+48
| | | | | | | | | | | | | | Summary: This makes it suitable for logging (which immediately found a bug, to be fixed in the next patch...) Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52715 llvm-svn: 343580
* [clangd] Query dex index using query-style trigrams, not identifier-style ↵Sam McCall2018-10-011-1/+1
| | | | | | trigrams llvm-svn: 343453
* [clangd] Fix header mapping for std::string. NFCEric Liu2018-10-011-0/+1
| | | | | | Some implementation has std::string declared in <iosfwd>. llvm-svn: 343448
* [clangd] Initial supoprt for cross-namespace global code completion.Eric Liu2018-09-273-1/+14
| | | | | | | | | | | | | | | | | | | | | Summary: When no scope qualifier is specified, allow completing index symbols from any scope and insert proper automatically. This is still experimental and hidden behind a flag. Things missing: - Scope proximity based scoring. - FuzzyFind supports weighted scopes. Reviewers: sammccall Reviewed By: sammccall Subscribers: kbobyrev, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52364 llvm-svn: 343248
* [clangd] Add more tracing to index queries. NFCEric Liu2018-09-273-2/+28
| | | | | | | | | | Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52611 llvm-svn: 343247
* [clangd] Fix bugs with incorrect memory estimate reportKirill Bobyrev2018-09-263-11/+12
| | | | | | | | | | | | | | * With the current implementation, `sizeof(std::vector<Chunk>)` is added twice to the `Dex` memory estimate which is incorrect * `Dex` logs memory usage estimation before `BackingDataSize` is set and hence the log report excludes size of the external `SymbolSlab` which is coupled with `Dex` instance Reviewed By: ioeric Differential Revision: https://reviews.llvm.org/D52503 llvm-svn: 343117
* [docs] Update PostingList string representation formatKirill Bobyrev2018-09-261-3/+2
| | | | | | | | | | | | Because `PostingList` objects are compressed, it is now impossible to see elements other than the current one and the documentation doesn't match implementation anymore. Reviewed By: ioeric Differential Revision: https://reviews.llvm.org/D52545 llvm-svn: 343116
* Removed extra semicolon to fix Wpedantic. (NFCI).Simon Pilgrim2018-09-261-1/+1
| | | | llvm-svn: 343083
* [clangd] Extract mapper logic from clangd-indexer into a library.Sam McCall2018-09-254-9/+120
| | | | | | | | | | | | | Summary: Soon we can drop support for MR-via-YAML. I need to modify some out-of-tree versions to use the library, first. Reviewers: kadircet Subscribers: mgorny, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D52465 llvm-svn: 343019
* [clangd] Fix reversed RIFF/YAML serializationSam McCall2018-09-251-2/+2
| | | | llvm-svn: 343017
* [clangd] Merge binary + YAML serialization behind a (mostly) common interface.Sam McCall2018-09-255-121/+130
| | | | | | | | | | | | | | | | | Summary: Interface is in one file, implementation in two as they have little in common. A couple of ad-hoc YAML functions left exposed: - symbol -> YAML I expect to keep for tools like dexp - YAML -> symbol is used for the MR-style indexer, I think we can eliminate this (merge-on-the-fly, else use a different serialization) Reviewers: kbobyrev Subscribers: mgorny, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52453 llvm-svn: 342999
* [clangd] NFC: Simplify code, enforce LLVM Coding StandardsKirill Bobyrev2018-09-256-73/+73
| | | | | | | | | | | | | | | | For consistency, functional-style code pieces are replaced with their simple counterparts to improve readability. Also, file headers are fixed to comply with LLVM Coding Standards. `static` member of anonymous namespace is not marked `static` anymore, because it is redundant. Reviewed By: sammccall Differential Revision: https://reviews.llvm.org/D52466 llvm-svn: 342974
* [clangd] Fix some buildbots after r342965Kirill Bobyrev2018-09-251-1/+1
| | | | | | Some compilers fail to parse struct default member initializer. llvm-svn: 342970
* [clangd] Implement VByte PostingList compressionKirill Bobyrev2018-09-253-38/+210
| | | | | | | | | | | | | | | | | | | | | | | | | | | This patch implements Variable-length Byte compression of `PostingList`s to sacrifice some performance for lower memory consumption. `PostingList` compression and decompression was extensively tested using fuzzer for multiple hours and runnning significant number of realistic `FuzzyFindRequests`. AddressSanitizer and UndefinedBehaviorSanitizer were used to ensure the correct behaviour. Performance evaluation was conducted with recent LLVM symbol index (292k symbols) and the collection of user-recorded queries (7751 `FuzzyFindRequest` JSON dumps): | Metrics | Before| After | Change (%) | ----- | ----- | ----- | ----- | Memory consumption (posting lists only), MB | 54.4 | 23.5 | -60% | Time to process queries, sec | 7.70 | 9.4 | +25% Reviewers: sammccall, ioeric Reviewed By: sammccall Differential Revision: https://reviews.llvm.org/D52300 llvm-svn: 342965
* [clangd] Fix uninit bool in r342888Sam McCall2018-09-241-1/+1
| | | | llvm-svn: 342903
* [clangd] Do bounds checks while reading data, otherwise var-length records ↵Sam McCall2018-09-241-139/+138
| | | | | | are too painful. NFC llvm-svn: 342888
* [clangd] Force Dex to respect symbol collector flagsKirill Bobyrev2018-09-241-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | `Dex` should utilize `FuzzyFindRequest.RestrictForCodeCompletion` flags and omit symbols not meant for code completion when asked for it. The measurements below were conducted with setting `FuzzyFindRequest.RestrictForCodeCompletion` to `true` (so that it's more realistic). Sadly, the average latency goes down, I suspect that is mostly because of the empty queries where the number of posting lists is critical. | Metrics | Before | After | Relative difference | ----- | ----- | ----- | ----- | Cumulative query latency (7000 `FuzzyFindRequest`s over LLVM static index) | 6182735043 ns | 7202442053 ns | +16% | Whole Index size | 81.24 MB | 81.79 MB | +0.6% Out of 292252 symbols collected from LLVM codebase 136926 appear to be restricted for code completion. Reviewers: ioeric Differential Revision: https://reviews.llvm.org/D52357 llvm-svn: 342866
* [clangd] Remember to serialize symbol origin in YAML.Eric Liu2018-09-211-0/+15
| | | | llvm-svn: 342730
* [clangd] Store preamble macros in dynamic index.Eric Liu2018-09-191-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Pros: o Loading macros from preamble for every completion is slow (see profile). o Calculating macro USR is also slow (see profile). o Sema can provide a lot of macro completion results (e.g. when filter is empty, 60k for some large TUs!). Cons: o Slight memory increase in dynamic index (~1%). o Some extra work during preamble build (should be fine as preamble build and indexAST is way slower). Before: {F7195645} After: {F7195646} Reviewers: ilya-biryukov, sammccall Reviewed By: sammccall Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52078 llvm-svn: 342529
* [clangd] Fix error handling for SymbolID parsing (notably YAML and dexp)Sam McCall2018-09-184-19/+21
| | | | llvm-svn: 342505
OpenPOWER on IntegriCloud