summaryrefslogtreecommitdiffstats
path: root/clang-tools-extra/clangd/index/Serialization.cpp
Commit message (Collapse)AuthorAgeFilesLines
* [clangd] Use our own relation kind.Haojian Wu2019-10-171-27/+2
| | | | | | | | | | | | | | | | Summary: Move the RelationKind from Serialization.h to Relation.h. This patch doesn't introduce any breaking changes. Reviewers: kadircet Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D68981 llvm-svn: 375117
* [clangd] Also accept flags from CLANGD_FLAGS variable.Sam McCall2019-07-251-2/+2
| | | | | | | | | | | | | | | | This simplifies various workflows, particularly in debugging/development. e.g. editors will tend to propagate flags, so you can run `env CLANGD_FLAGS=-input-mirror-file=/tmp/mirror vim foo.cc` rather than change the configuration in a persistent way. (This also gives us a generic lever when we don't know how to customize the flags in some particular LSP client). While here, add a test for this and other startup logging, and fix a couple of direct writes to errs() that should have been logs. Differential Revision: https://reviews.llvm.org/D65153 llvm-svn: 366991
* [clangd] Use xxhash instead of SHA1 for background index file digests.Sam McCall2019-07-081-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Currently SHA1 is about 10% of our CPU, this patch reduces it to ~1%. xxhash is a well-defined (stable) non-cryptographic hash optimized for fast checksums (like crc32). Collisions shouldn't be a problem, despite the reduced length: - for actual file content (used to invalidate bg index shards), there are only two versions that can collide (new shard and old shard). - for file paths in bg index shard filenames, we would need 2^32 files with the same filename to expect a collision. Imperfect hashing may reduce this a bit but it's well beyond what's plausible. This will invalidate shards on disk (as usual; I bumped the version), but this time the filenames are changing so the old files will stick around :-( So this is more expensive than the usual bump, but would be good to land before the v9 branch when everyone will start using bg index. Reviewers: kadircet Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, llvm-commits Tags: #llvm Differential Revision: https://reviews.llvm.org/D64306 llvm-svn: 365311
* [clangd] Add HadErrors field into shardsKadir Cetinkaya2019-07-041-3/+4
| | | | | | | | | | | | Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D64133 llvm-svn: 365122
* [clangd] Store hash of command line in index shards.Kadir Cetinkaya2019-07-041-0/+60
| | | | | | | | | | | | | | Summary: This is to enable cache invalidation when command line flags changes. Reviewers: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D64018 llvm-svn: 365121
* [clangd] Index API and implementations for relationsNathan Ridge2019-06-151-2/+4
| | | | | | | | | | | | | | | | | Summary: This builds on the relations support added in D59407, D62459, and D62471 to expose relations in SymbolIndex and its implementations. Reviewers: kadircet Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D62839 llvm-svn: 363481
* [clangd] Serialization support for RelationSlabNathan Ridge2019-06-031-2/+81
| | | | | | | | | | | | | | Summary: This builds on D59407 to provide YAML and RIFF serialization support. Reviewers: kadircet Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D62459 llvm-svn: 362353
* [clangd] Bump index version and get rid of wrong assertionKadir Cetinkaya2019-05-091-1/+1
| | | | | | | | | | | | | | | | | | | | Summary: After rL360344, BackgroundIndex expects symbols with zero refcounts. Therefore existing index files are no longer valid. Assertion regarding finding target of a reference was wrong, since background-index might still be going on or we might've loaded only some part of the shards and might be missing the declaring shards for the symbol. Reviewers: ilya-biryukov Subscribers: MaskRay, jkorous, arphaman, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D61734 llvm-svn: 360349
* [clangd] Fix serialization logic for Origin and Flags.Kadir Cetinkaya2019-04-291-2/+2
| | | | llvm-svn: 359470
* [clangd] Bump clangd-index version for TemplateArgument changesKadir Cetinkaya2019-04-151-1/+1
| | | | llvm-svn: 358383
* [clangd] Add TemplateArgumentList into SymbolKadir Cetinkaya2019-04-121-0/+2
| | | | | | | | | | | | | | | | | | | | | Summary: Part of re-landing rC356541 with D59599. Changes the way we store template arguments, previous patch was storing them inside Name field of Symbol. Which was violating the assumption: ```Symbol::Scope+Symbol::Name == clang::clangd::printQualifiedName``` which was made in multiple places inside codebase. This patch instead moves those arguments into their own field. Currently the field is meant to be human-readable, can be made structured if need be. Reviewers: ioeric, ilya-biryukov, gribozavr Subscribers: MaskRay, jkorous, arphaman, jdoerfert, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D59640 llvm-svn: 358273
* Moved SymbolOrigin into its own header and implementation fileDmitri Gribenko2019-02-281-0/+1
| | | | | | | | | | | | Reviewers: ioeric Subscribers: mgorny, jkorous, arphaman, kadircet, jdoerfert, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D58773 llvm-svn: 355086
* Moved SymbolLocation into its own header and implementation fileDmitri Gribenko2019-02-281-1/+1
| | | | | | | | | | | | Reviewers: ioeric Subscribers: mgorny, jkorous, arphaman, kadircet, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D58768 llvm-svn: 355082
* [clangd] NFC: fix clang-tidy warnings.Haojian Wu2019-01-251-3/+3
| | | | | | | Most are about llvm code style violation (found via readability-identifier-naming check). llvm-svn: 352205
* Update the file headers across all of the LLVM projects in the monorepoChandler Carruth2019-01-191-4/+3
| | | | | | | | | | | | | | | | | to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
* [clangd] Remove 'using namespace llvm' from .cpp files. NFCIlya Biryukov2019-01-071-66/+72
| | | | | | | | The new guideline is to qualify with 'llvm::' explicitly both in '.h' and '.cpp' files. This simplifies moving the code between header and source files and is easier to keep consistent. llvm-svn: 350531
* [clangd] Put direct headers into srcs section.Kadir Cetinkaya2018-11-271-11/+64
| | | | | | | | | | | | | | | | | | | Summary: Currently, there's no way of knowing about header files using compilation database, since it doesn't contain header files as entries. Using this information, restoring from cache using compile commands becomes possible instead of doing directory traversal. Also, we can issue indexing actions for out-of-date headers even if source files depending on them haven't changed. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D54817 llvm-svn: 347669
* [clangd] Collect and store expected types in the indexIlya Biryukov2018-11-261-1/+3
| | | | | | | | | | | | | | | | | | | | Summary: And add a hidden option to control whether the types are collected. For experiments, will be removed when expected types implementation is stabilized. The index size is almost unchanged, e.g. the YAML index for all clangd sources increased from 53MB to 54MB. Reviewers: ioeric, sammccall Reviewed By: sammccall Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52274 llvm-svn: 347560
* [clangd] Cleanup: stop passing around list of supported URI schemes.Eric Liu2018-11-221-6/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: Instead of passing around a list of supported URI schemes in clangd, we expose an interface to convert a path to URI using any compatible scheme that has been registered. It favors customized schemes and falls back to "file" when no other scheme works. Changes in this patch are: - URI::create(AbsPath, URISchemes) -> URI::create(AbsPath). The new API finds a compatible scheme from the registry. - Remove URISchemes option everywhere (ClangdServer, SymbolCollecter, FileIndex etc). - Unit tests will use "unittest" by default. - Move "test" scheme from ClangdLSPServer to ClangdMain.cpp, and only register the test scheme when lit-test or enable-lit-scheme is set. (The new flag is added to make lit protocol.test work; I wonder if there is alternative here.) Reviewers: sammccall Reviewed By: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D54800 llvm-svn: 347467
* Address comments.Kadir Cetinkaya2018-11-191-3/+4
| | | | llvm-svn: 347237
* Use digest size instead of hardcoding it.Kadir Cetinkaya2018-11-191-1/+1
| | | | llvm-svn: 347236
* [clangd] Store source file hash in IndexFile{In,Out}Kadir Cetinkaya2018-11-191-0/+13
| | | | | | | | | | | | | | Summary: Puts the digest of the source file that generated the index into serialized index and stores them back on load, if exists. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D54693 llvm-svn: 347235
* [clangd] Truncate SymbolID to 8 bytes.Haojian Wu2018-11-161-2/+2
| | | | | | | | | | | | | | Summary: This is our goal. It has a non-zero rick, but so far we haven't see any collision (externally and internally). Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D54622 llvm-svn: 347044
* [clangd] Replace StringRef in SymbolLocation with a char pointer.Haojian Wu2018-11-141-3/+6
| | | | | | | | | | | | | | | | | Summary: This would save us 8 bytes per ref, and buy us ~40MB in total for llvm index (from ~300MB to ~260 MB). The char pointer must be null-terminated, and llvm::StringSaver guarantees it. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53427 llvm-svn: 346852
* [clangd] Truncate SymbolID to 16 bytes.Sam McCall2018-10-241-2/+2
| | | | | | | | | | | | | | | Summary: The goal is 8 bytes, which has a nonzero risk of collisions with huge indexes. This patch should shake out any issues with truncation at all, we can lower further later. Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53587 llvm-svn: 345113
* [clangd] Namespace style cleanup in cpp files. NFC.Sam McCall2018-10-201-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Standardize on the most common namespace setup in our *.cpp files: using namespace llvm; namespace clang { namespace clangd { void foo(StringRef) { ... } And remove redundant llvm:: qualifiers. (Except for cases like make_unique where this causes problems with std:: and ADL). This choice is pretty arbitrary, but some broad consistency is nice. This is going to conflict with everything. Sorry :-/ Squash the other configurations: A) using namespace llvm; using namespace clang; using namespace clangd; void clangd::foo(StringRef); This is in some of the older files. (It prevents accidentally defining a new function instead of one in the header file, for what that's worth). B) namespace clang { namespace clangd { void foo(llvm::StringRef) { ... } This is fine, but in practice the using directive often gets added over time. C) namespace clang { namespace clangd { using namespace llvm; // inside the namespace This was pretty common, but is a bit misleading: name lookup preferrs clang::clangd::foo > clang::foo > llvm:: foo (no matter where the using directive is). llvm-svn: 344850
* [clangd] Clear the semantic of RefSlab::size.Haojian Wu2018-10-181-5/+5
| | | | | | | | | | | | | | | | | Summary: The RefSlab::size can easily cause confusions, it returns the number of different symbols, rahter than the number of all references. - add numRefs() method and cache it, since calculating it everytime is nontrivial. - clear misused places. Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53389 llvm-svn: 344745
* [clangd] Encode Line/Column as a 32-bits integer.Haojian Wu2018-10-181-4/+4
| | | | | | | | | | | | | | | | | | | | | | Summary: This would buy us more memory. Using a 32-bits integer is enough for most human-readable source code (up to 4M lines and 4K columns). Previsouly, we used 8 bytes for a position, now 4 bytes, it would save us 8 bytes for each Ref and each Symbol instance. For LLVM-project binary index file, we save ~13% memory. | Before | After | | 412MB | 355MB | Reviewers: sammccall Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D53363 llvm-svn: 344735
* [clangd] Print numbers of symbols and refs as well when loading theHaojian Wu2018-10-171-2/+7
| | | | | | index. llvm-svn: 344679
* [clangd] NFC: Migrate to LLVM STLExtras API where possibleKirill Bobyrev2018-10-071-1/+1
| | | | | | | | | | | | | | This patch improves readability by migrating `std::function(ForwardIt start, ForwardIt end, ...)` to LLVM's STLExtras range-based equivalent `llvm::function(RangeT &&Range, ...)`. Similar change in Clang: D52576. Reviewed By: sammccall Differential Revision: https://reviews.llvm.org/D52650 llvm-svn: 343937
* [clangd] clangd-indexer gathers refs and stores them in index files.Sam McCall2018-10-041-1/+63
| | | | | | | | | | Reviewers: ioeric Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52531 llvm-svn: 343778
* [clangd] Support refs() in dex. Largely cloned from MemIndex.Sam McCall2018-10-041-2/+3
| | | | | | | | | | Reviewers: hokein Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52726 llvm-svn: 343760
* [clangd] Fix bugs with incorrect memory estimate reportKirill Bobyrev2018-09-261-2/+8
| | | | | | | | | | | | | | * With the current implementation, `sizeof(std::vector<Chunk>)` is added twice to the `Dex` memory estimate which is incorrect * `Dex` logs memory usage estimation before `BackingDataSize` is set and hence the log report excludes size of the external `SymbolSlab` which is coupled with `Dex` instance Reviewed By: ioeric Differential Revision: https://reviews.llvm.org/D52503 llvm-svn: 343117
* [clangd] Fix reversed RIFF/YAML serializationSam McCall2018-09-251-2/+2
| | | | llvm-svn: 343017
* [clangd] Merge binary + YAML serialization behind a (mostly) common interface.Sam McCall2018-09-251-5/+62
| | | | | | | | | | | | | | | | | Summary: Interface is in one file, implementation in two as they have little in common. A couple of ad-hoc YAML functions left exposed: - symbol -> YAML I expect to keep for tools like dexp - YAML -> symbol is used for the MR-style indexer, I think we can eliminate this (merge-on-the-fly, else use a different serialization) Reviewers: kbobyrev Subscribers: mgorny, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D52453 llvm-svn: 342999
* [clangd] Fix uninit bool in r342888Sam McCall2018-09-241-1/+1
| | | | llvm-svn: 342903
* [clangd] Do bounds checks while reading data, otherwise var-length records ↵Sam McCall2018-09-241-139/+138
| | | | | | are too painful. NFC llvm-svn: 342888
* [clangd] Canonicalize include paths in clangd.Eric Liu2018-09-071-1/+1
| | | | | | Get rid of "../" and "../../". llvm-svn: 341645
* [clangd] Add "Deprecated" field to Symbol and CodeCompletion.Eric Liu2018-09-061-3/+4
| | | | | | | | | | | | | | Summary: Also set "deprecated" field in LSP CompletionItem. Reviewers: sammccall, kadircet Reviewed By: sammccall Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits Differential Revision: https://reviews.llvm.org/D51724 llvm-svn: 341576
* [clangd] make zlib compression optional for binary formatSam McCall2018-09-051-11/+29
| | | | llvm-svn: 341465
* [clangd] Fix buildbot failures on older compilers from r341375Sam McCall2018-09-051-3/+3
| | | | llvm-svn: 341451
* [clangd] Define a compact binary serialization fomat for symbol slab/index.Sam McCall2018-09-041-0/+366
Summary: This is intended to replace the current YAML format for general use. It's ~10x more compact than YAML, and ~40% more compact than gzipped YAML: llvmidx.riff = 20M, llvmidx.yaml = 272M, llvmidx.yaml.gz = 32M It's also simpler/faster to read and write. The format is a RIFF container (chunks of (type, size, data)) with: - a compressed string table - simple binary encoding of symbols (with varints for compactness) It can be extended to include occurrences, Dex posting lists, etc. There's no rich backwards-compatibility scheme, but a version number is included so we can detect incompatible files and do ad-hoc back-compat. Alternatives considered: - compressed YAML or JSON: bulky and slow to load - llvm bitstream: confusing model and libraries are hard to use. My attempt produced slightly larger files, and the code was longer and slower. - protobuf or similar: would be really nice (esp for back-compat) but the dependency is a big hassle - ad-hoc binary format without a container: it seems clear we're going to add posting lists and occurrences here, and that they will benefit from sharing a string table. The container makes it easy to debug these pieces in isolation, and make them optional. Reviewers: ioeric Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits Differential Revision: https://reviews.llvm.org/D51585 llvm-svn: 341375
OpenPOWER on IntegriCloud