In order to make tooling around dwarf health easier, I've added an `--verify-json` option to `llvm-dwarfdump --verify` that will spit out error summary data with counts to a JSON file.
I've added the same capability to `llvm-gsymutil` in a [different PR.](https://github.com/llvm/llvm-project/pull/81763)
The format of the json is:
``` json
{
"error-categories": {
"<first category description>": {"count": 1234},
"<next category description>": {"count":4321}
},
"error-count": 5555
}
```
for a clean run:
``` json
{
"error-categories": {},
"error-count": 0
}
```
---------
Co-authored-by: Kevin Frei <freik@meta.com>
We recently ran into some bad DWARF where the `DW_AT_stmt_list` of many
compile units was randomly set to invalid values and was causing LLDB to
crash due to an assertion about address sizes not matching. Instead of
asserting, we should return an appropriate recoverable `llvm::Error`.
Refactor the code that calculates the offsets for the various pieces of
the DWARF .debug_names index section, to make it easier to share the
code with other tools, such as LLD.
GsymUtil, like DwarfDump --verify, spews a *lot* of data necessary to
understand/diagnose issues with DWARF data. The trouble is that the kind
of information necessary to make the messages useful also makes them
nearly impossible to easily categorize. I put together a similar output
categorizer (https://github.com/llvm/llvm-project/pull/79648) that will
emit a summary of issues identified at the bottom of the (very verbose)
output, enabling easier tracking of issues as they arise or are
addressed.
There's a single output change, where a message "warning: Unable to
retrieve DWO .debug_info section for some object files. (Remove the
--quiet flag for full output)" was being dumped the first time it was
encountered (in what looks like an attempt to make something easily
grep-able), but rather than keep the output in the same order, that
message is now a 'category' so gets emitted at the end of the output.
The test 'tools/llvm-gsymutil/X86/elf-dwo.yaml' was changed to reflect
this difference.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
The DWARF 5 debug_str_offsets section starts with a header, which must
be skipped in order to access the underlying `strp`s.
However, the verifier supports some pre-standardization version of this
section (with the same section name), which does not have a header. In
this case, the offsets start on the first byte of the section. More in
[1] and [2] about this legacy section.
How does The DWARF verifier figure out which version to use? It manually
reads the **first** header in debug_info and uses that. This is wrong
when multiple debug_str_offset sections have been linked together, in
particular it is wrong in the following two cases:
1. A standard DWARF 4 object file (i.e. no debug_str_offsets) linked
with a standard DWARF 5 object file.
2. A non-standard DWARF 4 object file (i.e. containing the header-less
debug_str_offsets section) linked with a standard DWARF 5 object file.
Based on discussions in https://github.com/llvm/llvm-project/pull/81210,
the legacy version is only possible with dwo files, and dwo files cannot
mix the legacy version with the dwarf 5 version. As such, we change the
verifier to only check the debug_info header in the case of dwo files.
If it sees a dwarf 4 version, it handles it the legacy way.
Note: the modified test was technically testing an unsupported
combination of dwarf version + non-dwo sections. To see why, simply note
that the test contained no `debug_info.dwo` sections, so the call to
DWARFObject::forEachInfoDWOSections was doing nothing. We were finding
the error through the "standard version", which shouldn't happen.
[1]: https://gcc.gnu.org/wiki/DebugFission
[2]: https://gcc.gnu.org/wiki/DebugFissionDWP
This adds the following values to the CodeView.h enums (and updates the
various functions that use them):
* CPUType:
* Added `Unknown`
* This is not currently documented in the online documentation, but this
is present in `cvconst.h` in the latest DIA SDK (Visual Studio 2022,
17.7.6)
* `Unknown` is the CPUType that is emitted by `aliasobj.exe` in the
Compile3Sym records, and can be found in objects that link with
`oldnames.lib`
* SourceLanguage (All of these are documented at
https://learn.microsoft.com/en-us/visualstudio/debugger/debug-interface-access/cv-cfl-lang?view=vs-2022
and are present in `cvconst.h` in the latest DIA SDK (Visual Studio
2022, 17.7.6))
* Added Go
* Added AliasObj
* emitted by `aliasobj.exe` in certain records, can be found in PDBs
that link with `oldnames.lib`
* Changed Swift to the official Microsoft enumeration
* Added `OldSwift`
* The old Swift enumeration of `S` was changed to `OldSwift` to allow
pdb dumping utilities to continue to emit correct source language
information for old PDBs
### WARNING
The `Swift` change is a potentially breaking change, as the swift
compiler will now emit `0x13` for the SourceLanguage type in PDB records
instead of `S`. This could potentially break utilities that relied on
the old enum value.
* CallType
* Added Swift
* This is not currently documented in the online documentation, but this
is present in `cvconst.h` in the latest DIA SDK (Visual Studio 2022,
17.7.6)
A line table whose sole entry is an end sequence should not have the
entry's file index verified, as that value corresponds to the initial
value of the state machine, not to a real file index. In DWARF 5, this
is particularly problematic as it uses 0-based indexing, and the state
machine specifies a starting index of 1; in other words, you'd need to
have _two_ files before such index became legal "by default".
A previous attempt to fix this problem was done [1], but it was too
specific in its condition, and did not capture all possible cases where
this issue can happen.
[1]: https://github.com/llvm/llvm-project/pull/77004
The new experimental calling convention preserve_none is the opposite
side of existing preserve_all. It tries to preserve as few general
registers as possible. So all general registers are caller saved
registers. It can also uses more general registers to pass arguments.
This attribute doesn't impact floating-point registers. Floating-point
registers still follow the c calling convention.
Currently preserve_none is supported on X86-64 only. It changes the c
calling convention in following fields:
* RSP and RBP are the only preserved general registers, all other
general registers are caller saved registers.
* We can use [RDI, RSI, RDX, RCX, R8, R9, R11, R12, R13, R14, R15, RAX]
to pass arguments.
It can improve the performance of hot tailcall chain, because many
callee saved registers' save/restore instructions can be removed if the
tail functions are using preserve_none. In my experiment in protocol
buffer, the parsing functions are improved by 3% to 10%.
This stemps from conversatin in:
https://github.com/llvm/llvm-project/pull/77457#discussion_r1457889792.
Right now Abbrev code for abbrev is combination of DIE TAG and other
attributes.
In the future it will be changed to be an index. Since DenseSet does not
preserve an order, added a sort based on abbrev code. Once change to
index is
made, it will print out abbrevs in the order they are stored.
The amount and format of output from `llvm-dwarfdump --verify` makes it
quite difficult to know if a change to a tool that produces or modifies
DWARF is causing new problems, or is fixing existing problems. This diff
adds a categorized summary of issues found by the DWARF verifier, on by
default, at the bottom of the error output.
The change includes a new `--error-display` option with 4 settings:
* `--error-display=quiet`: Only display if errors occurred, but no
details or summary are printed.
* `--error-display=summary`: Only display the aggregated summary of
errors with no error detail.
* `--error-display=details`: Only display the detailed error messages
with no summary (previous behavior)
* `--error-display=full`: Display both the detailed error messages and
the aggregated summary of errors (the default)
I changed a handful of tests that were failing due to new output, adding
the flag to use the old behavior for all but a couple. For those two I
added the new aggregated output to the expected output of the test.
The `OutputCategoryAggregator` is a pretty simple little class that
@clayborg suggested to allow code to only be run to dump detail if it's
enabled, while still collating counts of the category. Knowing that the
lambda passed in is only conditionally executed is pretty important
(handling errors has to be done *outside* the lambda). I'm happy to move
this somewhere else (and change/improve it) to be more broadly useful if
folks would like.
---------
Co-authored-by: Kevin Frei <freik@meta.com>
FileCheck test added
```
./bin/llvm-lit -sv llvm/test/tools/llvm-gsymutil/X86/elf-dwo.yaml
```
Manual test steps:
- Create binary with split-dwarf:
```
clang++ -g -gdwarf-4 -gsplit-dwarf main.cpp -o main_split
```
- Remove or remane the dwo file to a different name so llvm-gsymutil can't find it
```
mv main_split-main.dwo main_split-main__.dwo
```
- Now run llvm-gsymutil conversion, it should print out warning with and
without the `--quiet` flag
```
$ ./bin/llvm-gsymutil --convert=./main_split
Input file: ./main_split
Output file (x86_64): ./main_split.gsym
warning: Unable to retrieve DWO .debug_info section for main_split-main.dwo
Loaded 0 functions from DWARF.
Loaded 12 functions from symbol table.
Pruned 0 functions, ended with 12 total
```
```
$ ./bin/llvm-gsymutil --convert=./main_split --quiet
Input file: ./main_split
Output file (x86_64): ./main_split.gsym
warning: Unable to retrieve DWO .debug_info section for some object files. (Remove the --quiet flag for full output)
Pruned 0 functions, ended with 12 total
```
TableEntry names are pointers into the string table section, and
accessing their
length requires a search for `\0`. However, 99% of the time we only need
to
compare the name against some other other, and such a comparison will
fail as
early as the first character.
This commit adds a method to the interface of TableEntry so that such a
comparison can be done without extracting the full name. It saves 10% in
the
time (1250ms -> 1100 ms) to evaluate the following expression.
```
lldb \
--batch \
-o "b CodeGenFunction::GenerateCode" \
-o run \
-o "expr Fn" \
-- \
clang++ -c -g test.cpp -o /dev/null &> output
```
The current implementation of DebugNames is _only_ using hashes to
compute the bucket number. Once inside the bucket, it reverts back to
string comparisons, even though not all hashes inside a bucket are
identical.
This commit changes the behavior so that we check the hash before
comparing strings. Such check is so important that it speeds up a simple
benchmark by 20%. In other words, the following expression evaluation
time goes from 1100ms to 850ms.
```
bin/lldb \
--batch \
-o "b CodeGenFunction::GenerateCode" \
-o run \
-o "expr Fn" \
-- \
clang++ -c -g test.cpp -o /dev/null &> output
```
(Note, these numbers are considering the usage of IDX_parent)
Summary:
FileEntry.Dir can be empty if debug info only contains relative path.
This caused an assertion failure when gsym segmentation is trying to
copy a file entry with empty dir. As the fitst entry of StringTable is
always empty (and is preserved), `StringOffsetMap` doesn't have key 0.
Hence, `find(0)` returns `End` and `operator->()` fails the assertion
Test Plan:
./bin/llvm-lit -sv llvm/test/tools/llvm-gsymutil/X86/elf-empty-dir.yaml
This commit introduces a helper function to DWARFAcceleratorTable::Entry
which follows DW_IDX_Parent attributes to returns the corresponding
parent Entry in the table.
It is tested by enhancing dwarfdump so that it now prints:
1. When data is corrupt.
2. When parent information is present, but the parent is not indexed.
3. The parent entry offset, when the parent is present and indexed. This
is printed in terms a real entry offset (the same that gets printed at
the start of each entry: "Entry @ 0x..."), instead of the encoded number
in the table (which is an offset from the start off the Entry list).
This makes it easy to visually inspect the dwarfdump and check what the
parent is.
This implements the ideas discussed in [1].
To summarize, this commit changes AsmPrinter so that it outputs
DW_IDX_parent information for debug_name entries. It will enable
debuggers to speed up queries for fully qualified types (based on a
DWARFDeclContext) significantly, as debuggers will no longer need to
parse the entire CU in order to inspect the parent chain of a DIE.
Instead, a debugger can simply take the parent DIE offset from the
accelerator table and peek at its name in the debug_info/debug_str
sections.
The implementation uses two types of DW_FORM for the DW_IDX_parent
attribute:
1. DW_FORM_ref4, which points to the accelerator table entry for the
parent.
2. DW_FORM_flag_present, when the entry has a parent that is not in the
table (that is, the parent doesn't have a name, or isn't allowed to be
in the table as per the DWARF spec). This is space-efficient, since it
takes 0 bytes.
The implementation works by:
1. Changing how abbreviations are encoded (so that they encode which
form, if
any, was used to encode IDX_Parent)
2. Creating an MCLabel per accelerator table entry, so that they may be
referred by IDX_parent references.
When all patches related to this are merged, we are able to show that
evaluating an expression such as:
```
lldb --batch -o 'b CodeGenFunction::GenerateCode' -o run -o 'expr Fn' -- \
clang++ -c -g test.cpp -o /dev/null
```
is far faster: from ~5000 ms to ~1500ms.
Building llvm-project + clang with and without this patch, and looking
at its impact on object file size:
```
ls -la $(find build_stage2_Debug_idx_parent_assert_dwarf5 -name \*.cpp.o) | awk '{s+=$5} END {printf "%\047d\n", s}'
11,507,327,592
-la $(find build_stage2_Debug_no_idx_parent_assert_dwarf5 -name \*.cpp.o) | awk '{s+=$5} END {printf "%\047d\n", s}'
11,436,446,616
```
That is, an increase of 0.62% in total object file size.
Looking only at debug_names:
```
$stage1_build/bin/llvm-objdump --section-headers $(find build_stage2_Debug_idx_parent_assert_dwarf5 -name \*.cpp.o) | grep __debug_names | awk '{s+="0x"$3} END {printf "%\047d\n", s}'
440,772,348
$stage1_build/bin/llvm-objdump --section-headers $(find build_stage2_Debug_no_idx_parent_assert_dwarf5 -name \*.cpp.o) | grep __debug_names | awk '{s+="0x"$3} END {printf "%\047d\n", s}'
369,867,920
```
That is an increase of 19%.
DWARF Linkers need to be changed in order to support this. This commit
already brings support to "base" linker, but it does not attempt to
modify the parallel linker. Accelerator entries refer to the
corresponding DIE offset, and this patch also requires the parent DIE
offset -- it's not clear how the parallel linker can access this. It may
be obvious to someone familiar with it, but it would be nice to get help
from its authors.
[1]:
https://discourse.llvm.org/t/rfc-improve-dwarf-5-debug-names-type-lookup-parsing-speed/74151/
Allow the dumping of .dwo files contents to show up when dumping an
executable with split DWARF.
Currently if you run llvm-dwarfdump on a binary that has skeleton
compile units, you only see the skeleton compile units. Since the main
binary has the linked addresses it would be nice to be able to dump
DWARF from the .dwo files and how the resolved addresses instead of
showing the address index and "<unresolved>" in the output. This patch
adds an option that can be specified to dump the non skeleton DIEs named
--dwo.
Added the ability to use the following options with split dwarf as well:
--name <name>
--lookup <addr>
--debug-info <die-offset>
If there is no file name in the prologue of a line table, the verifier
will try to verify the file index, which will be set to 1 by default.
This will cause the DWARF verifier to throw an error even if there is no
error.
rdar://114476503
rdar://114343624
GNU addr2line supports lookup by symbol name in addition to the existing
address lookup. llvm-symbolizer starting from
e144ae54dcb96838a6176fd9eef21028935ccd4f supports lookup by symbol name.
This change extends this lookup with possibility to specify optional
offset.
Now the address for which source information is searched for can be
specified with offset:
llvm-symbolize --obj=abc.so "SYMBOL func_22+0x12"
It decreases the gap in features of llvm-symbolizer and GNU addr2line.
This lookup now is supported for code only.
Migrated from: https://reviews.llvm.org/D139859
Pull request: https://github.com/llvm/llvm-project/pull/75067
This is a follow up patch after .debug_names can now emit local type
unit entries when we compile with type units + DWARF5 + .debug_names.
The pull request that added this functionality was:
https://github.com/llvm/llvm-project/pull/70515
This patch makes sure that the DebugNamesDWARFIndex in LLDB will not
manually need to parse type units if they have a valid index. It also
fixes the index to be able to correctly extract name entries that
reference type unit DIEs. Added a test to verify things work as
expected.
llvm-gsymutil allows address ranges to overlap. There was a bug where if
we had debug info for a function with a range like [0x100-0x200) and a
symbol at the same start address yet with a larger range like
[0x100-0x300), we would randomly get either only information from the
first or second entry. This could cause lookups to fail due to the way
the binary search worked.
This patch makes sure that when lookups happen we find the first address
table entry that can match an address, and also ensures that we always
select the first FunctionInfo that could match. FunctionInfo entries are
sorted such that the most debug info rich entries come first. And if we
have two ranges that have the same start address, the smaller range
comes first and the larger one comes next. This patch also adds the
ability to iterate over all function infos with the same start address
to always find a range that contains the address.
Added a unit test to test this functionality that failed prior to this
fix and now succeeds.
Also fix an issue when dumping an entire GSYM file that has duplicate address entries where it used to always print out the binary search match for the FunctionInfo, not the actual data for the address index.
The DWARFUnitVector class lives inside of the DWARFContextState. Prior
to this fix a non const reference was being handed out to clients. When
fetching the DWO units, there used to be a "bool Lazy" parameter that
could be passed that would allow the DWARFUnitVector to parse individual
units on the fly. There were two major issues with this approach:
- not thread safe and causes crashes
- the accessor would check if DWARFUnitVector was empty and if not empty
it would return a partially filled in DWARFUnitVector if it was
constructed with "Lazy = true"
This patch fixes the issues by always fully parsing the DWARFUnitVector
when it is requested and only hands out a "const DWARFUnitVector &".
This allows the thread safety mechanism built into the DWARFContext
class to work corrrectly, and avoids the issue where if someone
construct DWARFUnitVector with "Lazy = true", and then calls an API that
partially fills in the DWARFUnitVector with individual entries, and then
someone accesses the DWARFUnitVector, they would get a partial and
incomplete listing of the DWARF units for the DWOs.
DWARF produced by LTO and BOLT can sometimes be broken where file
indexes are beyond the end of the line table's file list in the
prologue. This patch allows llvm-gsymutil to convert this DWARF without
crashing, and emits errors when:
line table contains entries with an invalid file index (line entry will
be removed) inline functions that have invalid DW_AT_call_file file
indexes when there are no line table entries for a function and we fall
back to making a single line table entry from the functions
DW_AT_decl_file/DW_AT_decl_line attributes, we make sure the
DW_AT_decl_file attribute is valid before emitting it.
Fix line table lookups in line tables with multiple lines with the same
address.
Compilers emit line tables that have multiple line table entries with
the same address. When doing lookups, we always need to use the last
line entry if a lookup address matches the address of one or more line
entries. This is because the size of an address range for a line uses
the next line entry to figure out how big the current line entry is. If
the next line entry has the same address, that means the current line
entry has a size of zero, so no bytes correspond to the line entry.
This patch ensures that lookups will always pick the last matching line
entry when the lookup address matches more than one line entry.
llvm/include/llvm/Support/VersionTuple.h doesn't need anything from
llvm/Support/Endian.h, but llvm/lib/DebugInfo/BTF/BTFParser.cpp relies
on a transitive inclusion of llvm/Support/Endian.h.
Recent versions of GNU binutils starting from 2.39 support symbol+offset
lookup in addition to the usual numeric address lookup. This change adds
symbol lookup to llvm-symbolize and llvm-addr2line.
Now llvm-symbolize behaves closer to GNU addr2line, - if the value specified
as address in command line or input stream is not a number, it is treated as
a symbol name. For example:
llvm-symbolize --obj=abc.so func_22
llvm-symbolize --obj=abc.so "CODE func_22"
This lookup is now supported only for functions. Specification with
offset is not supported yet.
This is a recommit of 2b27948783e4bbc1132d3220d8517ef62607b558, reverted
in 39fec5457c0925bd39f67f63fe17391584e08258 because the test
llvm/test/Support/interrupts.test started failing on Windows. The test was
changed in 18f036d0105589c3175bb51a518c5d272dae61e2 and is also updated in
this commit.
Differential Revision: https://reviews.llvm.org/D149759
C++20 comes with std::erase to erase a value from std::vector. This
patch renames llvm::erase_value to llvm::erase for consistency with
C++20.
We could make llvm::erase more similar to std::erase by having it
return the number of elements removed, but I'm not doing that for now
because nobody seems to care about that in our code base.
Since there are only 50 occurrences of erase_value in our code base,
this patch replaces all of them with llvm::erase and deprecates
llvm::erase_value.
These files satisfy all of the following:
- misc-include-cleaner indicates that these files do not need
Endian.h.
- They do not mention "endian" anywhere.
- They do not include any *.inc or *.def, which could need
llvm::support::endian.
