Missing Swift Metadata on ELF Platforms
Swift code relies pretty heavily on runtime metadata for various runtime checks
like type identity, reflection, and runtime protocol conformance checks. This
data contains information like type layout, which tools like LLDB use as part of
formatting and interacting with variables. Other tools like
swift-reflection-dump extract this information for testing and analyzing
binaries.
This is the story of how working on Swift support for FreeBSD eventually pushed me to track down a bug that has affected Swift on ELF platforms for the better part of a decade. It ends up taking a pretty deep look at how ELF works and was pretty nasty to resolve across the various versions of linkers that we support.
The Problem
The Structure of Metadata
The compiler stores each kind of metadata in a separate section. There are several sections:
- swift5_protocols
- swift5_protocol_conformances
- swift5_type_metadata
- swift5_typeref
- swift5_reflstr
- swift5_fieldmd
- swift5_assocty
- swift5_replace
- swift5_replac2
- swift5_builtin
- swift5_capture
- swift5_mpenum
- swift5_accessible_functions
- swift5_runtime_attributes
On Mach-O (Apple) platforms, dyld calls a registered callback for each section, so the dynamic loader drives registering the metadata into the image as it’s loading. COFF (Microsoft) and ELF (Linux/Unix/*BSD/basically-everything-else) don’t have that luxury since the loaders aren’t owned by a company with invested interest in supporting Swift.
Instead, we have a constructor function that gathers the sections using the section bound symbols and injects them into the runtime that way.
__attribute__((__constructor__))
static void swift_image_constructor() {
#define SWIFT_SECTION_RANGE(name) \
{ reinterpret_cast<uintptr_t>(&__start_##name), \
static_cast<uintptr_t>(&__stop_##name - &__start_##name) }
const void *baseAddress = nullptr;
if (&__ehdr_start != nullptr) {
baseAddress = __ehdr_start;
}
::new (§ions) swift::MetadataSections {
swift::CurrentSectionMetadataVersion,
baseAddress,
nullptr,
nullptr,
SWIFT_SECTION_RANGE(swift5_protocols),
SWIFT_SECTION_RANGE(swift5_protocol_conformances),
SWIFT_SECTION_RANGE(swift5_type_metadata),
SWIFT_SECTION_RANGE(swift5_typeref),
SWIFT_SECTION_RANGE(swift5_reflstr),
SWIFT_SECTION_RANGE(swift5_fieldmd),
SWIFT_SECTION_RANGE(swift5_assocty),
SWIFT_SECTION_RANGE(swift5_replace),
SWIFT_SECTION_RANGE(swift5_replac2),
SWIFT_SECTION_RANGE(swift5_builtin),
SWIFT_SECTION_RANGE(swift5_capture),
SWIFT_SECTION_RANGE(swift5_mpenum),
SWIFT_SECTION_RANGE(swift5_accessible_functions),
SWIFT_SECTION_RANGE(swift5_runtime_attributes),
SWIFT_SECTION_RANGE(swift5_tests),
};
#undef SWIFT_SECTION_RANGE
swift_addNewDSOImage(§ions);
}
This portion supports the needs of the runtime in a fully loaded and executing process. This has generally worked, or at least appeared to work, for many years.
Static ELF Scanning
The static file extraction is a bit more complicated. Image::scanELF and the
paired templated Image::scanELFType functions effectively implement half of an
ELF dynamic loader.
for (auto &ph : *phdrs) {
if (ph.p_filesz == 0)
continue;
auto contents = O->getData().slice(ph.p_offset, ph.p_offset + ph.p_filesz);
if (contents.empty() || contents.size() != ph.p_filesz)
continue;
Segments.push_back({ph.p_vaddr, contents});
HeaderAddress = std::min(HeaderAddress, (uint64_t)ph.p_vaddr);
}
First it scans over the program headers, extracting out all of the portions of the files into loaded segments.
// Collect the dynamic relocations.
auto resolver = getRelocationResolver(*O);
auto resolverSupports = resolver.first;
auto resolve = resolver.second;
if (!resolverSupports || !resolve)
return;
auto machine = O->getELFFile().getHeader().e_machine;
auto relativeRelocType = llvm::object::getELFRelativeRelocationType(machine);
for (auto &S : static_cast<const llvm::object::ELFObjectFileBase *>(O)
->dynamic_relocation_sections()) {
bool isRela =
O->getSection(S.getRawDataRefImpl())->sh_type == llvm::ELF::SHT_RELA;
for (const llvm::object::RelocationRef &R : S.relocations()) {
// `getRelocationResolver` doesn't handle RELATIVE relocations, so we
// have to do that ourselves.
if (isRela && R.getType() == relativeRelocType) {
auto rela = O->getRela(R.getRawDataRefImpl());
DynamicRelocations.insert(
{R.getOffset(), {{}, HeaderAddress + rela->r_addend}});
continue;
}
if (!resolverSupports(R.getType()))
continue;
auto symbol = R.getSymbol();
auto name = symbol->getName();
if (!name) {
llvm::consumeError(name.takeError());
continue;
}
uint64_t offset = resolve(R.getType(), R.getOffset(), 0, 0, 0);
DynamicRelocations.insert({R.getOffset(), {*name, offset}});
}
}
Then it goes and records the dynamic relocations. The relocations aren’t actually applied to the addresses as they’re loaded though, so it’s kind of in this weird hybrid state of being static data chopped up into segments.
After all of that, we have this lovely little snippet that goes and sticks the entire file in a single segment at the very end with a lovely FIXME comment.
// FIXME: ReflectionContext tries to read bits of the ELF structure that
// aren't normally mapped by a phdr. Until that's fixed,
// allow access to the whole file 1:1 in address space that isn't otherwise
// mapped.
Segments.push_back({HeaderAddress, O->getData()});
So now we have to ask, what is the ReflectionContext doing that is so bad?
Reflection Context Extraction
The above didn’t actually extract any metadata, it just took a set of ELF objects and chopped them up into half-loaded segments while making a note of the dynamic relocations where it could.
To actually extract the metadata segments, we have to turn to
ReflectionContext::readELF. At a high level, it takes a memory reader, and for
each metadata section that we’re looking for, attempts to iterate over each
section header. readELF is a wrapper that mostly just calls down into
readELFSections with a template argument based on the bitness and endianess of
the ELF file. readELFSections is where the actual extraction and registration
takes place.
template <typename T>
std::optional<uint32_t> readELFSections(
RemoteAddress ImageStart,
std::optional<llvm::sys::MemoryBlock> FileBuffer,
llvm::SmallVector<llvm::StringRef, 1> PotentialModuleNames = {}) {
// ...
The function signature kind of gives away that it’s leaking some technical
details through the abstraction. We have a RemoteAddress pointing at the start
of the image that we’re loading. When operating on static object files on disk,
this RemoteAddress comes from the ObjectFileMemoryReader that we constructed
above, that involved chopping the object file into segments. From LLDB, that
points at the actual memory of the running process. But, LLDB actually does
something else, it populates that second FileBuffer field with a value. The
static object memory reader does not. PotentialModuleNames isn’t interesting
for this investgation, so I won’t go into that.
std::vector<MemoryReader::ReadBytesResult> ReadDataBuffer;
auto readData = [&](uint64_t Offset, uint64_t Size) -> const void * {
if (FileBuffer.has_value()) {
auto Buffer = FileBuffer.value();
if (Offset + Size > Buffer.allocatedSize())
return nullptr;
return (const void *)((uint64_t)Buffer.base() + Offset);
} else {
MemoryReader::ReadBytesResult Buf =
this->getReader().readBytes(ImageStart + Offset, Size);
if (!Buf)
return nullptr;
ReadDataBuffer.push_back(std::move(Buf));
return ReadDataBuffer.back().get();
}
};
The first construct declared in the function is a lambda to control whether we read the bytes from the image memory or the object file buffer. If we have a file buffer, we read from that, otherwise we read from the remote address that is supposed to point at the start of the image we’re extracting the metadata from.
Then we read the ELF header so that we can parse the ELF file to navigate to the section header table.
const void *Buf = readData(0, sizeof(typename T::Header));
if (!Buf)
return {};
auto Hdr = reinterpret_cast<const typename T::Header *>(Buf);
assert(Hdr->getFileClass() == T::ELFClass && "invalid ELF file class");
// From the header, grab information about the section header table.
uint64_t SectionHdrAddress = Hdr->e_shoff;
uint16_t SectionHdrNumEntries = Hdr->e_shnum;
uint16_t SectionEntrySize = Hdr->e_shentsize;
// ~~ error handling and non-standard fallback machinery here ~~
// Collect all the section headers, we need them to look up the
// reflection sections (by name) and the string table.
// We read the section headers from the FileBuffer, since they are
// not mapped in the child process.
std::vector<const typename T::Section *> SecHdrVec;
for (unsigned I = 0; I < SectionHdrNumEntries; ++I) {
uint64_t Offset = SectionHdrAddress + (I * SectionEntrySize);
auto SecBuf = readData(Offset, sizeof(typename T::Section));
if (!SecBuf)
return {};
const typename T::Section *SecHdr =
reinterpret_cast<const typename T::Section *>(SecBuf);
SecHdrVec.push_back(SecHdr);
}
Once we’ve collected the section headers, we define a new lambda function
findELFSectionByName. It iterates over each section in the header table that
we collected above.
// Skip unused headers
if (Hdr->sh_type == llvm::ELF::SHT_NULL)
continue;
uint32_t Offset = Hdr->sh_name;
const char *Start = (const char *)StrTab + Offset;
uint64_t StringSize = strnlen(Start, StrTabSize - Offset);
if (StringSize > StrTabSize - Offset) {
Error = true;
break;
}
std::string SecName(Start, StringSize);
if (SecName != Name)
continue;
It skips any uninitialized section headers. For anything that isn’t NULL, it
extracts the name from the header string table, and looks for the one that
matches the name we asked for.
RemoteAddress SecStart = ImageStart + Hdr->sh_addr;
auto SecSize = Hdr->sh_size;
MemoryReader::ReadBytesResult SecBuf;
if (FileBuffer.has_value()) {
// sh_offset gives us the offset to the section in the file,
// while sh_addr gives us the offset in the process.
auto Offset = Hdr->sh_offset;
if (FileBuffer->allocatedSize() < Offset + SecSize) {
Error = true;
break;
}
auto *Buf = malloc(SecSize);
SecBuf = MemoryReader::ReadBytesResult(
Buf, [](const void *ptr) { free(const_cast<void *>(ptr)); });
memcpy((void *)Buf,
(const void *)((uint64_t)FileBuffer->base() + Offset),
SecSize);
} else {
SecBuf = this->getReader().readBytes(SecStart, SecSize);
}
if (!SecBuf)
return {nullptr, 0};
auto SecContents = RemoteRef<void>(SecStart, SecBuf.get());
savedBuffers.push_back(std::move(SecBuf));
return {SecContents, SecSize};
If we find the section name we’re looking for, we find the start of the section, get the size of the section, and attempt to read the data for that section into a buffer.
SwiftObjectFileFormatELF ObjectFileFormat;
auto FieldMdSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::fieldmd), true);
auto AssocTySec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::assocty), true);
auto BuiltinTySec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::builtin), true);
auto CaptureSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::capture), true);
auto TypeRefMdSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::typeref), true);
auto ReflStrMdSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::reflstr), true);
auto ConformMdSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::conform), true);
auto MPEnumMdSec = findELFSectionByName(
ObjectFileFormat.getSectionName(ReflectionSectionKind::mpenum), true);
That process is done for each metadata section of interest. Now, there’s also a
mismatch in the section flags between what the compiler emits and what is
declared in swiftrt.o. The compiler emits the swift5_reflstr and
swift5_typeref sections with the ALLOC flag, but not the RETAIN flag,
while swiftrt.o declares all of the sections with the ALLOC and RETAIN
flag. Most modern linkers know to still merge these sections, but older versions
of gold do not, so there are actually two sections with the same name, so do
the search again looking for the copy of the section without the RETAIN flag.
ReflectionInfo info = {{FieldMdSec.first, FieldMdSec.second},
{AssocTySec.first, AssocTySec.second},
{BuiltinTySec.first, BuiltinTySec.second},
{CaptureSec.first, CaptureSec.second},
{TypeRefMdSec.first, TypeRefMdSec.second},
{ReflStrMdSec.first, ReflStrMdSec.second},
{ConformMdSec.first, ConformMdSec.second},
{MPEnumMdSec.first, MPEnumMdSec.second},
PotentialModuleNames};
return this->addReflectionInfo(info);
Once we find the data, it’s formed into a ReflectionInfo object and registered
with the ReflectionContext object so that tools can operate on it.
The Breakdown
This code path has broken roughly once a year for as long as I’ve worked on Swift. It’s usually something in LLD optimizing things differently and making our metadata disappear on us, but with enough flags and pain, we’ve been able to keep the old code mostly running.
FreeBSD was the straw the broke the proverbial camel’s back and finally got me to look into things more deeply. Lets dive into what the breakdown on FreeBSD looks like.
Ultimately, several reflection tests weren’t passing due to missing metadata and LLDB was struggling as well, saying it couldn’t find the metadata either.
Running swift-reflection-dump on one of these binaries says that there is no
metadata (others would crash).
FIELDS:
=======
ASSOCIATED TYPES:
=================
BUILTIN TYPES:
==============
CAPTURE DESCRIPTORS:
====================
CONFORMANCES:
=============
MULTI-PAYLOAD ENUM DESCRIPTORS:
===============================
Taking one of the instances where swift-reflection-dump was crashing and
running it on the mac, I was able to attach LLDB and get something more
interesting.
uint32_t Offset = Hdr->sh_name;
const char *Start = (const char *)StrTab + Offset;
uint64_t StringSize = strnlen(Start, StrTabSize - Offset);
The code was crashing in the call to strnlen.
[lldb] % p StrTab
(const char *&) 0x016fdfdb09
[lldb] % p StrTabSize
652
[lldb] % p Start
(char char *) 0x08fec1a3af
[lldb] % p Offset
(uint32_t) 4294967295
The string table pointer (StrTab) looks reasonable and it reports that it’s
652 bytes. The Start pointer, which should point somewhere inside of the
string table, and Offset that is used to calculate Start are massive.
In hex, Offset is 0xffffffff. That’s poison.
Offset comes directly from the section header though, so that’s really weird.
{
sh_name = 0xffffffff
sh_type = 0xffffffff
sh_flags = 0x0000000000000000
sh_addr = 0x0000000000000000
sh_offset = 0x0000000000000000
sh_size = 0x0000000000000000
sh_link = 0x0000001d
sh_info = 0x00000000
sh_addralign = 0x00000000000000b6
sh_entsize = 0x0000000000000001
}
Printing the whole section header, it’s pretty corrupted.
We started seeing the corruption in the fourth section header.
% readelf -h -S libL.so
ELF Header:
Magic: 7f 45 4c 46 02 01 01 09 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: FreeBSD
ABI Version: 0
Type: DYN (Shared object file)
Machine: Advanced Micro Devices x86-64
Version: 0x1
Entry point address: 0
Start of program headers: 64 (bytes into file)
Start of section headers: 11072 (bytes into file)
Flags: 0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 10
Size of section headers: 64 (bytes)
Number of section headers: 55
Section header string table index: 53
There are 55 section headers, starting at offset 0x2b40:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
...
[ 3] .dynsym DYNSYM 00000000000002b0 000002b0
00000000000000f0 0000000000000018 A 8 1 8
[ 4] .gnu.version SUNW_versym 00000000000003a0 000003a0
0000000000000014 0000000000000002 A 3 0 2
...
Looking at it in readelf, the header looks fine, so the failure is somewhere
between where we’re looking at the section header and where we read the file,
but the file itself is okay. I also confirmed that the file is fine by opening
the object in a hex editor and manually re-constructing that section header.
ELF
The most important part for understanding the failure is to realize that ELF has two views, the linking view and execution view.
Linking View
┌────────────────────────────────┐
│ │
│ ELF Header │
│ │
├────────────────────────────────┤
│ │
│ Program Header Table │
│ (optional) │
│ │
├────────────────────────────────┤
│ │
│ Section 1 │
│ │
├────────────────────────────────┤
│ │
│ ... │
│ │
├────────────────────────────────┤
│ │
│ Section n │
│ │
├────────────────────────────────┤
│ │
│ ... │
│ │
├────────────────────────────────┤
│ │
│ Section Header Table │
│ │
└────────────────────────────────┘
The linking view is the ELF object file at rest on disk. Operating on the file in the linking view is done using the relative offsets into the file. Extracting information is done through the section header table, which has an offset into the file, and the size of the section, along with some other information.
The program headers are optional according to the spec, but are practically included in the dynamic libraries (shared objects), and omitted from the object files.
Execution View
┌────────────────────────────────┐
│ │
│ ELF Header │
│ │
├────────────────────────────────┤
│ │
│ Program Header Table │
│ │
├────────────────────────────────┤
│ │
│ │
│ Segment 1 │
│ │
│ │
├────────────────────────────────┤
│ │
│ Segment 2 │
│ │
├────────────────────────────────┤
│ │
│ │
│ ... │
│ │
│ │
├────────────────────────────────┤
│ │
│ Section Header Table │
│ (optional) │
│ │
└────────────────────────────────┘
The execution view represents how the loaded image is laid out in memory. Note
that the roles change. All loaded images must have a program header table, but
do not guarantee that the section header table is available in this view.
Instead of sections, the image is now represented by segments, each of which are
described by a program header in the program header table. Note that segments
can overlap, and overlapping segments are allowed to have different flags. Yes,
with the appropriate fun, you can make your .rodata section writeable. Don’t
do that though.
This design makes it relatively easy to only load what the program actually needs into memory for execution. The section header table is generally not needed. Some implementations use the first section header as an overflow for when the ELF header can’t represent things like the number of segments due to size limitations. In these representations, that first section header is actually important in the loaded view, but the other section headers aren’t guaranteed to be loaded.
To uphold alignment guarantees and for performance reasons, ensuring that the
loaded data ends on a power-of-two is usually a requirement. This means that we
may pick up several section headers. Some linkers, like gold, seem to fairly
reliably pull in the entire file. Or at least with the tests that were enabled
on Linux. Relying on this is not well defined though, and LLD, which optimizes
things more vigorously than Gold did, exposes this.
The Fix
Okay, we understand the issue. Swift chops the object file into segments,
reflecting the execution view of an ELF image, and then attempts to locate the
section headers. ELF only guarantees that the section headers are available in
the linking view of the object at rest, not in the execution view. The FIXME
in scanELF tried to work around it by feeding the whole object file in as a
segment, but because the memory load was in one of the earlier segments, the
linear scan through the segments found an earlier segment that didn’t contain
the full section header table and we ran off of what was available in the
execution view.
How do we fix it while staying fully within the realm of well-defined ELF files? Linker sets!
Note Segment
My first attempt was to define a table of start/stop section bound pointers in a special segment. Yes, segment. Note segments are described in ELF to allow programs to add additional metadata. The table contains pointers, so it’s imperative for performance and memory use that the table be aligned.
Unfortunately, this is where the fun of ELF comes in.
The 2001 ELF draft specification says
In 64-bit objects, each entry is an array of 8-byte words in the format of the target processor. In 32-bit objects, each entry is an array of 4-byte words in the format of the target processor.
Great! This should give us that 8-byte alignment that we’re looking for to store our bounds pointers.
Unfortunately, that didn’t work out. The 2001 draft spec was never fully adopted anywhere that I have found.
The 1997 spec and earlier define the note segment as follows
The node information in sections and program header elements holds any number of entires, each of which is an array of 4-byte words in the format of the target processor.
This aligns with how Linux and FreeBSD behave. Of course, this is engrained in ABI, so unlikely to ever change, but it means that the nice idea of sticking the data in a note segment wasn’t what we wanted.
Table Behind a Symbol
After tussling with trying to align something that didn’t want aligning, I took a step back and thought about how we would access the data normally if it weren’t “special”. Normally accessing something from outside of a given file is done through a symbol.
The challenge to this design is that the dynamic symbols in a process address space should all be unique. Having more than one symbol with the same name in the address space would make it impossible for the dynamic linker to determine which one it should use, so it will just choose one at random, and thus it’s undefined which one you’ll get. You can kind of fudge around this if every copy of the symbol point at the same data since you won’t notice, but it’s still considered undefined behavior, which is BAD and something YOU SHOULD NEVER DO.
For this purpose, we would need this symbol to be defined and exposed on every image, pointing at the section boundaries for the metadata for each respective image. So the data is different and there are multiple different addresses with the same symbol name. All of this is BAD and something YOU SHOULD NEVER DO.
So that’s exactly what I’m going to do. :)
It’s undefined behavior because you don’t know which version of the data you’ll
get from the dynamic loader. It relies on unique names to determine which
address to use. In our case, readELFSections gets the ImageStart address as
an input, so we know which image we’re trying to extract the metadata from. So
long as the symbol name only appears once in that image, we can uniquely
extract the data from the image for the process and be okay. The section bounds
are resolved statically by the linker, so they won’t conflict either.
The MemoryReader API only exposed a mechanism for searching for a symbol name
across all images in the reader, not from a specific address, so that was the
first thing to add.
/// Look up the given public symbol name in the remote process.
virtual RemoteAddress getSymbolAddress(const std::string &name) = 0;
/// Lookup the given public symbol name from a given ImageStart address when
/// operating on multi-image processes.
///
/// Performs a full-process lookup if not overridden in a subclass.
virtual RemoteAddress getSymbolAddress(RemoteAddress ImageStart,
const std::string &name) {
return getSymbolAddress(name);
}
The default implementation is boring, but the interesting bit is in the
ObjectMemoryReader and in LLDB.
reflection::RemoteAddress
ObjectMemoryReader::getSymbolAddress(reflection::RemoteAddress imageStart,
const std::string &name) {
const Image *image;
uint64_t imageAddr;
std::tie(image, imageAddr) =
decodeImageIndexAndAddress(imageStart.getRawAddress());
if (!image)
return reflection::RemoteAddress();
auto symbolAddr = image->getSymbolAddress(name);
if (!symbolAddr)
return reflection::RemoteAddress();
return reflection::RemoteAddress(encodeImageIndexAndAddress(
image, remote::RemoteAddress(
*symbolAddr, reflection::RemoteAddress::DefaultAddressSpace)));
}
The
ObjectMemoryReader::getSymbolAddress(reflection::RemoteAddress, const std::string&)
implementation takes an address that has the image slide encoded into it, which
is what gets passed into readELFSections, decodes the image object.
std::optional<uint64_t> Image::getSymbolAddress(StringRef Name) const {
auto findIn = [&](auto &&Symbols) -> std::optional<uint64_t> {
for (const auto &Symb : Symbols) {
auto SymbName = Symb.getName();
if (!SymbName) {
llvm::consumeError(SymbName.takeError());
continue;
}
if (*SymbName != Name)
continue;
auto Value = Symb.getValue();
if (!Value) {
llvm::consumeError(Value.takeError());
continue;
}
return *Value;
}
return std::nullopt;
};
// Prefer the regular symbol table and then fall back to the dynamic symbol
// table.
if (auto Addr = findIn(O->symbols()))
return Addr;
if (const auto *ELF = dyn_cast<llvm::object::ELFObjectFileBase>(O))
if (auto Addr = findIn(ELF->getDynamicSymbolIterators()))
return Addr;
return std::nullopt;
}
Once we have the Image object, we search through the dynamic symbols in that image looking for the one that matches, and return the address if we find it.
The LLDB side is a similar idea. We use the image start address to get the resolve the image. Then find the symbol with the right name in the symbol table. Once we find it, return the address that the symbol resolves to.
swift::remote::RemoteAddress
LLDBMemoryReader::getSymbolAddress(swift::remote::RemoteAddress image_start,
const std::string &name) {
if (name.empty())
return swift::remote::RemoteAddress();
Log *log = GetLog(LLDBLog::Types);
Target &target = m_process.GetTarget();
Address image_addr;
if (!target.ResolveLoadAddress(image_start.getRawAddress(), image_addr)) {
LLDB_LOG(log, "[MemoryReader] could not resolve image start {0:x} for {1}",
image_start.getRawAddress(), name);
return swift::remote::RemoteAddress();
}
ModuleSP module_sp = image_addr.GetModule();
if (!module_sp) {
LLDB_LOG(log, "[MemoryReader] no module at image start {0:x} for {1}",
image_start.getRawAddress(), name);
return swift::remote::RemoteAddress();
}
ConstString name_cs(name.c_str(), name.size());
SymbolContextList sc_list;
module_sp->FindSymbolsWithNameAndType(name_cs, lldb::eSymbolTypeAny, sc_list);
SymbolContext sym_ctx;
for (size_t idx = 0, e = sc_list.GetSize(); idx < e; ++idx) {
if (!sc_list.GetContextAtIndex(idx, sym_ctx) || !sym_ctx.symbol)
continue;
if (sym_ctx.symbol->GetType() == lldb::eSymbolTypeUndefined)
continue;
auto load_addr = sym_ctx.symbol->GetLoadAddress(&target);
if (load_addr == LLDB_INVALID_ADDRESS)
continue;
LLDB_LOGV(log, "[MemoryReader] {0} resolved to {1:x} within image", name,
load_addr);
return swift::remote::RemoteAddress(
load_addr, swift::remote::RemoteAddress::DefaultAddressSpace);
}
LLDB_LOG(log, "[MemoryReader] symbol resolution failed for {0} in image",
name);
return swift::remote::RemoteAddress();
}
Next, we actually need to add the table to the images
extern "C" __attribute__((__used__, __retain__, __visibility__("default")))
const SwiftReflectionSections __swift5_reflection_sections = {
SWIFT_REFLECTION_SECTIONS_VERSION,
#define SWIFT_REFLECTION_SECTION_BOUNDS(name) \
{static_cast<const void *>(&__start_##name), \
static_cast<const void *>(&__stop_##name)}
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_fieldmd),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_assocty),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_builtin),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_capture),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_typeref),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_reflstr),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_protocol_conformances),
SWIFT_REFLECTION_SECTION_BOUNDS(swift5_mpenum),
#undef SWIFT_REFLECTION_SECTION_BOUNDS
};
Now we need to update readELFSections. We no longer need LLDB to attempt to
find the object file and pass that through, we should be able to follow the
symbol to this table and get everything we need that way.
/// On success returns the ID of the newly registered Reflection Info.
template <typename T>
std::optional<uint32_t> readELFSections(
RemoteAddress ImageStart,
llvm::SmallVector<llvm::StringRef, 1> PotentialModuleNames = {}) {
constexpr llvm::StringRef ELFReflectionSectionSymbol =
"__swift5_reflection_sections";
constexpr uint64_t ELFReflectionSectionsVersion = 1u;
RemoteAddress tableAddr = getReader().getSymbolAddress(
ImageStart, ELFReflectionSectionSymbol.str());
First we extract the pointer to the metadata table and verify that the versions match.
auto readTableSection = [&](unsigned index) -> ReflectionSection {
RemoteAddress startFieldAddr =
tableAddr + (1 + 2 * index) * T::PointerSize;
RemoteAddress stopFieldAddr =
tableAddr + (2 + 2 * index) * T::PointerSize;
auto start = getReader().readPointer(startFieldAddr, T::PointerSize);
auto stop = getReader().readPointer(stopFieldAddr, T::PointerSize);
if (!start || !stop)
return {nullptr, 0};
RemoteAddress sectionStart = start->getResolvedAddress();
RemoteAddress sectionStop = stop->getResolvedAddress();
if (!sectionStart || !sectionStop || sectionStart >= sectionStop)
return {nullptr, 0};
const uint64_t secSize =
sectionStop.getRawAddress() - sectionStart.getRawAddress();
if (secSize == 0)
return {nullptr, 0};
auto sectionBuffer = this->getReader().readBytes(sectionStart, secSize);
if (!sectionBuffer)
return {nullptr, 0};
auto secContents = RemoteRef<void>(sectionStart, sectionBuffer.get());
savedBuffers.push_back(std::move(sectionBuffer));
return {secContents, secSize};
};
From there, we define a new lambda for extracting the metadata sections. For a given index into the table, we extract the start and stop bounds for the associated metadata, read the contents of that data into a buffer, and send that into the reflection context.
#define SWIFT_REFLECTION_SECTION(Name, Field) \
sections[idx_##Name] = readTableSection(idx_##Name); \
populated |= bool(sections[idx_##Name]);
#include "ELFReflectionSections.def"
The ReflectionSection returned defined an explicit bool operator that means
that we can ‘or’ them together to determine if we actually found anything.
if (!populated)
return std::nullopt;
ReflectionInfo info{
#define SWIFT_REFLECTION_SECTION(Name, Field) \
{sections[idx_##Name].contents, sections[idx_##Name].size},
#include "ELFReflectionSections.def"
PotentialModuleNames,
};
return this->addReflectionInfo(info);
If we didn’t find any metadata, we don’t try to register anything. If we do find
anything, we construct the Reflectioninfo object like we did before and
register it with the runtime.
One More Thing
When talking about how the runtime extracted the runtime metadata, there was one sneaky bug that ran mostly silently that I also caught and fixed while digging through this.
When the gold linker doesn’t merge the metadata sections due to the
discrepancy between the ALLOC vs ALLOC + RETAIN flags, the section bound
symbols were binding to the ALLOC + RETAIN swift5_reflstr and
swift5_typeref sections declared in swiftrt.o instead of the ALLOC-only
sections by the same name defined by the compiler. Unfortunately, only the
sections declared by the compiler actually contained any data. The runtime was
silently pointing at empty metadata sections at least since 2023.
This likely wasn’t caught because other than in a few back-deployment scenarios, that metadata isn’t really used by the runtime, or if it is needed, there are often graceful ways for the runtime to attempt to access the data it needs.
Conclusion
With these changes, I was able to re-enable all of the reflection tests on Linux, Android, and FreeBSD.
We went from multiple levels of parsing ELF headers to extract the metadata to following a symbol pointer to a metadata table with the metadata section bounds encoded in it. This fixes a bug that has affected Swift on Linux and other ELF-based platforms since 2018 at least, and been an underlying issue to an unknown number of “weird” and “hard to reproduce” issues with missing metadata when dumping metadata and debugging Swift programs.
With the exception of defining the same symbol name in multiple images in the same process, this implementation should fall squarely in what is well defined for ELF of all versions and will hopefully mean I never have to touch this code again. Or at least not because we had missing metadata.
Related Pull Requests
- https://github.com/swiftlang/swift/pull/87285
- https://github.com/swiftlang/llvm-project/pull/13262
- https://github.com/swiftlang/swift/pull/90533
- https://github.com/swiftlang/swift/pull/90580