A network card is given a buffer to receive packets into. Its driver writes the buffer's physical address into the card, and the card writes there with DMA. But nothing stops the card writing anywhere else: a bug in its firmware, a malicious device plugged into an external port, or a corrupted descriptor, and it overwrites the kernel. Processes have had protected address spaces for decades. Devices, by default, have none.
How is a device limited to the memory it is supposed to use?
The obvious answer is a firewall with regions for each device (lesson 14.2). Region tables are coarse and few: a modern system has many devices, each using many scattered pages that change constantly as buffers are allocated and freed. What is needed is the same mechanism that protects processes: page tables, per device, checked on every access.
Each device (or function, or stream) gets its own page tables, set up by the operating system or hypervisor.
The device issues I/O virtual addresses; the IOMMU, in the path between the device and memory, translates each to a physical address using the device's tables.
Unmapped or wrongly permitted accesses fault: the access is refused (with an error response) and the fault is reported to software.
Identity comes from the fabric: the IOMMU selects the tables from the request's requester or stream ID (lesson 14.1). A device builds its own packets and could put any ID in them, so the fabric must check it: in PCIe, ACS Source Validation at the switch or root port rejects a request whose requester ID does not belong to the port it arrived on.
Translations are cached in the IOMMU's TLB; a miss walks the page tables in memory.
Cost | Mitigation |
|---|---|
Latency of a translation on each access | IOMMU TLBs; large pages |
Page-table walks on TLB misses | Bigger TLBs; prefetching; locality in buffer allocation |
Software cost of mapping and unmapping buffers | Long-lived mappings for buffer pools (with weaker protection between uses) |
Translation latency for high-bandwidth devices | Device-side translation caches (ATS): the device asks the IOMMU for translations and caches them, and the IOMMU invalidates them when mappings change |
Device-side caching keeps the protection only if invalidation is reliable: when the operating system unmaps a buffer, the IOMMU must invalidate the device's cached translation and wait for it to confirm, before the memory is reused.
System | Device translation |
|---|---|
Arm SMMU, Intel VT-d | Per-device (stream or requester ID) translation and fault reporting for DMA (fact A28) |
PCI Express ATS | Endpoints request and cache translations from the IOMMU, with invalidation messages when mappings change (fact P35) |
PCI Express PASID | Tags requests with an address-space ID so one device can work in several processes' address spaces (fact P35) |
A driver (illustrative) unmapped a receive buffer and freed the memory without waiting for the IOMMU's invalidation to complete. For a few microseconds the device still had a valid cached translation, and a late packet landed in memory the kernel had already given to another process. Waiting for the invalidation's completion before freeing the page closed the window.
The IOMMU selects tables by the requester or stream ID assigned by the fabric, not by anything the device chooses.
Lower translation latency; it requires the IOMMU to invalidate cached entries, and wait for confirmation, whenever mappings change.
Without an IOMMU, a DMA-capable device can access any physical memory.
An IOMMU translates and checks every device access with per-device page tables chosen by an unforgeable identity.
Cache translations for speed, but make invalidation and its completion part of every unmap.