Build XNU 6153.141.1 ARM64

Darwin Kernel Version 19.6.0: Mon Sep  7 21:53:47 EDT 2020; binwang:XNU/BUILD/obj/DEVELOPMENT_ARM64_BCM2837

It is similar to building XNU 4903, but a few things changed. I am just writing down my quick diff here:

  • Dependencies have version dump, take the latest version. dtrace/CTF Tools need to have macosx.internal SDK string replaced with the public macOS SDK string.
  • Missing a LLVM header file. Get it from here.
  • I also commented the embedded map thing out, but there’s one more place referencing it compared to 4903.
  • IODMACommand.iig generates a duplicated const qualifier. Remove the additional const qualifier at line 116.
  • Other steps are similar.


About the vGIC implementation on Qualcomm GICv2

Random Internet search gives you useful results:

With some induction, we can guess the GICH/GICV and vGIC maintenance interrupt for other QGIC2 SoC like MSM8994:

[04Ch 0076 8] Base Address : 00000000F9002000
[054h 0084 8] Virtual GIC Base Address : 00000000F9004000
[05Ch 0092 8] Hypervisor GIC Base Address : 00000000F9001000
[064h 0100 4] Virtual GIC Interrupt : 00000019

Bootstrapping the blue Windows when you have random AArch64 devices in the backyard

Last year I mentioned my attempt to bootstrap Windows 10 on Dragonboard 410c. This year I ported EDK2 to Nintendo Switch and successfully booted Windows 10 arm64 installation ramdisk (rs4, rs5, and 19H1 tested as of writing time). I will briefly introduce a common way to port EDK2 with existing codebase (e.g. U-Boot), as well as cases of booting in EL2 (hypervisor).


While this article applies to most ARM SoCs, the following content will use Tegra as the example. NVIDIA developed a few solutions for Windows on ARM in Windows 8 era: Tegra 3 (Tegra30) and 4 (Tegra114). No further model have official Windows BSP (Board Support Package) released publicly due to low market acceptance of those Windows RT products.

Despite of that, general AArch64 processors are capable to run Windows 10 without additional HAL extension library if the following conditions are satisfied:

  • Architecture Timer with ACPI GTDT table description. Either CP15 or MMIO clock is okay.
  • Generic Interrupt Controller v2/v3 (we are not yet aware of v4 support) with ACPI MADT (APIC) table description, or Broadcom Interrupt Controller
  • AArch64 instruction set (crypto extension is not required)
  • ARM Performance Monitor Unit with ACPI MADT (APIC) table description

One noticeable exception the initial generation of Qualcomm Kryo (Snapdragon 820, 821) due to the faulty cache design in large core cluster. Windows removed the required erratum for it due to the complication of patch maintenance.

In the case of Tegra X1, it satisfied all conditions outlined above. I used an old-bootrom Nintendo Switch as my experiment platform since it is much cheaper than Jetson TX1. Additionally, there is verified CoreBoot and U-Boot source code for these Tegra X1 devices including Nintendo Switch.

I assume you are familiar with the NVIDIA RCM Exploit (Fusee-Gelee) as well as Tegra Boot flow. If you are not familiar with Tegra Boot flow, please refer to Tegra Technical Reference Manual available on NVIDIA developer site.

Port U-Boot Code to EDK2

There are a few environment assumptions that need to be addressed while porting U-Boot device/driver code to EDK2:

  • While U-Boot runs in AArch64 context, it only utilizes little amount of memory at the memory bottom in most circumstances. EDK2/TianoCore loads everything as high as possible per UEFI specification. Certain peripheral operations are not 64-bit addressing aware. It’s okay to force converting 64-Bit pointers to 32-Bit without data loss in the U-Boot assumption, but in EDK2 this might lead to issues. One case is SDMA (single operation DMA). Tegra SDHCI controller SDMA operations are not 64-bit addressing aware. To address the issue, I slightly modified the DMA bounce buffer allocation library (also ported from U-Boot) to allocate bottom memory instead.
  • Syntax styles. U-Boot observes the Linux naming convention for functions and types; EDK2 observes the Windows style. It might be a good idea to write a shim to provide functions like readl/writel as well as udelay/mdelay.
  • There is probably no need for porting generic classes (e.g. udevice). You might not need them in EDK2 context.

To save myself some time bootstrapping the microSD slot, I ported the clock and device framework from U-Boot to EDK2. Here are a few suggestions while porting U-Boot code to EDK2:

  • Address issues mentioned above.
  • Put device specific definitions into “Include” directory, use PCD database when necessary.
  • Install these code services as DXE driver whenever possible. Invoke them using protocols.
  • For board/machine-dependent code library (e.g. mach-tegra), depends on the usage to integrate them with driver or use additional library instead.

From Device Tree to ACPI

Device Tree is the de-facto standard in ARM to describe the system and peripheral hierarchy. Windows RT introduces the intensive use of ACPI on ARM platforms. I will cover some required tables for a success Windows startup on ARM platforms. For tables such as CSRT and DSDT, check out the Microsoft documentation.

GTDT (Generic Timer Description Table)

For SoC with architecture timer, ARM defines GTDT table to describe platform timer information. In the device tree, an architectural timer may looks like this:

timer {
	compatible = "arm,armv8-timer";
	interrupt-parent = <&gic>;

And it looks like this in ACPI GTDT table:


[024h 0036   8]        Counter Block Address : FFFFFFFFFFFFFFFF
[02Ch 0044   4]                     Reserved : 00000000

[030h 0048   4]         Secure EL1 Interrupt : 0000001D
[034h 0052   4]    EL1 Flags (decoded below) : 00000002
                                Trigger Mode : 0
                                    Polarity : 1
                                   Always On : 0

[038h 0056   4]     Non-Secure EL1 Interrupt : 0000001E
[03Ch 0060   4]   NEL1 Flags (decoded below) : 00000002
                                Trigger Mode : 0
                                    Polarity : 1
                                   Always On : 0

[040h 0064   4]      Virtual Timer Interrupt : 0000001B
[044h 0068   4]     VT Flags (decoded below) : 00000002
                                Trigger Mode : 0
                                    Polarity : 1
                                   Always On : 0

[048h 0072   4]     Non-Secure EL2 Interrupt : 0000001A
[04Ch 0076   4]   NEL2 Flags (decoded below) : 00000002
                                Trigger Mode : 0
                                    Polarity : 1
                                   Always On : 0
[050h 0080   8]   Counter Read Block Address : FFFFFFFFFFFFFFFF

  • If your platform does not have MMIO architectural timer, write the address as 0xFFFFFFFFFFFFFFFF.
  • If you boot from EL2, you are required to supply all timer values. Otherwise only EL1 timers are needed.
  • PPI starts at 16. Plus 16 for all interrupt numbers you have in the device tree. The four interrupts are Secure EL1, Non-secure EL1, virtual timer and hypervisor in sequence.
  • You may have platform watchdog, supply it in the GTDT table too (see Qualcomm example). It is not mandatory for booting Windows though.

MADT (Multiple APIC Description Table)

Most AArch64 SoC systems have one or more GIC-compatible interrupt controllers. Windows has inbox GIC support, all needed is supplying proper information in the MADT table. The table also describes ARM Performance Monitor Unit information for system’s reference. In device tree, GIC and PMU look like this:

gic: interrupt-controller@50041000 {
	compatible = "arm,gic-400";
	#interrupt-cells = <3>;
	reg = <0x0 0x50041000 0x0 0x1000>,
	    <0x0 0x50042000 0x0 0x2000>,
            <0x0 0x50044000 0x0 0x2000>,
	    <0x0 0x50046000 0x0 0x2000>;
	interrupt-parent = <&gic>;
arm_pmu: arm-pmu {
	compatible = "arm,armv8-pmuv3";
	interrupts = <GIC_SPI 144 IRQ_TYPE_LEVEL_HIGH>,
            <GIC_SPI 146 IRQ_TYPE_LEVEL_HIGH>,

An example of the MADT table can be found here.

  • In MADT table, each processor core have an table entry. Make sure you have the same CPU object in DSDT table, with identical and unique UID and CPU interface ID.
  • If your platform supports ARM PSCI, parking address field can be ignored.
  • The four registers in GIC device tree are GIC distributor, GIC base address, hypervisor GIC base address and virtual GIC base address.
  • You might need to supply GIC redistributor address on GICv3 architecture.
  • SPI interrupt number starts at 32. Plus 32 for all performance interrupt number in MADT table.
  • MPIDR value needs to be referred from platform resources.

DBG2 (Microsoft Debug Table 2)

Microsoft defines DBG2 table for ARM platforms. Although Microsoft docs mark DBG2 table info as mandatory, you do not need to supply debug device information if you just want to boot Windows as a proof-of-concept :P. An empty DBG2 table is enough for booting.

For debug purposes, it is necessary to define at least one debug device (8250/16550 serial or USB) in DSDT and DBG2 table. More information can be found at here.

FADT (Fixed ACPI Description Table)

Indicates PSCI support and Hardware-reduced ACPI mode, then you are good to go.

Debugging ACPI

It’s incredibly difficult to debug early ACPI startup if you don’t have serial or debug access on the platform. Fortunately, Linux provides some utility for it. It is feasible to enable the UEFI FrameBuffer early printk support on 5.0+ kernels to simplify the debug process.


With much effort, Windows on ARM can run on a variety of AArch64 devices. There’s still much work between “just-booted” and “usable”, and it may cost you countless nights to achieve your marvel, even if there are always guys ask you “why”:


Fix broken Windows Management Instrumentation

A colleague told me a Windows Server 2016 node entered an inconsistent state after an abnormal shutdown. The following symptoms were observed:

  • Explorer hangs with “loading…” text
  • Hyper-V Management couldn’t connect to the local server
  • Group Policy Update consequently failed
  • Telemetry metrics disappeared
  • WMI Management reported “RPC: the requested object does not exist” for object Root

A quick diagnostics indicated a component failure with Windows Management Instrumentation. To determine the failure source, I ran WMIDiag from Microsoft. The log showed a metadata failure:

.1526 21:47:43 (1) !! ERROR: WMI CONNECTION errors occured for the following namespaces: ………………………………………….. 20 ERROR(S)!
 .1527 21:47:43 (0) ** - Root, 0x80010114 - The requested object does not exist..
 .1528 21:47:43 (0) ** - Root, 0x80010114 - The requested object does not exist..
 .1529 21:47:43 (0) ** - Root/subscription, 0x80010114 - The requested object does not exist..
 .1530 21:47:43 (0) ** - Root/DEFAULT, 0x80010114 - The requested object does not exist..
 .1531 21:47:43 (0) ** - Root/CIMV2, 0x80010114 - The requested object does not exist..
 .1532 21:47:43 (0) ** - Root/CIMV2/Security, 0x80010114 - The requested object does not exist..
 .1533 21:47:43 (0) ** - Root/CIMV2/TerminalServices, 0x80010114 - The requested object does not exist..
 .1534 21:47:43 (0) ** - Root/nap, 0x80010114 - The requested object does not exist..
 .1535 21:47:43 (0) ** - Root/SECURITY, 0x80010114 - The requested object does not exist..
 .1536 21:47:43 (0) ** - Root/STANDARDCIMV2, 0x80010114 - The requested object does not exist..
 .1537 21:47:43 (0) ** - Root/RSOP, 0x80010114 - The requested object does not exist..
 .1538 21:47:43 (0) ** - Root/RSOP/User, 0x80010114 - The requested object does not exist..
 .1539 21:47:43 (0) ** - Root/RSOP/Computer, 0x80010114 - The requested object does not exist..
 .1540 21:47:43 (0) ** - Root/WMI, 0x80010114 - The requested object does not exist..
 .1541 21:47:43 (0) ** - Root/directory, 0x80010114 - The requested object does not exist..
 .1542 21:47:43 (0) ** - Root/directory/LDAP, 0x80010114 - The requested object does not exist..
 .1543 21:47:43 (0) ** - Root/Policy, 0x80010114 - The requested object does not exist..
 .1544 21:47:43 (0) ** - Root/Microsoft, 0x80010114 - The requested object does not exist..
 .1545 21:47:43 (0) ** - Root/Microsoft/HomeNet, 0x80010114 - The requested object does not exist..
 .1546 21:47:43 (0) ** - Root/aspnet, 0x80010114 - The requested object does not exist..

The documentation suggested performing a metadata registration. The following script is utilized for the metadata repair:

@echo on
cd /d c:\temp
if not exist %windir%\system32\wbem goto TryInstall
cd /d %windir%\system32\wbem
net stop winmgmt
winmgmt /kill
if exist Rep_bak rd Rep_bak /s /q
rename Repository Rep_bak
for %%i in (*.dll) do RegSvr32 -s %%i
for %%i in (*.exe) do call :FixSrv %%i
for %%i in (*.mof,*.mfl) do Mofcomp %%i
net start winmgmt
goto End

if /I (%1) == (wbemcntl.exe) goto SkipSrv
if /I (%1) == (wbemtest.exe) goto SkipSrv
if /I (%1) == (mofcomp.exe) goto SkipSrv
%1 /RegServer

goto End

if not exist wmicore.exe goto End
wmicore /s
net start winmgmt

It will throw some errors. Ignore them. Then reboot the server.