Customizing the Go Toolchain for HarmonyOS: Conquering musl Emulated TLS on ARM64

Go’s support for linux/arm64 is remarkably robust—provided you are targeting a standard GNU C Library (glibc) environment.

HarmonyOS (Next), however, adopts musl libc, whose emulated Thread-Local Storage (TLS) mechanics fundamentally differ from glibc’s register-relative offsets. When compiling Go packages into c-shared dynamic libraries (.so) for native NAPI bindings inside HarmonyOS VPN sandboxes, standard Go binaries crash almost immediately upon load.

Here is the complete journey of customizing the Go 1.21.13 runtime across 12 targeted modifications, rewriting the ARM64 TLS addressing model, adapting Linux clone syscall flags, and overhauling cgo runtime initialization.

HarmonyOS Go Runtime Architecture

Bridging the Go runtime and HarmonyOS NAPI inside a musl libc sandbox


1. The Problem: Standard Go Crashing on HarmonyOS

1.1 Use Case & Build Target

To power the CocoProxy VPN client on HarmonyOS, our Go proxy engine needed to be compiled as a c-shared shared object (libcocoproxy.so), dynamically loaded through OpenHarmony NAPI by an ArkTS VpnExtensionAbility:

1
2
3
4
5
CGO_ENABLED=1 GOOS=linux GOARCH=arm64 \
CC=clang CGO_CFLAGS="--target=aarch64-linux-ohos --sysroot=..." \
go build -buildmode=c-shared -tags="with_gvisor,ohos" \
-ldflags="-extldflags '... napi_reg.o'" \
-o libcocoproxy.so

1.2 The Failure Signature

When loading the resulting .so via dlopen, execution crashed inside _rt0_arm64_linux_lib within .init_array. Because .init_array entries execute sequentially, subsequent NAPI module registration (RegisterCocoProxyModule) was never reached, yielding:

1
ArkCompiler: module libcocoproxy.so does not provide an export name 'getDebug'

1.3 The Root Cause

The crash occurred the very moment the Go runtime attempted to dereference the goroutine pointer (g) via TLS. Standard Go relies on glibc-style thread pointers, whereas HarmonyOS’s musl implementation employs an entirely divergent emulated TLS mechanism.


2. TLS Architecture: glibc vs. musl

2.1 The Standard glibc TLS Model

In glibc on ARM64, TLS variables are resolved directly via the thread pointer stored in TPIDR_EL0. The OS kernel initializes this register per thread. Each TLS variable resides at a static offset relative to TP:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
┌────────────────────────────────────────────────────────┐
│ glibc TLS Model │
│ │
│ TPIDR_EL0 ──→ ┌─────────────────────┐ │
│ │ TLS Var 1 │ │
│ │ TLS Var 2 │ │
│ │ ... │ │
│ │ tls_g slot (0x10) │ ← dynamic/fixed │
│ └─────────────────────┘ │
│ │
│ Go load_g: │
│ MRS TPIDR_EL0, R0 │
│ MOVD tls_g(SB), R11 │
│ MOVD (R0)(R11), g ← g = *(TP + tls_g) [1-level] │
└────────────────────────────────────────────────────────┘

This represents single-level indirection: *(TP + tls_g) = g.


2.2 The musl Emulated TLS Model

When running without hardware TLS descriptors, musl implements “emulated TLS” via Thread-Specific Data (TSD) keyed arrays managed via POSIX threads:

  • TLS storage does not use __tls_get_addr
  • Variables live in a TSD array accessed via pthread_key_t
  • pthread_setspecific(key, value) and pthread_getspecific(key) operate against a dynamic array positioned adjacent to the thread pointer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌────────────────────────────────────────────────────────┐
│ musl Emulated TLS Model │
│ │
│ TPIDR_EL0 ──→ ┌─────────────────────┐ │
│ │ TSD slot 0 │ │
│ │ TSD slot 1 │ │
│ tls_g ──→ │ TSD slot[k] ───────┼──→ ┌────────┐ │
│ │ ... │ │ g_slot │ │
│ └─────────────────────┘ └───┬────┘ │
│ ▼ │
│ g = *g_slot │
│ │
│ Key Difference: │
│ *(TP + tls_g) is NOT the g value! │
│ It is a pointer to the g_slot container. │
│ Requires *(*(TP + tls_g)) = g [2-level indirection] │
└────────────────────────────────────────────────────────┘

2.3 The Architectural Conflict

1
2
3
4
5
6
7
8
9
10
          glibc                          musl
┌──────────┐ ┌──────────┐
TP → │ slot[k] │ = g TP → │ slot[k] │ ──→ ┌────────┐
│ │ │ │ │ g_ptr │ = g
└──────────┘ └──────────┘ └────────┘

Access: Access:
*(TP + offset) = g *(TP + offset) = &g_ptr
(1-level indirection) *(*(TP + offset)) = g
(2-level indirection)

Because standard Go only performed single-level indirection, it read the address of g_slot as the g struct pointer. If g_slot was uninitialized, load_g pulled a garbage address, exploding on the next memory access.


3. Modification 1: Two-Level TLS Indirection

3.1 tls_arm64.h: Defining the MUSL_TLS Macro

File: src/runtime/tls_arm64.h

1
2
3
4
5
#ifdef GOOS_linux
#define TLS_linux
#define TLSG_IS_VARIABLE
#define MUSL_TLS // Enable musl 2-level TLS path
#endif

TLSG_IS_VARIABLE indicates tls_g is a runtime variable rather than a linker-computed TLS offset. MUSL_TLS acts as a build-time conditional in assembly.

3.2 tls_arm64.s: Rewriting load_g and save_g

File: src/runtime/tls_arm64.s

We restructured both routines to perform two sequential dereferences:

1
2
3
4
5
6
7
8
9
10
11
12
13
// ===== Standard load_g =====
TEXT runtime·load_g(SB),NOSPLIT,$0
MRS_TPIDR_R0 // R0 = TP
MOVD runtime·tls_g(SB), R27
MOVD (R0)(R27), g // g = *(TP + tls_g) [1-level]

// ===== Customized load_g =====
TEXT runtime·load_g(SB),NOSPLIT,$0
WORD $0xd53bd040 // Hardcoded: MRS TPIDR_EL0, R0
MOVD runtime·tls_g(SB), R27
ADD R27, R0, R27 // R27 = TP + tls_g (TSD slot address)
MOVD (R27), R27 // R27 = *(TP + tls_g) = g_slot pointer (1st level)
MOVD (R27), g // g = *g_slot (2nd level)
1
2
3
4
5
6
7
8
9
10
11
12
13
// ===== Standard save_g =====
TEXT runtime·save_g(SB),NOSPLIT,$0
MRS_TPIDR_R0
MOVD runtime·tls_g(SB), R27
MOVD g, (R0)(R27) // *(TP + tls_g) = g

// ===== Customized save_g =====
TEXT runtime·save_g(SB),NOSPLIT,$0
WORD $0xd53bd040 // MRS TPIDR_EL0, R0
MOVD runtime·tls_g(SB), R27
ADD R27, R0, R27 // R27 = TP + tls_g
MOVD (R27), R27 // R27 = slot pointer
MOVD g, (R27) // *slot = g (Write via 2nd level)

3.3 Explicit ARM64 Instruction Encoding

We replaced the macro with the raw opcode WORD $0xd53bd040 (MRS TPIDR_EL0, R0). This avoids compiler variations across targets and ensures clean instruction selection.

3.4 Global Variable Storage

1
2
3
4
5
6
7
8
// Standard Go: defined in TLS segment (glibc only)
GLOBL runtime·tls_g+0(SB), TLSBSS, $8

// Customized: standard global variable shared across threads
DATA runtime·tls_g+0(SB)/8, $0
GLOBL runtime·tls_g+0(SB), NOPTR, $8
DATA runtime·tls_g2+0(SB)/8, $0
GLOBL runtime·tls_g2+0(SB), NOPTR, $8

4. Modification 2: CLONE_SETTLS and New Thread Allocation

4.1 The Issue with Standard clone

Standard Go creates threads without passing a TLS parameter:

1
2
3
4
5
6
7
// Standard sys_linux_arm64.s
TEXT runtime·clone(SB),NOSPLIT|NOFRAME,$0
MOVW flags+0(FP), R0
MOVD stk+8(FP), R1
...
MOVD $SYS_clone, R8
SVC // clone(flags, stk, mp, gp, fn)

In musl, newly cloned threads must have their TPIDR_EL0 populated during the syscall via CLONE_SETTLS. Otherwise, newly spawned goroutine threads run with invalid thread pointers.

4.2 Updating Syscall Assembly & Flags

File: src/runtime/sys_linux_arm64.s

We updated runtime·clone to accept a sixth argument (tls in R3):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
TEXT runtime·clone(SB),NOSPLIT|NOFRAME,$0
MOVW flags+0(FP), R0
MOVD stk+8(FP), R1
...
+ MOVD tls+40(FP), R3 // TLS block pointer to x3
MOVD $SYS_clone, R8
SVC

CMP ZR, R0
BEQ child
+ MOVW R0, ret+48(FP) // Shift return offset for 6th parameter
RET

child:
...
MOVD R10, g_m(R11)
MOVD R11, g
+ CALL runtime·save_g(SB) // Save g to TLS immediately in child

File: src/runtime/os_linux.go

1
2
3
cloneFlags = _CLONE_VM | _CLONE_FS | _CLONE_FILES |
_CLONE_SIGHAND | _CLONE_SYSVSEM | _CLONE_THREAD |
_CLONE_SETTLS // Required for HarmonyOS musl

4.3 cocoAllocateTLS: Managing Thread Local Blocks

We introduced a custom allocator in Go runtime space to prepare the two-level pointer chain prior to thread creation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
const cocoTLSBlockSize = 16384  // 16KB TLS block

//go:nosplit
//go:nowritebarrier
func cocoAllocateTLS() unsafe.Pointer {
tlsG := getTLSg()
if tlsG == 0 {
return nil
}

block := sysAlloc(cocoTLSBlockSize, &memstats.other_sys)
if block == nil {
throw("cocoAllocateTLS: sysAlloc failed")
}

// Anchor TP in middle of block to guard against underflows/overflows
tpOffset := uintptr(cocoTLSBlockSize / 2)
tp := uintptr(block) + tpOffset

// Build the two-level indirection chain
slotAddr := tp + tlsG
gSlotAddr := slotAddr + 8
*(*uintptr)(unsafe.Pointer(slotAddr)) = gSlotAddr
*(*uintptr)(unsafe.Pointer(gSlotAddr)) = 0

return unsafe.Pointer(tp)
}

5. Modification 3: Rewriting cgo inittls

5.1 Probing TSD Offsets Safely

Because musl does not expose an ABI-stable constant for internal TSD array offsets relative to TP, we devised an active memory probing strategy with a fallback sigsetjmp safety net:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// Probe flow in custom gcc_linux_arm64.c:
// 1. Create pthread key
pthread_key_create(&k, NULL);

// 2. Set sentinel magic value
pthread_setspecific(k, (void*)0x23581321345589ULL);

// 3. Scan neighborhood of TP within [-4KB, +4KB]
for (intptr_t i = -4096; i < 4096; i += sizeof(void*)) {
if (sigsetjmp(inittls_jmpbuf, 1) == 0) {
if (*(uint64_t*)(tp + i + k*sizeof(void*)) == 0x23581321345589ULL) {
// Confirm with second probe to eliminate false positives
pthread_setspecific(k, (void*)0x9876543210ULL);
if (*(uint64_t*)(tp + i + k*sizeof(void*)) == 0x9876543210ULL) {
*tlsg = i + k * sizeof(void*);
break;
}
}
}
}

6. Modification 4: Routing Runtime Errors to hilog

6.1 Avoiding stderr Panics in Sandboxes

HarmonyOS’s VPN Extension Ability operates within a tightly locked sandbox where stderr (file descriptor 2) is closed or unmapped. Calling fprintf(stderr, ...) inside cgo/gcc_fatalf.c triggers an instant SIGSEGV.

6.2 Dynamic Linking against libhilog_ndk.z.so

Instead of static linkage, we dynamically bind to the OpenHarmony native logging facility:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#include <dlfcn.h>

typedef void (*OH_LogPrintFunc)(int type, int level, unsigned int domain,
const char *tag, const char *fmt, ...);

void fatalf(const char* format, ...) {
void *h = dlopen("libhilog_ndk.z.so", RTLD_NOW);
if (h) {
OH_LogPrintFunc fn = (OH_LogPrintFunc)dlsym(h, "OH_Log_Print");
if (fn) {
va_list ap;
va_start(ap, format);
fn(3 /* LOG_APP */, 6 /* LOG_FATAL */, 0, "runtime/cgo", format, ap);
va_end(ap);
}
}
abort();
}

7. Modification 5 & 6: Defensive Runtime Safeguards

7.1 Nil argv Guard in c-shared Mode

During NAPI initialization, ArkCompiler may invoke .init_array handlers without standard command-line vectors. In src/runtime/runtime1.go, we added an empty sentinel:

1
2
3
4
5
6
7
8
9
10
11
var csharedEmptyArgv [3]*byte

func args(c int32, v **byte) {
argc = c
if islibrary && v == nil {
argv = &csharedEmptyArgv[0]
} else {
argv = v
}
sysargs(c, argv)
}

7.2 Fallback for physPageSize

When HarmonyOS’s /proc/self/auxv omits the AT_PAGESZ entry, Go’s memory allocator previously panicked on physPageSize == 0. We enforced a safe default of 4096 bytes in src/runtime/os_linux.go.


8. Summary of All 12 Changes

# File Type Key Responsibility
1 src/runtime/tls_arm64.h Macros Define TLSG_IS_VARIABLE & MUSL_TLS
2 src/runtime/tls_arm64.s Assembly Rewrite load_g & save_g for 2-level indirection; add getTLSg
3 src/runtime/asm_arm64.s Assembly Hardcode MRS TPIDR_EL0; pass tls_g2 in cgo_init
4 src/runtime/race_arm64.s Assembly Align race detector macros with MUSL_TLS branching
5 src/runtime/sys_linux_arm64.s Assembly Accept tls in R3 during clone; call save_g on child entry
6 src/runtime/os_linux.go Go Add _CLONE_SETTLS; implement cocoAllocateTLS; enforce physPageSize fallback
7 src/runtime/proc.go Go Low-level startup tracing
8 src/runtime/mgc.go Go GC bootstrap tracing
9 src/runtime/runtime1.go Go Defend against null argv in c-shared initialization
10 src/runtime/cgo/gcc_fatalf.c C Replace crash-prone stderr with dlopen("libhilog_ndk.z.so")
11 src/runtime/cgo/gcc_libinit.c C Route thread failure diagnostics through fatalf
12 src/runtime/cgo/gcc_linux_arm64.c C Complete rewrite of inittls with safe TSD probing & slot setup

9. Dependency Architecture Graph

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
                 tls_arm64.h (MUSL_TLS, TLSG_IS_VARIABLE)
│
┌──────────────┼──────────────────┐
│ │ │
▼ ▼ ▼
tls_arm64.s race_arm64.s asm_arm64.s
(load_g/save_g (race load_g (rt0_go MRS,
2-level) branching) cgo_init tls_g2)
│ │ │
│ │ ▼
│ │ gcc_linux_arm64.c
│ │ (TSD probe, g-slot,
│ │ cgo threadentry)
│ │ │
└──────┬───────┘ │
│ │
▼ │
sys_linux_arm64.s │
(clone + CLONE_SETTLS │
child save_g) │
│ │
▼ │
os_linux.go │
(cocoAllocateTLS, │
newosproc clone+tls) │
│ │
└──────────┬───────────────┘
│
▼
gcc_fatalf.c ──→ gcc_libinit.c
(hilog logging) (safe aborts)
│
runtime1.go (argv null guard)
os_linux.go (physPageSize fallback)

10. Building and Verification

With our custom Go tree built at C:\Users\mateng\go-ohos:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$goRoot = "C:\Users\mateng\go-ohos"
$ndk = "D:\DevTools\Huawei\DevEcoStudio\sdk\default\openharmony\native"

$env:GOROOT = $goRoot
$env:CGO_ENABLED = "1"
$env:GOOS = "linux"
$env:GOARCH = "arm64"
$env:CC = "$ndk\llvm\bin\clang.exe"
$env:CGO_CFLAGS = "--target=aarch64-linux-ohos --sysroot=$ndk\sysroot -D__OHOS__=1 -D__MUSL__ -fPIC -Wno-error"
$env:CGO_LDFLAGS = "-target aarch64-linux-ohos -Wno-error --sysroot=$ndk\sysroot -lace_napi.z -lc -ldl"
$env:CGO_CFLAGS_ALLOW = ".*"
$env:CGO_LDFLAGS_ALLOW = ".*"

go build -a -tags="with_gvisor,ohos" -buildmode=c-shared `
-ldflags="-extldflags '--sysroot=$ndk\sysroot --target=aarch64-linux-ohos -Wl,--dynamic-list=dynamic_list.txt napi_reg.o'" `
-o libcocoproxy.so

Monitoring device output over OpenHarmony’s hdc:

1
2
3
4
5
6
7
hdc shell hilog | grep -i "cocoproxy\|napi\|runtime\.main"

# Expected output:
# runtime.main: entered
# runtime.main: gcenable done
# inittls: tls_g=112 tsd_off=...
# [NAPI] startTun called successfully

11. Final Thoughts

Porting a modern language runtime to an operating system with distinct libc conventions requires descending directly to the CPU’s memory abstraction layers.

The most subtle challenges were runtime-only crashes with zero stdout diagnostics, no core dumps, and strict sandbox memory permissions. Resolving this required peeling back ARM64 register behaviors, musl TSD allocations, and POSIX signal handling.

Hopefully, this guide saves other developers countless hours when bringing advanced Go networking engines and user-space network stacks into the growing HarmonyOS ecosystem.