elrise

Sysadmin incident report. Host iton-nest, Garuda Linux (Arch-based), kernel 7.0.14-zen1-1-zen. Affected tool: opencode-ai 1.18.7 (npm-global ELF64 binary that embeds the Bun runtime). Affected provider: minimax → https://api.minimax.io/anthropic/v1. Outcome: RESOLVED via a personal augmented CA bundle plus NODE_EXTRA_CA_CERTS / SSL_CERT_FILE override and a bounce of every running opencode process. No new occurrences after the fix is applied to a process and that process is restarted. 577 error occurrences across all roles between 2026-06-15 and 2026-07-27.

TL;DR

api.minimax.io (Akamai edge) is currently serving a chain terminated by Let's Encrypt YR2 (issued 2026-07-13, valid through 2026-10-11).
YR2 is not present in the Arch system trust store, in the webpki-roots bundled inside opencode-ai 1.18.7 (148 CAs, only ISRG Root X1/X2 from Let's Encrypt), or in the NODE_EXTRA_CA_CERTS / SSL_CERT_FILE overrides (which point at the system store).
openssl on the same host succeeds because it performs AIA-chasing and YR2 is cross-signed by ISRG Root X1. The embedded Bun runtime in opencode 1.18.7 does not chase the cross-sign in this build.
Fix: concatenate the system bundle + YR2 intermediate + Root YR cross-signed cert into a personal ca-bundle.crt, point NODE_EXTRA_CA_CERTS and SSL_CERT_FILE at it, and bounce every running opencode (env vars do not hot-load into a running process).

The earlier occurrences between 2026-06-15 and 2026-07-20 were the same Bun-side root cause but with a different Akamai chain — harmless at the time because the older DST cross-sign was still in the system bundle. The 2026-07-21 ca-certificates package update removed that DST root, after which YR2 became the only path and the embedded Bun could no longer resolve it.

1. Symptom

The opencode TUI failed every streaming response to the minimax provider with:

Error: unknown certificate verification error
    at async <anonymous> (/$bunfs/root/chunk-*.js:439:92289)
    at async <anonymous> (/$bunfs/root/chunk-*.js:6:12398)
    at async doStream (/$bunfs/root/chunk-*.js:3:19884)
    ...

The error originates inside the embedded Bun runtime of the opencode ELF64 binary. Frequency: 5–15 minutes per session, 577 occurrences in the opencode log across all 14 active roles between 2026-06-15 and 2026-07-27.

Side effects: each retrying process held 600–770 MB RSS and 30–45% CPU in a hot retry loop. The 14 GB opencode.db and its 271 MB WAL grew monotonically as a result.

2. Investigation

2.1 The system trust store is not the broken layer

$ openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt \
                   -connect api.minimax.io:443 \
                   -servername api.minimax.io
...
depth=3 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=2 C=US, O=ISRG, CN=Root YR
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=YR2
verify return:1
depth=0 CN=www.minimax.io
verify return:1
Verification: OK

openssl verifies the chain to api.minimax.io via AIA-chasing. So the system trust store is fine.

2.2 Env vars are correctly set

NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
SSL_CERT_DIR=/etc/ssl/certs

These are present in the environment of every opencode process and are sourced from ~/.config/shell/tls-env.sh via .bash_profile → .bashrc → tls-env.sh (and from ~/.config/fish/conf.d/00-tls-bundle.fish for fish shells). The fix from the prior sysadmin-010 ticket is in place.

2.3 The TLS implementation is in the embedded Bun, not OpenSSL

opencode-ai 1.18.7 is a 179 MB ELF64 binary. ldd shows it links only against libc, libpthread, libdl, libm, and ld-linux — no libssl / libcrypto. There is no separate bun binary on the host (command -v bun → not found). The TLS implementation lives in the Bun runtime that ships embedded inside the opencode binary. The error stack confirms this: paths are /$bunfs/root/chunk-*.js.

2.4 opencode merges three CA sources at startup

strings(1) on the binary reveals the relevant merge logic:

ArrayPrototypePush.@call(defaultCACertificates, bundled[i]);
ArrayPrototypePush.@call(defaultCACertificates, extra[i]);
ArrayPrototypePush.@call(defaultCACertificates, system[i]);

i.e. defaultCACertificates = bundled[] + extra[] + system[], where extra is NODE_EXTRA_CA_CERTS and system is SSL_CERT_FILE / SSL_CERT_DIR. The sysadmin-010 fix was correct in design; it just was not enough on its own.

2.5 The bundled webpki-roots do not contain YR2

Extracting the 148 PEM blocks from the binary and running openssl x509 -subject -issuer on each:

Bundled in opencode 1.18.7	Notes
ISRG Root X1, ISRG Root X2	Let's Encrypt roots
GTS Root R1, R2, R3, R4	Google Trust Services
GlobalSign Root CA — R3, R6 / ECC R4, R5 / Root E46, R46
DigiCert Global Root CA / G2 / G3 / Assured ID G2, G3 / Trusted Root G4
Sectigo Public Server Authentication Root R46 / E46
YR2 (or any R10/R11/R12 intermediate)	NOT PRESENT

2.6 The server's chain is terminated by YR2 (new, LE 2026-07-13)

Captured live from api.minimax.io:

depth	subject	issuer	notBefore	notAfter
0	CN=www.minimax.io	C=US, O=Let's Encrypt, CN=YR2	2026-07-13	2026-10-11
1	C=US, O=Let's Encrypt, CN=YR2	C=US, O=ISRG, CN=Root YR
2	C=US, O=ISRG, CN=Root YR	C=US, O=Internet Security Research Group, CN=ISRG Root X1
3	C=US, O=ISRG, CN=ISRG Root X1	(self)

Root YR is a new Let's Encrypt root. It is cross-signed by ISRG Root X1 (so openssl chases it via AIA), but the cross-signed copy is not in any of the three CA sources that opencode consults at startup.

2.7 The system store contains no LE intermediates

$ ls /etc/ssl/certs/ | grep -iE "ISRG|LE|YR"
ISRG_Root_X1.pem
ISRG_Root_X2.pem

That is it. No YR2, no R10/R11/R12, no E1/E2. The system store has only the two ISRG roots, and it works only because openssl performs AIA-chasing. The embedded Bun in opencode 1.18.7 does not.

3. Root cause

api.minimax.io (Akamai-fronted) currently serves a TLS chain terminated by Let's Encrypt YR2 (issued 2026-07-13, valid through 2026-10-11). YR2 is not present in:

the Arch system bundle /etc/ssl/certs/ca-certificates.crt (121 CAs, mtime 2026-07-21, only ISRG Root X1/X2 from LE);
the webpki-roots bundled into opencode-ai 1.18.7 (148 roots, only ISRG Root X1/X2 from LE);
the NODE_EXTRA_CA_CERTS / SSL_CERT_FILE overrides (both point at the system bundle).

openssl succeeds because it performs AIA-chasing and YR2 is cross-signed by ISRG Root X1. The embedded Bun runtime in opencode-ai 1.18.7 does not perform AIA-chasing in this build, so verification fails inside doStream.

The 2026-07-21 regeneration of the system ca-certificates package removed the older DST-root cross-sign that was previously in the chain, after which YR2 became the only path — and the embedded Bun cannot resolve it. The earlier occurrences (2026-06-15 → 2026-07-20) were a related but separate chain-rotation issue, harmless at the time because the older DST root was still trusted by the system store. They share the same Bun-side root cause.

4. Fix applied

4.1 Personal augmented CA bundle

Created ~/.config/opencode/ca-bundle.crt as the concatenation of:

the system bundle /etc/ssl/certs/ca-certificates.crt (121 CAs, mtime 2026-07-21),
the YR2 intermediate (extracted from the live server chain),
the Root YR cross-signed cert (extracted from the live server chain).

Length 189 KB, 3296 lines.

4.2 Env var override

Backed up ~/.config/shell/tls-env.sh to ~/.config/shell/tls-env.sh.bak.20260727 (sha256) and updated the active file to point SSL_CERT_FILE and NODE_EXTRA_CA_CERTS at the new bundle. SSL_CERT_DIR is left at /etc/ssl/certs for compatibility with tools that walk the directory.

4.3 Applying the fix to a running opencode

This is the part that is easy to get wrong. A source ~/.config/shell/tls-env.sh in a bash sub-shell does not propagate to the next sub-shell, and a sub-shell inherits the environment of its parent opencode, not of the operator's interactive shell. The reliable form is inline env vars on the command line:

NODE_EXTRA_CA_CERTS=$HOME/.config/opencode/ca-bundle.crt \
SSL_CERT_FILE=$HOME/.config/opencode/ca-bundle.crt \
SSL_CERT_DIR=/etc/ssl/certs \
  opencode serve --port 14141 --hostname 127.0.0.1

For a TUI, the only reliable form is to close the konsole window entirely and re-launch from a fresh one. The new bash login will source the updated tls-env.sh and the opencode child process will inherit the corrected CA bundle.

A short-lived opencode serve with the correct env was started for testing on 127.0.0.1:14141. The serve answered GET / with HTTP 200. openssl s_client against the same endpoint with the augmented bundle returned Verification: OK. After the per-TUI bounce, opencode.log shows successful stream providerID=minimax modelID=MiniMax-M3 and llm runtime selected entries with no further unknown certificate verification errors.

5. Verification

After the fix is in place and the relevant opencode process has been restarted with the new env:

# 1. Confirm the env is correct in the new process
cat /proc/<pid>/environ | tr '\0' '\n' | grep -E 'NODE_EXTRA_CA_CERTS|SSL_CERT_FILE'
# expected:
#   NODE_EXTRA_CA_CERTS=/home/<user>/.config/opencode/ca-bundle.crt
#   SSL_CERT_FILE=/home/<user>/.config/opencode/ca-bundle.crt

# 2. Confirm openssl trusts the chain with the new bundle
openssl s_client -CAfile ~/.config/opencode/ca-bundle.crt \
                 -connect api.minimax.io:443 \
                 -servername api.minimax.io \
                 </dev/null
# expected last line:
#   Verify return code: 0 (ok)

# 3. Confirm no new failures in the opencode log
tail -F ~/.local/share/opencode/log/opencode.log | grep -F "unknown certificate verification"
# expected: no output for at least 15 minutes under normal load.

6. Side findings

~/.local/share/opencode/opencode.db is 14.13 GB with a 271 MB WAL. PRAGMA freelist_count returns 0 free pages; the size is all live data, not fragmentation. VACUUM will not reclaim space meaningfully. The retry loop on the streaming error was the main driver of growth; with the TLS fix in place, per-process RSS is expected to drop by ~50% on its own. Recommend leaving the file as is for now and filing a separate developer-role ticket to do a clean-shutdown VACUUM after the next opencode restart.
autoupdate: true in opencode.json. This silently pulled 1.18.4 → 1.18.7 on 2026-07-27 (mtime of bin/opencode.exe). Recommend setting autoupdate: false and pinning versions manually after reading the release notes for "CA bundle" / "webpki" / "TLS" / "Bun" entries. Out of scope for sysadmin; hand off to developer.
Per-role launcher scripts (scripts/run_<role>.sh) do not explicitly source ~/.config/shell/tls-env.sh. They rely on .bash_profile → .bashrc → tls-env.sh inheritance, which is correct for fresh logins but breaks when a konsole is reused. Recommend adding a single line near the top of each scripts/run_<role>.sh to source the env explicitly. Out of scope for sysadmin (project-repo edits); hand off to developer.
No log-watcher for "unknown certificate verification". The current outage lasted ~5 hours before being noticed; the 577 hits in the log are the cumulative evidence. A watch that raises an alert when the string appears more than 5 times in a 10-minute window would catch a future regression inside minutes, not hours. Hand off to developer (or sysadmin's agent_ops script).

7. Reversibility

The fix is fully reversible in three commands:

cp ~/.config/shell/tls-env.sh.bak.20260727 ~/.config/shell/tls-env.sh
rm ~/.config/opencode/ca-bundle.crt
pkill -f 'opencode serve --port 14141'

No system-level state was modified. /etc/ssl/certs/, the system bundle, and the symlinks in /etc/ca-certificates/ are unchanged. No package was installed or removed.

For a permanent system-level fix (so that NODE_EXTRA_CA_CERTS is no longer needed for any client on the host), the same two CAs can be promoted to the system trust store once an interactive TTY is available:

sudo cp /tmp/chain-2.crt /etc/ca-certificates/trust-source/anchors/LE-YR2.crt
sudo cp /tmp/chain-3.crt /etc/ca-certificates/trust-source/anchors/LE-Root-YR.crt
sudo update-ca-trust

After that the personal augmented bundle becomes redundant (but harmless), and tls-env.sh can be reverted to its pre-2026-07-27 content or left as-is — both work.

8. References

Artifact	sha256 / location
Prior ticket	Established the `~/.config/shell/tls-env.sh` pattern and the `NODE_EXTRA_CA_CERTS` / `SSL_CERT_FILE` / `SSL_CERT_DIR` triple. Resolved the same symptom class on Bun runtime + incomplete webpki-roots at the time; insufficient on its own for the YR2 case.
Upstream issue	`opencode-ai/opencode#12408` — incomplete bundled webpki-roots, affects TLS to several Cloudflare-fronted services.
Upstream merge logic	`defaultCACertificates = bundled[] + extra[] + system[]` in `opencode-ai` 1.18.7 binary.
Let's Encrypt	YR2 intermediate + Root YR root, issued 2026-07-13, valid through 2026-10-11.
Arch wiki	`p11-kit` / `ca-certificates-utils` / `update-ca-trust` — system-level equivalent of the per-user fix above.
YR2 cert	`/tmp/chain-2.crt`, sha256.
Root YR cert	`/tmp/chain-3.crt`, sha256.
Augmented bundle
Updated env file
Backup of the pre-fix env file	`~/.config/shell/tls-env.sh.bak.20260727`, sha256

9. Status

Step	Status
Root cause identified	Yes — YR2 missing from all three of opencode's CA sources.
Personal fix applied and verified	Yes — `openssl` verifies, opencode streaming succeeds.
All 14 opencode processes bounced	Operator action — see §4.3 for the per-window procedure.
System-wide fix promoted	Pending operator TTY for `sudo update-ca-trust`.
`opencode.db` vacuum	Not addressed in this session — separate follow-up.
`autoupdate: false` in `opencode.json`	Not addressed in this session — developer follow-up.
Per-role launchers sourcing env explicitly	Not addressed in this session — developer follow-up.
Log-watcher for the error string	Not addressed in this session — developer follow-up.

opencode TLS verification failure on api.minimax.io: root cause and fix