feat(supervisor): NFLOG-based bypass detection to drop the CAP_SYSLOG requirement

[gaveezy]

Problem Statement

Sandbox pods/containers currently request CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and CAP_SYSLOG. With Kubernetes user namespaces (GA in 1.36, already supported via enable_user_namespaces) the first three become namespaced and lose their host-level power — but CAP_SYSLOG does not. The kernel checks syslog(2)//dev/kmsg access against the initial user namespace (capable(), not ns_capable()), so in a hostUsers: false pod the granted CAP_SYSLOG is a no-op and the bypass detection monitor silently degrades.

The only consumer of CAP_SYSLOG is the bypass detection monitor in openshell-supervisor-process, which tails dmesg --follow for nftables log entries (prefix openshell:bypass:<ns>:) emitted by the per-workload netns REJECT rules. Reading the host-global kernel ring buffer from inside a sandbox is also a mild information leak, and requires the nf_log_all_netns sysctl on the node.

Proposed Design

Switch bypass log delivery from kernel-printk to nfnetlink_log (NFLOG):

• During netns setup (privileged startup, before supervisor seccomp hardening), bind an AF_NETLINK/NETLINK_NETFILTER socket to an NFLOG group inside the workload netns. Netns affinity is fixed at socket() time; binding needs only CAP_NET_ADMIN over the netns owner, which works under user namespaces.
• Generate the nftables log rules as log prefix <p> group <N> when the socket bind succeeds; keep the existing log ... flags skuid printk rules as an automatic fallback when it fails (e.g. kernel without nfnetlink_log).
• The bypass monitor consumes NFLOG packet messages (prefix attribute for filtering, NFULA_UID for workload UID, raw IP payload for dst/ports/proto) and reports through the existing OCSF dual-emit path. The dmesg path remains as fallback (still useful in the VM driver, which runs with full capabilities).
• Drop CAP_SYSLOG from the Kubernetes, Docker, and Podman driver capability sets.

Enforcement is unaffected: the nftables REJECT rules do not depend on the log delivery mechanism. NFLOG delivery is netns-scoped, so the monitor no longer reads host-global kernel logs, and nf_log_all_netns is no longer needed on the NFLOG path.

Alternatives Considered

• Keep dmesg, relax kernel.dmesg_restrict on nodes — exposes host kernel logs to every pod on the node; strictly worse.
• eBPF-based detection — bpf() is also initial-userns-gated (CAP_BPF/CAP_SYS_ADMIN in init ns), so it has the same problem as CAP_SYSLOG, plus a new dependency.
• Netlink crate dependency (netlink-packet-netfilter / libnetfilter_log bindings) — the required protocol subset (group bind, copy mode, packet attr parsing) is ~stable kernel ABI and small; a hand-rolled libc implementation avoids new dependencies in a security-sensitive crate and matches its existing raw-FFI style.

Agent Investigation

• CAP_SYSLOG's only consumer is bypass_monitor::spawn (crates/openshell-supervisor-process/src/bypass_monitor/mod.rs); drivers request it in openshell-driver-kubernetes/src/driver.rs, openshell-driver-docker/src/lib.rs, openshell-driver-podman/src/container.rs.
• No eBPF usage anywhere in the workspace; netns/nftables management shells out to iproute2/nft, so NFLOG covers the whole detection path.
• The supervisor seccomp prelude does not restrict socket(); the workload runtime filter blocks non-NETLINK_ROUTE netlink sockets, so the sandboxed workload cannot bind the NFLOG group itself.
• The sidecar topology has no bypass log listener today (no CAP_SYSLOG there either); it keeps printk rules unchanged.
• Implementation is complete on a branch with unit tests (netlink message layout, datagram parsing, IPv4/IPv6 packet parsing, ruleset generation); Linux cargo check/clippy (pedantic+nursery)/tests pass. PR to follow.

[drew]

📋 triage-agent

Triage Assessment

Classification: feature-valid

Summary

The capability problem is real and the NFLOG proposal is technically sound. Current code tails dmesg --follow for bypass diagnostics while Kubernetes, Docker, and Podman request CAP_SYSLOG; under a pod user namespace the kernel permission check still uses capable(CAP_SYSLOG), so that grant cannot authorize access to the host-global kernel log.

Investigation

• crates/openshell-supervisor-process/src/bypass_monitor/mod.rs checks that dmesg exists, then runs dmesg --follow with stderr suppressed. An EPERM therefore ends monitoring without surfacing the permission failure.
• crates/openshell-supervisor-process/src/netns/nft_ruleset.rs keeps LOG commands non-required and REJECT commands required, so changing log delivery does not weaken enforcement.
• crates/openshell-supervisor-process/src/netns/mod.rs currently enables nf_log_all_netns, confirming the host kernel-log dependency.
• The three combined/container drivers request SYSLOG; no other runtime consumer was found.
• The architecture supports the proposal: network namespace setup occurs before supervisor startup hardening, and the workload seccomp policy blocks non-route netlink sockets. A supervisor-owned NFLOG socket can therefore be created during privileged setup and consumed afterward.
• No duplicate or merged fix was found. Proposal: Split Supervisor and Agent into Separate Pods with gVisor Isolation #981 is broader architecture work. PR feat(supervisor): deliver bypass detection logs via NFLOG and drop CAP_SYSLOG #2383 implements this proposal but was self-closed, unreviewed, and unmerged.

Recommendation

Proceed as a valid supervisor/sandbox security improvement. Before merge, require live-kernel E2E coverage that proves nft log group packet delivery and behavior when NFLOG support is unavailable. Also describe the fallback precisely: after removing CAP_SYSLOG, ring-buffer fallback cannot restore diagnostics in restricted Kubernetes, Docker, or Podman environments; it remains useful only where kernel-log access is independently available, such as the VM path.

Routing applied: area:supervisor, area:sandbox, and topic:security; removed state:triage-needed. state:agent-ready remains a human-only decision.

[gaveezy]

@drew thanks for the triage, I will submit the PR for the implementation soon

[drew]

Before implementing, can you talk a little more about how this would get configured and when we would use it? If understand this replaces an existing implementation and is used for all drivers.

[gaveezy]

@drew for the VM and Docker backend, no configuration changes, for Kubernetes if we want to opt-in user namespace then need to set pod.spec.hostUsers field to false