FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

Krish Agarwal¹, Zhuoming Chen¹, Yanyuan Qin², Zhenyu Gu², Atri Rudra³, Beidi Chen¹

¹ Carnegie Mellon University    ² AMD    ³ University at Buffalo

[Paper] | [Blog]

---

Real-time multimodal applications are hard to serve: each combines different models, sits at its own latency and throughput tradeoffs, and no shared serving stack fits them all. FlashRT is an agent harness that guides a coding agent to build a customized serving system for each application instead. A human writes a simple, strictly-sequential, single-GPU reference backend plus the frontend that drives it; the agent converts the reference into an explicit intermediate representation (IR), derives an efficient, possibly multi-GPU, streaming deployment from the IR's dependency structure, and validates it against the reference, all in a self-driven loop that never touches the human-written code. Across five applications, on both NVIDIA B200 and AMD MI355X GPUs, the agent-built deployments reach up to 70× lower latency and 3.6× higher throughput than the references, with no hand-written serving code. This repository contains the shared runtime, the harness, and the deployments the agent produced for the included applications.

Applications

• WorldPlay — drive through a live generated world with the keyboard.
• LiveAvatar — talk face-to-face with an avatar that answers out loud.
• Krea-Realtime + SAM3 — restyle your webcam background in real time while you stay unedited.
• Qwen3-Omni — type a prompt and hear the spoken response as it is generated.
• LongLive — steer a continuously generated video by speaking prompts.

Quick Start

Installation

conda create -n flashrt python=3.13 -y
conda activate flashrt
bash install.sh

install.sh installs the Python dependencies, flashrt and its kernels, and the third-party engines under third_party/. It targets CUDA by default. For ROCm, run FLASHRT_TARGET=rocm bash install.sh. See docs/setup.md for details and prerequisites.

Download model weights

Download the checkpoints for the applications you want to run:

python -m flashrt.weights <app>

Weights are placed under checkpoints/ at the repo root. See docs/setup.md for the details.

Run an application

Each application runs as a backend process (the model pipeline, serving shared memory) plus a frontend process (the user-facing surface). examples/<app>/README.md documents the full flow per application: setup, weight download, backend scripts per GPU budget, and the frontend. The pattern is always:

# backend: pick a script for your GPU budget (rocm/ for the ROCm build)
bash examples/<app>/cuda/run_4gpu.sh

# once it logs "<App> backend READY", in a second shell:
bash examples/<app>/run_frontend.sh

Every backend variant of an application speaks the same shared-memory contract (defined in flashrt/apps/<app>/adapter.py, configured by flashrt/apps/<app>/config.yaml), so the frontend attaches to whichever backend is running. docs/frontends.md covers frontend setup.

Layout

<repo-root>/
├── AGENTS.md            ← entry point for the optimization agent
├── assets/              ← shared static assets (images, clips)
├── docs/                ← setup and how-to guides
├── examples/<app>/      ← launch scripts per application
├── flashrt/             ← the installable package
│   ├── apps/<app>/      ← one directory per application:
│   │   ├── config.yaml      runtime config (shared buffer names, paths)
│   │   ├── adapter.py       frontend ↔ backend IPC contract
│   │   ├── reference/       human-written sequential reference backend
│   │   ├── frontend/        human-written frontend
│   │   └── backend/         agent-written optimized deployments + IR
│   ├── apps/_template/  ← copy this to start a new application
│   ├── ir/              ← IR framework (graphs, executor, analysis)
│   ├── channels/        ← shared-memory IPC buffers
│   ├── distributed/     ← process groups, collectives, parallel plans
│   ├── models/ runner/ layers/ memory/ loader/ ...
│   └── frontend/        ← shared frontend helpers
├── flashrt_kernels/     ← custom kernels
└── third_party/         ← engines cloned by install.sh

Adding your own

• New application: copy flashrt/apps/_template/, fill in the reference pipeline, adapter, and frontend, then run the optimization agent against it. Walkthrough: docs/adding_an_application.md.
• New model: add the model implementation and config preset to the shared runtime so pipelines can use it. Walkthrough: docs/adding_a_model.md.

Running the optimization agent

AGENTS.md at the repo root is the agent's entry point: the IR workflow, the optimization targets, the correctness and benchmarking requirements, and the boundary between human-written and agent-written code. Nested AGENTS.md files under flashrt/apps/, flashrt/ir/, and assets/ refine the rules per tree, and the closest one to a file wins. docs/adding_an_application.md includes example prompts for pointing an agent at an application.

Acknowledgements

The shared runtime builds on open-source serving infrastructure, in particular vLLM, vLLM-Omni, Flash Attention, and model loading adapted from FastVideo. The included applications build on the open releases of their respective models, linked from each application's README.

Citation

@misc{agarwal2026flashrtagentharnessguiding,
  title         = {FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications},
  author        = {Krish Agarwal and Zhuoming Chen and Yanyuan Qin and Zhenyu Gu and Atri Rudra and Beidi Chen},
  year          = {2026},
  eprint        = {2607.18171},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.18171}
}