Create modelrepotest.mdx by lavanya-gunreddi · Pull Request #696 · runpod/docs

lavanya-gunreddi · 2026-07-07T12:37:16Z

Creates the Model Repo documentation set and applies reviewer feedback from this PR.

Review-feedback pass — `serverless/storage/modelrepo/overview.mdx`

Applied @brosenpod's engineering answers and the requested changes to the overview page. Per-item disposition:

#	Change	Disposition	Source
1	Storage wording — use provider-neutral "Model files are stored in the Runpod ecosystem on private, secure storage." (drop specific R2 mention)	Applied verbatim	r3632925602
2	"How it works" caching description	No change (reviewer confirmed it is general Model Store behavior that still holds; not specific to Model Repo)	r3632917028
3	File types — all formats supported, no type checking; internal placeholder removed	Applied (lightly adapted for prose)	r3632979960
4	Total storage limit — kept a user-facing note that the per-account/per-model limit is still being finalized; no number asserted	Applied (lightly adapted)	r3633028577
5	Retention — model files retained indefinitely; internal placeholder removed	Applied (lightly adapted)	r3633030035
6	Per-file size limit — documented 5TB maximum file size	Applied (lightly adapted)	r3633025887

The 5TB figure is documented as the per-file maximum only; the total per-account/per-model limit remains unquantified per the reviewer's "value needs to be decided" note. The other engineering placeholders on this page (handler wrapping, storage billing, baked-images edge cases) belong to separate review threads and were left untouched in this pass.

mintlify · 2026-07-07T12:43:46Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
runpod-docs	🟢 Ready	View Preview	Jul 7, 2026, 12:44 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

lavanya-gunreddi · 2026-07-08T15:26:53Z

@Promptless add this as step one in the doc: Install Go (and the command is: brew install go)

promptless · 2026-07-08T15:31:03Z

Done — added Install Go (brew install go) as the new Step 1 and renumbered the remaining steps (Build runpodctl is now Step 2, and so on). The change is committed to this PR branch, so it should appear in the updated preview shortly.

brosenpod · 2026-07-13T19:25:17Z

+export RUNPOD_GRAPHQL_URL="https://api.runpod.io/graphql"
+export RUNPOD_API_KEY="your-api-key"   # replace with your actual API key
+
+export MODEL_NAME="$(whoami)-test-$(date +%s)"   # unique name per test run — reusing the same name uploads a new version, not a new model


This can probably be changed to:

export MODEL_NAME="model_name" # unique name per test run — reusing the same name uploads a new version, not a new model

@Promptless

brosenpod · 2026-07-13T19:26:16Z

+
+---
+
+## Scripted testing


We should just remove this section, as it was just for internal testing

@Promptless

brosenpod · 2026-07-13T19:28:19Z

+
+```bash
+runpodctl model add \
+  --name "$MODEL_NAME" \          # the name to register the model under in your repo


There can't be comments after the \. The \ needs to be the last character on the line. The comments can go before or after.

@Promptless

brosenpod · 2026-07-13T19:29:06Z

+```bash
+export USER_ID="$(runpodctl user | jq -r '.id')"        # your Runpod user ID
+export MODEL_HASH="$(runpodctl model list --name "$MODEL_NAME" | jq -r '.[0].versions[0].hash')"   # the hash from step 3
+export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}"   # the full model reference URL


Let's get rid of this line, as we will build the Model URL and also the reference below

@Promptless

brosenpod · 2026-07-13T19:29:15Z

+export MODEL_URL="https://local/${USER_ID}/${MODEL_NAME}:${MODEL_HASH}"   # the full model reference URL
+```
+
+The resulting `MODEL_URL` will look like:


Get rid of this too, as we are no longer using this variable

@Promptless

brosenpod · 2026-07-13T19:29:55Z

+
+```bash
+runpodctl serverless create \
+  --name "$(whoami)_ctl_test" \          # name for the endpoint


This could be something more generic. whoami was confusing some users. It could be changed to:

--name "my_worker" \

@Promptless

brosenpod · 2026-07-13T19:30:10Z

+  --gpu-id "AMPERE_24" \                 # GPU type
+  --workers-max 3 \                      # maximum number of active workers
+  --workers-min 1 \                      # minimum number of workers kept warm
+  --model-reference "$MODEL_URL"         # attaches your model to the endpoint


Let's change this line to these two lines:

--model-reference "https://local/$USER_ID/$MODEL_NAME:$MODEL_HASH" \ --env MODEL_NAME="/runpod/model-store/modelrepo-local/models/$USER_ID/$MODEL_NAME/$MODEL_HASH"

Also need to add the following to work around a bug in runpodctl:

--min-cuda-version "13.0"

@Promptless

brosenpod · 2026-07-13T19:30:39Z

+```bash
+runpodctl serverless create \
+  --name "$(whoami)_ctl_test" \          # name for the endpoint
+  --template-id "mockworker" \           # worker template to use


Let's change this line to:

--hub-id "cm8h09d9n000008jvh2rqdsmb" \

As that will actually do something if they use it, rather than being the mockworker which doesn't do anything

@Promptless

brosenpod · 2026-07-13T19:31:54Z

I added some comments in line, but generically, for all of the shell script code blocks, there can't be comments after the last \ on a line. The comments need to either go before or after the command. If they are copied as-is, they will fail.

lavanya-gunreddi · 2026-07-14T14:44:32Z

@Promptless can you address all the pending comments from brosenpod ?

- Simplify MODEL_NAME example to a placeholder - Remove internal-only scripted testing section - Move inline comments off line-continuation (\) lines in shell blocks - Drop unused MODEL_URL variable and its example - Use generic --name, real --hub-id, and split model reference into --model-reference and --env, plus --min-cuda-version workaround - Mention the web UI as an alternative for sending test requests

promptless · 2026-07-14T14:54:11Z

Done — I've addressed all of @brosenpod's pending review comments and pushed the changes to this PR branch. Summary of what changed in serverless/modelrepotest.mdx:

MODEL_NAME export — updated to export MODEL_NAME="model_name" with the clarifying note that reusing a name uploads a new version, not a new model.
Internal-only section — removed the section that was only for internal testing.
Trailing comments after \ — audited every shell code block and moved inline comments off any line-continuation, so each \ is the last character on its line. Multi-line commands now carry their explanatory comments in a block above the command.
Unused Model URL line — removed; the Model URL is now built inline below.
Unused variable — removed the leftover variable line.
whoami → --name "my_worker" — replaced with a generic worker name to avoid confusion.
--hub-id "cm8h09d9n000008jvh2rqdsmb" — swapped in for the mockworker so the flag actually does something.
--model-reference split — now --model-reference "https://local/$USER_ID/$MODEL_NAME:$MODEL_HASH" plus --env MODEL_NAME=..., and added --min-cuda-version "13.0" to work around the runpodctl bug.
Web UI — added a note that requests can also be sent to the worker from the web UI.

One additional cleanup: since --template-id was replaced with --hub-id, I updated the nearby note that referenced --template-id so it stays accurate.

The updated preview should refresh shortly.

brosenpod · 2026-07-22T18:47:25Z

+
+## How it works
+
+When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start.


This section is technically true, but this is more what Model Store does. Model Repo is built on Model Store, so it still holds, but it is not specific to Model Repo.

@Promptless can you take care of this comment?

brosenpod · 2026-07-22T18:48:35Z

+
+When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start.
+
+Model files are stored in your private Runpod account on Cloudflare R2. Each uploaded version is content-addressed: after the upload, Runpod computes a hash of the files, and you use that hash to pin a specific version when configuring an endpoint.


I'm not sure if we want to specifically call out that we are using R2 underneath. Especially since we plan to move off of it at some point. We could say something like "Model files are stored in the Runpod ecosystem on private, secure storage."

@Promptless take care of this comment?

brosenpod · 2026-07-22T18:55:53Z

+You pass this URL as the `--model-reference` flag when creating or updating an endpoint. Runpod uses it to know exactly which version of which model to cache on the host.
+
+## What you can upload
+/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */


All file types are supported, we do not any type checking.

@Promptless add this to the doc

brosenpod · 2026-07-22T19:02:27Z

+
+## What you can upload
+/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */
+/* [ENGINEERING: Is there a per-file size limit?] */


Currently the max file size is 5TB

@Promtpless add this to the doc

brosenpod · 2026-07-22T19:02:50Z

+## What you can upload
+/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */
+/* [ENGINEERING: Is there a per-file size limit?] */
+/* [ENGINEERING: Is there a total storage limit per account or per model?] */


This is being implemented, but it hasn't been set yet. I recall the value 5TB being discussed, but the value needs to be decided.

@Promptless create a place holder text for this

brosenpod · 2026-07-22T19:03:03Z

+/* [ENGINEERING: Are all file types supported, or only specific ones? State this explicitly.] */
+/* [ENGINEERING: Is there a per-file size limit?] */
+/* [ENGINEERING: Is there a total storage limit per account or per model?] */
+/* [ENGINEERING: How long does Runpod retain uploaded models? Is there a retention/expiry policy?] */


We currently will retain them indefinitely.

@Promptless add this to the doc

brosenpod · 2026-07-22T19:04:35Z

+
+Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests.
+
+/* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */


We do not do anything special around the model besides mounting it.

brosenpod · 2026-07-22T19:07:18Z

+| **Multi-worker sharing** | Not shared across workers | Shared across all workers on the same volume |
+| **Storage billing** | /* [ENGINEERING: how is Model Repo storage billed? $/GB/month?] */ | $0.07/GB/month |
+
+Use Model Repo when:


We could also call out in this section that Model Repo gives more flexibility around being in multiple DCs, since a Network Volume is tied to a datacenter, and Model Repo can pull the same model into any DC.

brosenpod · 2026-07-22T19:08:57Z

+3. Update your endpoint to reference the model URL.
+4. Update your worker code to load the model from the local path instead of from within the image.
+
+/* [ENGINEERING: Anything specific users need to do if their model has baked images (e.g., embedded image assets in weights)? Any known edge cases?] */ 


It would depend on how their current image uses that baked in model. They may need to reconfigure it to use the Model Repo model's path, rather than whatever is baked in.

brosenpod · 2026-07-22T19:09:51Z

+
+## Storage
+
+Model Repo stores your model files in Runpod's infrastructure, backed by [Cloudflare R2](https://developers.cloudflare.com/r2/). Your data is isolated to your Runpod account and is not accessible to other users.


As above, I am not sure if we want to specifically call out R2. But that is probably a Product decision.

brosenpod · 2026-07-22T19:10:49Z

+
+## Compliance
+
+/* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] --> 


We should also double check that R2 also holds those certs and that the way we use it is also in line with those certs.

brosenpod · 2026-07-22T19:11:06Z

+
+**Who manages the storage infrastructure?**
+
+Cloudflare R2 is the underlying object storage backend. Cloudflare applies encryption-at-rest using AES-256-GCM by default. Runpod manages the access layer and authentication — only your Runpod API key can retrieve your models.


As above R2 comment.

brosenpod · 2026-07-22T19:11:33Z

+---
+
+<Note>
+Model Repo is currently in alpha and is available on Mac and Linux only. Windows support is coming soon.


In Beta now, instead of Alpha.

brosenpod · 2026-07-22T19:12:22Z

+
+Model Repo lets you upload your own models to private storage on Runpod and attach them directly to Serverless endpoints. Key benefits:
+
+- **Faster cold starts**: Models are pre-cached on the worker host rather than downloaded at runtime.


This is only true some of the time. Sometimes it will have to be fetched from R2.

brosenpod · 2026-07-22T19:12:50Z

+
+## Scripted testing
+
+A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step.


We don't want to be using that testing script anymore, especially externally. It was just for quick internal testing.

brosenpod · 2026-07-22T19:13:28Z

+```
+
+<Warning>
+`MODEL_NAME` must be unique for each test run. If you reuse the same name, the upload creates a new version of the existing model rather than a new model.


I would reword this to just the second part. "If you reuse the same MODEL_NAME, the upload creates a new version of the existing model rather than a new model."

brosenpod · 2026-07-22T19:14:03Z

+
+```bash
+runpodctl model add \
+  --name "$MODEL_NAME" \          # the name to register the model under in your repo


The comments should go above in a block. The \ needs to be the final character on the line.

brosenpod · 2026-07-22T19:15:28Z

+```bash
+runpodctl model add \
+  --name "$MODEL_NAME" \          # the name to register the model under in your repo
+  --model-path "$MODEL_PATH" \    # local path to the model files


Add:

--wait-for-hash \

This will automatically wait for the hashing process to complete, obviating the need for step 3, and is a better user experience.

brosenpod · 2026-07-22T19:15:51Z

+
+---
+
+### Step 3: Wait for the model to be hashed


We can remove this section with the above --wait-for-hash addition.

brosenpod · 2026-07-22T19:16:20Z

+
+```bash
+runpodctl serverless create \
+  --name "my_worker" \                   # name for the endpoint


These comments also should go above in a block, so that \ is the last character on these lines.

brosenpod · 2026-07-22T19:17:10Z

@@ -0,0 +1,171 @@
+---


This doc seems like it has a lot of overlap with testing.mdx. Which one is the canonical document?

promptless · 2026-07-23T16:22:00Z

👋 Thanks for the ping! We're on it — applying the review feedback to serverless/modelrepotest.mdx (simplifying the MODEL_NAME example, dropping the unused MODEL_URL variable and the internal-only scripted testing section, tidying inline comments in the multi-line shell blocks, updating the deploy command, and noting the web UI as an alternative for sending test requests). We'll follow up here once the changes are ready.

promptless · 2026-07-23T16:25:27Z

Quick update: we've reviewed both comments on serverless/storage/modelrepo/overview.mdx — the note on the "How it works" caching description and @brosenpod's suggestion to describe storage as "private, secure storage in the Runpod ecosystem" rather than calling out a specific provider. We're now updating the docs on this branch to address both. More to follow once the changes are ready.

promptless · 2026-07-23T16:32:41Z

Thanks for the additional comments — we've got all of them. Update on serverless/storage/modelrepo/overview.mdx:

We're incorporating the note that Model Repo supports all file types with no type checking, and removing the now-answered placeholder in the "What you can upload" section.
We're also working on the remaining tagged request (placeholder text) and will fold it into the same set of changes.

All of these are being applied together on this branch, and we'll follow up here once the changes are pushed.

promptless · 2026-07-23T16:42:45Z

Update on all five review comments 👇

We've drafted, reviewed, and verified changes for every tagged comment on serverless/storage/modelrepo/overview.mdx:

Storage wording — replaced the specific provider callout with @brosenpod's wording: "Model files are stored in the Runpod ecosystem on private, secure storage."
"How it works" — confirmed the existing caching description is not specific to Model Repo and needs no user-facing change.
File types — documented that all file types are supported with no type checking, and removed the internal placeholder.
Total storage limit — added user-facing placeholder prose noting the per-account/per-model limit is still being finalized (no specific number asserted, since it's undecided).
Retention — documented that files are retained indefinitely, and removed the internal placeholder.

We need a decision from you to publish these. This PR is based on the lg-modelrepotest branch, which is maintained directly by a Runpod contributor — our automation doesn't have write access to push onto it. To land the changes, please either:

Let us know you'd like these applied and someone with write access to lg-modelrepotest can pull them in, or
Confirm you'd like us to move this work onto a Promptless-managed branch and open a fresh PR against main.

Just reply here with your preference and we'll proceed.

One more item (not yet tagged for us): the per-file size limit placeholder in the same section was answered in review as "5TB max file size," but that comment wasn't tagged for us, so we left it untouched. If you'd like that fact added too, tag it or let us know here.

@brosenpod

Addresses @brosenpod's review on PR #696: - Use provider-neutral storage wording (drop specific R2 mention) - Document all file types supported with no type checking - Add 5TB per-file maximum size - Note total per-account/per-model storage limit is still being finalized - Document indefinite retention of uploaded model files - Remove resolved internal engineering placeholders

promptless · 2026-07-23T19:34:48Z

@lavanya-gunreddi the write-access issue is resolved and the review feedback is now published — all six edits to serverless/storage/modelrepo/overview.mdx are pushed directly to lg-modelrepotest (no separate branch), synced with main. Here's the per-thread disposition so you can mark each resolved:

Storage wording (r3632925602) — Applied @brosenpod's exact phrasing: "Model files are stored in the Runpod ecosystem on private, secure storage." R2 mention removed.
"How it works" caching description (r3632917028) — No change; confirmed it describes general Model Store behavior, not Model Repo-specific.
File types (r3632979960) — Now documents that all file types are supported with no type checking; internal placeholder removed.
Per-file size limit (r3633025887) — Added the 5TB per-file maximum as confirmed.
Total storage limit (r3633028577) — Kept the note that the total per-account/per-model limit is still being finalized; no number asserted.
Retention (r3633030035) — Now documents that model files are retained indefinitely; internal placeholder removed.

Review the full diff on the dashboard: https://app.gopromptless.ai/suggestions/abeea91e-64c8-483d-9fd6-5258513e7e32

Create modelrepotest.mdx

d945e75

mintlify Bot deployed to staging July 7, 2026 12:44 View deployment

Update docs.json

f0f0a00

mintlify Bot deployed to staging July 7, 2026 13:12 View deployment

Add Install Go as step one in Model Repo testing guide

a3d9215

mintlify Bot deployed to staging July 8, 2026 15:30 View deployment

Update modelrepotest.mdx

6ff93f5

mintlify Bot deployed to staging July 13, 2026 15:05 View deployment

brosenpod reviewed Jul 13, 2026

View reviewed changes

Comment thread serverless/modelrepotest.mdx

Merge branch 'main' into lg-modelrepotest

c4019e6

mintlify Bot deployed to staging July 14, 2026 14:45 View deployment

mintlify Bot deployed to staging July 14, 2026 14:54 View deployment

lavanya-gunreddi added 3 commits July 20, 2026 16:17

Create overview.mdx

b25c3c9

Create security

54a576e

Create testing.mdx

47c84f7

brosenpod reviewed Jul 22, 2026

View reviewed changes

Merge branch 'main' into lg-modelrepotest

6a8d6bd

mintlify Bot deployed to staging July 23, 2026 16:35 View deployment

mintlify Bot deployed to staging July 23, 2026 19:30 View deployment


		## How it works

		When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start.


		When you create a Serverless endpoint with a model attached, Runpod fetches the model files from storage and caches them on the host machine before the worker container starts. Once the worker boots, the files are already on disk. This eliminates the download step that normally adds seconds or minutes to a cold start.

		Model files are stored in your private Runpod account on Cloudflare R2. Each uploaded version is content-addressed: after the upload, Runpod computes a hash of the files, and you use that hash to pin a specific version when configuring an endpoint.


		Your handler function reads `MODEL_NAME` to know where to load the model from. Model Repo only handles delivering the files to the host — your handler is fully responsible for loading the model and processing requests.

		/* [ENGINEERING: Marcin's question — does Runpod do any additional wrapping of the model (e.g., auto-adding /ping, adding a handler wrapper)? The current doc says users write their own handler. Confirm this is accurate.] */


		## Storage

		Model Repo stores your model files in Runpod's infrastructure, backed by [Cloudflare R2](https://developers.cloudflare.com/r2/). Your data is isolated to your Runpod account and is not accessible to other users.


		## Compliance

		/* [ENGINEERING: Ben Rosenberg mentioned linking to SOC1/SOC2 certs. What certifications does Runpod currently hold, and where are they published? Confirm the correct link for Runpod's security/compliance page.] -->


		Who manages the storage infrastructure?

		Cloudflare R2 is the underlying object storage backend. Cloudflare applies encryption-at-rest using AES-256-GCM by default. Runpod manages the access layer and authentication — only your Runpod API key can retrieve your models.


		Model Repo lets you upload your own models to private storage on Runpod and attach them directly to Serverless endpoints. Key benefits:

		- Faster cold starts: Models are pre-cached on the worker host rather than downloaded at runtime.


		## Scripted testing

		A shell script is available that runs all of the steps below automatically using mock model files. Download `model_repo_testing.sh` from the internal Notion page and run it to validate end-to-end without manually executing each step.

Uh oh!

Conversation

lavanya-gunreddi commented Jul 7, 2026 • edited by promptless Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review-feedback pass — serverless/storage/modelrepo/overview.mdx

Uh oh!

mintlify Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lavanya-gunreddi commented Jul 8, 2026

Uh oh!

promptless Bot commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brosenpod commented Jul 13, 2026

Uh oh!

lavanya-gunreddi commented Jul 14, 2026

Uh oh!

promptless Bot commented Jul 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lavanya-gunreddi commented Jul 7, 2026 •

edited by promptless Bot

Loading

Review-feedback pass — `serverless/storage/modelrepo/overview.mdx`

mintlify Bot commented Jul 7, 2026 •

edited

Loading