Replace the three upload Workflows with inline retries and throttling

[Jbithell]

Why

Cloudflare are introducing per-step billing for Workflows. The weather station reports roughly every 30 seconds, and every observation created three workflow instances — Windguru, Met Office and Windy — which is around 255k instances a month on the free plan.

Each of those workflows was a single step.do() wrapping one fetch, existing purely to get durable retry. The observation is already saved to D1 before the workflow is created, so a failed upload only ever costs a data point on a third-party site.

OvernightSaveToR2 is untouched — it runs a handful of times a day.

What changed

Uploads run inline under ctx.waitUntil. The three uploader bodies moved verbatim into website/workers/uploads/{windguru,metoffice,windy}.ts — same URLs, same params, same MD5 salt scheme, same success/backoff detection. NonRetryableError from cloudflare:workflows became a local NonRetryableUploadError, and the step's timeout: "5 seconds" became AbortSignal.timeout on each fetch.

withRetry in workers/uploads/retry.ts does 3 attempts spread over ~18s (0s, +3s, +15s), giving a ~33s worst case. That is less patient than the workflows' 3 attempts over ~3 minutes, but retries can no longer outlive the invocation — and the next observation arrives ~30s later regardless. Sleeping costs no CPU time, which matters against the free plan's 10ms per-invocation limit.

Each service is throttled to its own minimum interval: Windguru 2 minutes (matching the interval=120 already sent), Met Office and Windy 5 minutes. Windy rejects anything sent inside 5 minutes, so most of its uploads were previously created only to be discarded. This cuts outbound calls from ~8,500/day to ~1,300/day and removes nearly all of the rate-limit rejections. The trade-off is fewer data points on the three public sites.

New upload_state D1 table (migration 0004_wealthy_wildside.sql) holds last_attempt_at, last_success_at and last_error per service. The attempt timestamp is written before the uploads run, so an observation arriving while a slow retry is still in flight can't dispatch to the same service again. last_error also replaces the visibility the Workflows dashboard used to give.

D1 rather than KV for this state: KV reads are cached for a minimum of 60 seconds, which would break a 2-minute throttle, and the free plan allows only 1,000 KV writes/day against the ~1,300 this needs.

Bug fix: double-encoded dateutc

The Met Office dateutc value was escaped by hand (.replace("T","+").replace(/:/g,"%3A")) and then escaped again by URLSearchParams, so WOW was receiving the literal string 2026-07-26+12%3A00%3A00 instead of a timestamp. It now passes the plain YYYY-mm-DD HH:mm:ss value and lets URLSearchParams do the escaping:

	sent on the wire	what WOW decodes
before	dateutc=2026-07-26%2B12%253A00%253A00	2026-07-26+12%3A00%3A00
after	dateutc=2026-07-26+12%3A00%3A00	2026-07-26 12:00:00

Verification

Ran locally against react-router dev with a stub standing in for the three services:

Case	Result
All three succeed	200 in 0.09s, one request each, last_success_at set, last_error cleared
Second observation straight after	200 in 0.06s, observation still inserted, all three skipped as throttled, zero outbound fetches
Met Office returns 429	Exactly one request — NonRetryableUploadError short-circuits the retry loop
Windy returns 500	Three requests at +0s, +3s, +15s, then last_error recorded
Mixed outcomes in one dispatch	Targets fully isolated — Windguru recorded success while the other two recorded errors
dateutc after the fix	Stub decodes "2026-07-26 12:00:00" from an observation timestamped 2026-07-26T12:00:00.000Z

The outgoing query strings were diffed against what the workflows sent and are identical apart from the dateutc fix above.

npm run build passes. npm run types:check reports only the pre-existing app/root.tsx(73,10) process.env error, which is untouched by this branch.

Note

Removing the three entries from wrangler.jsonc stops new instances but leaves the workflows registered in the Cloudflare account — they need deleting there separately. Two orphans from an earlier refactor (PSCWeather-HandleReceivedObservation, PSCWeather-DisregardReceivedObservation) appear to still be registered too.

[Copilot]

Pull request overview

This PR removes the three “upload” Cloudflare Workflows (Windguru / Met Office / Windy) and replaces them with inline uploads executed under ctx.waitUntil, adding local retry/backoff logic and per-target throttling backed by a new D1 upload_state table to reduce workflow instance costs and outbound requests.

Changes:

• Remove the three upload workflow bindings and workflow entrypoint classes; keep OvernightSaveToR2 unchanged.
• Add inline upload modules for Windguru/Met Office/Windy with a shared withRetry helper and fetch timeouts.
• Add upload_state D1 table + throttle/dispatch logic to skip uploads until each target’s minimum interval elapses.

Reviewed changes

Copilot reviewed 12 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
website/wrangler.jsonc	Removes the three upload workflow bindings to stop new workflow instances.
website/workers/app.ts	Deletes the three upload workflow entrypoints; keeps scheduled workflow behavior.
website/app/routes/uploadFromWeatherStation.ts	Switches from creating three workflows to a single inline dispatchObservation call in waitUntil.
website/workers/uploads/index.ts	New dispatch/throttling orchestration and persistence of attempt/success/error state.
website/workers/uploads/retry.ts	New local retry/backoff helper and non-retryable error type.
website/workers/uploads/windguru.ts	New inline Windguru uploader with timeout and non-retryable credential failure.
website/workers/uploads/metoffice.ts	New inline Met Office uploader with dateutc fix and timeout/backoff handling.
website/workers/uploads/windy.ts	New inline Windy uploader with timeout/backoff handling.
website/database/schema/UploadState.ts	Adds Drizzle schema/types for upload_state and uploadTargets.
website/database/schema.d.ts	Re-exports UploadState schema/types.
website/database/migrations/0004_wealthy_wildside.sql	Adds the upload_state table.
website/database/migrations/meta/0004_snapshot.json	Drizzle migration snapshot for the new table.
website/database/migrations/meta/_journal.json	Records the new migration in the migration journal.
website/worker-configuration.d.ts	Regenerated Wrangler types reflecting removed workflow bindings and updated runtime types.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.