Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
52e64ce
feat(platform): add shared buildkite extension client
roychying Jul 17, 2026
7826743
refactor(submitqueue): rebase buildkite adapter onto shared platform …
roychying Jul 17, 2026
4f58d41
feat(stovepipe): add buildkite-backed BuildRunner adapter
roychying Jul 17, 2026
a31f44d
feat(platform): add shared githubactions extension client
roychying Jul 20, 2026
fa7875f
refactor(submitqueue): rebase githubactions adapter onto shared platf…
roychying Jul 20, 2026
85fc006
feat(stovepipe): add githubactions-backed BuildRunner adapter
roychying Jul 20, 2026
3c10122
fix(stovepipe): match reversed BuildRunner.Trigger arg order in backends
roychying Jul 20, 2026
f10d7d3
Extract platform/http.SendRequest and add buildkite.ErrNotFound to de…
roychying Jul 23, 2026
d1d1a08
Propagate DecodeMetadataEnv's decode error instead of silently swallo…
roychying Jul 23, 2026
8042513
Extract a buildJSONWithEnv test helper that asserts the JSON marshal …
roychying Jul 23, 2026
f2a70a6
Check the w.Write error in TestCreateBuild instead of silently discar…
roychying Jul 23, 2026
e8d7e12
Let stovepipe's Buildkite BuildRunner panic on nil wiring instead of …
roychying Jul 23, 2026
e605145
Propagate the metadata marshal error in stovepipe's Buildkite Trigger…
roychying Jul 23, 2026
84f125d
Stop asserting on error message text in stovepipe's GitHub Actions Tr…
roychying Jul 23, 2026
9a475f8
Propagate the URI and metadata marshal errors in submitqueue's Buildk…
roychying Jul 23, 2026
e0aac07
Let submitqueue's Buildkite BuildRunner panic on nil wiring instead o…
roychying Jul 23, 2026
0e6f18b
Let submitqueue's GitHub Actions BuildRunner panic on nil wiring inst…
roychying Jul 23, 2026
504c9f0
Return a githubactions.ErrNotFound sentinel from do()/CancelRun inste…
roychying Jul 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions platform/extension/buildrunner/buildkite/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "go_default_library",
srcs = ["client.go"],
importpath = "github.com/uber/submitqueue/platform/extension/buildrunner/buildkite",
visibility = ["//visibility:public"],
deps = ["//platform/http:go_default_library"],
)

go_test(
name = "go_default_test",
srcs = ["client_test.go"],
embed = [":go_default_library"],
deps = [
"//platform/http:go_default_library",
"@com_github_stretchr_testify//assert:go_default_library",
"@com_github_stretchr_testify//require:go_default_library",
],
)
27 changes: 27 additions & 0 deletions platform/extension/buildrunner/buildkite/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Buildkite client

Shared HTTP client and Buildkite-specific facts for every domain's Buildkite-backed `BuildRunner`. There is no `BuildRunner` interface here by design — each domain (`submitqueue`, `stovepipe`, ...) defines its own `BuildRunner` and its own `BuildStatus`, and adapts this package's `State` to it. See [`doc/rfc/stovepipe/steps/build.md`](../../../../doc/rfc/stovepipe/steps/build.md#alternatives-considered-for-sharing-the-contract) for why the contract stays per-domain while the backend is shared.

## What lives here

- `Client`: `CreateBuild` / `GetBuild` / `CancelBuild` against the Buildkite REST API, plus `EncodeBuildNumber` / `ParseBuildNumber` for the build-number-as-id convention. Construct one with `NewClient(httpClient)`, where `httpClient` already has the pipeline's base URL (via `platform/http.BaseURLTransport`) and auth configured.
- `State` / `ParseState`: Buildkite's own build-state vocabulary (`creating`, `scheduled`, `running`, `blocked`, `passed`, `failed`, `canceling`, `canceled`, `skipped`, `not_run`), collapsed into the five states every domain's `BuildStatus` already distinguishes. Each domain still does its own trivial `State` → its own `BuildStatus` switch — this package does not know either domain's entity types.
- `DecodeMetadataEnv`: recovers a JSON-encoded `map[string]string` from a build's echoed env vars, given the env key the caller used at trigger time.

## Who consumes it

- `submitqueue/extension/buildrunner/buildkite` — batch-identity `BuildRunner`, resolves changes via `changeset.Resolver` before triggering.
- `stovepipe/extension/buildrunner/buildkite` — URI-identity `BuildRunner`, triggers directly from `headURI`/`baseURI`.

Both wrap a `*Client` built at the wiring layer; this package never constructs one from raw config (no credentials, no pipeline slug) itself.

## How the same `Client` stays safe to share across two different checkout strategies

SubmitQueue's build materializes state that doesn't exist yet — the runner resolves a batch DAG into composite base/head commits by applying patches. Stovepipe's build checks out a commit that already exists on trunk and, for an incremental build, diffs it against a baseline. These are different problems at the CI-pipeline level (see [build.md's "Why separate contracts"](../../../../doc/rfc/stovepipe/steps/build.md#why-separate-contracts)), yet both go through the same `Client.CreateBuild` call — the `Client` never inspects `CreateBuildRequest.Env` or picks a strategy, so there is nothing here that needs to "know" which pattern applies.

The split happens entirely outside this package, at two layers below it:

1. **Each domain's own adapter shapes a different `Env` payload.** SubmitQueue's runner sends `SQ_BASE_URIS`/`SQ_HEAD_URIS` as JSON-encoded arrays of resolved change URIs; stovepipe's sends `STOVEPIPE_HEAD_URI`/`STOVEPIPE_BASE_URI` as single plain strings. Both just call `Client.CreateBuild` with their own `Env` map — this package treats it as an opaque `map[string]string`.
2. **Each domain's `Client` targets a different Buildkite pipeline**, bound once at wiring time via the `httpClient`'s `BaseURLTransport.BaseURL` (e.g. `.../pipelines/submitqueue-go-code` vs `.../pipelines/stovepipe-go-code`). Each pipeline has its own Buildkite pipeline script (owned by Buildkite config, outside this repo) written against its domain's env-var contract — one applies patches into composite commits, the other checks out `STOVEPIPE_HEAD_URI` and diffs against `STOVEPIPE_BASE_URI`.

So the checkout strategy is a static, wiring-time binding — one `Client` instance ↔ one pipeline ↔ one pipeline script ↔ one env-var contract ↔ one domain's adapter — never a runtime decision made anywhere in Go code.
219 changes: 219 additions & 0 deletions platform/extension/buildrunner/buildkite/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
// Copyright (c) 2025 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

// Package buildkite provides the HTTP client and Buildkite-specific facts
// (build state vocabulary, the env-var metadata round-trip convention, and
// build number id encoding) shared by every domain's Buildkite-backed
// BuildRunner. It intentionally holds no BuildRunner interface or domain
// entity types — each domain (submitqueue, stovepipe, ...) defines its own
// BuildRunner and its own BuildStatus, and adapts this package's State to it.
// See doc/rfc/stovepipe/steps/build.md's "Alternatives considered for
// sharing the contract" for the rationale.
package buildkite

import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"strconv"

phttp "github.com/uber/submitqueue/platform/http"
)

// ErrNotFound is returned when the Buildkite API responds with 404 to a
// request for a resource by ID (e.g. GetBuild on an unknown build number).
var ErrNotFound = errors.New("buildkite: resource not found")

// Client is a thin wrapper around the Buildkite REST endpoints a BuildRunner
// needs: create, get, and cancel a build.
type Client struct {
httpClient *http.Client
}

// NewClient wraps a pre-configured *http.Client as a Buildkite Client. The
// caller is responsible for the base URL (via platform/http.BaseURLTransport)
// and auth (via an Authorization-header transport).
func NewClient(httpClient *http.Client) *Client {
return &Client{httpClient: httpClient}
}

// CreateBuildRequest is the payload for POST /builds.
type CreateBuildRequest struct {
Message string `json:"message"`
Env map[string]string `json:"env"`
}

// BuildResponse is the subset of fields callers need from a Buildkite build
// object.
type BuildResponse struct {
Number int `json:"number"`
State string `json:"state"`
WebURL string `json:"web_url"`
Env map[string]string `json:"env"`
}

// CreateBuild creates a new Buildkite build.
func (c *Client) CreateBuild(ctx context.Context, req CreateBuildRequest) (BuildResponse, error) {
body, err := json.Marshal(req)
if err != nil {
return BuildResponse{}, fmt.Errorf("marshal request: %w", err)
}
var build BuildResponse
if err := c.do(ctx, http.MethodPost, "/builds", body, &build); err != nil {
return BuildResponse{}, err
}
return build, nil
}

// GetBuild fetches a build by its Buildkite build number.
func (c *Client) GetBuild(ctx context.Context, number int) (BuildResponse, error) {
u := fmt.Sprintf("/builds/%d", number)
var build BuildResponse
if err := c.do(ctx, http.MethodGet, u, nil, &build); err != nil {
return BuildResponse{}, err
}
return build, nil
}

// CancelBuild requests cancellation. Returns nil when the build is already
// terminal (HTTP 422) — the Buildkite API uses that status to indicate a
// non-cancellable build, which the BuildRunner contract treats as a no-op.
func (c *Client) CancelBuild(ctx context.Context, number int) error {
u := fmt.Sprintf("/builds/%d/cancel", number)
status, respBody, err := phttp.SendRequest(ctx, c.httpClient, http.MethodPut, u, nil, c.setHeaders)
if err != nil {
return err
}

switch status {
case http.StatusOK:
return nil
case http.StatusUnprocessableEntity:
// Already terminal — no-op per BuildRunner.Cancel contract.
return nil
default:
return fmt.Errorf("unexpected status %d from cancel: %s", status, respBody)
}
}

// do sends an HTTP request with the standard Buildkite headers and, on a 2xx
// response, decodes the body into out (when non-nil). A 404 is reported as
// ErrNotFound so callers can distinguish it from other failures.
func (c *Client) do(ctx context.Context, method, rawURL string, body []byte, out any) error {
status, respBody, err := phttp.SendRequest(ctx, c.httpClient, method, rawURL, body, c.setHeaders)
if err != nil {
return err
}

if status == http.StatusNotFound {
return ErrNotFound
}
if status < 200 || status >= 300 {
return fmt.Errorf("API returned status %d: %s", status, respBody)
}

if out != nil {
if err := json.Unmarshal(respBody, out); err != nil {
return fmt.Errorf("unmarshal response: %w", err)
}
}
return nil
}

func (c *Client) setHeaders(req *http.Request) {
req.Header.Set("Content-Type", "application/json")
}

// EncodeBuildNumber encodes a Buildkite build number as an opaque build id
// string.
func EncodeBuildNumber(number int) string {
return strconv.Itoa(number)
}

// ParseBuildNumber is the inverse of EncodeBuildNumber.
func ParseBuildNumber(id string) (int, error) {
n, err := strconv.Atoi(id)
if err != nil {
return 0, fmt.Errorf("invalid build ID %q", id)
}
return n, nil
}

// State is Buildkite's own build-state vocabulary, interpreted from the raw
// state string every domain's Buildkite adapter receives on a build object.
// Buildkite's raw states are: creating, scheduled, running, blocked, passed,
// failed, canceling, canceled, skipped, not_run.
type State string

const (
// StateUnknown is returned for a raw state this package does not
// recognize. Not terminal — callers should keep polling rather than
// treat it as a final outcome.
StateUnknown State = ""
// StateAccepted means the build has been accepted for execution but has
// not started running yet (Buildkite: creating, scheduled).
StateAccepted State = "accepted"
// StateRunning means the build is currently executing, including while
// blocked on a manual block step (Buildkite: running, blocked).
StateRunning State = "running"
// StateSucceeded means the build completed successfully (Buildkite:
// passed).
StateSucceeded State = "succeeded"
// StateFailed means the build did not produce a passing result
// (Buildkite: failed, not_run, skipped).
StateFailed State = "failed"
// StateCancelled means the build was cancelled, or is in the process of
// being cancelled (Buildkite: canceling, canceled).
StateCancelled State = "cancelled"
)

// ParseState maps a raw Buildkite build-state string to a State. An
// unrecognized raw string maps to StateUnknown rather than being assumed
// terminal.
func ParseState(raw string) State {
switch raw {
case "creating", "scheduled":
return StateAccepted
case "running", "blocked":
return StateRunning
case "passed":
return StateSucceeded
case "failed", "not_run", "skipped":
return StateFailed
case "canceling", "canceled":
return StateCancelled
default:
return StateUnknown
}
}

// DecodeMetadataEnv recovers a JSON-encoded map[string]string from env[key].
// Buildkite echoes env vars back on the build object, so a caller that seeded
// a JSON-encoded map at Trigger time can recover it here at Status time
// without any local state. Returns an empty non-nil map when the key is
// absent. When the value is present but cannot be decoded, returns a nil map
// and the decode error for the caller to log or act on as it sees fit.
func DecodeMetadataEnv(env map[string]string, key string) (map[string]string, error) {
meta := make(map[string]string)
raw, ok := env[key]
if !ok || raw == "" {
return meta, nil
}
if err := json.Unmarshal([]byte(raw), &meta); err != nil {
return nil, fmt.Errorf("decode metadata env %q: %w", key, err)
}
return meta, nil
}
Loading
Loading