Introduction

This book is a collection of guides on how to work with the monorepo. It is intended for core developers and regular contributors who work often with the monorepo. If you're not a regular contributor, you'll probably find our docs more useful.

Mise

Our build environment is managed through a tool called mise. Mise provides a convenient way to install and manage all necessary dependencies for building and testing the packages in this repository. It helps guarantee that every job in the pipeline runs using the same set tools, which leads to more deterministic builds and fewer environment-related issues.

Mise defines a list of required software (like Go, Foundry, etc.) in a configuration file called mise.toml. The CI pipeline then runs mise at the beginning of every job in order to ensure that dependencies are installed at their proper versions. Therefore, all build-time dependencies must be defined in mise.toml. For this reason, we recommend developers use mise for local development as well.

Adding New Mise Dependencies

For the most part, adding a new Mise dependency is as straightforward as adding a new package to mise.toml. However, some packages require some additional configuration.

Aliased Packages

GitHub packages that expose multiple executables or where the executable name is different from the package name will require an alias to be defined in mise.toml. To do this:

Add an alias named after the package to the [alias] stanza.
Configure the alias to point to the package repository and specify the executable at the end as [exe=<your-executable>], e.g.ubi:goreleaser/goreleaser-pro[exe=goreleaser].
Add your package and its version to the list of tools in the [tools] stanza.

PR Authorization

Jobs running on self-hosted infrastructure will not run on forked PRs. This is a hard limitation of CircleCI. Additionally, any jobs that use secrets or contexts will not run if the PR's owner is not part of the ethereum-optimism GitHub org. This is a security policy: without these requirements, anyone could open a forked PR and run untrusted code on our executors, or exfiltrate secrets.

To get around this, forked PRs must be authorized by pushing them to the main repository. We run a bot called bailiff to do this.

To use Bailiff, you must be a member of the engineering team on GitHub. Then, to authorize a PR, post a comment that looks like this:

/ci authorize <full commit hash>

- or -

/ci authorize <full URL to commit>

The commit hash is compared with the head of the forked PR. If they don't match, Bailiff will refuse to push the PR. This prevents malicious users from ninja-pushing to PRs after they've been authorized, and ensures that we know exactly what is running on our infrastructure. New commits will have to be authorized again.

Note: Bailiff does not push to the source branch on the fork. It pushes to its own branch based on a hash of the source branch. It'll look like external-fork/<sha256 hash>. For this reason, re-running CI jobs on the source branch will not work. Instead, re-run the commit on the external-fork branch or push a new commit and re-authorize.

Bailiff will post a PR status if it succeeded. That status looks like this:

Requirements for Authorization

All of the following must be true for a PR to be authorized:

The authorizing user must be a member of the engineering group on GitHub.
The /ci command must be valid. The commit hash must be the full hash, not a shortened version. It is too easy to spoof commit short hashes.
The commit hash must match the head of the PR.
The PR must be on a fork.

Re-Running Failed PRs

If a PR fails due to a flake, you can re-run it from CircleCI's UI. However, you must re-run the commit that Bailif pushed. For example, you may see the following in the CircleCI UI:

A pipeline from pull/12345 (i.e., the one the user pushed to their fork)
A pipeline from external-fork/abcdef123456 (i.e., the one Bailiff pushed)

Re-run the pipeline on the external-fork branch. If you re-run the wrong branch:

The job will always fail, since OSS builds cannot be run on self-hosted runners.
The failing job's test results will override Bailiff's.

You will have to push a new commit and re-authorize the PR to resolve the issue. You can see which branch Bailiff pushed by looking at the Bailiff status check.

Troubleshooting

If you're having trouble with Bailiff, see below.

Check for the PR status. If that exists, bailiff ran and the problem is elsewhere.
Check for the existence of an external-fork branch. If that exists, bailiff ran and CI results are pending.
Wait a few minutes. Sometimes bailiff needs time to process a large queue of authorizations.
Ensure your /ci command is correct.
Check the bailiff logs here, and report the issue in the #platforms-general channel on Discord.

Self Hosted Runners

We use self-hosted runners to run some of our CI jobs. Self-hosted runners let us run jobs on bigger boxes for less money, and allow us to share build caches and the like between job runs on the same machine. Jobs using self-hosted runners will use machine executors and one of the following resource types:

latitude-1: Generic runner. Used for smaller jobs and Go builds. Has up to 16x concurrency.
latitude-1-go-e2e: Go E2E test runner. Has up to 2x concurrency to avoid overloading the machine, since the e2e tests are parallelized across all the machine's cores.
latitdue-fps-1: Dedicated fault proofs builder. Only allows 1 job to run at a time due to how heavy the fault proof tests are.

Generally, you should use the default CircleCI Docker runners for new jobs unless you're making heavy use of Go tools.

Forking Monorepo CI

It's possible to fork the monorepo and have functioning CI on your own CircleCI account, but it requires some extra steps.

OP Labs maintains a set of Ansible playbooks that provision the self-hosted runners. These playbooks are closed-source since they contain secrets. However, as a result of using mise, there is nothing special installed on the self-hosted runners that isn't already available in CircleCI's base Docker images. For example, here's the Dockerfile we use to configure the runners:

FROM circleci/runner-agent:machine-3.0.25-6554-32567e6

ARG GO_VERSION=1.22.8

USER root

RUN apt-get update && \
    apt-get install -y \
    curl \
    vim \
    git \
    build-essential \
    clang \
    jq \
    lld \
    binutils \
    ca-certificates \
    parallel

COPY wrapper.sh /var/opt/circleci/wrapper.sh
RUN chmod +x /var/opt/circleci/wrapper.sh

USER circleci:circleci

# Everything below this line is installed locally as the CircleCI user. The CCI user does not have sudo for security
# reasons, so don't put anything here that needs root.

WORKDIR /home/circleci

ENV PATH="$PATH:/usr/local/go/bin:/home/circleci/.foundry/bin:/home/circleci/go/bin:/home/circleci/.cargo/bin:/home/circleci/.local/bin"
ENV CIRCLECI_RUNNER_COMMAND_PREFIX="['/var/opt/circleci/wrapper.sh']"

wrapper.sh is equally simple:

#!/bin/bash

set -e

task_agent_cmd=${@:1}

echo "Running CircleCI task agent with command: ${task_agent_cmd}"
# Set up PATH
export PATH="$PATH:/usr/local/go/bin:/home/circleci/.foundry/bin:/home/circleci/go/bin:/home/circleci/.cargo/bin"
# Run the command
$task_agent_cmd
# Collect exit code
exit=$?
echo "CircleCI task agent finished."
exit $exit

Using the files above, you can either provision your own self-hosted runners in your monorepo fork or replace usages of the self-hosted runners in config.yaml with CircleCI's Docker runners.

Optimism Monorepo Ops