Audit Process

Context

P1: Audit Process

When teams need an audit, there should be a clear process with owners for all required steps: defining requirements and invariants, getting internal approvals, working with program management, talking to auditors, determining how many audits to get, what kinds of audits, negotiating audit prices, scheduling the audit, determining if a fix review is needed, and what to do with the results of an audit.

This document describes the use of software audits at Op Labs. In includes:

The resulting process integrates with the SDLC andenlists PgM and EVM Safety to help the Tech Lead execute the steps that are common to all audits, so that effort and uncertainty are minimised.

For further context on this process you can read this companion document and the references.

Summary

Audit Procurement

The audit requirements are established during the project FMAs in the Design Review phase of the SDLC and captured in the Security Readiness Document. Both the audit procurement and the feature implementation can start in parallel once the design is reviewed.

The Security Readiness Document is one of the deliverables from the design review and the primary artifact needed by PMO to schedule an audit. This document will be updated as necessary during the delivery lifecyle. It contains:

  • A summary of the project (or a link to a suitable summary if it already exists).
  • All relevant links to the project documentation, including specs and FMAs.
  • The scope for the audit.

We use Spearbit as our preferred auditing services provider and have a established a retainer with them to streamline approval. However, the feature team can choose a different provider from this list, from past engagements, or from any other source if they have a strong reason to go outside of Spearbit. Program Management (PMO) is available in the #pmo slack channel for assistance with anything related to engaging auditor services.

We don’t want to agree to exact audit dates too early, as that will compromise the quality of the software delivery. Instead, we will agree with auditors on a high-level schedule to confirm availability and ensure they are kept up to date on the implementation timeline and process, choosing an exact audit date close to the release date. Auditors not wishing to agree to this process should not be selected.

Auditors must agree to review the fixes to the vulnerabilities reported. Auditors not wishing to agree to this step should not be selected.

Once the Security Readiness Document and auditor preference has been submitted, PMO will request a SOW from the vendor for approval on Zip by:

  • Choosing "Request a Purchase/Vendor Onboarding/Purchase Renewal".
  • Under "What are you looking to purchase?" select "Other".
  • If the auditors have not been engaged in the past they will need to supply legal agreements, which will be also included in the Zip request.

The audit can only be executed once the Zip request is approved. PMO will coordinate any administrative back and forth, scheduling, or meetings that need to happen leading up the audit approval and kickoff.

Audit Execution

A devnet deployment is a requirement for the audit execution. As the date for the alphanet deployment is known with certainty, a date for the audit can be agreed so that the audit can be executed in parallel with the alphanet and betanet deployments and acceptance testing, and concluded before the testnet deployment.

We prefer to communicate with auditors over Slack during the audit. Questions from auditors should be answered promptly and carefully. These questions reveal gaps in the specifications or the scope, which should be amended accordingly.

Each vulnerability disclosed will be considered separately, fixed on an individual commit, and reviewed again by the auditors on the repo.

For all audit findings that we will fix as part of a later feature, create an issue for each finding in the monorepo. The issue title should be the finding title, the description links to the audit report, and apply the TBD label.

After Each Audit

Once all the fixes are applied and reviewed, the project lead should upload the final audit report to our repo.

If a valid high severity vulnerability was found, and this is the last expected audit for the project, **a post-mortem must be conducted and another audit of the same type must be scheduled**. These new audits follow the same process as any other audit.

Emergency Process

The audit process is tied to the SDLC process. A fast-track audit process would only be needed if we find out that we need audits later in the SDLC process, most likely as a result of updates to the risk modelling or excessive vulnerabilities in the last scheduled audit. The process described above is still applicable in these cases.

If the audit process is started in later stages of the SDLC, the documentation will be ready and can be put together as the Security Readiness Document by including a summary of the project, if that didn’t exist yet.

We already know that we need an audit, and we can safely assume that an external audit by Spearbit will fulfil the requirements.

The audit request still need to be approved via the Zip process above. If time doesn't allow for this then you should speak with your manager & PMO about your options to fast-track an audit as an exception.

Updating This Process

This process will be reviewed if SEV0 or SEV1 incidents are revealed during production, reported through a bug bounty, or caught in the last audit before production. The post-mortem might recommend updating this process.

Conversely, this process can also be reviewed with the goal of relaxing its requirements if no SEV1 or SEV0 bugs or incidents have happened in production, the bug bounty, or any last audit for at least six months.

References

Next Steps

  • Update the Failure Mode Analyses (FMAs)
  • Update the Security <> Developer Interface
  • Refactor the docs so that they point to the github repo with the reports, instead of pointing at individual reports.
  • The success of this initiative depends partially on the SDLC process being adopted and respected.
  • Include in the SDLC process that other feature teams and EVM Safety should review specs and scope.