Filecoin — Sector State Management

It’s been a while since I look at Go code. Lately I got some free time and wanted to switch gears to Sector state management implemented with pure Go. Reading code design in big projects is beneficial to both coding and design architecture. The latest Go code of Lotus uses modular design. Lotus code evolves gradually, from all logic coupling together in first release to modularizing in later releases.

Lotus Miner “pack” data that users need to store into Sectors, and process each Sector. Thie article will go over the Sector state management.

Modular Framework

Modules related to Sector state management are listed below:

Sector state management is based on state machine. Lotus implements general statemachine in go-statemachine pack. Built upon StateMachine, StateGroup is an abstract object that manages multiple StateMachines at same time. StateMachine only changes the state; the specific state is stored through go-statestore. Beyond StateMachine, the specific business includes defining state change rules and corresponding functions to states. SectorState is the specific business for Lotus managing Sector. When we have these low-level modules, it is easier to know which Lotus functions to call. Start from the low-level state management module, StateMachine:

StateMachine

StateMachine in defined in go-statemachine at machine.go:

type StateMachine struct {
     planner  Planner
     eventsIn chan Event
 
     name      interface{}
     st        *statestore.StoredState
     stateType reflect.Type
 
     stageDone chan struct{}
     closing   chan struct{}
     closed    chan struct{}
 
     busy int32
 }

planner is an abstract function for statemachine state transfer. Planner receives Event, then decides next step of processing based on current state user.

type Planner func(events []Event, user interface{}) (interface{}, uint64, error)

st is saving state. stateType is type for saving state.

The core of StateMachine is run function. It has three parts: receiving Event, processing state, and calling next function. The key step is processing state:

err := fsm.mutateUser(func(user interface{}) (err error) {
    nextStep, processed, err = fsm.planner(pendingEvents, user)
    ustate = user
    if xerrors.Is(err, ErrTerminated) {
        terminated = true
        return nil
    }
    return err
})

mutateUser checks current StoredState, executes planner function, then stores the state of planner function result. planner function returns next method, number of processed Events, and possible error. Run function triggers another go routine to execute nextStep.

mutateUser implementation uses reflect property of Go to abstract and modularize. Folks interested in this logic can look it up.

StateGroup

Business usually needs lots of StateMachine. For example, Lotus Miner saves multiple Sectors, each Sector is an independent state maintained by aStateMachine. StateGroup instantiates multiple “identical” StateMachine. This is defined in group.go:

type StateGroup struct {
     sts       *statestore.StateStore
     hnd       StateHandler
     stateType reflect.Type
 
     closing      chan struct{}
     initNotifier sync.Once
 
     lk  sync.Mutex
     sms map[datastore.Key]*StateMachine
 }

sms is an array of StateMachine. StateHandler is an interface for StateMachine state process function:

type StateHandler interface {
     Plan(events []Event, user interface{}) (interface{}, uint64, error)
 }

In another word, we need an interface for StateHandler (Plan function for state transfer) on StateMachine.

StoredState

StoredState is an abstract Key-Value storage. The Get/Put methods are straight forward and easy to understand.

SectorState

storage-fsm implements Sector state related business logic. It also means state defination, state transfer functions are implemented in this pack. The complete Sector information is defined in storage-fsm/types.go:

type SectorInfo struct {
     State        SectorState
     SectorNumber abi.SectorNumber // TODO: this field's name should be changed to SectorNumber
     Nonce        uint64           // TODO: remove
 
     SectorType abi.RegisteredProof
 
     // Packing
     Pieces []Piece
 
     // PreCommit1
     TicketValue   abi.SealRandomness
     TicketEpoch   abi.ChainEpoch
     PreCommit1Out storage.PreCommit1Out
 
     // PreCommit2
     CommD *cid.Cid
     CommR *cid.Cid
     Proof []byte
 
     PreCommitMessage *cid.Cid
 
     // WaitSeed
     SeedValue abi.InteractiveSealRandomness
     SeedEpoch abi.ChainEpoch
 
     // Committing
     CommitMessage *cid.Cid
     InvalidProofs uint64 // failed proof computations (doesn't validate with proof inputs)
 
     // Faults
     FaultReportMsg *cid.Cid
 
     // Debug
     LastErr string
 
     Log []Log
 }

SectorInfo includes Sector state, data for Precommit1/2, Committing data, etc. SectorState describes Sector state in details. All Sector states are defined in sector_state.go:

const (
     UndefinedSectorState SectorState = ""
 
     // happy path
     Empty          SectorState = "Empty"
     Packing        SectorState = "Packing"       // sector not in sealStore, and not on chain
     PreCommit1     SectorState = "PreCommit1"    // do PreCommit1
     PreCommit2     SectorState = "PreCommit2"    // do PreCommit1
     PreCommitting  SectorState = "PreCommitting" // on chain pre-commit
     WaitSeed       SectorState = "WaitSeed"      // waiting for seed
     Committing     SectorState = "Committing"
     CommitWait     SectorState = "CommitWait" // waiting for message to land on chain
     FinalizeSector SectorState = "FinalizeSector"
     Proving        SectorState = "Proving"
     // error modes
     FailedUnrecoverable SectorState = "FailedUnrecoverable"
     SealFailed          SectorState = "SealFailed"
     PreCommitFailed     SectorState = "PreCommitFailed"
     ComputeProofFailed  SectorState = "ComputeProofFailed"
     CommitFailed        SectorState = "CommitFailed"
     PackingFailed       SectorState = "PackingFailed"
     Faulty              SectorState = "Faulty"        // sector is corrupted or gone for some reason
     FaultReported       SectorState = "FaultReported" // sector has been declared as a fault on chain
     FaultedFinal        SectorState = "FaultedFinal"  // fault declared on chain
 )

For those of you know SDR algorithm, you can find many familiar terms hers: PreCommit1，PreCommit2，Commiting, etc. Now with the context of state process function, we can thoroughly understand what each state means and what they process.

In fsm.go, Plan function in Sealing is state processing function for Sector:

func (m *Sealing) Plan(events []statemachine.Event, user interface{}) (interface{}, uint64, error) {
     next, err := m.plan(events, user.(*SectorInfo))
     if err != nil || next == nil {
         return nil, uint64(len(events)), err
     }
 
     return func(ctx statemachine.Context, si SectorInfo) error {
         err := next(ctx, si)
         if err != nil {
             log.Errorf("unhandled sector error (%d): %+v", si.SectorNumber, err)
             return nil
         }
 
         return nil
     }, uint64(len(events)), nil // TODO: This processed event count is not very correct
 }

Plan function is also pretty straight forward. Call plan() to process curresnt state, and return the next function. We can look at the state processing in two 0arts: 1/ defination of state transfer 2/ state process.

Sector state transfer is defined in fsmPlanners:

var fsmPlanners = map[SectorState]func(events []statemachine.Event, state *SectorInfo) error{
     UndefinedSectorState: planOne(on(SectorStart{}, Packing)),
     Packing:              planOne(on(SectorPacked{}, PreCommit1)),
     PreCommit1: planOne(
         on(SectorPreCommit1{}, PreCommit2),
         on(SectorSealPreCommitFailed{}, SealFailed),
         on(SectorPackingFailed{}, PackingFailed),
     ),
     PreCommit2: planOne(
         on(SectorPreCommit2{}, PreCommitting),
         on(SectorSealPreCommitFailed{}, SealFailed),
         on(SectorPackingFailed{}, PackingFailed),
     ),
...

For instance, we receive SectorPreCommit1 event at PreCommit1 state, now we are about to enter PreCommit2 state. All Sector state transfer is illustrated below:

Then we call the corresponding function to process state. Take PreCommit1 as an example, the corresponding function would be handlePreCommit1.

func (m *Sealing) handlePreCommit1(ctx statemachine.Context, sector SectorInfo) error {
     tok, epoch, err := m.api.ChainHead(ctx.Context())
     ...
     pc1o, err := m.sealer.SealPreCommit1(ctx.Context(), m.minerSector(sector.SectorNumber), ticketValue, sector.pieceInfos())
     if err != nil {
         return ctx.Send(SectorSealPreCommitFailed{xerrors.Errorf("seal pre commit(1) failed: %w", err)})
     }
   ...
 }

We can clearly see that Precommit1 calculation is dones by calling rust-fil-proofs from SealPrecommit1 in handlePreCommit1(). Lastly let’s look at definition of each state:

Empty — empty state

Packing — packing state, to fill multiple Pieces into one Sector

PreCommit1 — PreCommit1 calculation

PreCommit2 — PreCommit2 calculation

PreCommitting — submit Precommit2 result to blockchain

WaitSeed — wait for random seed (given 10 blocks time, make the random seed unpredictable)

Committing — calculate Commit1/Commit2 and submit proof to blockchain

CommitWait — wait for confirmation from blockchain

FinalizeSector — Sector state confirmed and clean temp

Summary：

Sector state management is based on statemachine. General statemachine is implemented by using go-statemachine. State storage is implemented by using go-statestore. Built upon these modules, , storage-fsm implements Sector state defination and state handle functions.