/biz/ - Business & Finance

>>29841334
Fantom network incident analysis
Development
On Thursday, February 25 2021 at 3.04 PM UTC, Fantom Opera mainnet halted new block confirmations, which resulted in a temporary outage.

The core developer team and Fantom validator community immediately responded and successfully resumed the network within 7 hours.

This feat required consensus across 39 validators in many different time zones; while the community quickly diagnosed the issue, most outage time was due to waiting for sufficient stake weight to come back online.

To be clear, no funds were at risk as a result of the network halt.

Cause
One of the biggest validators slowed down the block emission, which caused a second big validator to slow down as well. The other validators kept producing blocks, but the two lagging ones were not able to catch up. These two validators are big enough to represent more than 1/3W of stake, and they caused a domino effect that halted new block confirmations.

Recovery
Our core developer team coordinated with the validators to apply a temporary patch to slow down the event emission rate to allow lagging nodes to catch up. Once all the nodes were synced again, they started to confirm blocks and finalize epochs.

Validator nodes size
The incident has highlighted that the network needs to make some changes regarding the validator nodes’ requirements.

We identified two issues:

The recent increase in the value of FTM made creating new nodes very prohibitive. While this is not directly related to the incident, it accentuated point 2.
Most of the staking power is concentrated among six nodes due to the concentration of delegations.
We’ll make a proposal for stakers and delegators to vote on the on-chain governance. We’re still defining the parameters but will likely involve lowering the minimum amount of FTM required to run a node and putting a cap on the node size.