After a technical issue that prevented users from trading crypto or transferring assets on-chain, infrastructure operators of the Solana network raced to fix the problem on Saturday.
According to Solana’s Discord server, the blockchain began “forking” (creating conflicting versions its transaction history) around 00:53 New York Time. Soon after, the validators’ RAM started increasing and the chain’s transaction throughput dropped off a cliff.
These tech problems combined to effectively stop all on-chain activity from the Solana network. According to Solana Explorer, the network was processing 93 transactions per second (TPS) by 02:00. This is well below the rate of 5000 TPS just 15 minutes earlier.
This was a similar crisis to the chain’s rocky technology incidents in 2022 when a series halts and slowdowns led reforms to Solana’s management of inbound traffic. Although Saturday’s incident was not immediately caused by any fault, validator operators and network engineers suspect that there was a bug in Solana’s new code. This code had been online for hours.
Some validators started downgrading to the older version without a clear bug to squash. This was in the hope of revitalizing Solana’s throughput, according to SolBlaze (a pseudonymous developer who manages a liquid stake pool and is active within developer circles). Later, Solana employees started advocating the downgrade.
In an attempt to restore Solana’s operations, a majority of validators switched back to the older software within hours. It did not solve the problem that was slowing down performance. It was then decided to try a more drastic approach: re-starting the chain at the point just before the forking.
SolBlaze stated that coordination of a restart effort means that the chain will go offline entirely, which is always an option.
UPDATE: [4:09 UTC] Additional details.