Ethereum Gas Reprice Hard Fork Was A Success

Posted on October 21, 2016 by bok

Originally posted Oct 19 2016, updated 02:49 Oct 21 2016 UTC.

Table of contents

Summary

The Gas Reprice hard fork successfully stopped all the previous attacks since the start of Devcon2.

Update 14:26 Oct 19 2016 UTC – The attacker is now conducting a new set of smaller impact attacks using another an underpriced EXP opcode and a BALANCE operation that need some cache improvements – see the updated section at the bottom of this page to see the impact on the block processing. The next planned hard fork will address these new attack vectors – see ATTENTION MINERS: Recommending miners lower the gas limit target to 2 million. What a dynamic and responsive Ethereum ecosystem this is!

Update 02:49 Oct 19 2016 UTC – The attacks have now ceased, at least temporarily. The attacker may have given up as the latest attacks do not have much impact on the running of the network.

And the attacker’s accounts and transactions has ?accidentally? revealed that they pool mined with EthPool and DwarfPool. These pool miners would have recorded the IP address(es) of the attacker’s miners. Read all about it in next exciting blog post Ethereum Network Attacker’s IP Address Is Traceable.

The Attacks

The Ethereum blockchain has been undergoing a attack since the start of the Devcon2 conference a few weeks ago (beginning Sep 19 2016).

The first attack started on block 2,283,416 at 01:04:56 Sep 19 Shanghai time and targeted the go-ethereum geth clients, causing a memory error and shutting down these nodes across the network. Luckily the Ethereum network has separately implemented node clients written in Rust (Parity) and Java (EthereumJ) – these clients did not suffer from the same memory bug and kept the network up and running. The Ethereum developers in Shanghai had to get up early before the start of the conference and shortly after released a version of geth with the memory bug fixed. This ended up delaying the start of the conference by half an hour.

My 125 Mhs solo miner stopped working for the week while I was in Shanghai as I had not set up a VPN connection to my node due to security reasons.

Soon after the first attack, the attacker then regularly sent spam transactions (see here and here) to slow down the Ethereum network. Using cheaply priced opcodes (Ethereum Virtual Machine instruction codes), the attacker sent transactions that caused heavy computational and disk input/output loads on the Ethereum nodes. This was later followed by spam transactions that created many empty accounts on the blockchain, causing a “state-bloat” on the blockchain.

Worst affected users seem to be those trying to run a full sync of the Ethereum blockchain on machines with limited memory and running on a hard disk drive (HDD) instead of a solid state drive (SSD). See Is something wrong with the network right now?, sync isn’t working for 0.8.5 been trying for days and 5th time downloading Blockchain 15 days for example.

Non-spam transactions were failing as quite a number of mining pools decided not to process transactions as these would slow down their mining node operations, and the non-spam transactions were getting lost in the ether.

The Hard Forks

The Ethereum Foundation announced two hard forks to stop the spam transaction attacks. As listed in the previous link, you have to install the latest Ethereum node clients containing the hard fork code – geth 1.4.18, Parity 1.3.8, EthereumJ 1.3.6 and Ethereum Wallet 0.8.6 (remember never to trust links to the node clients from unofficial sites). If you have run an earlier version of the node client after the first Gas Repricing hard fork (see below), you will have to re-sync your blockchain data from scratch.

The First Hard Fork – Gas Repricing

The first EIP 150 Long-term gas cost changes for IO-heavy operations to mitigate transaction spam attacks hard fork successfully activated at block 2,463,000 at Oct-18-2016 01:19:31 PM +UTC to reprice the gas for some of the opcodes. This raises the cost for the attacker to send their spam transactions.

The Attacks Have Stopped

The attackers transactions started failing (source) immediately after the hard fork. This account spend 9 – 3.899521491 = 5.100478509 ethers (~ USD 64.10) to send 1,183 spam transactions to the network.

The Gas Limit

The gasLimit represents the total amount of gas for the computation and storage that can be spent within each block (14 seconds average block time). This gasLimit is a figure that is voted on by the miners.

You can see the Gas Limit chart increasing on the bottom right of the chart below from https://ethstats.net/ around 7+ hours after the hard fork:
screen-shot-2016-10-19-at-07-45-35

Following are some gasLimit statistics. The gasLimit was around 4.7 million before the attack. As advised by the Ethereum Foundation, miners voted to reduce the gasLimit to 1.5 million during the attack to limit the damage by the spam transactions, and this was reduced further to 0.5 million at the later stages of the attack. You can now see it rising to 2.5 million:

The Gas Price

The gasPrice increased during the attacks as a portion of miners stopped accepting transactions and the remaining miners that accepted transactions had their blocks full from the spam transactions. Non-spam transactions were getting dropped and unprocessed by the Ethereum network. For non-spam transactions to be mined, the gasPrice for these transactions had to be increased. From https://etherscan.io/charts/gasprice:
screen-shot-2016-10-19-at-10-25-52

Checking the current gasPrice with the following code, the gasPrice has fallen back to the pre-attack levels:

Regular account to account transactions cost 21,000 gas, which now equates to around 0.000459187432998 ETH or ~ USD 0.005365596876048108, or around half a cent, at 1 ETH = 12.543 USD. At the peak, someone paid a gasPrice of 0.00000091, ~ 45x the normal amount, to push their transactions through the network.

The Uncles

The rate of uncles per 1,000 blocks has been higher than average during the attack period but this is now falling:

The higher than average rate of uncles during the attack period was most likely due to mining nodes being slowed down by the heavy computational and disk input/output burden caused by the spam transactions.

The Empty Blocks

During the attack, some miners decided not to include any transactions in their mined blocks as processing these transactions would slow down the initial preparation of the block before mining on the block commences – see Dwarfpool Empty Block Mining? and Wall of fame / wall of shame – miners who are doing nothing for the network right now. Here are some statistics that show that empty blocks have not yet reduced to the pre-attack period:

A Successful Hard Fork

Congratulations to the developers and the Ethereum Foundation for a successful hard fork! The spam transactions have stopped. For a while at least. Update 14:26 Oct 19 2016 UTC – Less than a day of calm.

The Second Hard Fork – State Clearing

There will be the EIP #158 State Clearing hard fork shortly to remove many of the empty accounts filling up the blockchain.

Ethereum Classic Classic?

This gas repricing hard fork was not as contentious as the hard fork to refund the USD 50 million hack on The DAO’s smart contract code with a bug. The Ethereum Classic (ETC) chain arose from this The DAO hard fork as about 10% of miners (who insist on immutable blockchains) continued mining on the non-hard forked chain. And the It-Who-Must-Not-Be-Named exchange listed ETC on their exchange by surprise.

The ETC chain has recently been undergoing similar spam transaction attack and there is a proposal to perform the same hard fork at block 2,500,000 on Oct 25 2016. As this hard fork violates the principle of an immutable blockchain that is a concern for some part of that community, there are calls for a vote before hard forking. There is a small chance that an Ethereum Classic Classic (ETCC) chain may arise from this community split.

A New Set Of Attacks (Update 14:26 Oct 19 2016 UTC)

A new set of attacks using underpriced EXP and BALANCE opcodes is currently occurring – see Lol, I think its another attack contract – Burns fees like a champ. Somone is very persistent here 🙂. The effects on the network are apparently not severe like the previous attacks. And this opcode will be repriced in the next hard fork.

The following logs shows that some blocks are taking longer than usual to process.

Update 09:13 Oct 20 2016 UTC – From FAQ: Upcoming Ethereum Hard Fork – Ethereum Blog, vitalik stated before the new attacks that:

It’s worth noting that an increase in the gas cost of the EXP opcode is likely in HF round 2; this is the one arithmetic opcode that we’ve found that is underpriced (though by a substantially smaller factor than what’s getting fixed in HF round 1).

The Good News (Update 22:03 Oct 19 2016 UTC)

From Lol, I think its another attack contract – Burns fees like a champ. Somone is very persistent here 🙂, gustav_simonsson writes:

Correcting the gas cost of op codes makes Ethereum more efficient as it enables more txs and real throughput by disincentivizing “artificial” use of the platform – simple loops over op codes which has no utility other than DoS.

In fact, the attack surface of Ethereum is surprisingly small given that it’s much more powerful & flexible than other live blockchain systems. This is much due to having cross-client specifications & test suits and multiple implementations in different programming languages.

Claiming the attacker has “unlimited runway to continue” is ignorant FUD.

The vulnerabilities fixed over the past weeks have all been in very specific subsets of the Ethereum protocols:

1. gas cost of a subset of the VM op codes.
2. logic around “empty” accounts
3. performance of state handling of accounts.
4. unconfirmed tx (mempool) queuing & broadcast logic.

(3) and (4) have been vastly improved in geth, parity and other implementations which is why the attacker is no longer targeting these.

(1) was too a large degree fixed by the HF, although it’s clear that EXP (and possibly a few other arithmetic op codes) also need to be adjusted.

(1) and (2) will be further addressed by a 2nd HF (https://github.com/ethereum/EIPs/issues/158)

With the current gas system, there will always be some combination of op codes requiring the most resource usage for the least cost.

Prior to this HF, the block gas limit was voted to 0.5M to ensure the network remained operational and processed txs during attacks.

Now we see that the network is chugging along during attack txs with a at 2M gas limit (although older, slower systems without SSD drives will need patches that are currently being worked on in both geth and parity).

Overall, this HF + patches in clients have improved the bottleneck of Ethereum by more than an order of magnitude. In a few weeks, it’ll be another order of magnitude.

And in the long term all txs we currently describe as “attack” or “DoS” txs will become indistinguishable from regular txs as their tx fees will be proportional to their resource usage.

This entry was posted in Blog and tagged , . Bookmark the permalink.