besu/ethereum/evmtool

EVM Tool

EVM Tool is a stand alone EVM executor and test execution tool. The principal purpose of the tool is for testing and validation of the EVM and enclosing data structures.

Using EVM Tool in execution-specification-tests

To use EVM Tool in Execution Spec tests it is recommended that you use the GraalVM build, as the framework incurs significant startup penalties for each invocation when run via the Java runtime.

Building Execution Tests on macOS

Current as of 24 Jun 2023.

MacOS users will typically encounter two problems,one relating to the version of Python used and one relating to zsh.

Homebrew will only install the most recent version of Python as python3, and that is 3.11. The execution tests require 3.10. The solution is to use a 3.10 version of python to set up the virtual environment.

python3.10 -m venv ./venv/

Zsh requires braces to be escaped in the command line, so the step to install python packages needs to escape the brackets

pip install -e .\[docs,lint,test\]

An all-in-one script, including homebrew, would look like

brew install ethereum solidity
git clone https://github.com/ethereum/execution-spec-tests
cd execution-spec-tests
python3 -m venv ./venv/
source ./venv/bin/activate
pip install -e .[docs,lint,test]

Building EvmTool with GraalVM on macOS

First you need a GraalVM JDK installed, if not already installed.

It is recommended you install SDKMAN to manage the graalVM install, homebrew has issues with native attributes and code signing.

sdk install java 22.3.r17-grl 
sdk use java 22.3.r17-grl

You can also manually install GraalVM from the GraalVM website..

Once GraalVM is installed you use the nativeCompile target.

./gradlew nativeCompile 

The resulting binary is ./ethereum/evmtool/build/native/nativeCompile/evmtool

If the testing repository and besu are installed in the same parent directory, the command to run the execution tests is

fill -v tests --evm-bin ../besu/ethereum/evmtool/build/install/evmtool/bin/evm 

Assuming homebrew and SDKMan are both installed, the complete script is

sdk install java 22.3.r17-grl 
sdk use java 22.3.r17-grl
git clone https://github.com/hyperledger/besu
cd besu
./gradlew nativeCompile
cd ..

brew install ethereum solidity
git clone https://github.com/ethereum/execution-spec-tests
cd execution-spec-tests
python3 -m venv ./venv/
source ./venv/bin/activate
pip install -e .[docs,lint,test]

fill -v tests --evm-bin ../besu/ethereum/evmtool/build/install/evmtool/bin/evm

If you don't want to use the GraalVM tool the binary that is compatible is generated by the ethereum:evmtool:installdist target and is located at ./ethereum/evmtool/build/install/evmtool/bin/evm

Why not GraalVM for everything?

Using GraalVM in execution-spec-tests results in over 20x performance increase in execution. It will be faster to build GraalVM from scratch and run the execution-spec-tests than to run just the Java version.

It is demonstrably faster to run the Java version for a node. All the test execution gains are the result of reduced startup penalties. Larger benchmarks will show that performance intensive EVM code will be slower in GraalVM than the Java version due to the adaptive compiler.

For contracts that execute 30 million gas in small operations it is often faster to run the Java EVMTool than the GraalVM EVMTool, including startup penalty. The execution tests focus on smaller VM tests that demonstrate specification conformance.

We would also need to reconsider some library choices. GraalVM does not work with Log4J, and we would have to ban that library across Besu and all dependents. Libraries such as Netty also have some problematic entry points that interact poorly with how SLF4J needs to initialize.