* implementation of Bonsai snapshots based BonsaiWorldStateArchive
includes: try-with-resources and AutoCloseable WorldState in order to release snapshots when we are done with them
Signed-off-by: garyschulte <garyschulte@gmail.com>
Due to subtle differences between Bytes32 and UInt256 the changes failed
burn-in testing and are being reverted.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
The current internal storage APIs implement UInt256 rather than Bytes32,
which is an accident of prior EVM design. This migrates all internal
handling of storage addresses and values to Bytes32.
The main performance gain is reduced GC pressure, in that UInt256
stores the data as an array of ints, whereas Bytes32 keeps a sliced or
wrapped array of bytes. Since database APIs are all byte array based
this should remove unneeded conversions when going to/from the DB.
Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com>
* use optimistictransactiondb for mutable isolated snapshots
* plumbing necessary to have a snapshot specific updater.
* snapshot rolling working
* implement AutoCloseable on BonsaiSnapshotWorldState to ensure we can correctly dispose of snapshots
* add snapshot transaction cloning, change snapshot based worldstate to extend persisted worldstate rather than in-memory worldstate
Signed-off-by: garyschulte <garyschulte@gmail.com>
Add java module info to the manifests of the following libraries used by the Besu EVM
./crypto - org.hyperledger.besu.internal.Crypto
./ethereum/rlp - org.hyperledger.besu.internal.Rlp
./util - org.hyperledger.besu.internal.Util
./datatypes - org.hyperledger.besu.Datatypes
./evm - org.hyperledger.besu.Evm
./plugin-api - org.hyperledger.besu.PluginApi
Signed-off-by: Mustafa Uzun <mustafa.uzun@limechain.tech>
* Backward Sync is now remembering recent finalized blocks.
When syncing using BWS we now consider finalized blocks reported by consensus layer.
When no finalized block is specified, nothing strange will go on. When Finalized block is specified
But Besu is not aware of any finalized block yet, we will sync as usual until the finalized block is about to get imported.
A new invariant is introduced when importing blocks using BWS.
All imported blocks has to descent from latest finalized block or a TTD block if we did not finalize yet.
* Importing a finalized block updates the information in Besu Mutable Blockchain.
* It is no longer possible to import blocks into the blockchain below a previously finalized block when using BWS
* When a new finalized block gets announced while BWS is in progress and Besu already has it imported, the Mutable Blockchain gets updated, and the chain gets checked for possible pruning
* When trying to import blocks of equal height as newly announced finalized block, then only the new announced finalized block will be possible to import using BWS
* When importing a new block using BWS after finalized, we can now guarantee that the block descends from latest finalized block
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* BWS now has explicit parametrized check for how deep to sync
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* addressing review comments
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Addressing review comment
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Start external services before we start mining
Reasons:
- you may wish to query the blockchain using rpc before you start
processing
- you may wish to register a websocket e.g to listen for blocks
currently you will probably miss one because we start processing
before we start the ws service
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Add beforeExternalServices hook for plugins
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* move the acceptance test plugins
move the test plugins that are used in the acceptance tests into the same
package namespace as the acceptance tests so that they are ignored by
sonar for code coverage
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Fix typo in log
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
This is a re-implementation of the initial POC done in https://github.com/PegaSysEng/pantheon/pull/1909/ by Danno Ferrin <danno.ferrin@gmail.com>
* Only enable plugin rpc api when enabled on --rpc-http-api or --rpc-ws-apis
* Only allow new rpc endpoints to be defined
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
The labelled gauge functionality will allow monitoring of gauge type data without creating many gauges, which can be particularly handy in tracking related data.
Signed-off-by: Paul Harris <paul.harris@consensys.net>
Signed-off-by: Adrian Sutton <adrian.sutton@consensys.net>
EIP-1702 account versioning was a candidate for the Istanbul hard fork
but was removed prior to the first testnet. Other versioning techniques
have greater core dev mind share and one preparatory step landed in
London, EIP-3541, making EIP-1702 very unlikely to make it to Mainnet.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Refactor: PrivacyBlockProcessor to clarify intent
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Wire up privacy genesis options into PrivacyParameters
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Refactor private state genesis into it's own class
- pass through in privacyParameters
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Refactor: inject PrivateStateGenesis into PrivacyPrecompiles
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Private Genesis initialisation
- set code, balance and storage from private-genesis.json
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Check on-chain with private genesis in acceptance tests
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Remove unused EthGetCodeCall
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Use a plugin based aproach for privacy genesis
- if the plugin is registered it will be used to apply private genesis
state
- if onchain flexible privacy groups is enable that will be applied
after
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* PrivateGenesisAcceptanceTest::createPrivacyGroup can be private
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* PrivateStateGenesis add debug logs
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Warn if genesis account allocation is in reserved precompile range
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Add sender into unsignedPrivateMarkerTransaction for plugin to make descision
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Remove locking solution as may not be needed
- if this is required we should first evaluate actual use cases
and test scenarios
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Rename PrivateStateGenesis -> PrivateStateGenesisAllocator
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Tidy up naming for getPrivateStateGenesisAllocator
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Privacy Plugin javadocs
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Update spotless to 5.14.2 and java-google-format to 1.10.0.
Both of these are needed to support Java 16 and Gradle 7.x.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Refactor: clarify intent of arg when testing permissioning plugin
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Refactor: remove duplicate createPrivateMarkerTransaction
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* PrivateMarkerTransactionFactory: add option to delegate to plugin
if defined the plugin will be responsible for creating the pmt
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Implement locking strategy for eea_sendRawTransaction
- this will lock per address being sent to prevent nonce
too low errors
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Refactor plugin integration to use a more explicit poco
- the plugin will need to sign the transaction with the data it's given
- it will not need to query anything e.g nonce etc
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* refactor: update naming and tidy up based on comments
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Isolate code for calculating gas limit when using privacy plugin
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Co-authored-by: Vijay Michalik <vijay.michalik@consensys.net>
* Upgrade to Apache Tuweni 2.0
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove intermediate repository
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove all occurrences of toBytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Migrate to tuweni-bytes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* add changelog
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* correct reference tests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Initial API changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* more changes
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Change APIs for VM ops
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Use constant UInt256.ONE
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Optimize a bit address <> word transformation
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* spotless
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Hook up an empty plugin for future unrestricted privacy encryption
Skeleton implementation for PrivacyPayloadEncryptionProvider
Wire up unencrypted serialization for private transactions
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* wire up simple privacy plugin in acceptance tests
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Pass the marker transaction through to plugin
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Move getters/setters into consistent place
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Rename plugin methods and config to be more generic
- remove protections around transaction.restriction
- the plugin is responsible for this
- make plugin more generic so could be used for restricted/unrestricted
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Provide more meaningful error message
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Fix up test naming to match impl naming
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Rename UNRESTRICTED_PRIVACY to PLUGIN_PRIVACY
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix typo in PrivacyParmeters::toString
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Fix exception messages to be PrivacyPlugin
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Remove blockNumber from getPrivateTransactionFromPayload
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* allow transaction pool to accept transactions with maxFeePerGas less than base fee
* make 1559 fields return short hex format instead of 8 bytes zero padded
* update reference tests to latest
Signed-off-by: garyschulte <garyschulte@gmail.com>
* address assumption that if 1559 is enabled there is a london fork block config
fixes#2192
Signed-off-by: garyschulte <garyschulte@gmail.com>
* omit null chainId from transaction json serialization, and fix merge regression
Signed-off-by: garyschulte <garyschulte@gmail.com>
* update json-rpc responses for 1559, use current feeCap and gasPremium field names
Signed-off-by: garyschulte <garyschulte@gmail.com>
* add pr number to changelog
Signed-off-by: garyschulte <garyschulte@gmail.com>
* fix transaction decoder
Signed-off-by: Gary Schulte <garyschulte@gmail.com>
* rebase and fix merge conflicts
Signed-off-by: garyschulte <garyschulte@gmail.com>
* remove null chainId from json-rpc test fixtures
Signed-off-by: garyschulte <garyschulte@gmail.com>
* the great 1559 rename
Signed-off-by: garyschulte <garyschulte@gmail.com>
* re-remove gasTarget
Signed-off-by: garyschulte <garyschulte@gmail.com>
* add serialization tests for 1559 transaction fields
Signed-off-by: garyschulte <garyschulte@gmail.com>
* add json property ordering annotations to Transaction*Result and spotless
Signed-off-by: garyschulte <garyschulte@gmail.com>
* rebase on master, update error message feedback from Karim
Signed-off-by: garyschulte <garyschulte@gmail.com>
* Permissioning: Add plugin extension point
Added plugin extension point to allow developers to write their
own implementation of `NodePermissioningProvider::isPermitted`
This will allow developers to implement their own interpretations of
things like on-chain permissioning.
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* refactor: rename NodePermissioningProvider::isPermitted
Interface will be used for other pemissioning needs
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Permissioning: added hook for NodeMessagePermissioning
All message sent to a peer will call into isMessagePermitted
if providers have been registered through the plugin api
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* AcceptanceTests: test node nodePermissioningProvider
4 node cluster with permissioning blocking a direct between two nodes
and permissioning blocking transaction messages for a single node
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: unit tests for NodePermissioningControllerFactory
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: fat finger typo
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: reduce likely hood of flakey test
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: remove comment
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: typos
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: remove jitpack references
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: tidy up EthPeerTest args
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: update plugin hash check
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* fix: improve test reliability
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* refactor: move test-plugins out from besu/main into acceptance-tests
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Update Signature hash preimage
* Update json rpc to list fields for access list
* Update what is considered "raw" hash
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Since the EIP-1559 transition is going away, simply use a set of
accepted transactions for transaction validator that we'll check
against.
Don't assume that there are two transaction types and instead check
what type the transactions are.
Use guessType when we're deserializing from json.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* add doomed key check (busy-waiting for now)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* optional and logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove logging
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* sleeping and hardening
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename segments
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move away from atomic references to regular vars
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove hardened segment parameter
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* increase sleep
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* spotless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary interface
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* rename
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move commit waiting outside of timer
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "rename segments"
This reverts commit 184eefaaa0ccc857b0caff2b382f8338ff225d5d.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add documentation
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* bump up sleep to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* (POC) Add lock to ensure that we don't prune while comitting
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary persist (#569)
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* flesh out @mbaxter's idea and remove my code
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* iterator changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* hybridize with doomed key
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move doomed key unset to after node added listener
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* update instead of getting and setting doomedKeyRef in commit
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* comment
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* invert condition
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove locks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove `removeAllKeysUnless`
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* more remove removeAllKeysUnless
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* reuse streamKeys
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove test
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* set default lock timeout to 1ms
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add default lock timeout to tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* fix jmh compilation error
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert back to locks instead of doomedkey
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* change delete to not guarantee deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* plugin hash
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "change delete to not guarantee deletion"
This reverts commit 2289bb34cf.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* skip key deletion on timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* clear in rollback
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "fix jmh compilation error"
This reverts commit b64ecf8656.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "add default lock timeout to tests"
This reverts commit aff6aa6065.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Revert "set default lock timeout to 1ms"
This reverts commit 267fe0a642.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use noSlowDown write option instead of global timeout
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back tests
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* close tryDeleteOptions
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* remove unnecessary lock
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* move increment inside try
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* use StorageException subclass instead of field
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* revert accidental deletion in javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tryDelete javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add trace for skipping key deletion
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* merge catch and finally try blocks
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* switch from exception to boolean return value
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* tweak
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* changelog changes
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add api back
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* add back throws javadoc
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Co-authored-by: Meredith Baxter <meredith.baxter@consensys.net>
Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>
Introduced Security Module Plugin API. This allows to switch to a different security module provider to provide cryptographic function that can be used by NodeKey (such as sign, ECDHKeyAgreement etc.). By default register KeyPairSecurityModule otherwise attempt to load Security Module via plugin API.
CLI Options:
--security-module=<name>. (defaults to localfile)
Signed-off-by: Usman Saleem <usman@usmans.info>