Two updates to Bonsai Tries
* Log Rolling is implemented on top of the existing Persisted head. When
Besu is at chain head and the new best head makes the current head an
orphan branch, the Bonsai TrieLogs are used to roll back to a common
block and roll forward to the needed base block. Goerli is known to
maintain sync. There are still some issues with frontier era block
receipts.
* Non-mutable reads can be done off of the persisted block. These are
accurate for all reads that were performed in the block. If a read is
not known it proceeds through a fallback series of calls to prior
layers until it hits the persisted block. These layered reads are
driven off of the TrieLogs.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Bonsai tries will require storing state by block hash rather than by
state root. To accommodate both forest mode and bonsai mode all state
queries will pass in both the block hash and state root. This also
permits parallel forest/bonsai modes for private state with bonsai
public state.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
The tests were brittle when I started to change BlockDataGenerator to
randomly generate transactions of different types. This makes sure that
we're always under the limit so we can test it more resiliently.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Remove all but 4 log4j2.xml config files
* The main config for the besu CLI app
* The config for the evmTool CLI app
* The config for acceptance tests
* A config in testUtil
If any tests depend on a log4j file they should import testutil.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* Add tracing support for internals and JSON-RPC
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove rocksdb tracing as it slows down execution too much
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add B3 headers extraction on JSON-RPC requests
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Remove traces around trie tree as they slow down syncing significantly
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing to fast sync pipeline
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add tracing for all pipelines
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Address code review
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Add acceptance tests and break out the shaded dependency
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Fix tracer id
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Revert changes to trie
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Upgrade otel to latest, remove old tracing
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
* Code review comments
Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>
Since the EIP-1559 transition is going away, simply use a set of
accepted transactions for transaction validator that we'll check
against.
Don't assume that there are two transaction types and instead check
what type the transactions are.
Use guessType when we're deserializing from json.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The change from create to DEFAULT was something that I saw was needed
to keep the reference tests performant in the upcoming EIP-2718 changes.
The change from reading the whole mainnet genesis to reading just the
config portion came from OutOfMemoryErrors in the reference tests when
running normalizeKeys.
Also moved them to ProtocolTestFixture since they're only used by tests.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Adding the Log4j "jul" (java.util.logging) adapter resulted in many
messages like this at startup:
`main INFO Registered Log4j as the java.util.logging.LogManager.`
These come from the Log4j status logger. We can get rid of those by
setting the status attribute on all configurations to a higher logging
level. WARN is the next higher level.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* #1561 - Create/store/update ENR when local node is created
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1561 - spotlessApply
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update discovery library to prod version 0.4.1
Signed-off-by: David Mechler <david.mechler@consensys.net>
This is a not-fully-functional prototype of Bonsai Tries.
Bonsai tries is a flat leaf storage, branch-by-location, and diff based reorgs
refactoring of the existing forest based trie storage mechanism aimed at
creating sustainable performance at mainnet loads.
* Since it is experimental a feature flag of --Xdata-storage-format=BONSAI
controls activation.
Some required changes have a long reach:
* To accommodate location based storage many Trie operations accept both a
location and hash value. Each data storage format is keyed off of only
one of the fields, so many tests will pass in null to the other field.
* MutableWorldStateUpdater.persist now takes an argument of a block hash.
If this is a natural progression of blocks the hash of the new block is
passed in. Otherwise null should be passed in.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
This implements the encoding/decoding logic and because it's backwards
compatible we can introduce it immediately. You can see that some of
the typed-transaction specific encoding/decoding logic is tested where
there are EIP-1559 encoding/decoding tests but there'll be more tests
included in the EIP-2930 PR.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
This PR adds a waiting list for NewPooledTransactionHashesMessage in order to group several hashes into a single GetPooledTransactionsFromPeerTask
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Add the protocols section to admin_peers. This involved plumbing
EthPeers through where we were previously using the P2PNetwork to get
our data. Hence most of the PR is DI refactoring.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Moves TransactionInvalidReason to top level so it can be accessed by
privacy code while also allowing the TransactionValidator to be
collapsed as it's only used on the permissionless side.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
The reason this existed in the first place was only because the author
wanted to impact the existing code as little as possible and so copied
analogous classes. However, this class didn't make sense in the context
of the eth65 changes because there isn't size-in-bytes limiting logic,
only an implementaion-specific max-count logic that can easily fit in
the sender class. The tests were deleted because we already have
coverage of the 4096 batch size in PendingTransactionsMessageSenderTest.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* #1066 Switched to use unprefixed hex strings for memory and stack values
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Disable flaky tests per Ben Burns(Yeti) request
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Revert last commit and enable ignored tests.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1157 - updated to create 2 agents so that proper bonding can occur
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1162 - Updated test to mock the local peer PING packet creation so that the hash can be managed.
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Added admin_logsRepairCache end point
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove p2p network code per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Updates from PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Spotless Apply fixes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Admin force cache refresh when called through end point per PR comments
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Pr updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update changelog for 1.5.1
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Remove check for 0x prefix on addresses to match expectations
Signed-off-by: David Mechler <david.mechler@consensys.net>
* Update graphql pending to allow for sorting of transactions
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Add Miner data endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 Added tests for new miner endpoints
Signed-off-by: David Mechler <david.mechler@consensys.net>
* #1408 - PR updates
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
* SpotlessApply updtes
Signed-off-by: David Mechler <david.mechler@consensys.net>
Co-authored-by: David Mechler <davemec@users.noreply.github.com>
To provide more context to admins and to add more meaning the full sync
log lines report the amount of gas processed in MegaGas/Second. Some
blocks are fuller than others and this provides a gauge as to wether the
import is slowing down or if the blocks are filling up.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Prioritize high gas prices during mining. Previously we ordered only by
the order in which the transactions were received. This will increase
expected profit when mining.
Co-authored-by: Joshua Melton <jmelton@lawlogix.com>
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
If a peer exceeds the authorized number of pending blocks, Besu will replace the lowest priority block in the cache from this peer by this new one until the local node sync a new block and maybe purges one of the blocks of this peer.
The highest priority blocks are those that are lowest in block height and then higher priority if they were sent more recently.
Other peers will not be impacted and will be able to continue sending pending blocks.
The cache size limit is the distance between the minimum and maximum value of the BlockPropagationRange parameter. Besu automatically purges blocks outside this range.
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Enable eth65 by default for the RC period. This commit should be
reverted if issues are found before the release date.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
* Prefer `EvmAccount` in interfaces and methods
* Rename `DefaultEvmAccount` to `WrappedEvmAccount`
* Move `UpdateTrackingAccount` to a top level class from an inner class
* Re-type `getTouchedAcounts()` to not use `UpdateTrackingAccount`
* Extract `WorldStateArchive` interface, rename old class
`DefaultWorldStateArchive`
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Fix some issues which caused some log to be missing when calling the eth_getLogs method
- Setting up a cache version
- Add a check integrity of the cache
- Fix a lock issue
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
We were attempting to use hashes from a queue we frequently evict from
and not handling that case in an iterator that walks over that
collection.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
This solves the same problem as 3a39fbb but better. The attempt
there has an issue where it still emulates sending the request and
therefore only 5 requests max can be aborted. Thus the problem of the
peer becoming stuck in a busy state remains. Here we do what we
should've done all along, which is proactively abort these requests
when a disconnect event occurs.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Any process that depends on pre-assigned peer tasks is liable to grow
our pendingRequests out of control.
When we attempt to execute a pending transaction, we check for that
peers availability:
PendingPeerRequest.java
```
final Optional<EthPeer> selectedPeer =
leastBusySuitablePeer.filter(EthPeer::hasAvailableRequestCapacity);
selectedPeer.ifPresent(this::sendRequest);
return selectedPeer.isPresent();
}
}
```
However, if that peer was disconnected while there were max outstanding
requsts, we'll never reattempt it as selectedPeer will be empty.
Furthermore, we'll return false and keep that pending request in our
list of requests.
With this change we indicate that we aren't waiting for any outstanding
requests when we disconnect from a peer, allowing the existing logic
that triggers when peers are disconnected to be hit.
The getter I added just for testing purposes was to get around the
inadequacies of the MockPeerConnection. There's a whole lot of callback
plumbing that's needed for a simple ethPeer.disconnect to do the same
thing it does when we call it in our production code.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
Move key referenceTest classes relating to reading the JSON data into a
"main" java package so that it can be included in other production
classes. Some classes were renamed to make their intent clearer. Other
smaller changes that bring classes in line with current coding
standards were done.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
CompletableFuture already has its own atomicity under the hood that
allows for threadsafe cancelations.
Hides the AtomicReference that gives us idempotency in executing the
task so that inheritors don't need to care about it.
Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com>
This PR fixes an error when downloading chain (Async operation failed)
When there are several blocks which are very large (> 12M) and which are requested in the same segment (by default 200) we can have timeouts and never manage to synchronize. This modification will make it possible to gradually reduce the size of the segment with each attempt. Then the segment resumes its default size for the next blocks
If the reduction is not enough at the last attempt we try with a single block
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
- Added `--rpc-tx-feecap` command line flag.
- Maximum transaction fees (in Wei) accepted for transaction submitted through RPC.
- Defaulted to 1 ether.
- Updated `TransactionPool.addLocalTransaction` method: performs an additional check to verify if transaction fees don't exceed user defined fee cap (if cap is set to 0 then it is ignored and means no cap).
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
* rename more whitelist occurrences; change allowlisted to allowed and reword where we ended up with allowlisting
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Upgrade to ErrorProne 2.4.0
* public constructors on abstract classes are removed
* Javadoc must have meaningfull documentation
* lambdas should not be variables
* Added to the list of confusing inner class names (Entry and Type)
* no assert keyword in tests
* Obsolete JDK classes produce errors now
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Before this pull request Besu was using the latest known fork id to create status message for Ethereum P2P protocol handshake.
This latest known fork id was created based on a list of forks retrieved from the Genesis.
For private networks it is possible that all fork blocks number are set to 0.
The algorithm to compute the valid fork hashes excludes 0 values.
As a result, the list was empty and the `getLatestForkId` was returning `null`. This is an issue when you support capabilities >= to Eth/64 sub protocol because other peers expect the fork id value in the `RLP` encoded message.
Moreover, the algorithm to compute the fork id should be aware of the chain head number and update `CRC` value only for fork blocks below the current head.
This pull request fixes this issue by fetching the chain head number and update accordingly the `CRC` value.
Unit tests have been extended to cover an exhaustive list of possible combinations on named networks (`goerli`, `rinkeby`, `ropsten` and `mainnet`).
Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>
ProtocolContext uses a generic for the consensus state, which has a very
large footprint across the code to accomplish what it intends to
accomplish. For every call there are about 61 other lines per call that
need to be updated, over 1300 lines total.
Instead replace it with java.lang.Class#cast, which provides runtime
security, and use generics to provide the compile time sugar that
allows for chained methods of the appropriate type. Then remove its
(quite large) footprint from the rest of the code.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
Danno Ferrin
Ratan (Rai) Sur
Antoine Toulme
David Mechler
matkt
Abdelhamid Bakhta
Sally MacFarlane
Adrian Sutton