* Fast sync should traverse the world state depth first
1. The pending requests queue in the world state downloader is now a different data structure. We now use a list of priority queues.
2. The node requests now know about the parent request that was responsible for spawning them.
3. When a node has data available and is asked to persist before its children are persisted, the node will not do anything. Instead, it will wait for all its children to persist first and will persist together with the last child.
Storing children before parents gives us the following benefits:
* We don't need to store pending requests on disk any more.
* When restarting download from a new pivot, we do not need to walk and check the whole tree any more.
And the following drawbacks:
* We now have pending nodes in memory for which we already downloaded data, but we do not store them in the database yet.
Overall expectations on performance:
We still need to download every single state node at least once, so there is no improvement there. We will save a significant amount of time in case we change pivots. And we save lots of read/writes on filesystem because tasks are not needed to be written to disk any more.
We want to avoid having too many pending unsaved nodes in memory, not to run out of it. If we were always handling only one request to our peers at the same time, we would not need to be worried, and we would just use a simple depth first search. Because we batch our requests, we might produce too many pending unprocessed nodes in memory if we are not careful about the order of processing requests. That is where the priority on node request comes from. We want to always process nodes lower in the tree before nodes higher in the tree, and preferably we want to first process children from the same parent so that we can save the current unsaved parent as soon as possible.
At the moment, I still left in the code several artefacts that I use for debugging the behaviour. I am planning to get rid of most of these counters, feel free to point them out in the review. There is for instance a weird counter in the NodeDataRequest class that I am using to monitor the total amount of unsaved nodes. In case pending unsaved node count rises too high, there is a warning printed into the logs. At the moment of writing, I would expect the counter to stay below 10 000 generally and not rise above 20 000 nodes. If you saw the number rise to for instance 100 000 that would signify a bug.
Similarly, because of the order of processing of the nodes, we do not need to store huge number of requests on the disk any more and the whole list fits comfortably into the memory. Without batching, we would not have more than a thousand requests around waiting. Because of the batching, we can see the number of requests occasionally rises all the way up to 300 000, but usually should be under 200 000.
Note that at any time there should not be more pending unsaved blocks than pending requests. Such a situation would be a bug to be reported.
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Addressing review comments
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Fixed failing test
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Improving test coverage
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Addressing review comments
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
Message sizes greater than the 16KB TLS record limit were being truncated.
Implementation based on 06f94311d8 but removed Protobufs
Note, this is an early access feature and behind a feature flag.
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
* Add missing Magneto ETC hard fork entry in test
This corresponds to 61bf0d9ca
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Change ETC bootnode public keys
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Add ETC Mystique hard fork spec (ECIP-1104)
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* 6.0.1
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Removing includeUninitialized parameter
New version of the client-java-api-6.0.1.jar does not have the includeUnitialized parameter on the method.
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* 6.0.1 compiles successfully
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Co-authored-by: Jiri Peinlich <jiri.peinlich@gmail.com>
Implemented getPositiveInt function in JsonUtil to validate a JSON positive number value. Using this function, the blockperiodseconds should now be validated whenever retrieved from the genesis config, including transitions.
Signed-off-by: George Patterson <g-patt@outlook.com>
* Stream JSON RPC responses to avoid creating big JSON string in memory for large responses
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Adapt code to last development on result with Optionals
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Log an error if there is an IOException during the streaming of the response
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Remove the intermediate String object creation, writing directly to a Buffer
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Implement response streaming for web socket
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix log messages
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Move inner classes to outer level, to avoid too big class files
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* Fix copyright
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
* lots of errorprone fixes
* some license updates
* some mockito updates
* upgrade the rocksdb version
* Prometheus left at 0.9.0 as 0.10.0+ introduces OpenMetrics
related changes that break unit tests.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* initial commit of executePayload to main
* additional coverage, change findLatestValidAncestor to take Hash rather than Block, fix comments
Signed-off-by: garyschulte <garyschulte@gmail.com>
* compile issues sorted, some tests failing
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* closing runnerBehind closes the vertx shared with runnerAhead, which now throws an exception
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* checkpoint when 4/5 websocket login tests pass
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* exp moved to attribute from principal
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* fixed more tests
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* fixed more tests
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* exception handling test improvement
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* exception handling test improvement
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* static renamed
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* We want old implementation of the host()
Newly vertex handles the forward headers and modifies host(). In the
process vert.x loses track of port from Host header in case the port was
not a string.
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* adding dependency on jackson-databind for tests
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* making sure changes are spotless
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* Dealing with regression
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* fixing last failing vert.x test hopefully
Signed-off-by: Jiri Peinlich <jiri.peinlich@gmail.com>
* removed commented out code
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* reverts debugging adjustment
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* removed commented out code
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* minor whitespace cleanup
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* (internal) Refactor 'onchain' to 'flexible' where applicable (#3075)
* CLI option name change
Signed-off-by: Frank Li <b439988l@gmail.com>
* refactor privacyparameters.java and add deprecation warning
Signed-off-by: Frank Li <b439988l@gmail.com>
* more refactoring
Signed-off-by: Frank Li <b439988l@gmail.com>
* add to everything.toml
Signed-off-by: Frank Li <b439988l@gmail.com>
* bugs
Signed-off-by: Frank Li <b439988l@gmail.com>
* more missing variable names
Signed-off-by: Frank Li <b439988l@gmail.com>
* more classes
Signed-off-by: Frank Li <b439988l@gmail.com>
* more classes
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix compile error
Signed-off-by: Frank Li <b439988l@gmail.com>
* add new test to invalidate passing both commands
Signed-off-by: Frank Li <b439988l@gmail.com>
* more refactoring + more tests
Signed-off-by: Frank Li <b439988l@gmail.com>
* new batch
Signed-off-by: Frank Li <b439988l@gmail.com>
* final batch?
Signed-off-by: Frank Li <b439988l@gmail.com>
* failing unit test
Signed-off-by: Frank Li <b439988l@gmail.com>
* revert incorrect refactoring back to onchain
Signed-off-by: Frank Li <b439988l@gmail.com>
* fix unit test
Signed-off-by: Frank Li <b439988l@gmail.com>
* comment
Signed-off-by: Frank Li <b439988l@gmail.com>
* comment
Signed-off-by: Frank Li <b439988l@gmail.com>
* support both privx methods
Signed-off-by: Frank Li <b439988l@gmail.com>
* add to changelog
Signed-off-by: Frank Li <b439988l@gmail.com>
* address comment
Signed-off-by: Frank Li <b439988l@gmail.com>
* add plugin privacy
Signed-off-by: Frank Li <b439988l@gmail.com>
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* checkpoint when 4/5 websocket login tests pass
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* trying to figure out how to decouple this test
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* spotless
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* removes Orion from integration test
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* spotless
Signed-off-by: Justin Florentine <justin+github@florentine.us>
* Add jackson dependency to merge module
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Migrate JWTAuthOptions creations for public keys
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Check http client response status
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Replace Orian with Tessera in tests
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Change Tessera expected error messages in tests
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Change executor of integrationTests to allow spanning Docker processes
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Justin Florentine <justin+github@florentine.us>
Co-authored-by: Jiri Peinlich <jiri.peinlich@gmail.com>
Co-authored-by: Frank Li <39414003+frankisawesome@users.noreply.github.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* the changelog got a bit out of sync. this fixes it
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* download links for 21.10.5
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* RC2
Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* Start external services before we start mining
Reasons:
- you may wish to query the blockchain using rpc before you start
processing
- you may wish to register a websocket e.g to listen for blocks
currently you will probably miss one because we start processing
before we start the ws service
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Add beforeExternalServices hook for plugins
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* move the acceptance test plugins
move the test plugins that are used in the acceptance tests into the same
package namespace as the acceptance tests so that they are ignored by
sonar for code coverage
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
* Fix typo in log
Signed-off-by: Antony Denyer <git@antonydenyer.co.uk>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
* log data path on error
Signed-off-by: Frank Li <b439988l@gmail.com>
* missing full stop
Signed-off-by: Frank Li <b439988l@gmail.com>
* lint
Signed-off-by: Frank Li <b439988l@gmail.com>
* address comments
Signed-off-by: Frank Li <b439988l@gmail.com>
* line length
Signed-off-by: Frank Li <b439988l@gmail.com>
* grammar
Signed-off-by: Frank Li <b439988l@gmail.com>
* grammar
Signed-off-by: Frank Li <b439988l@gmail.com>
* Move discovery options to genesis definition (#3113)
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Set default network through PicoCLI
Signed-off-by: Diego López León <dieguitoll@gmail.com>
* Change implementation to use native JDK instead of Guava
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Change the docker base images from debian (implied by openjdk image) to
ubuntu explicitly. Ubuntu has quicker reaction time to base image
security vulnerabilities.
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
* exclude ATs from sonar cloud
Signed-off-by: Frank Li <b439988l@gmail.com>
* add debug line
Signed-off-by: Frank Li <b439988l@gmail.com>
* remove sonar property (should fail)
Signed-off-by: Frank Li <b439988l@gmail.com>
* try different class
Signed-off-by: Frank Li <b439988l@gmail.com>
* actual non covered code
Signed-off-by: Frank Li <b439988l@gmail.com>
* add sonar coverage flag
Signed-off-by: Frank Li <b439988l@gmail.com>
* delete debug entries
Signed-off-by: Frank Li <b439988l@gmail.com>
* trigger bad coverage
Signed-off-by: Frank Li <b439988l@gmail.com>
* try different file
Signed-off-by: Frank Li <b439988l@gmail.com>
* remove coverage trigger
Signed-off-by: Frank Li <b439988l@gmail.com>
Co-authored-by: Sally MacFarlane <sally.macfarlane@consensys.net>
Shift operation expect the shift amount to be an unsigned int 256, even
though only 8 bits provides meaningful value. Besu optimized this by
casting this to an int, but it was signed, resulting in a bad shift
amount. To ensure that such errors don't cause issues again test the
binary data types as a potentially negative along the major int
types (8,16,32,64,128,256).
Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>
matkt
Mohit Saxena
Jiri Peinlich
Simon Dudley
Diego López León
Sally MacFarlane
taccatisid
fab-10
George Patterson
Frank Li
Danno Ferrin
Antoine Toulme
garyschulte
Fredrik Svantes
Antony Denyer
Daniel Lehrner
Adrian Sutton
Justin Florentine