Burn in build of 23.1.0 (#5093)

* rebase off of burn-in release, remove unreleased 23.1.0-RC2 from CHANGELOG

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

CHANGELOG.md

@@ -1,15 +1,26 @@
 # Changelog
 
-## 23.1.0-RC2
+## 23.1.0
+Besu 23.1.0 is a recommended update for Mainnet users. Thank you all for your patience as we crafted this quarterly release.
+
+This is a rather large release with some breaking changes, so please be sure to read these notes carefully before you upgrade any Besu instances. We are including a move to Java 17 LTS. To build and run Besu, please make sure you have Java 17 on the host machine. Additionally, there are a host of spec compliance changes that change existing formats, so please check the specific RPC updates. Lastly, this release formalizes a deprecation notice for GoQuorum privacy modes and IBFT1.0 in Besu. These will be removed in the 23.4 series, unless otherwise stated.
+
+From the improvements and fixes side, we have a host of execution performance improvements and fixes for defects with bonsai storage. We have also included an error detection and auto-heal capability for nodes that encounter state issues. This should keep nodes online and validating that may have previously required a resync.
+
+One final note. 23.1.0 is not a Shanghai ready release. If you intend to test Besu on the long-lived testnets like Zhejiang, please follow the instructions here. We will have more to share on our official Shanghai releases soon.
 
 ### Breaking Changes
 - Add a new CLI option to limit the number of requests in a single RPC batch request. Default=1 [#4965](https://github.com/hyperledger/besu/pull/4965)
 - Change JsonRpc http service to return the error -32602 (Invalid params) with a 200 http status code
 - Besu requires minimum Java 17 and up to build and run [#3320](https://github.com/hyperledger/besu/issues/3320)
-- PKCS11 with nss module (PKCS11 based HSM can be used in DevP2P TLS and QBFT PKI) does not work with RSA keys 
-in Java 17. SoftHSM is tested manually and working. (Other PKCS11 HSM are not tested). The relevant unit and acceptance 
+- PKCS11 with nss module (PKCS11 based HSM can be used in DevP2P TLS and QBFT PKI) does not work with RSA keys
+in Java 17. SoftHSM is tested manually and working. (Other PKCS11 HSM are not tested). The relevant unit and acceptance
 tests are updated to use EC private keys instead of RSA keys.
 - Change eth_feeHistory parameter `blockCount` to accept hexadecimal string (was accepting plain integer) [#5047](https://github.com/hyperledger/besu/pull/5047)
+- Default configurations for the deprecated Ropsten, Kiln, Shandong, and Astor networks have been removed from the CLI network list. These networks can currently be accessed but will require a user-provided genesis configuration. [#4869](https://github.com/hyperledger/besu/pull/4869)
+- GoQuorum-compatible privacy is deprecated and will be removed in 23.4
+- IBFT 1.0 is deprecated and will be removed in 23.4
+- Optimize SSTORE Operation execution time (memoize current and original value) [#4836](https://github.com/hyperledger/besu/pull/4836)
 
 ### Additions and Improvements
 - Support for new DATAHASH opcode as part of EIP-4844 [#4823](https://github.com/hyperledger/besu/issues/4823)
@@ -25,11 +36,23 @@ tests are updated to use EC private keys instead of RSA keys.
 - Besu requires minimum Java 17 and up to build and run [#3320](https://github.com/hyperledger/besu/issues/3320)
 - Add worldstate auto-heal mechanism [#5059](https://github.com/hyperledger/besu/pull/5059)
 - Support for EIP-4895 - Withdrawals for Shanghai fork
+- Improve SLOAD and SSTORE performance by caching empty slots [#4874](https://github.com/hyperledger/besu/pull/4874)
+- RPC methods that lookup block by hash will now return an error response if no block found [#4582](https://github.com/hyperledger/besu/pull/4582)
+- Added support for `safe` and `finalized` strings for the RPC methods using defaultBlock parameter [#4902](https://github.com/hyperledger/besu/pull/4902)
+- Added post-execution state logging option to EVM Tool [#4709](https://github.com/hyperledger/besu/pull/4709)
+- Add access list to Transaction Call Object [#4802](https://github.com/hyperledger/besu/issues/4801)
+- Add timestamp fork support, including shanghaiTime and cancunTime forks [#4743](https://github.com/hyperledger/besu/pull/4743)
+- Optimization:  Memoize transaction size and hash at the same time [#4812](https://github.com/hyperledger/besu/pull/4812)
+- Add chain data pruning feature with three experimental CLI options: `--Xchain-pruning-enabled`, `--Xchain-pruning-blocks-retained` and `--Xchain-pruning-frequency` [#4686](https://github.com
+/hyperledger/besu/pull/4686)
+  - Note that chain pruning is hidden and disabled by default. Once you choose to enable chain pruning, a new column family will be added to the db and you cannot roll back to a previous versi
+on of Besu.
 
 ### Bug Fixes
 - Mitigation fix for stale bonsai code storage leading to log rolling issues on contract recreates [#4906](https://github.com/hyperledger/besu/pull/4906)
-- Ensure latest cached layered worldstate is subscribed to storage, fix problem with RPC calls using 'latest' [#5039](https://github.com/hyperledger/besu/pull/5039)  
-
+- Ensure latest cached layered worldstate is subscribed to storage, fix problem with RPC calls using 'latest' [#5076](https://github.com/hyperledger/besu/pull/5076)
+- Fix for segmentation faults on worldstate truncation, snap-sync starts [#4786](https://github.com/hyperledger/besu/pull/4786)
+- Fix for worldstate mismatch on failed forkchoiceUpdate [#4862](https://github.com/hyperledger/besu/pull/4862)
 
 ## 23.1.0-RC1
 

ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/query/BlockchainQueries.java

@@ -855,12 +855,19 @@ public class BlockchainQueries {
         .getBlockHeader(blockHash)
         .flatMap(
             blockHeader -> {
-              try (var ws =
+              try (final var worldState =
                   worldStateArchive
                       .getMutable(blockHeader.getStateRoot(), blockHeader.getHash(), false)
+                      .map(
+                          ws -> {
+                            if (!ws.isPersistable()) {
+                              return ws.copy();
+                            }
+                            return ws;
+                          })
                       .orElse(null)) {
-                if (ws != null) {
-                  return Optional.ofNullable(mapper.apply(ws));
+                if (worldState != null) {
+                  return Optional.ofNullable(mapper.apply(worldState));
                 }
               } catch (Exception ex) {
                 LOG.error("failed worldstate query for " + blockHash.toShortHexString(), ex);

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/AbstractTrieLogManager.java

@@ -97,16 +97,17 @@ public abstract class AbstractTrieLogManager<T extends MutableWorldState>
 
   @VisibleForTesting
   TrieLogLayer prepareTrieLog(
-      final BlockHeader blockHeader,
-      final Hash worldStateRootHash,
+      final BlockHeader forBlockHeader,
+      final Hash forWorldStateRootHash,
       final BonsaiWorldStateUpdater localUpdater,
       final BonsaiWorldStateArchive worldStateArchive,
       final BonsaiPersistedWorldState forWorldState) {
-    debugLambda(LOG, "Adding layered world state for {}", blockHeader::toLogString);
-    final TrieLogLayer trieLog = localUpdater.generateTrieLog(blockHeader.getBlockHash());
+    debugLambda(LOG, "Adding layered world state for {}", forBlockHeader::toLogString);
+    final TrieLogLayer trieLog = localUpdater.generateTrieLog(forBlockHeader.getBlockHash());
     trieLog.freeze();
-    addCachedLayer(blockHeader, worldStateRootHash, trieLog, worldStateArchive, forWorldState);
-    scrubCachedLayers(blockHeader.getNumber());
+    addCachedLayer(
+        forBlockHeader, forWorldStateRootHash, trieLog, worldStateArchive, forWorldState);
+    scrubCachedLayers(forBlockHeader.getNumber());
     return trieLog;
   }
 

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiInMemoryWorldState.java

@@ -167,12 +167,7 @@ public class BonsaiInMemoryWorldState extends BonsaiPersistedWorldState
     final Hash newWorldStateRootHash = rootHash(localUpdater);
     archive
         .getTrieLogManager()
-        .saveTrieLog(
-            archive,
-            localUpdater,
-            newWorldStateRootHash,
-            blockHeader,
-            (BonsaiPersistedWorldState) this.copy());
+        .saveTrieLog(archive, localUpdater, newWorldStateRootHash, blockHeader, this);
     worldStateRootHash = newWorldStateRootHash;
     worldStateBlockHash = blockHeader.getBlockHash();
     isPersisted = true;

ethereum/core/src/main/java/org/hyperledger/besu/ethereum/bonsai/BonsaiLayeredWorldState.java

@@ -40,8 +40,13 @@ import org.apache.tuweni.bytes.Bytes32;
 import org.apache.tuweni.units.bigints.UInt256;
 
 /** A World State backed first by trie log layer and then by another world state. */
-public class BonsaiLayeredWorldState implements MutableWorldState, BonsaiWorldView, WorldState {
-  private Optional<BonsaiWorldView> nextWorldView;
+public class BonsaiLayeredWorldState
+    implements MutableWorldState,
+        BonsaiWorldView,
+        WorldState,
+        BonsaiWorldStateKeyValueStorage.BonsaiStorageSubscriber {
+  private Optional<BonsaiWorldView> nextWorldView = Optional.empty();
+  private Optional<Long> newtWorldViewSubscribeId = Optional.empty();
   protected final long height;
   protected final TrieLogLayer trieLog;
   private final Hash worldStateRootHash;
@@ -58,7 +63,7 @@ public class BonsaiLayeredWorldState implements MutableWorldState, BonsaiWorldVi
       final TrieLogLayer trieLog) {
     this.blockchain = blockchain;
     this.archive = archive;
-    this.nextWorldView = nextWorldView;
+    this.setNextWorldView(nextWorldView);
     this.height = height;
     this.worldStateRootHash = worldStateRootHash;
     this.trieLog = trieLog;
@@ -76,21 +81,28 @@ public class BonsaiLayeredWorldState implements MutableWorldState, BonsaiWorldVi
   }
 
   public void setNextWorldView(final Optional<BonsaiWorldView> nextWorldView) {
-    maybeUnSubscribe();
+    maybeUnSubscribe(); // unsubscribe the old view
     this.nextWorldView = nextWorldView;
+    maybeSubscribe(); // subscribe the next view
+  }
+
+  private void maybeSubscribe() {
+    nextWorldView
+        .filter(BonsaiPersistedWorldState.class::isInstance)
+        .map(BonsaiPersistedWorldState.class::cast)
+        .ifPresent(
+            worldState -> {
+              newtWorldViewSubscribeId = Optional.of(worldState.worldStateStorage.subscribe(this));
+            });
   }
 
   private void maybeUnSubscribe() {
     nextWorldView
-        .filter(WorldState.class::isInstance)
-        .map(WorldState.class::cast)
+        .filter(BonsaiPersistedWorldState.class::isInstance)
+        .map(BonsaiPersistedWorldState.class::cast)
         .ifPresent(
-            ws -> {
-              try {
-                ws.close();
-              } catch (final Exception e) {
-                // no-op
-              }
+            worldState -> {
+              newtWorldViewSubscribeId.ifPresent(worldState.worldStateStorage::unSubscribe);
             });
   }
 

gradle.properties

@@ -1,4 +1,4 @@
-version=23.1.0-SNAPSHOT
+version=23.1.0
 
 org.gradle.welcome=never
 # Set exports/opens flags required by Google Java Format and ErrorProne plugins. (JEP-396)