blockscout

Commit Graph

Author	SHA1	Message	Date
nikitosing	da6c71d911	Fix zero balances coming via WS (#9510 )	9 months ago
Alexander Kolotov	51d82f1dbf	zksync chain type support (#9631 ) * zkSync customizations * Insert placeholders instead of deriving current token balances * ZkSync Batches status tracking (#9080) * initial version of batch tracking * missed file added * attempt to add DB migration * Finalized L1 txs tracking * keep batches in DB * Batches statuses tracker introduction * rpc endponts to get batches data * extended views for blocks and transactions * Refactoring of fetchers * Fetch historical blocks * handle_info calls simplified * Ability to recover missed blocks * zksync info in a separate sub-map * added doc comments, part 1 * finalized doc comments * actual bathes count instead of the last imported batch * fix formatting * credo fixes * Address dialyzer warnings * Fix spelling * remaining issues with spelling and dialyzer * Attempt to address BlockScout Web Tests issue * review comments addressed, part 1 * review comments addressed, part 2 * collection all_options for import module reworked to get rid of dialyzer findings * removed unnecessary functionality * proper import * Credo fixes * Add CHAIN_TYPE=zksync to image generation workflow * Proper handling of empty transactions list in etc_getBlockByNumber * Merge master * Address merge issues * Fix format * Refactoring of chain type specific code for block and transaction views * Consistent name for functions * add exceptions for Credo.Check.Design.AliasUsage * Fix rebasing conflicts * Fix rebase conflicts * fix issue with stability fees in tx view * make Stability related tests dependent on chain type in compile time * move zksync related migration * Changelog updated * removal of duplicated migration * List r,s,v as optional attributes for transaction --------- Co-authored-by: Viktor Baranov <baranov.viktor.27@gmail.com> Co-authored-by: Qwerty5Uiop <alex000010@bk.ru>	9 months ago
nikitosing	546b732ac1	Add stability validators (#9390 ) * Add stability validators * Process review comments * Fix tests	9 months ago
Victor Baranov	9819522ea1	Optimism chain type (#9460 ) * Transaction page L1 fields * Path fix * Reduce the number of files from 19 to 5 in logs rotate config * Customize optimism-goerli deployment * Optimism branding * Remove testnet logo text. OG uses customized label * Fix Circles theme * L1 tx fields fix for Optimism BedRock update * Remove redundant line * Add gas_price handling for elixir_to_params and change function ordering * Remove l1TxOrigin handling for another version of RPC * Add GA * Fix realtime fetcher test * Update Changelog * Fix internal transactions processing for non-consensus blocks * Lose consensus only for consensus=true blocks * Fix handling transaction RPC responses without some fields * Fix tests except for indexer module * Add Optimism BedRock support (Txn Batches, Output Roots, Deposits, Withdrawals) (#6980) * Add op_output_roots table * Add OptimismOutputRoots runner * Add initial code for output roots fetcher * Add checks to init function * Partially add logs and L1 reorgs handling * Add reorgs handling * Add RPC retries * Write output roots to database * Log output roots handling * Update indexer README * Add API v2 for Optimism Output Roots * Add op_withdrawals table * Add OptimismWithdrawals runner * Prepare realtime optimism withdrawals fetching * Add realtime optimism withdrawals fetching * Define checks in init function * log.first_topic can be nil * Show total count of output roots in API v2 * Add msg_nonce gaps filler * Refactoring * Intermediate refactoring * Add historical withdrawals handling and refactor * Finish op_withdrawals table filling * Small refactoring * Add op_withdrawal_events table * Add OptimismWithdrawalEvents runner * Add OptimismWithdrawalEvent fetcher * Update indexer README * Add API v2 for Optimism Withdrawals * Add env variables to common-blockscout.env and Makefile * Set `from` as address object instead of just address hash for withdrawal * mix format * Add op_transaction_batches table * Add OptimismTxnBatches runner * Add a draft for OptimismTxnBatch fetcher * Add a draft for OptimismTxnBatch * Extend a draft for OptimismTxnBatch * Extend OptimismTxnBatch * Finish OptimismTxnBatch (without reorgs handling yet) * Optimize OptimismTxnBatch fetcher * Remove duplicated txn batches * Add zlib_inflate_handler for empty case * Add reorgs handling for txn batches * Fix reorgs handling for txn batches * Small refactor * Finish Indexer.Fetcher.OptimismTxnBatch (without refactoring yet) * Apply new ex_rlp version * Add API v2 for Optimism Txn Batches * Add env variables to common-blockscout.env and Makefile * Refactor OptimismTxnBatch fetcher for mix credo * Replace binary_slice with binary_part function to run with Elixir 1.13 * Update changelog * Update indexer readme * Rename op_withdrawals.l2_tx_hash field to l2_transaction_hash * Rename l1_tx_hash fields to l1_transaction_hash * Rename tx fields to transaction fields * Rename env variables * Rename env variables * Add an indexer helper * Add an indexer helper * Small refactoring * Fix tx_count for txn batches view * Use EthereumJSONRPC.Block.ByHash instead of the raw call * Infinity timeout for blocks query * Small refactoring * Refactor init function for two modules * Small refactoring * Rename l1_transaction_timestamp field to l1_timestamp * Rename withdrawal_hash field to hash * Refactor for decode_data function * Refactor for mix credo * Add INDEXER_OPTIMISM_L1_BATCH_BLOCKS_CHUNK_SIZE env and small refactoring * Add INDEXER_OPTIMISM_L1_BATCH_BLOCKS_CHUNK_SIZE env to other files * Add an index for l1_block_number field * Add an index for l1_block_number field * Remove redundant :ok * Use faster way to count rows in a table * Refactor reorgs monitor functions * Clarify frame structure * Reduce storage consumption for optimism transaction batches * Reuse CacheHelper.estimated_count_from function * Bedrock optimism deposits (#6993) * Create `op_deposits` table * Add OptimismDeposit runner * WIP Fetcher * Finish fetcher * Integrate deposits into APIv2 * Add envs * Fix requests * Remove debug * Update envs names * Rename `tx` -> `transaction` * Reuse `decode_data/2` * Fix review * Add `uninstall_filter` * Fix formatting * Switch to realtime mode more carefully * Fix review Allow nil in timestamp Add progress logging Improve check_interval calculation * Fix logging and env * Fix Association.NotLoaded error * Replace switching to realtime mode log * Remove excess start_block * Fix reorg logging * Fix `from_block` > `to_block` and add realtime logging * Fix block boundaries --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * mix format * Return total count of L2 entity by a separate API request * Filter by consensus blocks * Parallelize tx count operation and small refactoring * Use read replica for L2 entities in API * Parse block_number and tx_hash for Optimism Deposits module * Return page_size back to 50 * Small fixes and refactoring * Update apps/block_scout_web/lib/block_scout_web/api_router.ex Co-authored-by: Maxim Filonov <53992153+sl1depengwyn@users.noreply.github.com> * Small optimization * Use ecto association instead of explicit join for txn batches * Refactoring * Use Stream inspead of Enum * Small refactoring * Add assoc for transaction batches in OptimismFrameSequence * Use common reorg monitor for Optimism modules * Rename Explorer.Helpers to Explorer.Helper * Don't start an optimism module unless the main optimism module is not started * Don't start reorg monitor for optimism modules when it is not needed * Small refactoring * Remove debug broadcasting * Add Optimism BedRock Deposits to the main page in API (#7200) * Add Optimism BedRock Deposits to the main page in API * Update changelog * Pass the number of deposits instead of only one item per once --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Refactor for credo * Output L1 fields in API v2 for transaction page * Update changelog * Use helper * Refactor Indexer.Fetcher.Optimism * Fix l1_timestamp issue in OptimismTxnBatch fetcher * Reset Logger metadata before Indexer.Transform.OptimismWithdrawals.parse function finishes * Fix IDs ordering in remove_duplicates function of Indexer.Fetcher.OptimismTxnBatch * Consider rewriting of the first frame in Indexer.Fetcher.OptimismTxnBatch * Fix Indexer.Fetcher.OptimismTxnBatch (consider chunking) * Fix Indexer.Fetcher.OptimismTxnBatch * Fix handling invalid frame sequences in Indexer.Fetcher.OptimismTxnBatch * Read Optimism finalization period from a smart contract * Fixes for dialyzer * Fix for EthereumJSONRPC tests * Fixes for Explorer tests * Fixes for Explorer tests * Fix of block/realtime/fetcher_test.exs * mix format and small fixes for block_scout_web tests * Reset GA cache * Fix handling nil in PendingBlockOperation.estimated_count() --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> Co-authored-by: Maxim Filonov <53992153+sl1depengwyn@users.noreply.github.com> * Fix autocomplete * Fix merging conflicts * Add exit handler to Indexer.Fetcher.OptimismWithdrawal * Fix transactions ordering in Indexer.Fetcher.OptimismTxnBatch * Update changelog * Refactor to fix credo * Mix credo fix * Fix transaction batches module for L2 OP stack (#7827) * Fix mixed transactions handling in Indexer.Fetcher.OptimismTxnBatch * Ignore duplicated frame * Update changelog * Add sorting to the future frames list * Change list order --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Remove unused aliases * Ignore previously handled frame by OP transaction batches module (#8122) * Ignore duplicated frame * Update changelog --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Return alias for Explorer.Chain.Cache.Helper in chain.ex * Ignore invalid frame by OP transaction batches module (#8208) * Update changelog * Ignore invalid frame * Update changelog --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Fix Indexer.Fetcher.OptimismTxnBatch * Fix API v2 for OP Withdrawals * Refactor optimism fetchers init * Add log for switching from fallback url * Fix for Indexer.Fetcher.OptimismTxnBatch * Add OP withdrawal status to transaction page in API (#8702) * Add OP withdrawal status to transaction page in API * Update changelog * Small refactoring * Update .dialyzer-ignore --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Add start pause to `Indexer.Fetcher.OptimismTxnBatch` * Small refactor of `Indexer.Fetcher.OptimismTxnBatch` * Consider consensus block only when retrieving OP withdrawal transaction status (#8811) * Consider consensus block only when retrieving OP withdrawal transaction status * Update changelog * Clear GA cache --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> * Hotfix for optimism_withdrawal_transaction_status function * Return all OP Withdrawals bound to L2 transaction * Try to import config * Remove unused functions from Explorer.Chain * Refactor for mix credo * Fix order of proxy standards: 1167, 1967 * Fixes in Optimism due to changed log topics type * Fix for EthereumJSONRPC tests * Clear GA cache and update cspell.json * Fix indexer tests * Return current exchange rate in api/v2/stats * Fix log decoding bug * Temp disable build of image for arm64 * Rewrite Indexer.Fetcher.OptimismTxnBatch module * Add handling of span batches * Add support of latest block for Optimism modules * Update changelog and spelling * Rewrite Indexer.Fetcher.OptimismTxnBatch module * Add handling of span batches * Add support of latest block for Optimism modules * Refactoring * Partially add specs and docs for public functions * Refactoring * add an entry to CHANEGELOG.md * apply review (use origin entity instead of joined entity in with tx status) * Fixes after rebase * Remove old UI sustomizations * Optimism chain type * Change structure of folders * Fixes after review * Fix CHANGELOG * Fixes after 2nd review * Process 3d review: add tests for fee/2 function * Process 4th review * Review fix: move Op related functions from chain.ex * Review fix: make OptimismFinalizationPeriod configurable * Process review comment * System.get_env("CHAIN_TYPE") => Application.get_env(:explorer, :chain_type) --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> Co-authored-by: Qwerty5Uiop <alex000010@bk.ru> Co-authored-by: varasev <33550681+varasev@users.noreply.github.com> Co-authored-by: Maxim Filonov <53992153+sl1depengwyn@users.noreply.github.com> Co-authored-by: rlgns98kr <rlgns98kr@gmail.com>	9 months ago
Qwerty5Uiop	42425edef8	Null round handling (#9403 ) * Null round handling * Add repo for filecoin chain type * Add repo for filecoin chain type * Modify gas price constraint for Filecoin as it for PolygonEdge * Fix null round heights db type * Add filecoin to chain-type matrix --------- Co-authored-by: Viktor Baranov <baranov.viktor.27@gmail.com>	9 months ago
POA	b6859d3547	Define chain type	10 months ago
Kirill Fedoseev	f857b6b39b	fix: too many connections in tests	10 months ago
nikitosing	a47d83f0be	Add bridged tokens functionality, could be enabled by compile time en… (#9169 ) * Add bridged tokens functionality, could be enabled by compile time env var * Process reviewer's comment * Fix credo * Process review comments * Reset GA cache * Fix warning	10 months ago
varasev	c7724f51df	Shibarium Bridge indexer and API v2 extension (#8929 ) * Define shibarium_bridge table * Add ShibariumBridgeOperations runner * Add init for Indexer.Fetcher.Shibarium.L1 * Draft for Indexer.Fetcher.Shibarium.L1 * mix format for Indexer.Fetcher.Shibarium.L1 * Fix Indexer.Fetcher.Shibarium.L1 * Refactor Indexer.Fetcher.Shibarium.L1 * Add draft (incomplete) for Indexer.Fetcher.Shibarium.L2 * Extend draft (incomplete) for Indexer.Fetcher.Shibarium.L2 * Extend draft (incomplete) for Indexer.Fetcher.Shibarium.L2 * Complete unrefactored Indexer.Fetcher.Shibarium.L2 * Improve Indexer.Fetcher.Shibarium.L2 * Prepare Indexer.Fetcher.Shibarium.L2 for transformer * Add unrefactored Indexer.Transform.Shibarium.Bridge * Small refactoring of Shibarium modules * Refactoring * Add API v2 for Shibarium Deposits and Withdrawals * Small fixes * Update changelog * Fixes and refactoring * Partially add specs and docs * Partially add specs and docs * Log topic type changed to bytea * Small improvements * Cache deposits and withdrawals counters for Shibarium * Fixes for credo * Fixes for Chain.import related to CHAIN_TYPE * Reset GA cache * Small refactoring * Update mix.exs * Remove unnecessary credo ignore --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> Co-authored-by: Viktor Baranov <baranov.viktor.27@gmail.com>	10 months ago
Kirill Fedoseev	e91b2c7ac0	fix: fmt and test config	10 months ago
Kirill Fedoseev	37e968f76c	fix: tests	10 months ago
Viktor Baranov	e6b39be251	Manage proxy implementation ttl via avg block time	11 months ago
Nikita Pozdniakov	5615d01e46	Add new events to addresses channel: eth_bytecode_db_lookup_started and smart_contract_was_not_verified	1 year ago
nikitosing	ca1bd8314d	API v2: address nft (#8634 ) * Add /api/v2/addresses/{address_hash}/nft and /api/v2/{address_hash}/nft/collections * Add filter to /api/v2/tokens/{address_hash}/instances?holder_address_hash={holder_address} * Changelog * Revert log disabling * Fix warn * Add @spec and @doc * Process review comments	1 year ago
nikitosing	f9d5377e64	Add endpoint url to the block_scout_web logging (#8609 ) * Replace :console logger backend with LoggerJSON; Add endpoint url to the block_scout_web logging * Move logger_json dependency * Turn json logs only for prod environment * Disable debug ecto logs for test environment * Return HTTP method to logs	1 year ago
varasev	e9522deb1e	Suave functional (#8556 ) * Add execution_node field indexing * Add `wrapped` field (and its subfields) indexing * Add allowed_peekers field for transaction to API v2 response * Add execution_node field for transaction to API v2 response * Add Suave fields for regular transaction to API v2 response * Add API v2 endpoint for transactions filtered by execution node * Add index for execution_node field * Small fixes * Update changelog * Fix for CHAIN_TYPE, backward compatibility * Update spelling * Fix for dyalizer * Fix default_on_conflict * Fix test.exs * Fix view for Suave transaction * Add Suave-specific comments * mix format * Set pool_size to 1 for suave * Small refactoring for repo * mix format, mix dialyzer --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com>	1 year ago
POA	2836ccf017	Small refactoring	1 year ago
POA	5ea5da2ba4	Define chain type	1 year ago
Maxim Filonov	80c99447dd	Add RSK chain type	1 year ago
varasev	3235ab1caa	Deposits and Withdrawals for Polygon Edge (#8180 ) * Prepare tables for Polygon Supernet Withdrawals module * Prepare Indexer.Fetcher.PolygonSupernetWithdrawal * Prepare Indexer.Fetcher.PolygonSupernetWithdrawalExit * Prepare tables for Polygon Supernet Deposits module * Partially add Polygon Supernet Deposits module * Add Polygon Supernet Deposits module * Extend transaction data in API v2 for Polygon Supernet Deposits and Withdrawals * Extend API v2 for list of Polygon Supernet Deposits * Extend API v2 for list of Polygon Supernet Withdrawals * Modify collated_gas_price constraint to support Immutable test chain * Update number of retries * Fix Indexer.Fetcher.PolygonSupernetWithdrawal * Fix API v2 for Indexer.Fetcher.PolygonSupernetWithdrawal * Return page_size to 50 * Fix API v2 for Indexer.Fetcher.PolygonSupernetDeposit * Fix Explorer.Validator.MetadataRetriever to prevent sending request to nil address * Refactor PolygonSupernet* modules * Refactor PolygonSupernet* modules * Small refactoring of PolygonSupernet* modules * Update changelog * Fixes for dialyzer and cspell * Fix of Explorer tests * Add env vars for docker * Add env variable for eth_getLogs * Share event signatures between modules * Refactoring for fill_block_range function * Small refactoring of transaction actions module * Move polygon_supernet_* modules to a separate folder * Add specs for public functions * Remove redundant function * Reuse decode_data function * Move polygon_supernet_* files into a separate folder * Move polygon_supernet_* files into a separate folder * Partially disband Explorer.Chain module * Define chain type * Remove infinite waiting for safe block * max format * Rename Polygon Supernet to Polygon Edge * Set INDEXER_INTERNAL_TRANSACTIONS_TRACER_TYPE to polygon_edge if CHAIN_TYPE is polygon_edge * Fix chain type definition * Fix repos list definition in releases_tasks * INDEXER_POLYGON_SUPERNET_ -> INDEXER_POLYGON_EDGE_ --------- Co-authored-by: POA <33550681+poa@users.noreply.github.com> Co-authored-by: Viktor Baranov <baranov.viktor.27@gmail.com>	1 year ago
Nikita Pozdniakov	40d71d88c7	Forward API v2 to read replica	2 years ago
nikitosing	1a232e11f4	Add /smart-contracts and /state-changes endpoints (#6973 ) * Add /smart-contracts endpoint to API v2 * Add transactions/{hash}/state-changes endpoint; State changes code refactoring * Add tests * Disable CheckBytecodeMatchingOnDemand fetcher in test environment * CHANGELOG.md	2 years ago
nikitosing	708a6b0036	Improve working with contracts implementations (#5561 ) * Fix error on invalid response from node while fetching implementation address; Improve storing the implementation name * Add time out on implementation fetching and storing implementation info in DB * Add tests for contracts implementation fetching; Fixes for SmartContract module * Fixes after rebase * Refactor	2 years ago
Victor Baranov	3cac89f15c	API v2 for frontend (#6379 ) * Core BlockScout API V2 (#6164) API endpoints: /transactions /transactions/{tx_hash} /transactions/{tx_hash}/token-transfers /transactions/{tx_hash}/internal-transactions /transactions/{tx_hash}/logs /transactions/{tx_hash}/raw-trace * Block initial commit * Finish Block API * Address API initial commit * Fix some issue with API * Add tags * transaction -> coin_transfer * Some changes * Add /transactions and /token-transfers for addresses * Fix test * Fix tests * Fix block rewards * Add /json-rpc-url API endpoint * Create method_id index concurrently * Fix for concurrent index creation * BS core API V2: addresses, stats, main page, websockets (#6361) * socket/v2 * Refactor: reuse fees counter fucntion, remove unecessary clause of do_token_transfer_amount_for_api * Improve token transfers preload * Done with channels * Add some endpoints to /address * Fix credo * Add main page controller * Add stats controller * Move api search to API v2 * Fix some addresses methods; Rename gas_price * Improve logs view * Fix tests * Brush and finalize websockets * Add API_V2_ENABLED env * Fix credo * Add is_smart_contract clause * Add CHANGELOG entry Co-authored-by: nikitosing <32202610+nikitosing@users.noreply.github.com> Co-authored-by: Никита Поздняков <nikitosing4@mail.ru>	2 years ago
nikitosing	d7eef32074	Migrate account data to separate DB (#67 ) * Config changes * Separate migrations * Remain to fix preloads * Some changes * All is working, need to fix tests migrations * mix tasks * Fix tests and tune config * Add migration script to migrate account data * Fix after rebase * Add branch to config.yml * Add fix for broken sequence id counter * Add drop dblink * Add envs to makefile and common-bs.env * Fix credo	2 years ago
Qwerty5Uiop	e45d3a9676	Fix runtime test conf	2 years ago
Qwerty5Uiop	e269f90c95	Refactor runtime config	2 years ago
Viktor Baranov	969c222d43	Remove Staking dapp logic from Blockscout core	2 years ago
Никита Поздняков	c97afb501d	Disable by default fetching btc_value (add env variable)	2 years ago
Viktor Baranov	bf3a7869e4	Manage indexer memory limit	3 years ago
nikitosing	7088dca160	Fix vulnerability	3 years ago
Viktor Baranov	29aa23e213	An ability to separate Ecto Repo DB endpoint for the API endpoints	3 years ago
Victor Baranov	cc348ef824	Price for bridged tokens	4 years ago
Paul Tsupikoff	6a5916764d	Consolidate POSDAO contract reading (#2371 ) Previously, staking contract reading was arbitrarily spread across Explorer.Staking.ContractState, Explorer.Staking.PoolsReader and Indexer.Fetcher.StakingPools. Using async fetcher infrastructure for staking contracts is inadequate, as blocks may arrive out of order, while we want fetching to be triggered by newer block arrival. Also, contract calls were not batched enough, and it was hard to follow their sequence across involved modules. Now, on each incoming block, which is newer than the last seen, we fully update not only global contract state, but all validators, pools and delegators. All requests are intelligently compiled into four batches. These are defined together to give better overview of performed actions. Chain.import infrastructure is still used for inserting data into DB to leverage smart batching of INSERT queries. Likelihood calculation is added: it zips results of getPoolsToBeElected and getPoolsLikelihood calls to make a map of addresses. Only POS_STAKING_CONTRACT env variable is required now. Other contract addresses are fetched from it during launch. ABIs are concatenated together during launch as well, as we don't support batch requests against distinct ABIs yet. This should be addressed in further PRs. Up-to-date ABIs copied from build artifacts of posdao-contracts repo. File names now correspond to contract names, README directs to their origin. * Fetch inactive delegators, mark pools and delegators deleted (#2205) * Fetch min stakes and token contract address from StakingAuRa (#2313) * Fetch block reward ratio using validatorRewardPercent getter (#2424) * Add missing `banned_until` field in ON CONFLICT clause for staking pools.	4 years ago
Victor Baranov	cce0ff7aca	Add CHANGELOG entry	5 years ago
Victor Baranov	f9e853de97	Eliminate test secret too	5 years ago
Victor Baranov	5539ad2a77	Revert refactor to generalized cache solution	5 years ago
Victor Baranov	1dda3e50c7	Estimated address count on the main page	5 years ago
Ayrat Badykov	ad68c42fe0	add simple sender	5 years ago
Victor Baranov	455d6d647f	Addresses counter	5 years ago
saneery	ea0d47cfa4	Add mock to send events directly to listener	5 years ago
slightlycyborg	6af33827d8	fixed formatting	5 years ago
slightlycyborg	49dde3e239	Enbl historians in config.exs, disbl in test.exs; add to Application.ex	5 years ago
pasqu4le	da775b07b7	Group Explorer caches Problem: a bunch of different caches are under the Explorer.Chain namespaces, cluttering the codebase Solution: create a subdirectory and collect them all under the Explorer.Chain.Cache namespace	5 years ago
maxgrapps	5892fa6de6	content issues, test runned	6 years ago
Ayrat Badykov	390bb24d64	fix tests	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT() of current_token_balances to eliminate TokenHoldersCounter `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT()` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT() FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT()` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
fvictorio	616658adbf	Show token price in USD	6 years ago
Luke Imhoff	903d322553	Use index and COUNT() for address_to_validation_count BlockValidationCounter is not needed once the query is switched to `COUNT()` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT()` and `COUNT(hash)` (the old usage before `76fc8e0377` in `92f99cac22`) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT() FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT()` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT() FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT()` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT()` takes about 20ms plus latency.	6 years ago

1 2

77 Commits (18ef2c0ef5c3b6a405f3364dda76a0991f038422)