505 Commits (35e4df17bd98a404c081b625b4be844ef25173ff)

Author	SHA1	Message	Date
Ayrat Badykov	1068c5f775	start cache in repo process	6 years ago
Ayrat Badykov	98461ecd04	cache max and min block number	6 years ago
Ayrat Badykov	c71d49b4f3	update transaction even if internal transaction are inserted	6 years ago
Luke Imhoff	294c9f37e4	Convert UncatalogedRewards.Processor into a BufferedTask ... * Makes it works like the rest of the fetchers for consistency. * Allow it to participate in memory monitoring Bug Fixes * Addresses are created as needed for block rewards. * Coin Balances are created for each block and address mentioned. * Coin Balances are queues in CoinBalance.Fetcher. Enhancements * Fetch beneficiary errors no longer kill the block catchup, but instead only causes them to be async fetched while the rest of the block imports	6 years ago
Luke Imhoff	00e136e20c	Calculate gas payments for all blocks in batch in SQL	6 years ago
Luke Imhoff	d7c494435f	Use alias consistently	6 years ago
Luke Imhoff	ace3abf8df	Don't generate wei^2 from Wei.mult ... Wei.mult required both parameters to be Wei.t, but that meant it was actually yielding `wei^2` units. Fix the units by requiring the first parameter to be wei and the second to be a unitless integer, which makes more sense for the `gas_price * gas_used` usage as we expect that to be measured in wei and not wei^2.	6 years ago
Ayrat Badykov	722650faab	add Indexer.ReplacedTransaction.FetcherTest	6 years ago
Ayrat Badykov	a4f2aee213	add ReplacedTRansaction fetcher to supervision tree	6 years ago
Ayrat Badykov	cf78b8cb80	move replaced transactions update to fetcher	6 years ago
Ayrat Badykov	20c7a1ba21	use or_where statement to update replaced transactions	6 years ago
Ayrat Badykov	3c1dd1d71f	remove replaced transactions fetcher	6 years ago
Ayrat Badykov	fbcf7571b1	fix CR issues	6 years ago
Ayrat Badykov	6ec2776681	fix tests	6 years ago
Luke Imhoff	1e5507adbc	Set Token holder_count to 0 for new tokens ... Fixes #1359 If there is a conflict, then the token is no new and existing holder_count is not overriden. This ensures that pre-existing tokens that haven't been data migrated are kept as `holder_count` `NULL` and migrated tokens or new tokens passing through this change leave the `holder_count` unchanged.	6 years ago
zachdaniel	c68d59a011	feat: render addresses according to eip-55 ... For more information, see: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-55.md#specification	6 years ago
Luke Imhoff	4f1254b1be	Import.Runner.CurrentTokenBalances update_token_holder_counts ... When `CurrentTokenBalance`s are directly changed 1. Determine which rows will be deleted in `deleted_address_current_token_balances` using same `where` as `on_conflict` for `address_current_token_balances` 2. Upsert rows using pre-existing `address_current_token_balances`. 3. Calculate delta of holder count using `deleted_address_current_token_balances` and `address_current_token_balances`. If the delta is non-zero, update the `tokens.holder_count`.	6 years ago
Luke Imhoff	4144373367	Import.Runner.Block.update_token_holder_counts ... When `CurrentTokenBalance`s change due to blocks being reorged in `delete_address_current_token_balances` and `derive_address_current_token_balances`, whether an `address_hash` is a holder can change, which means the token holder_count can change. `update_token_holder_counts` checks if there is a net change in holders or does not update for that token.	6 years ago
Luke Imhoff	4556ee2c01	Fix tests of transactions on non-consensus blocks ... Tests for BlockTransactionController mistakenly simulated transactions on non-consensus blocks by making the `transaction.block` `consensus` `false`, but this is unrealistic - the importing only sets the consensus block to `transaction.block` and non-consensus blocks show up as `transaction.transaction_forks` `transaction_fork.uncle`. To prevent this from happening again, `Explorer.Factory.with_block` now checks the `block` has `consensus: true`. With this setup fixed, the tests needed to be updated to show that *no* _transactions_ would be shown for non-consensus blocks (because there are only transactions forks) and whether the non-consensus blocks showed as "above tip" only had to do with the number being above the consensus number and not to do with it being non-consensus. Floki selectors are used to improve error messages when messages are unexpected.	6 years ago
Luke Imhoff	57808d6b23	Separate usage of max consensus block number and block height ... The block height should be used when calculating confirmations while the max consensus block number when a block number is needed and you want to tell if the chain is empty or only the genesis block is available. The block height is 0 if there are no blocks or if there is only the genesis block.	6 years ago
Luke Imhoff	cb7bc384e8	Calculate indexed ratio in SQL ... By calculating the indexed ratio in SQL, we only have 1 round trip to the database instead of 1 to get the max and 1 to get the min.	6 years ago
Luke Imhoff	083e5d3278	Ensure min and max block number is for consensus blocks ... Adding `WHERE consensus = true` both fixes the bug that `min_block_number` and `max_block_number` were including non-consensus blocks, but it also makes the query faster because there is an index for consensus blocks. Times from Eth Mainnet production: ``` ether=> SELECT min(b0."number") FROM "blocks" AS b0; min ----- 0 (1 row) Time: 414.218 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0; QUERY PLAN ----------------------------------------------------------------------------------------------- Finalize Aggregate (cost=223454.95..223454.96 rows=1 width=8) -> Gather (cost=223454.73..223454.95 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=222454.73..222454.74 rows=1 width=8) -> Parallel Seq Scan on blocks b0 (cost=0.00..215051.99 rows=2961099 width=8) (5 rows) Time: 1.040 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------- Result (cost=0.45..0.46 rows=1 width=8) InitPlan 1 (returns $0) -> Limit (cost=0.43..0.45 rows=1 width=8) -> Index Only Scan using one_consensus_block_at_height on blocks b0 (cost=0.43..158050.90 rows=7045520 width=8) Index Cond: (number IS NOT NULL) (5 rows) Time: 3.350 ms ether=> SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; min ----- 0 (1 row) Time: 1.059 ms ```	6 years ago
Luke Imhoff	43fce861c0	Additional tests to show #1317 is no reproducible	6 years ago
Luke Imhoff	51f181db15	Failing regression tests for #1320	6 years ago
Luke Imhoff	2ec6c85c14	Failing regression test for #1310	6 years ago
zachdaniel	1c77ef0e19	feat: render block time with up to one decimal point of accuracy	6 years ago
Luke Imhoff	5524a49725	Don't fetch internal transactions for simple token transfers ... Resolves #1303 Don't fetch internal transactions for transactions that have token transfers from the logs if the status is `:ok` and the transaction did not create a contract.	6 years ago
Luke Imhoff	a571a4edb3	Don't fetch or import internal transactions for value transfers ... Fixes #1287 For simple value transfers, the UI already doesn't show the redundant single internal transaction, but we can speed up the indexing by determine the internal transactions aren't needed when the transacion is a successful simple value transfer as determined by status being `:ok` and the input being empty (`0x`).	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	6165f6757f	Move uniquifiation of address current token balances out of runner ... For all other runners, it is the responsibility of the caller to ensure unique entries. This ensures that the least amount of data as possible is sent to `Explorer.Chain.Import.all`, which speeds up the whole import process.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Andrew Cravenho	f3ab5e4c58	Update Parity Sokol node tests	6 years ago
fvictorio	616658adbf	Show token price in USD	6 years ago
William Sanches	dfc600b1c9	Add tests	6 years ago
William Sanches	590f81004c	Add emission tiles to address transactions	6 years ago
Amanda Sposito	bf46d3ee51	Add BlockScoutWeb.RealtimeEventHandler and Explorer.Chain.Events.Publisher ... * Separate EventHandler according to broadcast_type event, so the frontend realtime does not need to ignore the catchup events.	6 years ago
Gustavo Santos Ferreira	af8b92aa0d	new queries for blocks and transactions	6 years ago
Gustavo Santos Ferreira	f139317f88	multiply two wei values	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Luke Imhoff	f67241cfc0	Use COUNT(*) for count_token_transfers_from_token_hash ... By using `COUNT(*)` instead of `COUNT(column)`, Postgres optimizes the count to use the row count recorded in the index. The ETS counter appeared necessary because in 95de1ebcb0, `COUNT(id)` was being used (95de1ebcb0/apps/explorer/lib/explorer/chain/token_transfer.ex (L138)), which seems reasonable in Ecto as you need some column to count, but in Postgres, it is more efficient to count everything with `*` as it implies "count rows". `COUNT(column)` requires Postgres check if any of the values are `NULL` and it does not optimize the query even if the column is `NOT NULL` in most cases like when the `id` was used. Testing on eth60-test database, `COUNT(*)` took 62ms while `COUNT(column)` (in this case `amount` since the table no longer has `id) took 16s demonstrating the vast speed improvement of `*` counting. ``` sql> SELECT COUNT(*) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:02] 1 row retrieved starting from 1 in 70 ms (execution: 62 ms, fetching: 8 ms) sql> SELECT COUNT(amount) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:43] 1 row retrieved starting from 1 in 16 s 862 ms (execution: 16 s 855 ms, fetching: 7 ms) ```	6 years ago
Luke Imhoff	07cb543948	Failing regression test for #1266 ... When the max block number exceeds the search range by more than 1 it causes extra ranges to be calculated and they depend on the max block number instead of the search range.	6 years ago
Luke Imhoff	786f89164b	Failing regression test for #1262	6 years ago
Luke Imhoff	2b60cba8a1	Remove unused import	6 years ago
Luke Imhoff	ef02ae0504	Fix missed pinning ... Showed up as unused variables	6 years ago
Amanda Sposito	de3520667f	Add reward value of a block in the tile and overview section	6 years ago
William Sanches	2b48235898	Remove opcodes from CodeCompiler ... Really old versions of solidity compiler, e.g. v0.1.3, have different output schemas. This data was never used by our application, so instead of having to deal with different outputs, we just don't try to get this item anymore.	6 years ago
William Sanches	c1e582e85c	Add better error messages	6 years ago
Gustavo Santos Ferreira	ebf0aa6860	module that imports validators metadata into the DataBase	6 years ago
Gustavo Santos Ferreira	be397fbd7b	retriever module that queries the contracts to generate validator metadata	6 years ago
fvictorio	cd51e7cb29	Use address_current_token_balances in address tokens tab	6 years ago