726 Commits (37121c1d250d7dd364e2ec906214b5b6cde97f24)

Author	SHA1	Message	Date
Amanda Sposito	bf46d3ee51	Add BlockScoutWeb.RealtimeEventHandler and Explorer.Chain.Events.Publisher ... * Separate EventHandler according to broadcast_type event, so the frontend realtime does not need to ignore the catchup events.	6 years ago
Gustavo Santos Ferreira	af8b92aa0d	new queries for blocks and transactions	6 years ago
Gustavo Santos Ferreira	f139317f88	multiply two wei values	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Luke Imhoff	f67241cfc0	Use COUNT(*) for count_token_transfers_from_token_hash ... By using `COUNT(*)` instead of `COUNT(column)`, Postgres optimizes the count to use the row count recorded in the index. The ETS counter appeared necessary because in 95de1ebcb0, `COUNT(id)` was being used (95de1ebcb0/apps/explorer/lib/explorer/chain/token_transfer.ex (L138)), which seems reasonable in Ecto as you need some column to count, but in Postgres, it is more efficient to count everything with `*` as it implies "count rows". `COUNT(column)` requires Postgres check if any of the values are `NULL` and it does not optimize the query even if the column is `NOT NULL` in most cases like when the `id` was used. Testing on eth60-test database, `COUNT(*)` took 62ms while `COUNT(column)` (in this case `amount` since the table no longer has `id) took 16s demonstrating the vast speed improvement of `*` counting. ``` sql> SELECT COUNT(*) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:02] 1 row retrieved starting from 1 in 70 ms (execution: 62 ms, fetching: 8 ms) sql> SELECT COUNT(amount) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:43] 1 row retrieved starting from 1 in 16 s 862 ms (execution: 16 s 855 ms, fetching: 7 ms) ```	6 years ago
Luke Imhoff	2a4223c2c6	Allow block numbers to exceed missing block search range by > 1 ... Fixes #1266 Issue #1266 was triggered when the recorded block number exceeded the reported max block number as can happen temporary when the node that gives the latest block number is farther behind than the node that gave the block data. The previous `Explorer.Chain.missing_block_number_ranges` query added in #1230 did not account for the max block number in the database exceeding the range max. When this occurs, an errant range was produced by the #1230 query because it chose the greatest of the range and the max block when trying to find the end point for the gap search. The testing did not catch #1266 because the doctests only used block numbers that exceeded the search range, by 1 and the bug did not show up until it was 2 or more. The new query is split into 4 subqueries: 1. missing prefix when range min < min(block.number) 2. missing infix when the blocks aren't in the range at all 3. missing suffix when max(block.number) < range max 4. gaps between existent block numbers This version is also more efficient as there is only a single sort of the outer `UNION ALL` result instead of a sort of real blocks and fake end points for the range. It is able to move the `UNION ALL` up because it doesn't need to produce fake end cap blocks. This positioning of the `UNION ALL` and `ORDER BY` on the outer most query allows for all 4 queries to use `one_consensus_block_at_height` index and either a limit (for `min`/`max`) or a window aggregate (for `lag`), which has minimal, single row cost.	6 years ago
Luke Imhoff	07cb543948	Failing regression test for #1266 ... When the max block number exceeds the search range by more than 1 it causes extra ranges to be calculated and they depend on the max block number instead of the search range.	6 years ago
Luke Imhoff	1579db1aec	Add ON CONFLICT to derive_transaction_forks ... Fixes #1262	6 years ago
Luke Imhoff	786f89164b	Failing regression test for #1262	6 years ago
Luke Imhoff	81179f9459	Use :timer functions to make timeout values clearer	6 years ago
Luke Imhoff	1a17658866	Increase DB transaction timeout to 4 minutes ... This prevents DB timeouts against Eth Mainnet for stretches of 20 minutes when testing with block concurrency of 5 and block batch size of 5 while production is simultaneously running at 12x12.	6 years ago
Luke Imhoff	2b60cba8a1	Remove unused import	6 years ago
Luke Imhoff	ef02ae0504	Fix missed pinning ... Showed up as unused variables	6 years ago
Luke Imhoff	04d3644839	Use where clause for on_conflict to only write tuples that are different ... When the `ON CONFLICT` clause is triggered, it still causes a new tuple in Postgres to be written even if the value of all the columns is the same. Making a new tuple means that the table needs to be vacuumed sooner and leads to more work and churn for the database and its storage. But, there is a way to prevent this unnecessary churn: Postgres supports a `WHERE` clause on the `ON CONFLICT DO UPDATE` that allows us to not write a new row if the data would be the same. While testing this, it already showed it can prevent unnecessary updates to the `tokens` table when the MetadataUpdater is running.	6 years ago
Luke Imhoff	3c0adace16	Include import_id and transaction_id in logger metadata ... When `import_id` and `transaction_id` was added in Logger calls, they were not added to the metadata allowed to be shown and so weren't appearing in the logs.	6 years ago
Amanda Sposito	6cb8f3f2ba	Add :dev to plug_cowboy dependency in explorer ... * The iex was not initiating due to this dependency	6 years ago
Amanda Sposito	de3520667f	Add reward value of a block in the tile and overview section	6 years ago
Luke Imhoff	8e1bfc0589	Convert DBConnection.ConnectionError tcp recv: closed to {:error, :timeout} ... Resolves #1251	6 years ago
William Sanches	2b48235898	Remove opcodes from CodeCompiler ... Really old versions of solidity compiler, e.g. v0.1.3, have different output schemas. This data was never used by our application, so instead of having to deal with different outputs, we just don't try to get this item anymore.	6 years ago
William Sanches	c1e582e85c	Add better error messages	6 years ago
William Sanches	3ff530c22e	Update JS solidity compiler ... The old lib would produce errors with solidity versions newer than v0.5.0. This will allow more recent codes to be verified.	6 years ago
Gustavo Santos Ferreira	f2cbf37200	genserver that executes validator data imports every 24 hours	6 years ago
Gustavo Santos Ferreira	ebf0aa6860	module that imports validators metadata into the DataBase	6 years ago
Gustavo Santos Ferreira	be397fbd7b	retriever module that queries the contracts to generate validator metadata	6 years ago
Gustavo Santos Ferreira	7549b8f8c8	prepare configs and new dependencies for the retriever	6 years ago
Gustavo Santos Ferreira	d563a2568a	add metadata column to address_names	6 years ago
fvictorio	cd51e7cb29	Use address_current_token_balances in address tokens tab	6 years ago
Luke Imhoff	d36dac0dc2	Import stages ... Stages will be between Import and the Import.Runner. Unlike Runners, which use 1 common Ecto.Multi, a Stage can produce 1 or more independent Ecto.Multi that will run in separate transactions. This allows addresses to be chunked into separate transactions, which releases the locks between transactions unlike the INSERT-level chunking that was all in 1 transaction.	6 years ago
Luke Imhoff	29d945b9b6	Use reraise and __STACKTRACE__ to reraise safe_insert_all errors ... Fixes #1238 `reraise` is the current accepted idiom for reraising, not `raise` alone as it preserves the original stacktrace.	6 years ago
Luke Imhoff	75f9af7f56	Use lag windowing function to reduce missing block time ... Ecto 3.0 gets us easy access to partitions and the windowing function `lag`. `lag` is able to keep track of the previous row without a self join. Importantly, the new query with `lag` and no more `generate_series` completes in ~2.5 seconds on Ethereum mainnet using `:timer.tc(Explorer.Chain, :missing_block_number_ranges, [6889126..0])` on ETH Mainnet Test DB 40 while the old version doesn't return after 5+ minutes. I've heavily commented the new version of the query since this took awhile to figure out and most importantly finding the proper terminology: "gaps and islands" was key to find the starting query using `lag`. The query in SQL is as follows: ```sql SELECT island.last_number + 1, island.next_number - 1 FROM ( SELECT lag(land.number) OVER "w" AS last_number, land.number AS next_number FROM ( SELECT blocks.number AS number FROM blocks WHERE blocks.consensus = TRUE UNION ALL ( SELECT LEAST(min_blocks.number, before_range_min) FROM ( SELECT min(blocks.number) AS number FROM blocks WHERE blocks.consensus = TRUE) ) AS min_blocks WHERE min_blocks.number IS NULL OR min_blocks.number != before_range_min ) UNION ALL ( SELECT GREATEST(max_blocks.number, after_range_max) FROM ( SELECT max(blocks.number) AS number FROM blocks WHERE blocks.consensus = TRUE ) AS max_blocks WHERE max_blocks.number IS NULL OR max_blocks.number != after_range_max ) AS land WINDOW "w" AS (ORDER BY land.number) ) AS islands WHERE islands.last_number != islands.next_number - 1 ORDER BY islands.last_number ```	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
William Sanches	a754a72390	Add block_reward import	6 years ago
William Sanches	23bd2b0049	Rename block_rewards to emission_rewards	6 years ago
Luke Imhoff	65197f5913	Add shrunk to metadata ... Allow separating shrunk vs unshrunk to monitor how often shrinking occurs.	6 years ago
Luke Imhoff	2569d2595e	Move missing_block_range_count and missing_block_count out of prose ... Being in metadata will allow it to be graphed by tools.	6 years ago
Luke Imhoff	1af8f89001	Move step out of prose and into metadata	6 years ago
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Felipe Renan	3ce83278fb	Improve docs in Address.TokenBalance and Address.CurrentTokenBalance schemas	6 years ago
Amanda Sposito	4833c6c962	Fix Address Trasaction query ... * When we removed the Transaction dynamic query in favor of a union approach, we removed the guarantee we had with the joins and the subquery. * The bug was that only the first transaction that had token transfers was being shown.	6 years ago
Amanda Sposito	607be27612	Remove unused function from Transaction module.	6 years ago
fvictorio	be36adbc8e	Improve token balances fetching for specific addresses	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Sebastian Abondano	c762719ba0	Ensures consensus is valid for parent blocks ... Why: * We noticed some blocks with invalid consensus values on Goerli. * Issue link: https://github.com/poanetwork/blockscout/issues/1181 This change addresses the need by: * Editing the realtime fetcher to check, after insertion to DB, a block's parent consensus value and trigger a refetch for that block number if consensus is `FALSE`. * Creating `Indexer.Block.Realtime.ConsensusEnsurer` module, where logic to ensure a given block has consensus lives. * Adding `list_block_number_with_invalid_consensus/0` to `Explorer.Chain` to get block numbers with invalid consensus from DB. * Adding `Indexer.Block.InvalidConsensus.Worker` to find blocks with invalid consensus and queue them for refetching.	6 years ago
Luke Imhoff	c7d614e2ec	Include date in addition to time in log format	6 years ago
Luke Imhoff	4d778f032d	Stream token holders into ETS table	6 years ago
Luke Imhoff	e0921ff3a9	Stream block validations into ETS table	6 years ago
Luke Imhoff	e99850c65d	Stream token transfers into ETS table	6 years ago