679 Commits (8d4493280397cbdbd6a91993f0e3ac00c1c15000)

Author	SHA1	Message	Date
Luke Imhoff	294c9f37e4	Convert UncatalogedRewards.Processor into a BufferedTask ... * Makes it works like the rest of the fetchers for consistency. * Allow it to participate in memory monitoring Bug Fixes * Addresses are created as needed for block rewards. * Coin Balances are created for each block and address mentioned. * Coin Balances are queues in CoinBalance.Fetcher. Enhancements * Fetch beneficiary errors no longer kill the block catchup, but instead only causes them to be async fetched while the rest of the block imports	6 years ago
Luke Imhoff	7098f4b78d	Use Explorer.Chain.import in Indexer.Block.UncatalogedRewards.Importer ... Ensures logic is consistent across Catchup, Realtime, and Uncatalogued fetchers.	6 years ago
Luke Imhoff	00e136e20c	Calculate gas payments for all blocks in batch in SQL	6 years ago
Luke Imhoff	4c634eb9a6	Use Ecto 3 coalesce, sum, +, and * support	6 years ago
Luke Imhoff	ace3abf8df	Don't generate wei^2 from Wei.mult ... Wei.mult required both parameters to be Wei.t, but that meant it was actually yielding `wei^2` units. Fix the units by requiring the first parameter to be wei and the second to be a unitless integer, which makes more sense for the `gas_price * gas_used` usage as we expect that to be measured in wei and not wei^2.	6 years ago
Ayrat Badykov	a4f2aee213	add ReplacedTRansaction fetcher to supervision tree	6 years ago
Ayrat Badykov	cf78b8cb80	move replaced transactions update to fetcher	6 years ago
Ayrat Badykov	98e0529d3f	fix map access keys	6 years ago
Ayrat Badykov	20c7a1ba21	use or_where statement to update replaced transactions	6 years ago
Ayrat Badykov	3c1dd1d71f	remove replaced transactions fetcher	6 years ago
Ayrat Badykov	b780876b3c	fix migration	6 years ago
Ayrat Badykov	915d053dd5	update replaced transaction on import	6 years ago
Ayrat Badykov	fb597cd6e3	exclude replaced transaction from pending transactions list	6 years ago
Ayrat Badykov	fbcf7571b1	fix CR issues	6 years ago
Luke Imhoff	efe9a020e8	telemetry 0.2.0 => 0.3.0 ... ecto_sql 3.0.3 => 3.0.4	6 years ago
Luke Imhoff	14d7c9badd	Block.SecondDegreeRelation needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `nephew_relations` on `Explorer.Chain.Block`. Ecto expected the Explorer.Chain.Block.SecondDegreeRelation schema to have at least one primary key field ``` It was a previous bug that was hidden that Block.SecondDegreeRelation did not have primary key.	6 years ago
Luke Imhoff	82813d39a5	Tokens needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `token` on `Explorer.Chain.Address`. Ecto expected the Explorer.Chain.Token schema to have at least one primary key field ``` It was a previous bug that was hidden that Token did not have primary key.	6 years ago
Ayrat Badykov	5bbb8c9b2e	add replaced transaction fetcher	6 years ago
Ayrat Badykov	57bc693174	add view helper to use both transactions and internal_transactions	6 years ago
Ayrat Badykov	19cd720ed3	remove compilation warning	6 years ago
zachdaniel	45bf31c2dc	address view/test inconsistencies	6 years ago
Ayrat Badykov	f9e63cff47	fix Code.Fetcher	6 years ago
Ayrat Badykov	33cbc6e4ab	set created_contract_code_indexed_at	6 years ago
Luke Imhoff	1e5507adbc	Set Token holder_count to 0 for new tokens ... Fixes #1359 If there is a conflict, then the token is no new and existing holder_count is not overriden. This ensures that pre-existing tokens that haven't been data migrated are kept as `holder_count` `NULL` and migrated tokens or new tokens passing through this change leave the `holder_count` unchanged.	6 years ago
zachdaniel	c68d59a011	feat: render addresses according to eip-55 ... For more information, see: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-55.md#specification	6 years ago
Luke Imhoff	4f1254b1be	Import.Runner.CurrentTokenBalances update_token_holder_counts ... When `CurrentTokenBalance`s are directly changed 1. Determine which rows will be deleted in `deleted_address_current_token_balances` using same `where` as `on_conflict` for `address_current_token_balances` 2. Upsert rows using pre-existing `address_current_token_balances`. 3. Calculate delta of holder count using `deleted_address_current_token_balances` and `address_current_token_balances`. If the delta is non-zero, update the `tokens.holder_count`.	6 years ago
Luke Imhoff	4144373367	Import.Runner.Block.update_token_holder_counts ... When `CurrentTokenBalance`s change due to blocks being reorged in `delete_address_current_token_balances` and `derive_address_current_token_balances`, whether an `address_hash` is a holder can change, which means the token holder_count can change. `update_token_holder_counts` checks if there is a net change in holders or does not update for that token.	6 years ago
Luke Imhoff	20f50d0dd4	Token holder_count ... `Token` `holder_count` stores `Explorer.Chain.count_token_holders_from_token_hash`, so it does not need to be recalculate for all runs. The `holder_count` is `nil` and must be initialized with a data migration. Since we know `count_token_holders_from_token_hash` across all tokens takes ~90 minutes the `token.holder_count` will be checked first, but if it is `nil`, `count_token_holders_from_token_hash` will still be called, which means the code can be deployed and the data migrations run in the background without blocking further deploys.	6 years ago
Ayrat Badykov	9852abae95	add catchup code fetcher	6 years ago
Luke Imhoff	25598e4b7b	Document Explorer.Chain.SmartContract ... Fixes #1342 Document `t:Explorer.Chain.SmartContract.t/0` to clarify difference between `t:Explorer.Chain.SmartContract.t/0` `contract_source_code` and `t:Explorer.Chain.Address.t/0` `contract_code`. Include Solidity ABI Spec docs so domain knowledge is available with the source.	6 years ago
Ayrat Badykov	fbbdf78804	do not index internal transactions with simple token transfers ... 0xa9059cbb - signature of the transfer(address,uint256) function from the ERC-20 token specification. Although transaction input data can be faked we use this heuristics to filter simple token transfer internal transactions from indexing because they slow down realtime fetcher	6 years ago
zachdaniel	e3c4703a99	fixup! fix: poll on import completion and only new blocks	6 years ago
Luke Imhoff	4556ee2c01	Fix tests of transactions on non-consensus blocks ... Tests for BlockTransactionController mistakenly simulated transactions on non-consensus blocks by making the `transaction.block` `consensus` `false`, but this is unrealistic - the importing only sets the consensus block to `transaction.block` and non-consensus blocks show up as `transaction.transaction_forks` `transaction_fork.uncle`. To prevent this from happening again, `Explorer.Factory.with_block` now checks the `block` has `consensus: true`. With this setup fixed, the tests needed to be updated to show that *no* _transactions_ would be shown for non-consensus blocks (because there are only transactions forks) and whether the non-consensus blocks showed as "above tip" only had to do with the number being above the consensus number and not to do with it being non-consensus. Floki selectors are used to improve error messages when messages are unexpected.	6 years ago
Luke Imhoff	57808d6b23	Separate usage of max consensus block number and block height ... The block height should be used when calculating confirmations while the max consensus block number when a block number is needed and you want to tell if the chain is empty or only the genesis block is available. The block height is 0 if there are no blocks or if there is only the genesis block.	6 years ago
Luke Imhoff	cb7bc384e8	Calculate indexed ratio in SQL ... By calculating the indexed ratio in SQL, we only have 1 round trip to the database instead of 1 to get the max and 1 to get the min.	6 years ago
Luke Imhoff	083e5d3278	Ensure min and max block number is for consensus blocks ... Adding `WHERE consensus = true` both fixes the bug that `min_block_number` and `max_block_number` were including non-consensus blocks, but it also makes the query faster because there is an index for consensus blocks. Times from Eth Mainnet production: ``` ether=> SELECT min(b0."number") FROM "blocks" AS b0; min ----- 0 (1 row) Time: 414.218 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0; QUERY PLAN ----------------------------------------------------------------------------------------------- Finalize Aggregate (cost=223454.95..223454.96 rows=1 width=8) -> Gather (cost=223454.73..223454.95 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=222454.73..222454.74 rows=1 width=8) -> Parallel Seq Scan on blocks b0 (cost=0.00..215051.99 rows=2961099 width=8) (5 rows) Time: 1.040 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------- Result (cost=0.45..0.46 rows=1 width=8) InitPlan 1 (returns $0) -> Limit (cost=0.43..0.45 rows=1 width=8) -> Index Only Scan using one_consensus_block_at_height on blocks b0 (cost=0.43..158050.90 rows=7045520 width=8) Index Cond: (number IS NOT NULL) (5 rows) Time: 3.350 ms ether=> SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; min ----- 0 (1 row) Time: 1.059 ms ```	6 years ago
Luke Imhoff	8718ca4069	Update address_current_token_balances and address_token_balances for reorgs ... Fixes #1320 1. Delete any `address_token_balances` with matching consensus block number. 2. Delete any `address_current_token_balances` with matching consensus block number. 3. Derive `address_current_token_balances` for any deleted in (2).	6 years ago
Luke Imhoff	f016475f1d	Order derive_tranactions_forks SELECT to match transaction inserts ... Fixes #1311 Prevent deadlocks due to order in the `SELECT transaction` for `Explorer.Chain.Import.Runner.Blocks.derived_transaction_forks` differing from order in `Explorer.Chain.Import.Runner.Transactions.insert` by sorting both by hash ascending.	6 years ago
Luke Imhoff	75376f43e3	Delete block_rewards for non-consensus or old consensus blocks ... Fixes #1308 1. If a block is non-consensus delete rewards for its hash as the hash should no longer have rewards. 2. If a block is consensus delete rewards for its number as rewards currently in the database were from before a reorg.	6 years ago
Luke Imhoff	aab51cf2f4	Restrict Explorer.Chain.Block.get_blocks_without_reward to consensus ... Fixes #1310	6 years ago
zachdaniel	1c77ef0e19	feat: render block time with up to one decimal point of accuracy	6 years ago
Luke Imhoff	5524a49725	Don't fetch internal transactions for simple token transfers ... Resolves #1303 Don't fetch internal transactions for transactions that have token transfers from the logs if the status is `:ok` and the transaction did not create a contract.	6 years ago
Luke Imhoff	40004fdb4b	Rename file to match naming convention	6 years ago
Luke Imhoff	a571a4edb3	Don't fetch or import internal transactions for value transfers ... Fixes #1287 For simple value transfers, the UI already doesn't show the redundant single internal transaction, but we can speed up the indexing by determine the internal transactions aren't needed when the transacion is a successful simple value transfer as determined by status being `:ok` and the input being empty (`0x`).	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	df8673242e	Order current token balance inserts to prevent deadlock ... Fixes #1295	6 years ago
Luke Imhoff	b69d437b06	Include block number when ordering token balances ... The block number matters for ordering as a token can be updated for an address multiple times across different blocks in one insert.	6 years ago
Luke Imhoff	6165f6757f	Move uniquifiation of address current token balances out of runner ... For all other runners, it is the responsibility of the caller to ensure unique entries. This ensures that the least amount of data as possible is sent to `Explorer.Chain.Import.all`, which speeds up the whole import process.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Gustavo Santos Ferreira	34dd660c1f	deal with coin market cap's pagination situation	6 years ago