1283 Commits (9f3ba2ca02ff098ca0b68a1799a9fde6dca67923)

Author	SHA1	Message	Date
Luke Imhoff	aa00cf7600	credo 0.10.2 => 1.0.0	6 years ago
Luke Imhoff	613577ebd4	bypass 0.9.0 => 1.0.0	6 years ago
Luke Imhoff	14d7c9badd	Block.SecondDegreeRelation needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `nephew_relations` on `Explorer.Chain.Block`. Ecto expected the Explorer.Chain.Block.SecondDegreeRelation schema to have at least one primary key field ``` It was a previous bug that was hidden that Block.SecondDegreeRelation did not have primary key.	6 years ago
Luke Imhoff	82813d39a5	Tokens needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `token` on `Explorer.Chain.Address`. Ecto expected the Explorer.Chain.Token schema to have at least one primary key field ``` It was a previous bug that was hidden that Token did not have primary key.	6 years ago
Luke Imhoff	5a576fcb04	Update to Phoenix 1.4 + Cowboy 2 for HTTP 2 ... Fixes #1322	6 years ago
Ayrat Badykov	5bbb8c9b2e	add replaced transaction fetcher	6 years ago
Ayrat Badykov	6ec2776681	fix tests	6 years ago
Ayrat Badykov	57bc693174	add view helper to use both transactions and internal_transactions	6 years ago
Ayrat Badykov	19cd720ed3	remove compilation warning	6 years ago
zachdaniel	45bf31c2dc	address view/test inconsistencies	6 years ago
Luke Imhoff	a89fe692a7	Make insert_address_current_token_balances_in_batches only insert new ... The previous version of `insert_address_current_token_balances_in_batches` could only be run once because it assumed that `address_current_token_balances` rows did not exist. I could have added an `ON CONFLICT` to to allow `insert_address_current`insert_address_current` to be rerun, but that would have attempted to rebuild the whole table, which would be very costly as the the complex join and aggregate calculations would need to be done for all rows even if they eventually conflict and we do `ON CONFLICT DO NOTHING`. Instead, `insert_address_current_token_balances_in_batches` searched for `address_token_balances` with `(address_hash, token_contract_address_hash)` pairs that should have an `address_current_token_balance` because the `address_token_balances` `value` is non-`NULL`. From this set of `address_token_balances_requiring_current`, the set of `address_current_token_balances` is removed, leaving only those `address_token_balances_without_required_current`, which are those affected by #1345. The rest of the script is rewritten to follow the convention from the new scripts like `update_new_tokens_holder_count_in_batches`, which include more logging with timing informations and estimates.	6 years ago
Ayrat Badykov	9a30c0c850	add index	6 years ago
Ayrat Badykov	f9e63cff47	fix Code.Fetcher	6 years ago
Ayrat Badykov	33cbc6e4ab	set created_contract_code_indexed_at	6 years ago
Luke Imhoff	1e5507adbc	Set Token holder_count to 0 for new tokens ... Fixes #1359 If there is a conflict, then the token is no new and existing holder_count is not overriden. This ensures that pre-existing tokens that haven't been data migrated are kept as `holder_count` `NULL` and migrated tokens or new tokens passing through this change leave the `holder_count` unchanged.	6 years ago
zachdaniel	c68d59a011	feat: render addresses according to eip-55 ... For more information, see: https://github.com/ethereum/EIPs/blob/master/EIPS/eip-55.md#specification	6 years ago
Luke Imhoff	28695f3d75	Data migration to populate tokens holder_count ... `update_new_tokens_holder_count_in_batches` will `UPDATE` any `tokens` where `holders_count` is `NULL` will have it set using the same query as `Explorer.Chain.count_token_holders_from_token_hash/1`, so after `holders_count` is filled in, it will just be a faster version of `Explorer.Chain.count_token_holders_from_token_hash/1`. `holder_counts` is updated with deltas by `Explorer.Chain.Import.Runner.{Block, Address.CurrentTokenBalance}` as `address_current_token_balances` is changed in a way that would change `Explorer.Chain.count_token_holders_from_token_hash/1` without the need to run the expensive `COUNT` query again.	6 years ago
Luke Imhoff	4f1254b1be	Import.Runner.CurrentTokenBalances update_token_holder_counts ... When `CurrentTokenBalance`s are directly changed 1. Determine which rows will be deleted in `deleted_address_current_token_balances` using same `where` as `on_conflict` for `address_current_token_balances` 2. Upsert rows using pre-existing `address_current_token_balances`. 3. Calculate delta of holder count using `deleted_address_current_token_balances` and `address_current_token_balances`. If the delta is non-zero, update the `tokens.holder_count`.	6 years ago
Luke Imhoff	4144373367	Import.Runner.Block.update_token_holder_counts ... When `CurrentTokenBalance`s change due to blocks being reorged in `delete_address_current_token_balances` and `derive_address_current_token_balances`, whether an `address_hash` is a holder can change, which means the token holder_count can change. `update_token_holder_counts` checks if there is a net change in holders or does not update for that token.	6 years ago
Luke Imhoff	20f50d0dd4	Token holder_count ... `Token` `holder_count` stores `Explorer.Chain.count_token_holders_from_token_hash`, so it does not need to be recalculate for all runs. The `holder_count` is `nil` and must be initialized with a data migration. Since we know `count_token_holders_from_token_hash` across all tokens takes ~90 minutes the `token.holder_count` will be checked first, but if it is `nil`, `count_token_holders_from_token_hash` will still be called, which means the code can be deployed and the data migrations run in the background without blocking further deploys.	6 years ago
Ayrat Badykov	2e94a0d2a6	return InternalTransaction.Fetcher	6 years ago
Ayrat Badykov	9852abae95	add catchup code fetcher	6 years ago
Luke Imhoff	25598e4b7b	Document Explorer.Chain.SmartContract ... Fixes #1342 Document `t:Explorer.Chain.SmartContract.t/0` to clarify difference between `t:Explorer.Chain.SmartContract.t/0` `contract_source_code` and `t:Explorer.Chain.Address.t/0` `contract_code`. Include Solidity ABI Spec docs so domain knowledge is available with the source.	6 years ago
Ayrat Badykov	fbbdf78804	do not index internal transactions with simple token transfers ... 0xa9059cbb - signature of the transfer(address,uint256) function from the ERC-20 token specification. Although transaction input data can be faked we use this heuristics to filter simple token transfer internal transactions from indexing because they slow down realtime fetcher	6 years ago
Luke Imhoff	8a1c526cf8	Optimize update_address_current_token_balances_in_batches ... Fixes #1335 1. Calculate only the `address_current_token_balances` that need to be updated in the temporary table instead of all `address_current_token_balances`. 2. Add lot of `RAISE NOTICE` to print 1. Start Times 2. End Times 3. Elapsed Times 4. Estimate Time of Completion 3. Only update address_current_token_balances if the block number is less than the temporary tables value, in case other transactions occurred to fix the value while the script is running.	6 years ago
zachdaniel	e3c4703a99	fixup! fix: poll on import completion and only new blocks	6 years ago
Luke Imhoff	4556ee2c01	Fix tests of transactions on non-consensus blocks ... Tests for BlockTransactionController mistakenly simulated transactions on non-consensus blocks by making the `transaction.block` `consensus` `false`, but this is unrealistic - the importing only sets the consensus block to `transaction.block` and non-consensus blocks show up as `transaction.transaction_forks` `transaction_fork.uncle`. To prevent this from happening again, `Explorer.Factory.with_block` now checks the `block` has `consensus: true`. With this setup fixed, the tests needed to be updated to show that *no* _transactions_ would be shown for non-consensus blocks (because there are only transactions forks) and whether the non-consensus blocks showed as "above tip" only had to do with the number being above the consensus number and not to do with it being non-consensus. Floki selectors are used to improve error messages when messages are unexpected.	6 years ago
Luke Imhoff	57808d6b23	Separate usage of max consensus block number and block height ... The block height should be used when calculating confirmations while the max consensus block number when a block number is needed and you want to tell if the chain is empty or only the genesis block is available. The block height is 0 if there are no blocks or if there is only the genesis block.	6 years ago
Luke Imhoff	cb7bc384e8	Calculate indexed ratio in SQL ... By calculating the indexed ratio in SQL, we only have 1 round trip to the database instead of 1 to get the max and 1 to get the min.	6 years ago
Luke Imhoff	083e5d3278	Ensure min and max block number is for consensus blocks ... Adding `WHERE consensus = true` both fixes the bug that `min_block_number` and `max_block_number` were including non-consensus blocks, but it also makes the query faster because there is an index for consensus blocks. Times from Eth Mainnet production: ``` ether=> SELECT min(b0."number") FROM "blocks" AS b0; min ----- 0 (1 row) Time: 414.218 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0; QUERY PLAN ----------------------------------------------------------------------------------------------- Finalize Aggregate (cost=223454.95..223454.96 rows=1 width=8) -> Gather (cost=223454.73..223454.95 rows=2 width=8) Workers Planned: 2 -> Partial Aggregate (cost=222454.73..222454.74 rows=1 width=8) -> Parallel Seq Scan on blocks b0 (cost=0.00..215051.99 rows=2961099 width=8) (5 rows) Time: 1.040 ms ether=> EXPLAIN SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------- Result (cost=0.45..0.46 rows=1 width=8) InitPlan 1 (returns $0) -> Limit (cost=0.43..0.45 rows=1 width=8) -> Index Only Scan using one_consensus_block_at_height on blocks b0 (cost=0.43..158050.90 rows=7045520 width=8) Index Cond: (number IS NOT NULL) (5 rows) Time: 3.350 ms ether=> SELECT min(b0."number") FROM "blocks" AS b0 WHERE b0.consensus = true; min ----- 0 (1 row) Time: 1.059 ms ```	6 years ago
Luke Imhoff	99e6947d62	Update `address_current_token_balances` that are out-of-date ... Fixes #1317 Some bug in the past causes `address_current_token_balances` to not always be up-to-date. The verifier below show the number of incorrect `address_current_token_balances` not growing, so the bug itself is fixed, but the data is still wrong and need to be updated. Run `apps/explorer/priv/repo/migrations/scripts/update_address_current_token_balances_in_batches.sql` to fix any `address_current_token_balances` that have the incorrect `block_number` or `value`. Run the following to verify everything is fixed: ```sql SELECT COUNT(*) FROM (SELECT 1 FROM address_token_balances INNER JOIN address_current_token_balances ON address_current_token_balances.address_hash = address_token_balances.address_hash AND address_current_token_balances.token_contract_address_hash = address_token_balances.token_contract_address_hash GROUP BY address_token_balances.address_hash, address_token_balances.token_contract_address_hash, address_current_token_balances.block_number HAVING MAX(address_token_balances.block_number) != address_current_token_balances.block_number) AS address_incorrect_token_balances; ``` Which should return `0`. after the script is run. WARNING: the check SQL above will take 15-20 minutes on Eth Mainnet.	6 years ago
Luke Imhoff	43fce861c0	Additional tests to show #1317 is no reproducible	6 years ago
Luke Imhoff	8718ca4069	Update address_current_token_balances and address_token_balances for reorgs ... Fixes #1320 1. Delete any `address_token_balances` with matching consensus block number. 2. Delete any `address_current_token_balances` with matching consensus block number. 3. Derive `address_current_token_balances` for any deleted in (2).	6 years ago
Luke Imhoff	51f181db15	Failing regression tests for #1320	6 years ago
Luke Imhoff	f016475f1d	Order derive_tranactions_forks SELECT to match transaction inserts ... Fixes #1311 Prevent deadlocks due to order in the `SELECT transaction` for `Explorer.Chain.Import.Runner.Blocks.derived_transaction_forks` differing from order in `Explorer.Chain.Import.Runner.Transactions.insert` by sorting both by hash ascending.	6 years ago
Luke Imhoff	75376f43e3	Delete block_rewards for non-consensus or old consensus blocks ... Fixes #1308 1. If a block is non-consensus delete rewards for its hash as the hash should no longer have rewards. 2. If a block is consensus delete rewards for its number as rewards currently in the database were from before a reorg.	6 years ago
Luke Imhoff	aab51cf2f4	Restrict Explorer.Chain.Block.get_blocks_without_reward to consensus ... Fixes #1310	6 years ago
Luke Imhoff	2ec6c85c14	Failing regression test for #1310	6 years ago
zachdaniel	1c77ef0e19	feat: render block time with up to one decimal point of accuracy	6 years ago
Luke Imhoff	5524a49725	Don't fetch internal transactions for simple token transfers ... Resolves #1303 Don't fetch internal transactions for transactions that have token transfers from the logs if the status is `:ok` and the transaction did not create a contract.	6 years ago
Luke Imhoff	40004fdb4b	Rename file to match naming convention	6 years ago
Luke Imhoff	a571a4edb3	Don't fetch or import internal transactions for value transfers ... Fixes #1287 For simple value transfers, the UI already doesn't show the redundant single internal transaction, but we can speed up the indexing by determine the internal transactions aren't needed when the transacion is a successful simple value transfer as determined by status being `:ok` and the input being empty (`0x`).	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	df8673242e	Order current token balance inserts to prevent deadlock ... Fixes #1295	6 years ago
Luke Imhoff	b69d437b06	Include block number when ordering token balances ... The block number matters for ordering as a token can be updated for an address multiple times across different blocks in one insert.	6 years ago
Luke Imhoff	6165f6757f	Move uniquifiation of address current token balances out of runner ... For all other runners, it is the responsibility of the caller to ensure unique entries. This ensures that the least amount of data as possible is sent to `Explorer.Chain.Import.all`, which speeds up the whole import process.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Andrew Cravenho	f3ab5e4c58	Update Parity Sokol node tests	6 years ago
Gustavo Santos Ferreira	34dd660c1f	deal with coin market cap's pagination situation	6 years ago
fvictorio	616658adbf	Show token price in USD	6 years ago