420 Commits (0a3f29eb5289e74d388d2661b5fac2b60b47da14)

Author	SHA1	Message	Date
Ayrat Badykov	b780876b3c	fix migration	6 years ago
Ayrat Badykov	915d053dd5	update replaced transaction on import	6 years ago
Ayrat Badykov	0497a9daa7	mix format	6 years ago
Ayrat Badykov	fb5d4b20d9	modify status and error db constraints	6 years ago
Luke Imhoff	14d7c9badd	Block.SecondDegreeRelation needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `nephew_relations` on `Explorer.Chain.Block`. Ecto expected the Explorer.Chain.Block.SecondDegreeRelation schema to have at least one primary key field ``` It was a previous bug that was hidden that Block.SecondDegreeRelation did not have primary key.	6 years ago
Luke Imhoff	82813d39a5	Tokens needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `token` on `Explorer.Chain.Address`. Ecto expected the Explorer.Chain.Token schema to have at least one primary key field ``` It was a previous bug that was hidden that Token did not have primary key.	6 years ago
Luke Imhoff	a89fe692a7	Make insert_address_current_token_balances_in_batches only insert new ... The previous version of `insert_address_current_token_balances_in_batches` could only be run once because it assumed that `address_current_token_balances` rows did not exist. I could have added an `ON CONFLICT` to to allow `insert_address_current`insert_address_current` to be rerun, but that would have attempted to rebuild the whole table, which would be very costly as the the complex join and aggregate calculations would need to be done for all rows even if they eventually conflict and we do `ON CONFLICT DO NOTHING`. Instead, `insert_address_current_token_balances_in_batches` searched for `address_token_balances` with `(address_hash, token_contract_address_hash)` pairs that should have an `address_current_token_balance` because the `address_token_balances` `value` is non-`NULL`. From this set of `address_token_balances_requiring_current`, the set of `address_current_token_balances` is removed, leaving only those `address_token_balances_without_required_current`, which are those affected by #1345. The rest of the script is rewritten to follow the convention from the new scripts like `update_new_tokens_holder_count_in_batches`, which include more logging with timing informations and estimates.	6 years ago
Ayrat Badykov	9a30c0c850	add index	6 years ago
Luke Imhoff	28695f3d75	Data migration to populate tokens holder_count ... `update_new_tokens_holder_count_in_batches` will `UPDATE` any `tokens` where `holders_count` is `NULL` will have it set using the same query as `Explorer.Chain.count_token_holders_from_token_hash/1`, so after `holders_count` is filled in, it will just be a faster version of `Explorer.Chain.count_token_holders_from_token_hash/1`. `holder_counts` is updated with deltas by `Explorer.Chain.Import.Runner.{Block, Address.CurrentTokenBalance}` as `address_current_token_balances` is changed in a way that would change `Explorer.Chain.count_token_holders_from_token_hash/1` without the need to run the expensive `COUNT` query again.	6 years ago
Luke Imhoff	20f50d0dd4	Token holder_count ... `Token` `holder_count` stores `Explorer.Chain.count_token_holders_from_token_hash`, so it does not need to be recalculate for all runs. The `holder_count` is `nil` and must be initialized with a data migration. Since we know `count_token_holders_from_token_hash` across all tokens takes ~90 minutes the `token.holder_count` will be checked first, but if it is `nil`, `count_token_holders_from_token_hash` will still be called, which means the code can be deployed and the data migrations run in the background without blocking further deploys.	6 years ago
Ayrat Badykov	2e94a0d2a6	return InternalTransaction.Fetcher	6 years ago
Ayrat Badykov	9852abae95	add catchup code fetcher	6 years ago
Luke Imhoff	8a1c526cf8	Optimize update_address_current_token_balances_in_batches ... Fixes #1335 1. Calculate only the `address_current_token_balances` that need to be updated in the temporary table instead of all `address_current_token_balances`. 2. Add lot of `RAISE NOTICE` to print 1. Start Times 2. End Times 3. Elapsed Times 4. Estimate Time of Completion 3. Only update address_current_token_balances if the block number is less than the temporary tables value, in case other transactions occurred to fix the value while the script is running.	6 years ago
Luke Imhoff	99e6947d62	Update `address_current_token_balances` that are out-of-date ... Fixes #1317 Some bug in the past causes `address_current_token_balances` to not always be up-to-date. The verifier below show the number of incorrect `address_current_token_balances` not growing, so the bug itself is fixed, but the data is still wrong and need to be updated. Run `apps/explorer/priv/repo/migrations/scripts/update_address_current_token_balances_in_batches.sql` to fix any `address_current_token_balances` that have the incorrect `block_number` or `value`. Run the following to verify everything is fixed: ```sql SELECT COUNT(*) FROM (SELECT 1 FROM address_token_balances INNER JOIN address_current_token_balances ON address_current_token_balances.address_hash = address_token_balances.address_hash AND address_current_token_balances.token_contract_address_hash = address_token_balances.token_contract_address_hash GROUP BY address_token_balances.address_hash, address_token_balances.token_contract_address_hash, address_current_token_balances.block_number HAVING MAX(address_token_balances.block_number) != address_current_token_balances.block_number) AS address_incorrect_token_balances; ``` Which should return `0`. after the script is run. WARNING: the check SQL above will take 15-20 minutes on Eth Mainnet.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Gustavo Santos Ferreira	d563a2568a	add metadata column to address_names	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
William Sanches	a754a72390	Add block_reward import	6 years ago
William Sanches	23bd2b0049	Rename block_rewards to emission_rewards	6 years ago
Amanda Sposito	4833c6c962	Fix Address Trasaction query ... * When we removed the Transaction dynamic query in favor of a union approach, we removed the guarantee we had with the joins and the subquery. * The bug was that only the first transaction that had token transfers was being shown.	6 years ago
Luke Imhoff	6ec9afb4ba	Gradual migration support for call_has_call_type, call_has_input, and create_has_init ... Fixes #1112	6 years ago
Luke Imhoff	39b6eccfda	Separate schema and data migration for EIP-6 ... Fixes #1109 Use `NOT VALID` on constraint and don't put data updates in schema migration. Instead, migrate data using script based on @amandasposito's scripts.	6 years ago
Amanda Sposito	8ad5a3103e	Add index to addresses table ... Creates a partial index using the fetched_coin_balance and hash columns	6 years ago
Amanda Sposito	40a9da3022	Add script to migrate nonce to addresses ... This script copy nonce from the transactions' table to addresses' table in batches.	6 years ago
svenski123	a3364385c6	Comment out update query in migration script due to run length concerns on a large existing database.	6 years ago
svenski123	29199eb54e	Add nonce field to Address object and addresses table. ... This caches the highest nonce value seen for sent transactions for a given address. The top addresses query no longer needs to scan the transactions table for nonce values. The database migration script adds the nonce field to the addresses table and sets the nonce value according to the contents of the transactions table.	6 years ago
Felipe Renan	2ad1042de2	Add script to migrate block_number to token transfers ... This script copy block_number from the transactions' table to token transfers' table in batches.	6 years ago
Felipe Renan	31e1090138	Add block_number to token transfers ... This column is a denormalization from transactions' table.	6 years ago
Amanda Sposito	9843f2bba6	Add migration scripts that we've been using to migrate data	6 years ago
Luke Imhoff	0c4c80c880	Add missing constraints for internal transactions ... Calls didn't enforce that `call_type` and `input` was not `NULL`. Creates didn't enforce that `init` was not `NULL`.	6 years ago
Luke Imhoff	9cce7ab661	Adopt EIP-6 ... Geth has EIP-6, while Parity does not, but the Ethereum Yellow Paper includes EIP-6, so it is better to use EIP-6 when looking up community docs.	6 years ago
Felipe Renan	233d539a00	Add AddressCurrentTokenBalance schema	6 years ago
Gustavo Santos Ferreira	acf74e6267	migration to change type of field decimals	6 years ago
Luke Imhoff	7f7cdf753d	Switch token_transfers to composite primary key ... * Drop id. * Make (transaaction_hash, log_index) the primary key. This ports the change from `internal_transactions`.	6 years ago
Luke Imhoff	96b97f0dc1	Switch logs to composite primary key ... * Drop id. * Make (transaaction_hash, index) the primary key. This ports the change from `internal_transactions`.	6 years ago
Luke Imhoff	95e3774e9a	Switch internal_transactions to composite primary key ... * Drop id. * Make (transaaction_hash, index) the primary key.	6 years ago
Amanda Sposito	5ee4df3714	Alter 'Chain.address_to_internal_transactions' query ... * The query was taking too long and one of the reasons was because it was performing a sub-query in the internal_transactions table to identify if the transaction had siblings. This was made because for every parent transactions there is one equal internal_transaction, so this check was made to only show internal transactions that are different from the parent transaction. * There is another way to check this that removes the need to perform a sub-query in the internal transactions table. We need to check the indexes >`0` for `type`=`call`. * Also created new indexes to improve the 'where' match and the 'order by' clause.	6 years ago
Amanda Sposito	373248dbee	Update 'internal_transactions'.'block_number' after block reorg ... * The block_number may change during the block reorg so it needs to be updated at the denormalized column in internal_transaction.	6 years ago
Amanda Sposito	1acebdc7f7	Add 'block_number' and 'transaction_index' in 'internal_transaction' ... * Denormalize the internal_transaction table so we don't need to join the 'transactions' table to know the block_number and index	6 years ago
Alex Garibay	c60bd06e5e	Add bare admin panel (#927) ... * Add router for admin routes * Add mechanism for having a generated admin recovery key * Add admin panel to web interface Web portion now includes a set of routes exclusive for managing a BlockScout instance. Whenever an instance hasn't been configuration, a user will be prompted with a page to setup their instance. Once they've followed the steps and registered, the user will be able to perform administrative tasks.	6 years ago
Luke Imhoff	818f84a424	Transaction.Fork ... A Transaction.Fork is a Transaction recorded in a non-consensus block, such an uncle.	6 years ago
Luke Imhoff	4b140c87df	Block in the consensus chain marked consensus true	6 years ago
Luke Imhoff	96d5aea311	Block.SecondDegreeRelation ... The database table block_second_degree_relations contains links between the `uncle_hash` of the uncle and the `nephew_hash` of the block that said `uncle_hash` was an uncle. It also has a `uncle_fetched_at` that is `null` until the `uncle_hash` is confirmed as fetched in the database. This does not contain such a fetcher.	6 years ago
Amanda Sposito	fc38d7e2bf	Alter Token decimals column to bigint ... Fixes #758 There are some decimals with bigger values than the supported by the 'smallint' type. https://www.postgresql.org/docs/9.1/static/datatype-numeric.html	6 years ago
Stamates	742e543cdb	Incorporate update for blocks gas_used and gas_limit in initial migration	6 years ago
Stamates	c6af8d363e	Create recent collated indexes by address with hash on transaction table and remove address/hash indexes	6 years ago
Stamates	bd755fe469	Add transaction hash to address indexes for transaction and token_transfer tables to optimize count query	6 years ago
Luke Imhoff	102f467948	Denormalize last internal transaction error to transaction ... Allow UI and APIs to get the overall transaction error easily when status is `:error` without have to preload all the internal transactions.	6 years ago
Luke Imhoff	b32f263eda	Allow status to be nil until internal transactions are indexed ... For pre-Byzantium or Ethereum Classic (which never adopted EIP-658), the status in receipts will be `nil`, but we still want to ensure the status is eventually set, so when setting internal transactions, in addition to setting `internal_transactions_indexed_at` and `created_contract_address_hash`, the `status` will be updated. * If the `status` is already set, leave it as is * If the `status` is not set * If the last internal transaction for a transaction has an error, `status` is `:error` * If the last internal transaction for a transaction has no error, `status` is `:ok`	6 years ago