356 Commits (5a3ed4b4d40e09a2013e83880cdf15deeca78dac)

Author	SHA1	Message	Date
Ayrat Badykov	5191085052	add indexes required for blocks runner	6 years ago
Ayrat Badykov	caa40cc017	add index on block number to address token balances	6 years ago
Ayrat Badykov	6925f0c34e	add index for address_current_token_balances	6 years ago
Ayrat Badykov	ff0db8c32b	add unique index on address_hash, block_hash and address_type for block_rewards	6 years ago
Ayrat Badykov	ee814636b5	pass external libraries to the Solidity compiler	6 years ago
Ayrat Badykov	da9d6e5f31	add constructor arguments field to smart contract	6 years ago
Ayrat Badykov	8bfe0a33f8	use bytecode field instead of deployedBytecode in compiler output	6 years ago
Ayrat Badykov	7e8ad096dd	compile smart contracts with external libraries	6 years ago
Andrew Cravenho	4a5bf27060	Only add confirmed dropped/replaced txs to temp table	6 years ago
Andrew Cravenho	c0ac367960	Fix block_hash is null statement	6 years ago
Andrew Cravenho	f71228884c	Update migration script for dropped/replaced transactions	6 years ago
Ayrat Badykov	359b2608fe	fix dialyzer	6 years ago
Ayrat Badykov	d3c7c8636f	add replaced_transaction update script	6 years ago
Ayrat Badykov	b780876b3c	fix migration	6 years ago
Ayrat Badykov	915d053dd5	update replaced transaction on import	6 years ago
Ayrat Badykov	0497a9daa7	mix format	6 years ago
Ayrat Badykov	fb5d4b20d9	modify status and error db constraints	6 years ago
Luke Imhoff	14d7c9badd	Block.SecondDegreeRelation needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `nephew_relations` on `Explorer.Chain.Block`. Ecto expected the Explorer.Chain.Block.SecondDegreeRelation schema to have at least one primary key field ``` It was a previous bug that was hidden that Block.SecondDegreeRelation did not have primary key.	6 years ago
Luke Imhoff	82813d39a5	Tokens needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `token` on `Explorer.Chain.Address`. Ecto expected the Explorer.Chain.Token schema to have at least one primary key field ``` It was a previous bug that was hidden that Token did not have primary key.	6 years ago
Luke Imhoff	a89fe692a7	Make insert_address_current_token_balances_in_batches only insert new ... The previous version of `insert_address_current_token_balances_in_batches` could only be run once because it assumed that `address_current_token_balances` rows did not exist. I could have added an `ON CONFLICT` to to allow `insert_address_current`insert_address_current` to be rerun, but that would have attempted to rebuild the whole table, which would be very costly as the the complex join and aggregate calculations would need to be done for all rows even if they eventually conflict and we do `ON CONFLICT DO NOTHING`. Instead, `insert_address_current_token_balances_in_batches` searched for `address_token_balances` with `(address_hash, token_contract_address_hash)` pairs that should have an `address_current_token_balance` because the `address_token_balances` `value` is non-`NULL`. From this set of `address_token_balances_requiring_current`, the set of `address_current_token_balances` is removed, leaving only those `address_token_balances_without_required_current`, which are those affected by #1345. The rest of the script is rewritten to follow the convention from the new scripts like `update_new_tokens_holder_count_in_batches`, which include more logging with timing informations and estimates.	6 years ago
Ayrat Badykov	9a30c0c850	add index	6 years ago
Luke Imhoff	28695f3d75	Data migration to populate tokens holder_count ... `update_new_tokens_holder_count_in_batches` will `UPDATE` any `tokens` where `holders_count` is `NULL` will have it set using the same query as `Explorer.Chain.count_token_holders_from_token_hash/1`, so after `holders_count` is filled in, it will just be a faster version of `Explorer.Chain.count_token_holders_from_token_hash/1`. `holder_counts` is updated with deltas by `Explorer.Chain.Import.Runner.{Block, Address.CurrentTokenBalance}` as `address_current_token_balances` is changed in a way that would change `Explorer.Chain.count_token_holders_from_token_hash/1` without the need to run the expensive `COUNT` query again.	6 years ago
Luke Imhoff	20f50d0dd4	Token holder_count ... `Token` `holder_count` stores `Explorer.Chain.count_token_holders_from_token_hash`, so it does not need to be recalculate for all runs. The `holder_count` is `nil` and must be initialized with a data migration. Since we know `count_token_holders_from_token_hash` across all tokens takes ~90 minutes the `token.holder_count` will be checked first, but if it is `nil`, `count_token_holders_from_token_hash` will still be called, which means the code can be deployed and the data migrations run in the background without blocking further deploys.	6 years ago
Ayrat Badykov	2e94a0d2a6	return InternalTransaction.Fetcher	6 years ago
Ayrat Badykov	9852abae95	add catchup code fetcher	6 years ago
Luke Imhoff	8a1c526cf8	Optimize update_address_current_token_balances_in_batches ... Fixes #1335 1. Calculate only the `address_current_token_balances` that need to be updated in the temporary table instead of all `address_current_token_balances`. 2. Add lot of `RAISE NOTICE` to print 1. Start Times 2. End Times 3. Elapsed Times 4. Estimate Time of Completion 3. Only update address_current_token_balances if the block number is less than the temporary tables value, in case other transactions occurred to fix the value while the script is running.	6 years ago
Luke Imhoff	99e6947d62	Update `address_current_token_balances` that are out-of-date ... Fixes #1317 Some bug in the past causes `address_current_token_balances` to not always be up-to-date. The verifier below show the number of incorrect `address_current_token_balances` not growing, so the bug itself is fixed, but the data is still wrong and need to be updated. Run `apps/explorer/priv/repo/migrations/scripts/update_address_current_token_balances_in_batches.sql` to fix any `address_current_token_balances` that have the incorrect `block_number` or `value`. Run the following to verify everything is fixed: ```sql SELECT COUNT(*) FROM (SELECT 1 FROM address_token_balances INNER JOIN address_current_token_balances ON address_current_token_balances.address_hash = address_token_balances.address_hash AND address_current_token_balances.token_contract_address_hash = address_token_balances.token_contract_address_hash GROUP BY address_token_balances.address_hash, address_token_balances.token_contract_address_hash, address_current_token_balances.block_number HAVING MAX(address_token_balances.block_number) != address_current_token_balances.block_number) AS address_incorrect_token_balances; ``` Which should return `0`. after the script is run. WARNING: the check SQL above will take 15-20 minutes on Eth Mainnet.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
William Sanches	2b48235898	Remove opcodes from CodeCompiler ... Really old versions of solidity compiler, e.g. v0.1.3, have different output schemas. This data was never used by our application, so instead of having to deal with different outputs, we just don't try to get this item anymore.	6 years ago
William Sanches	3ff530c22e	Update JS solidity compiler ... The old lib would produce errors with solidity versions newer than v0.5.0. This will allow more recent codes to be verified.	6 years ago
Gustavo Santos Ferreira	7549b8f8c8	prepare configs and new dependencies for the retriever	6 years ago
Gustavo Santos Ferreira	d563a2568a	add metadata column to address_names	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
William Sanches	a754a72390	Add block_reward import	6 years ago
William Sanches	23bd2b0049	Rename block_rewards to emission_rewards	6 years ago
Amanda Sposito	4833c6c962	Fix Address Trasaction query ... * When we removed the Transaction dynamic query in favor of a union approach, we removed the guarantee we had with the joins and the subquery. * The bug was that only the first transaction that had token transfers was being shown.	6 years ago
Luke Imhoff	6ec9afb4ba	Gradual migration support for call_has_call_type, call_has_input, and create_has_init ... Fixes #1112	6 years ago
Luke Imhoff	39b6eccfda	Separate schema and data migration for EIP-6 ... Fixes #1109 Use `NOT VALID` on constraint and don't put data updates in schema migration. Instead, migrate data using script based on @amandasposito's scripts.	6 years ago
Amanda Sposito	8ad5a3103e	Add index to addresses table ... Creates a partial index using the fetched_coin_balance and hash columns	6 years ago
Amanda Sposito	40a9da3022	Add script to migrate nonce to addresses ... This script copy nonce from the transactions' table to addresses' table in batches.	6 years ago
svenski123	a3364385c6	Comment out update query in migration script due to run length concerns on a large existing database.	6 years ago
svenski123	29199eb54e	Add nonce field to Address object and addresses table. ... This caches the highest nonce value seen for sent transactions for a given address. The top addresses query no longer needs to scan the transactions table for nonce values. The database migration script adds the nonce field to the addresses table and sets the nonce value according to the contents of the transactions table.	6 years ago
Felipe Renan	2ad1042de2	Add script to migrate block_number to token transfers ... This script copy block_number from the transactions' table to token transfers' table in batches.	6 years ago
Felipe Renan	31e1090138	Add block_number to token transfers ... This column is a denormalization from transactions' table.	6 years ago
Amanda Sposito	9843f2bba6	Add migration scripts that we've been using to migrate data	6 years ago
Luke Imhoff	0c4c80c880	Add missing constraints for internal transactions ... Calls didn't enforce that `call_type` and `input` was not `NULL`. Creates didn't enforce that `init` was not `NULL`.	6 years ago
Luke Imhoff	9cce7ab661	Adopt EIP-6 ... Geth has EIP-6, while Parity does not, but the Ethereum Yellow Paper includes EIP-6, so it is better to use EIP-6 when looking up community docs.	6 years ago
Felipe Renan	233d539a00	Add AddressCurrentTokenBalance schema	6 years ago
Gustavo Santos Ferreira	acf74e6267	migration to change type of field decimals	6 years ago