141 Commits (cd6d981502382a1620d07f70afc4121952d146a2)

Author	SHA1	Message	Date
Ayrat Badykov	b70b0bdeca	add index to value_fetched_at	6 years ago
zachdaniel	48f363e0c6	feat: decoding candidates for unverified contracts	6 years ago
Ayrat Badykov	b2ec433aa8	add index to blocks' consensus	6 years ago
Ayrat Badykov	81587c4c51	mix format	6 years ago
Ayrat Badykov	cb4bfc2cad	remove duplicated indexes	6 years ago
zachdaniel	99f7858d0b	fix: constructor_arguments must be type `text`	6 years ago
Ayrat Badykov	59e1b997f8	Rename 20190228102650_add_index_created_contract_address_has.exs to 20190228102650_add_index_created_contract_address_hash.exs	6 years ago
Ayrat Badykov	6d5b9510c1	add index for created_contract_address_hash in transactions	6 years ago
zachdaniel	8b46986c88	feat: show an approximation of transaction time	6 years ago
Ayrat Badykov	a9a5935e19	add indexes for block_reward query	6 years ago
Andrew Cravenho	fc41be0798	format	6 years ago
Andrew Cravenho	72e38aab39	Remove up/0 and down/0 add alter to change/0	6 years ago
Andrew Cravenho	f31ed167d1	Update change to down	6 years ago
Andrew Cravenho	c9ae5fc8e7	Fix constructor arguments migration	6 years ago
Ayrat Badykov	5191085052	add indexes required for blocks runner	6 years ago
Ayrat Badykov	caa40cc017	add index on block number to address token balances	6 years ago
Ayrat Badykov	6925f0c34e	add index for address_current_token_balances	6 years ago
Ayrat Badykov	ff0db8c32b	add unique index on address_hash, block_hash and address_type for block_rewards	6 years ago
Ayrat Badykov	ee814636b5	pass external libraries to the Solidity compiler	6 years ago
Ayrat Badykov	da9d6e5f31	add constructor arguments field to smart contract	6 years ago
Ayrat Badykov	8bfe0a33f8	use bytecode field instead of deployedBytecode in compiler output	6 years ago
Ayrat Badykov	7e8ad096dd	compile smart contracts with external libraries	6 years ago
Andrew Cravenho	4a5bf27060	Only add confirmed dropped/replaced txs to temp table	6 years ago
Andrew Cravenho	c0ac367960	Fix block_hash is null statement	6 years ago
Andrew Cravenho	f71228884c	Update migration script for dropped/replaced transactions	6 years ago
Ayrat Badykov	359b2608fe	fix dialyzer	6 years ago
Ayrat Badykov	d3c7c8636f	add replaced_transaction update script	6 years ago
Ayrat Badykov	b780876b3c	fix migration	6 years ago
Ayrat Badykov	915d053dd5	update replaced transaction on import	6 years ago
Ayrat Badykov	0497a9daa7	mix format	6 years ago
Ayrat Badykov	fb5d4b20d9	modify status and error db constraints	6 years ago
Luke Imhoff	14d7c9badd	Block.SecondDegreeRelation needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `nephew_relations` on `Explorer.Chain.Block`. Ecto expected the Explorer.Chain.Block.SecondDegreeRelation schema to have at least one primary key field ``` It was a previous bug that was hidden that Block.SecondDegreeRelation did not have primary key.	6 years ago
Luke Imhoff	82813d39a5	Tokens needs primary key to be preloaded ... ecto_sql 3.0.6 added a check to enforce this. ``` cannot preload through association `token` on `Explorer.Chain.Address`. Ecto expected the Explorer.Chain.Token schema to have at least one primary key field ``` It was a previous bug that was hidden that Token did not have primary key.	6 years ago
Luke Imhoff	a89fe692a7	Make insert_address_current_token_balances_in_batches only insert new ... The previous version of `insert_address_current_token_balances_in_batches` could only be run once because it assumed that `address_current_token_balances` rows did not exist. I could have added an `ON CONFLICT` to to allow `insert_address_current`insert_address_current` to be rerun, but that would have attempted to rebuild the whole table, which would be very costly as the the complex join and aggregate calculations would need to be done for all rows even if they eventually conflict and we do `ON CONFLICT DO NOTHING`. Instead, `insert_address_current_token_balances_in_batches` searched for `address_token_balances` with `(address_hash, token_contract_address_hash)` pairs that should have an `address_current_token_balance` because the `address_token_balances` `value` is non-`NULL`. From this set of `address_token_balances_requiring_current`, the set of `address_current_token_balances` is removed, leaving only those `address_token_balances_without_required_current`, which are those affected by #1345. The rest of the script is rewritten to follow the convention from the new scripts like `update_new_tokens_holder_count_in_batches`, which include more logging with timing informations and estimates.	6 years ago
Ayrat Badykov	9a30c0c850	add index	6 years ago
Luke Imhoff	28695f3d75	Data migration to populate tokens holder_count ... `update_new_tokens_holder_count_in_batches` will `UPDATE` any `tokens` where `holders_count` is `NULL` will have it set using the same query as `Explorer.Chain.count_token_holders_from_token_hash/1`, so after `holders_count` is filled in, it will just be a faster version of `Explorer.Chain.count_token_holders_from_token_hash/1`. `holder_counts` is updated with deltas by `Explorer.Chain.Import.Runner.{Block, Address.CurrentTokenBalance}` as `address_current_token_balances` is changed in a way that would change `Explorer.Chain.count_token_holders_from_token_hash/1` without the need to run the expensive `COUNT` query again.	6 years ago
Luke Imhoff	20f50d0dd4	Token holder_count ... `Token` `holder_count` stores `Explorer.Chain.count_token_holders_from_token_hash`, so it does not need to be recalculate for all runs. The `holder_count` is `nil` and must be initialized with a data migration. Since we know `count_token_holders_from_token_hash` across all tokens takes ~90 minutes the `token.holder_count` will be checked first, but if it is `nil`, `count_token_holders_from_token_hash` will still be called, which means the code can be deployed and the data migrations run in the background without blocking further deploys.	6 years ago
Ayrat Badykov	2e94a0d2a6	return InternalTransaction.Fetcher	6 years ago
Ayrat Badykov	9852abae95	add catchup code fetcher	6 years ago
Luke Imhoff	8a1c526cf8	Optimize update_address_current_token_balances_in_batches ... Fixes #1335 1. Calculate only the `address_current_token_balances` that need to be updated in the temporary table instead of all `address_current_token_balances`. 2. Add lot of `RAISE NOTICE` to print 1. Start Times 2. End Times 3. Elapsed Times 4. Estimate Time of Completion 3. Only update address_current_token_balances if the block number is less than the temporary tables value, in case other transactions occurred to fix the value while the script is running.	6 years ago
Luke Imhoff	99e6947d62	Update `address_current_token_balances` that are out-of-date ... Fixes #1317 Some bug in the past causes `address_current_token_balances` to not always be up-to-date. The verifier below show the number of incorrect `address_current_token_balances` not growing, so the bug itself is fixed, but the data is still wrong and need to be updated. Run `apps/explorer/priv/repo/migrations/scripts/update_address_current_token_balances_in_batches.sql` to fix any `address_current_token_balances` that have the incorrect `block_number` or `value`. Run the following to verify everything is fixed: ```sql SELECT COUNT(*) FROM (SELECT 1 FROM address_token_balances INNER JOIN address_current_token_balances ON address_current_token_balances.address_hash = address_token_balances.address_hash AND address_current_token_balances.token_contract_address_hash = address_token_balances.token_contract_address_hash GROUP BY address_token_balances.address_hash, address_token_balances.token_contract_address_hash, address_current_token_balances.block_number HAVING MAX(address_token_balances.block_number) != address_current_token_balances.block_number) AS address_incorrect_token_balances; ``` Which should return `0`. after the script is run. WARNING: the check SQL above will take 15-20 minutes on Eth Mainnet.	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
William Sanches	2b48235898	Remove opcodes from CodeCompiler ... Really old versions of solidity compiler, e.g. v0.1.3, have different output schemas. This data was never used by our application, so instead of having to deal with different outputs, we just don't try to get this item anymore.	6 years ago
William Sanches	3ff530c22e	Update JS solidity compiler ... The old lib would produce errors with solidity versions newer than v0.5.0. This will allow more recent codes to be verified.	6 years ago
Gustavo Santos Ferreira	7549b8f8c8	prepare configs and new dependencies for the retriever	6 years ago
Gustavo Santos Ferreira	d563a2568a	add metadata column to address_names	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
William Sanches	a754a72390	Add block_reward import	6 years ago
William Sanches	23bd2b0049	Rename block_rewards to emission_rewards	6 years ago