160 Commits (667b0675a64ec5bb82ab8e7a9a7b2820926e60be)

Author	SHA1	Message	Date
Ayrat Badykov	be1623b07e	fix System.get_env("BLOCK_COUNT_CACHE_TTL") type	6 years ago
saneery	7aafc8914f	fix test	6 years ago
Victor Baranov	df2228e7ae	Fix mix format test	6 years ago
saneery	7bf6759646	add cache for block counter	6 years ago
Ayrat Badykov	da2c79de7c	make coinmarketcap's number of pages configurable	6 years ago
saneery	5acda9df98	add staking pools importer	6 years ago
zachdaniel	8c1bb01fb1	feat: Add RSK support	6 years ago
Ayrat Badykov	02f3d922fe	use seconds	6 years ago
Ayrat Badykov	6d21e8529e	fix converting to integer	6 years ago
Ayrat Badykov	dd2dbbfddb	allow setting update interval for addresses with balances counter	6 years ago
Ayrat Badykov	245a19121f	replace source_module	6 years ago
zachdaniel	b02383a80b	feat: store average block time in a genserver	6 years ago
Luke Imhoff	8ffbe6418d	Use COUNT(*) of current_token_balances to eliminate TokenHoldersCounter ... `TokenHolderCounter` was created 20 days before the current_token_balances table and wasn't redesigned to make use of `current_token_balances`. With `current_token_balances` in place, the problematic query over `token_balances` that necessitated `TokenHoldersCounter` can use `current_token_balances`. Using `current_token_balances` with `COUNT(*)` is when tested on eth60-test, the new query for `Explorer.Chain.count_token_holders_from_token_hash/1` still took 11 seconds. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:04:26] 1 row retrieved starting from 1 in 11 s 554 ms (execution: 11 s 463 ms, fetching: 91 ms) ``` Using `EXPLAIN`, it should this was using a full sequential scan of table, so like in the other `COUNT(*)` optimizations PRs, I added an index to enable an index scan. Running the migration to add the index took only 1 minute on eth60-test: ``` 2019-01-02T08:40:16.220 application=ecto_sql [info] == Running 20190102141900 Explorer.Repo.Migrations.IndexCurrentTokenHolders.change/0 forward 2019-01-02T08:40:16.220 application=ecto_sql [info] create index address_current_token_balances_token_contract_address_hash_index 2019-01-02T08:41:13.746 application=ecto_sql [info] == Migrated 20190102141900 in 57.5s ``` After the migration ran, Index Scan was enabled and the query took only 837ms. ``` sql> SELECT COUNT(*) FROM address_current_token_balances WHERE token_contract_address_hash = decode('f230b790e05390fc8295f4d3f60332c93bed42e2', 'hex') AND address_hash != decode('0000000000000000000000000000000000000000', 'hex') AND value > 0 [2019-01-02 08:42:44] 1 row retrieved starting from 1 in 866 ms (execution: 837 ms, fetching: 29 ms) ```	6 years ago
fvictorio	616658adbf	Show token price in USD	6 years ago
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Luke Imhoff	f67241cfc0	Use COUNT(*) for count_token_transfers_from_token_hash ... By using `COUNT(*)` instead of `COUNT(column)`, Postgres optimizes the count to use the row count recorded in the index. The ETS counter appeared necessary because in 95de1ebcb0, `COUNT(id)` was being used (95de1ebcb0/apps/explorer/lib/explorer/chain/token_transfer.ex (L138)), which seems reasonable in Ecto as you need some column to count, but in Postgres, it is more efficient to count everything with `*` as it implies "count rows". `COUNT(column)` requires Postgres check if any of the values are `NULL` and it does not optimize the query even if the column is `NOT NULL` in most cases like when the `id` was used. Testing on eth60-test database, `COUNT(*)` took 62ms while `COUNT(column)` (in this case `amount` since the table no longer has `id) took 16s demonstrating the vast speed improvement of `*` counting. ``` sql> SELECT COUNT(*) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:02] 1 row retrieved starting from 1 in 70 ms (execution: 62 ms, fetching: 8 ms) sql> SELECT COUNT(amount) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:43] 1 row retrieved starting from 1 in 16 s 862 ms (execution: 16 s 855 ms, fetching: 7 ms) ```	6 years ago
Luke Imhoff	81179f9459	Use :timer functions to make timeout values clearer	6 years ago
Luke Imhoff	3c0adace16	Include import_id and transaction_id in logger metadata ... When `import_id` and `transaction_id` was added in Logger calls, they were not added to the metadata allowed to be shown and so weren't appearing in the logs.	6 years ago
Gustavo Santos Ferreira	f2cbf37200	genserver that executes validator data imports every 24 hours	6 years ago
Gustavo Santos Ferreira	7549b8f8c8	prepare configs and new dependencies for the retriever	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
Luke Imhoff	65197f5913	Add shrunk to metadata ... Allow separating shrunk vs unshrunk to monitor how often shrinking occurs.	6 years ago
Luke Imhoff	2569d2595e	Move missing_block_range_count and missing_block_count out of prose ... Being in metadata will allow it to be graphed by tools.	6 years ago
Luke Imhoff	1af8f89001	Move step out of prose and into metadata	6 years ago
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Luke Imhoff	c7d614e2ec	Include date in addition to time in log format	6 years ago
Amanda Sposito	7eef405a36	Add BlockValidationCounter config to disable it in test env.	6 years ago
Amanda Sposito	845b399281	Add a cache to count the addresses with balance > 0.	6 years ago
Locke	ad606e4df4	Conform to code review. ... Only enable Explorer.ExchangeRates when it's needed. Also changed the required store based on the app needed. IE: BlockScoutWeb tests didn't care about the store but Explorer tests did.	6 years ago
Lokraan	582345cfb7	Create tests for new dependency injection model.	6 years ago
Lokraan	2a603be370	Merge with master.	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
Sebastian Abondano	1bdb8c1166	Ganache ignores internal transactions ... Why: * We were using the `EthereumJSONRPC.Geth` variant for Ganache but we recently learned that it doesn't support the `tracer` option for [debug_traceTransaction](https://github.com/ethereum/go-ethereum/wiki/Management-APIs#debug_tracetransaction). This `tracer` option is what we currently use to fetch internal transactions for Geth. This was the cause of errors as shown on issue 1152 (link below). * Issue link: https://github.com/poanetwork/blockscout/issues/1152 This change addresses the need by: * Creating `EthereumJSONRPC.Ganache`. It ignores fetching beneficiaries, internal transactions, and pending transactions. * Editing ganache config to use the `EthereumJSONRPC.Ganache` variant instead of `EthereumJSONRPC.Geth`. * Updating inaccurate doc in `EthereumJSONRPC.Geth`. It previously said fetching internal transactions was not supported for Geth which is inaccurate.	6 years ago
Andrew Cravenho	2cb7f4b0c7	Update README and change default variant to ganache	6 years ago
Andrew Cravenho	c1d1ad996c	Change the sokol node to localhost	6 years ago
Andrew Cravenho	72480f2389	Remove old sokol node from prod env	6 years ago
Andrew Cravenho	0595c8cbaa	Replace dev sokol node with personal node	6 years ago
Felipe Renan	43b55ef688	Log tokens that gave errors interacting with their functions	6 years ago
Felipe Renan	0fbda6c72f	Add retry fetching while fetching Token's fields	6 years ago
fvictorio	73f70745dd	Fix loading of secrets file for test	6 years ago
Konstantin Zolotarev	99b70a1b65	Revert dev settings	6 years ago
Konstantin Zolotarev	c1d1385abb	Added docker files	6 years ago
Konstantin Zolotarev	a9fd475df4	Unified config variables for all JSON RPC variants	6 years ago
Gustavo Santos Ferreira	1b988eeb83	allow to change source and supply via config	6 years ago
Sebastian Abondano	1cbaafd894	Adds log rotation ... Why: * To limit the total size of the logs retained while still allowing analysis of recent events. * Issue link: n/a This change addresses the need by: * Editing the logger backends to do file rotation every 50mb and keep the last 19 log files per logger backend. 50mb * 20 = ~1gb	6 years ago
Gustavo Santos Ferreira	92cf06ef3e	module to calculate market cap	6 years ago