199 Commits (ec5017e2388577ebfccc9280f70e3060054caf02)

Author	SHA1	Message	Date
Luke Imhoff	903d322553	Use index and COUNT(*) for address_to_validation_count ... BlockValidationCounter is not needed once the query is switched to `COUNT(*)` and an index is added for `blocks.miner_hash`. Creating the index on eth60-test took only 12 seconds. ``` 2018-12-21T08:32:46.375 application=ecto_sql [info] == Running 20181221143000 Explorer.Repo.Migrations.CreateBlocksMinerHashIndex.change/0 forward 2018-12-21T08:32:46.375 application=ecto_sql [info] create index blocks_miner_hash_index 2018-12-21T08:32:59.211 application=ecto_sql [info] == Migrated 20181221143000 in 12.8s ``` Before the index both `COUNT(*)` and `COUNT(hash)` (the old usage before 76fc8e0377 in 92f99cac22) take > 800ms, which would have a noticable impact on the UI. ``` sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:27:43] 1 row retrieved starting from 1 in 906 ms (execution: 889 ms, fetching: 17 ms) sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:28:04] 1 row retrieved starting from 1 in 819 ms (execution: 811 ms, fetching: 8 ms) ``` After the index both queries benefited because `COUNT(hash)` was able to use Bitmap Index Scan, but `COUNT(*)` was much better using a normal Index Scan: ``` sql> SELECT COUNT(hash) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:43] 1 row retrieved starting from 1 in 776 ms (execution: 768 ms, fetching: 8 ms) sql> SELECT COUNT(*) FROM blocks WHERE blocks.miner_hash = ( SELECT blocks.miner_hash FROM blocks LIMIT 1 ) [2018-12-21 08:33:55] 1 row retrieved starting from 1 in 130 ms (execution: 120 ms, fetching: 10 ms) ``` The `SELECT` to get a miner_hash takes 54ms alone, so the `COUNT(*)` takes ~70ms, but EXPLAINs take just as long so ~50ms is my latency to eth60-test, meaning the `SELECT COUNT(*)` takes about 20ms plus latency.	6 years ago
Luke Imhoff	f67241cfc0	Use COUNT(*) for count_token_transfers_from_token_hash ... By using `COUNT(*)` instead of `COUNT(column)`, Postgres optimizes the count to use the row count recorded in the index. The ETS counter appeared necessary because in 95de1ebcb0, `COUNT(id)` was being used (95de1ebcb0/apps/explorer/lib/explorer/chain/token_transfer.ex (L138)), which seems reasonable in Ecto as you need some column to count, but in Postgres, it is more efficient to count everything with `*` as it implies "count rows". `COUNT(column)` requires Postgres check if any of the values are `NULL` and it does not optimize the query even if the column is `NOT NULL` in most cases like when the `id` was used. Testing on eth60-test database, `COUNT(*)` took 62ms while `COUNT(column)` (in this case `amount` since the table no longer has `id) took 16s demonstrating the vast speed improvement of `*` counting. ``` sql> SELECT COUNT(*) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:02] 1 row retrieved starting from 1 in 70 ms (execution: 62 ms, fetching: 8 ms) sql> SELECT COUNT(amount) FROM token_transfers WHERE token_contract_address_hash = ( SELECT token_contract_address_hash FROM token_transfers LIMIT 1) [2018-12-21 07:51:43] 1 row retrieved starting from 1 in 16 s 862 ms (execution: 16 s 855 ms, fetching: 7 ms) ```	6 years ago
Luke Imhoff	81179f9459	Use :timer functions to make timeout values clearer	6 years ago
Luke Imhoff	3c0adace16	Include import_id and transaction_id in logger metadata ... When `import_id` and `transaction_id` was added in Logger calls, they were not added to the metadata allowed to be shown and so weren't appearing in the logs.	6 years ago
Gustavo Santos Ferreira	f2cbf37200	genserver that executes validator data imports every 24 hours	6 years ago
Gustavo Santos Ferreira	7549b8f8c8	prepare configs and new dependencies for the retriever	6 years ago
Luke Imhoff	ea58dad12b	Update to Ecto 3.0	6 years ago
Luke Imhoff	65197f5913	Add shrunk to metadata ... Allow separating shrunk vs unshrunk to monitor how often shrinking occurs.	6 years ago
Luke Imhoff	2569d2595e	Move missing_block_range_count and missing_block_count out of prose ... Being in metadata will allow it to be graphed by tools.	6 years ago
Luke Imhoff	1af8f89001	Move step out of prose and into metadata	6 years ago
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Luke Imhoff	c7d614e2ec	Include date in addition to time in log format	6 years ago
Amanda Sposito	7eef405a36	Add BlockValidationCounter config to disable it in test env.	6 years ago
Amanda Sposito	845b399281	Add a cache to count the addresses with balance > 0.	6 years ago
Locke	ad606e4df4	Conform to code review. ... Only enable Explorer.ExchangeRates when it's needed. Also changed the required store based on the app needed. IE: BlockScoutWeb tests didn't care about the store but Explorer tests did.	6 years ago
Lokraan	582345cfb7	Create tests for new dependency injection model.	6 years ago
Lokraan	2a603be370	Merge with master.	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
Sebastian Abondano	1bdb8c1166	Ganache ignores internal transactions ... Why: * We were using the `EthereumJSONRPC.Geth` variant for Ganache but we recently learned that it doesn't support the `tracer` option for [debug_traceTransaction](https://github.com/ethereum/go-ethereum/wiki/Management-APIs#debug_tracetransaction). This `tracer` option is what we currently use to fetch internal transactions for Geth. This was the cause of errors as shown on issue 1152 (link below). * Issue link: https://github.com/poanetwork/blockscout/issues/1152 This change addresses the need by: * Creating `EthereumJSONRPC.Ganache`. It ignores fetching beneficiaries, internal transactions, and pending transactions. * Editing ganache config to use the `EthereumJSONRPC.Ganache` variant instead of `EthereumJSONRPC.Geth`. * Updating inaccurate doc in `EthereumJSONRPC.Geth`. It previously said fetching internal transactions was not supported for Geth which is inaccurate.	6 years ago
Andrew Cravenho	2cb7f4b0c7	Update README and change default variant to ganache	6 years ago
Andrew Cravenho	c1d1ad996c	Change the sokol node to localhost	6 years ago
Andrew Cravenho	72480f2389	Remove old sokol node from prod env	6 years ago
Andrew Cravenho	0595c8cbaa	Replace dev sokol node with personal node	6 years ago
Felipe Renan	43b55ef688	Log tokens that gave errors interacting with their functions	6 years ago
Felipe Renan	0fbda6c72f	Add retry fetching while fetching Token's fields	6 years ago
fvictorio	73f70745dd	Fix loading of secrets file for test	6 years ago
Konstantin Zolotarev	99b70a1b65	Revert dev settings	6 years ago
Konstantin Zolotarev	c1d1385abb	Added docker files	6 years ago
Konstantin Zolotarev	a9fd475df4	Unified config variables for all JSON RPC variants	6 years ago
Gustavo Santos Ferreira	1b988eeb83	allow to change source and supply via config	6 years ago
Sebastian Abondano	1cbaafd894	Adds log rotation ... Why: * To limit the total size of the logs retained while still allowing analysis of recent events. * Issue link: n/a This change addresses the need by: * Editing the logger backends to do file rotation every 50mb and keep the last 19 log files per logger backend. 50mb * 20 = ~1gb	6 years ago
Gustavo Santos Ferreira	92cf06ef3e	module to calculate market cap	6 years ago
Lucas Narciso	ea3da7fd07	Consolidate Token Holders count info ... The number of token holders is now available in the token page again. It is cached by the TokenHoldersCounter GenServer that periodically updateds an ETS with all pairs of tokens and token holders.	6 years ago
Konstantin Zolotarev	0604f7a234	Fixes for pr comments	6 years ago
Konstantin Zolotarev	f89a27adf4	Fixes for ganache integration	6 years ago
Lokraan	76fc8e0377	Store validation counts for each address in an ets table. ... Use `realtime` and `catchup` events to differentiate what the client and counter have to listen to. Store validation counts for each address in an ets table.	6 years ago
Amanda Sposito	2d02ed6e24	Add a counter to store the token tranfers count info ... Co-authored-by: Lucas Narciso <lucas.narciso@plataformatec.com.br>	6 years ago
Andrew Cravenho	9892d1bd17	Add env variables to the production parity files	6 years ago
Andrew Cravenho	3e5896396a	Add environment variables to the config files	6 years ago
Luke Imhoff	32380ba1e9	prometheus_ecto ... Ecto query duration metrics for Prometheus.	6 years ago
Andrew Cravenho	2d230686d9	add fields back and provide better docs	6 years ago
Andrew Cravenho	45bd6965e0	update readme with deployment fixes	6 years ago
Alaradia	8962aa9fa7	inculde an example of the config ... includes an example of the config in dev.secret.exs.example.	6 years ago
Luke Imhoff	9a8cdc4031	Use Path.absname so that log paths are stable during reload ... Fixes #628 The Phoenix.CodeReloader causes the file loggers to restart and reload their config, but when they reload, the current working directory has shifted from the project root to the OTP app root, so it produces log files under each OTP app (`apps/APP_NAME/logs`). We don't want this, so compute the absolute path in the config files, so that the loaded `Application` environment has an absolute path in it that can't shift when the working directory changes.	6 years ago
Igor Florian	c2ff17756c	Get TokensBalance in the Blockchain ... We're now able to get the balances from a Address given it's address_hash, block_number and token_contract_address_hash.	6 years ago
Luke Imhoff	fdd1090c45	Re-enable Ecto loggers ... Allow Ecto loggers to log to the application-specific log **file**, `log/<Mix.env>/ecto.log` but raise `:console`'s level to `:info`, so that Ecto output does not flood `iex` and make it unusable.	6 years ago