91 Commits (652c536521abd659066acafa484b1fc3a489c4fe)

Author	SHA1	Message	Date
Luke Imhoff	ac35c4680f	Move error_count out of prose and into metadata ... Having count and error_count in the metadata would allow us to graph one vs the other to see how often we achieve partial retries vs full retries.	6 years ago
Luke Imhoff	63e646a5ff	Move count out of prose and into metadata ... Record the number of things being fetched in metadata instead of in text so that the count can be graphed.	6 years ago
Luke Imhoff	2010e90a71	Move block range out of prose and into metadata ... The catchup block range `first_block_number..last_block_number` will set `Logger.metadata(first_block_number: first_block_number, last_block_number: last_block_number)` for errors in `BoundIntervalSupervisor` and tasks in `Catchup.Fetcher`.	6 years ago
Luke Imhoff	0243726f9b	Move block number out of prose and into metadata ... Realtime block number will set `Logger.metadata(block_number: ...)` for each block number is fetches separately after getting the `newHead` websocket message.	6 years ago
Luke Imhoff	fed7d304e2	Include fetcher in logger metadata ... For each fetcher set the `Logger.metadata(fetcher: ...)` to a short semantic name. For Fetchers that use `BufferedTask`, `BufferedTask` now has an option to set `metadata` passed to `Logger.metadata/1` in 1. GenServer process 2. The `c:init/3` Task 3. The `c:run/2` Tasks	6 years ago
Luke Imhoff	c7d614e2ec	Include date in addition to time in log format	6 years ago
zachdaniel	5d12613f70	feat: support tracing via spandex	6 years ago
Sebastian Abondano	1bdb8c1166	Ganache ignores internal transactions ... Why: * We were using the `EthereumJSONRPC.Geth` variant for Ganache but we recently learned that it doesn't support the `tracer` option for [debug_traceTransaction](https://github.com/ethereum/go-ethereum/wiki/Management-APIs#debug_tracetransaction). This `tracer` option is what we currently use to fetch internal transactions for Geth. This was the cause of errors as shown on issue 1152 (link below). * Issue link: https://github.com/poanetwork/blockscout/issues/1152 This change addresses the need by: * Creating `EthereumJSONRPC.Ganache`. It ignores fetching beneficiaries, internal transactions, and pending transactions. * Editing ganache config to use the `EthereumJSONRPC.Ganache` variant instead of `EthereumJSONRPC.Geth`. * Updating inaccurate doc in `EthereumJSONRPC.Geth`. It previously said fetching internal transactions was not supported for Geth which is inaccurate.	6 years ago
Andrew Cravenho	2cb7f4b0c7	Update README and change default variant to ganache	6 years ago
Andrew Cravenho	c1d1ad996c	Change the sokol node to localhost	6 years ago
Andrew Cravenho	72480f2389	Remove old sokol node from prod env	6 years ago
Andrew Cravenho	0595c8cbaa	Replace dev sokol node with personal node	6 years ago
William Sanches	3528e4dc51	Add metadata updater	6 years ago
Alex Garibay	e23ebab069	Add transform functions for block indexing	6 years ago
Konstantin Zolotarev	99b70a1b65	Revert dev settings	6 years ago
Konstantin Zolotarev	c1d1385abb	Added docker files	6 years ago
Konstantin Zolotarev	a9fd475df4	Unified config variables for all JSON RPC variants	6 years ago
Luke Imhoff	cae25195e5	Indexer.Memory.Monitor ... `Indexer.Memory.Monitor` checks if the BEAM memory usage exceeds a set limit (defaults to 1 GiB) and if it does, it asks the process with the most memory that is registered as shrinkable to shrink.	6 years ago
Sebastian Abondano	1cbaafd894	Adds log rotation ... Why: * To limit the total size of the logs retained while still allowing analysis of recent events. * Issue link: n/a This change addresses the need by: * Editing the logger backends to do file rotation every 50mb and keep the last 19 log files per logger backend. 50mb * 20 = ~1gb	6 years ago
Sebastian Abondano	2b8ceb4e32	Update block reward contract beneficiary balances ... Why: * "Parity’s consensus engine allows using a smart contract for block reward calculation." - see https://wiki.parity.io/Block-Reward-Contract for details. This means that we need to use `trace_block` (https://wiki.parity.io/JSONRPC-trace-module#trace_block) to fetch the block reward contract beneficiaries and update their balances. Currently this doesn't happen so there are addresses (block contract reward beneficiaries) with inaccurate balances and missing rows in the `address_coin_balances` table. * Issue link: https://github.com/poanetwork/blockscout/issues/767 This change addresses the need by: * Adding `EthereumJSONRPC.fetch_beneficiaries/2` to help us fetch the block reward contract beneficiaries of a given block range. This only works with Parity. * Editing Indexer's dev and prod config by adding `trace_block` config to `:method_to_url`. * Adding `block_reward_contract_beneficiaries` to `Indexer.AddressExtraction` for it to know how to get addresses in this scenario. * Editing `Indexer.Block.Fetcher.fetch_and_import_range/2` to get block reward contract beneficiaries, create their addresses if they don't yet exist, and update their coin balances.	6 years ago
Konstantin Zolotarev	0604f7a234	Fixes for pr comments	6 years ago
Konstantin Zolotarev	f89a27adf4	Fixes for ganache integration	6 years ago
Nam Chu Hoai	d17917ce30	Allow unsecure WebSocket connections ... Some setup like ours will have the WebSocket connection in a private network to a local node and thus do not require a secure connection. This change allows to specify ws:// instead of wss:// endpoints as well as reading from the ENV var for the geth config of the indexer Co-authored-by: Luke Imhoff <luke.imhoff@dockyard.com>	6 years ago
Andrew Cravenho	9892d1bd17	Add env variables to the production parity files	6 years ago
Igor Florian	99012b2c4c	Add a new log file for token_balances fetching errors	6 years ago
Luke Imhoff	9a8cdc4031	Use Path.absname so that log paths are stable during reload ... Fixes #628 The Phoenix.CodeReloader causes the file loggers to restart and reload their config, but when they reload, the current working directory has shifted from the project root to the OTP app root, so it produces log files under each OTP app (`apps/APP_NAME/logs`). We don't want this, so compute the absolute path in the config files, so that the loaded `Application` environment has an absolute path in it that can't shift when the working directory changes.	6 years ago
Luke Imhoff	fdd1090c45	Re-enable Ecto loggers ... Allow Ecto loggers to log to the application-specific log **file**, `log/<Mix.env>/ecto.log` but raise `:console`'s level to `:info`, so that Ecto output does not flood `iex` and make it unusable.	6 years ago
Luke Imhoff	c032ea2291	Improved logging ... Have separate loggers for errors and per-application. * `:console` will continue to have all applications and levels. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` * `:error` is only for errors across all environments. It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/error.log`. * `:block_scout_web` is only when `:application` metadata is `:block_scout_web`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/block_scout_web.log`. * `:ethereum_jsonrpc` is only when `:application` metadata is `:ethereum_jsonrpc`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/ethereum_jsonrpc.log`. * `:explorer` is only when `:application` metadata is `:exploer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/explorer.log`. * `:indexer` is only when `:application` metadata is `:indexer`. Level is - `:debug` for `dev` - `:warn` for `test` - `:info` for `prod` It uses `LoggerFileBackend` and writes to `logs/<Mix.env>/indexer.log`.	6 years ago
Luke Imhoff	cddd810a58	Use :websocket_client for EthereumJSONRPC.WebSocket ... :websocket_client can properly use :ssl_verify_hostname.verify_fun/3 for :verify_fun and allow for the handshake to both be secure and pass for infura.	6 years ago
Luke Imhoff	c8c53405fc	Subscribe to newHeads for realtime indexer ... Separate Catchup and Realtime indexer's fetching, so that Catchup continues to use the old polling, `EthereumJSONRPC.json_rpc` while Realtime subscribes with `EthereumJSONRPC.subscribe`. Because Realtime gets notified of new blocks through the subscription, it no longer has a Sequence, so Sequence has been removed from `%BlockFetcher{}`. There is also no block concurrency or block batch size for Realtime, so that was removed from `%BlockFetcher{}` too and moved to `%BlockFetcher.Catchup{}`. Because Realtime requires a websocket to be connected, there is now an `Indexer.BlockFetcher.Realtime.Supervisor` that supervises the `Indexer.BlockFetcher.Realtime.WebSocket` and `Indexer.BlockFetcher.Realtime`. `Indexer.BlockFetcher.Realtime` because it is waiting on notifications from is subscription becomes a `GenServer` insteado of a `Task`, so it can be supervised normally. BlockFetcher.Supervisor becomes a normal `Supervisor` of `Catchup.Supervisor` and `Realtime.Supervisor`. As Realtime is using `EthereumJSONRPC.subscribe`, `subscribe_named_arguments` has been added as a sibling of `json_rpc_named_arguments` in the `:indexer` `config`s.	6 years ago
Peter Romfeld	d9e37c44a6	fix formatting	6 years ago
Peter Romfeld	b473420df3	use ETHEREUM_JSONRPC_HTTP_URL instead	6 years ago
Peter Romfeld	2ddc1b9747	get geth_url from environment and fix `variant` for {dev,prod}.exs	6 years ago
Luke Imhoff	fe49a753d3	block_rate -> block_interval ... block_rate is measured in milliseconds / block, but a rate would blocks / unit-time, so rename block_rate to block_interval to reflect it is the interval between blocks.	6 years ago
Amanda Sposito	3335817d90	Fix prod config file to use prod config instead of dev	6 years ago
Luke Imhoff	8f827ecfa6	Use Mox by default for local integration tests ... How to run without Mox is in the root `README.md`. Run all combinations of Variant and Transport on CircleCI to ensure that Mox matches HTTP behavior, so Mox can be used for fast local testing: * Geth, HTTP * Geth, Mox * Parity, HTTP * Parity, Mox Using Mox locally also lessens load on Sokol and the Infura API key the tests use for Geth under the assumption that we do far more local testing than CI testing. May help with #340, but does not fix the root cause, which is now known as `Sequence` not chunking ranges correctly on retry.	6 years ago
Luke Imhoff	466c1e76f6	Request transport module and options for all JSONRPC calls ... To lessen load on Sokol chain nodes during testing and to allow substituting different transports than HTTP, such as IPC, in the future, pass the transport module and its options to any call to `EthereumJSONRPC.json_rpc`. This will allow us to use a `mox` module in local testing, so we only do true chain integration testing in CircleCI.	6 years ago
Luke Imhoff	6b3fd80916	Extract Explorer.Indexer to Indexer ... Resolves #304	7 years ago
Luke Imhoff	27b4bf62d5	mix new indexer --sup	7 years ago