hyperlane-monorepo

The home for Hyperlane core contracts, sdk packages, and other infrastructure

blockchain hyperlane interchain monorepo

History

Mattie Conover b3a1ea5d3a RPC Error Handling (#2021 ) ### Description We have known for a while that our rules around when to retry certain calls has been a bit rough around the edges. This PR attempts to better handle a number of cases by retrying errors that are server errors and not retrying errors that we expect to have re-occur consistently on retry. I don't expect this to be perfect, but I hope it will get us about 90%+ of the way there. We can followup in the future to capture additional cases. My log investigation: [gcp link](https://console.cloud.google.com/logs/query;cursorTimestamp=2023-02-24T15:24:38.674835362Z;query=severity%3DWARNING%0AjsonPayload.target%3D~%22hyperlane_ethereum::%2528retrying%7Cfallback%2529%22%0A%2528jsonPayload.span.method%3D~%22.eth_.%22%20OR%20jsonPayload.fields.method%3D~%22.eth_.%22%2529%0A-jsonPayload.fields.text%3D%22%7B%5C%22message%5C%22:%5C%22no%20Route%20matched%20with%20those%20values%5C%22%7D%22%0A-jsonPayload.fields.error%3D~%22.connection%20closed%20before%20message%20completed.%22%0A-jsonPayload.fields.error%3D~%22.tcp%20connect%20error.%22%0A-jsonPayload.fields.text%3D%22%3Chtml%3E%5Cr%5Cn%3Chead%3E%3Ctitle%3E502%20Bad%20Gateway%3C%2Ftitle%3E%3C%2Fhead%3E%5Cr%5Cn%3Cbody%3E%5Cr%5Cn%3Ccenter%3E%3Ch1%3E502%20Bad%20Gateway%3C%2Fh1%3E%3C%2Fcenter%3E%5Cr%5Cn%3C%2Fbody%3E%5Cr%5Cn%3C%2Fhtml%3E%5Cr%5Cn%22%0A-jsonPayload.fields.error%3D%22%2528code:%20429,%20message:%20Your%20app%20has%20exceeded%20its%20compute%20units%20per%20second%20capacity.%20If%20you%20have%20retries%20enabled,%20you%20can%20safely%20ignore%20this%20message.%20If%20not,%20check%20out%20https:%2F%2Fdocs.alchemy.com%2Freference%2Fthroughput,%20data:%20None%2529%22%0A-jsonPayload.fields.text%3D%22default%20backend%20-%20404%22%0A-jsonPayload.fields.text%3D%22Bad%20Gateway%22%0A-jsonPayload.fields.error%3D~%22.Connection%20reset%20by%20peer%20%2528os%20error%20104%2529%22%0A-jsonPayload.fields.text%3D%22API%20call%20rejected%20because%20chain%20is%20not%20done%20bootstrapping%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32601,%20message:%20the%20method%20eth_feeHistory%20does%20not%20exist%2Fis%20not%20available,%20data:%20None%2529%22%0A-jsonPayload.target%3D%22hyperlane_ethereum::fallback%22%0A-jsonPayload.fields.error%3D~%22operation%20timed%20out%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20404%20Not%20Found:%20%7B%5C%22jsonrpc%5C%22:%5C%222.0%5C%22,%5C%22error%5C%22:%7B%5C%22code%5C%22:404,%5C%22message%5C%22:%5C%22arb1-sequencer%20rate%20limit%20hit.%20%20Try%20again%201%20minute%5C%22%7D%7D,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20429%20Too%20Many%20Requests:%20%7B%5C%22jsonrpc%5C%22:%5C%222.0%5C%22,%5C%22error%5C%22:%7B%5C%22code%5C%22:429,%5C%22message%5C%22:%5C%22Public%20RPC%20Rate%20Limit%20Hit,%20limit%20will%20reset%20in%2060%20seconds%5C%22%7D%7D,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D~%22unable%20to%20get%20local%20issuer%20certificate%22%0A-jsonPayload.fields.error%3D%22EOF%20while%20parsing%20a%20value%20at%20line%201%20column%200%22%0A-jsonPayload.fields.text%3D%22internal%20service%20failure%5Cn%22%0A-jsonPayload.fields.text%3D%22%3Chtml%3E%5Cr%5Cn%3Chead%3E%3Ctitle%3E503%20Service%20Temporarily%20Unavailable%3C%2Ftitle%3E%3C%2Fhead%3E%5Cr%5Cn%3Cbody%3E%5Cr%5Cn%3Ccenter%3E%3Ch1%3E503%20Service%20Temporarily%20Unavailable%3C%2Fh1%3E%3C%2Fcenter%3E%5Cr%5Cn%3Chr%3E%3Ccenter%3Enginx%3C%2Fcenter%3E%5Cr%5Cn%3C%2Fbody%3E%5Cr%5Cn%3C%2Fhtml%3E%5Cr%5Cn%22%0A-jsonPayload.fields.text%3D%22%3Chtml%3E%5Cr%5Cn%3Chead%3E%3Ctitle%3E502%20Bad%20Gateway%3C%2Ftitle%3E%3C%2Fhead%3E%5Cr%5Cn%3Cbody%3E%5Cr%5Cn%3Ccenter%3E%3Ch1%3E502%20Bad%20Gateway%3C%2Fh1%3E%3C%2Fcenter%3E%5Cr%5Cn%3Chr%3E%3Ccenter%3Enginx%3C%2Fcenter%3E%5Cr%5Cn%3C%2Fbody%3E%5Cr%5Cn%3C%2Fhtml%3E%5Cr%5Cn%22%0A-jsonPayload.fields.error%3D~%22error%20trying%20to%20connect:%20error:14094410:SSL%20routines:ssl3_read_bytes:sslv3%20alert%20handshake%20failure:..%2Fssl%2Frecord%2Frec_layer_s3.c:1543:SSL%20alert%20number%2040%22%0A-jsonPayload.fields.text%3D~%22The%20gateway%20cannot%20get%20a%20response,%20please%20try%20again%20or%20contact%20the%20administrator%22%0A-jsonPayload.fields.error%3D~%22Connection%20reset%20by%20peer%20%5C%2528os%20error%20104%5C%2529%22%0A-jsonPayload.fields.text%3D%22%3Chtml%3E%5Cr%5Cn%3Chead%3E%3Ctitle%3E502%20Bad%20Gateway%3C%2Ftitle%3E%3C%2Fhead%3E%5Cr%5Cn%3Cbody%3E%5Cr%5Cn%3Ccenter%3E%3Ch1%3E502%20Bad%20Gateway%3C%2Fh1%3E%3C%2Fcenter%3E%5Cr%5Cn%3Chr%3E%3Ccenter%3Ecloudflare%3C%2Fcenter%3E%5Cr%5Cn%3C%2Fbody%3E%5Cr%5Cn%3C%2Fhtml%3E%5Cr%5Cn%22%0A-jsonPayload.fields.text%3D%22%7B%5C%22jsonrpc%5C%22:%5C%222.0%5C%22,%5C%22error%5C%22:%7B%5C%22code%5C%22:0,%5C%22message%5C%22:%5C%22we%20can't%20execute%20this%20request%5C%22%7D,%5C%22id%5C%22:null%7D%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20transaction%20underpriced,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20replacement%20transaction%20underpriced,%20data:%20None%2529%22%0A-jsonPayload.fields.text%3D%22%7B%5C%22jsonrpc%5C%22:%5C%222.0%5C%22,%5C%22result%5C%22:%7B%5C%22code%5C%22:429,%5C%22message%5C%22:%5C%22Total%20number%20of%20requests%20exceeded.%20Want%20higher%20rate%20limit%3F%20Contact%20us%20at%20sales@gateway.fm%5C%22%7D,%5C%22id%5C%22:%5C%22%5C%22%7D%22%0A-jsonPayload.fields.text%3D%22%3Chtml%3E%5Cr%5Cn%3Chead%3E%3Ctitle%3E502%20Bad%20Gateway%3C%2Ftitle%3E%3C%2Fhead%3E%5Cr%5Cn%3Cbody%3E%5Cr%5Cn%3Ccenter%3E%3Ch1%3E502%20Bad%20Gateway%3C%2Fh1%3E%3C%2Fcenter%3E%5Cr%5Cn%3Chr%3E%3Ccenter%3Enginx%2F1.20.2%3C%2Fcenter%3E%5Cr%5Cn%3C%2Fbody%3E%5Cr%5Cn%3C%2Fhtml%3E%5Cr%5Cn%22%0A-jsonPayload.fields.text%3D~%22504%20ERROR%22%0A-jsonPayload.fields.text%3D%22Gateway%20Timeout%22%0A-jsonPayload.fields.text%3D%22404%20page%20not%20found%5Cn%22%0A-jsonPayload.fields.error%3D~%22error%20trying%20to%20connect:%20unexpected%20EOF%22%0A-jsonPayload.fields.error%3D~%22execution%20reverted:%20No%20router%20enrolled%20for%20domain.%20Did%20you%20specify%20the%20right%20domain%20ID%3F%22%0A-jsonPayload.fields.error%3D~%22VM%20Exception%20while%20processing%20transaction:%20revert%20No%20router%20enrolled%20for%20domain.%20Did%20you%20specify%20the%20right%20domain%20ID%3F%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20invalid%20transaction:%20nonce%20too%20low,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20already%20known,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32010,%20message:%20AlreadyKnown,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D~%22message:%20nonce%20too%20low%22%0A-jsonPayload.fields.error%3D~%22message:%20execution%20reverted:%20delivered%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32603,%20message:%20already%20known,%20data:%20None%2529%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32000,%20message:%20ALREADY_EXISTS:%20already%20known,%20data:%20None%2529%22%0A-jsonPayload.fields.text%3D%22%7B%5Cn%20%20%5C%22message%5C%22:%5C%22An%20invalid%20response%20was%20received%20from%20the%20upstream%20server%5C%22%5Cn%7D%22%0A-jsonPayload.fields.error%3D%22%2528code:%20-32601,%20message:%20Method%20not%20found,%20data:%20Some%2528Object%20%7B%5C%22method%5C%22:%20String%2528%5C%22'eth_getLogs'%20is%20not%20available%20on%20our%20public%20API.%20Head%20over%20to%20https:%2F%2Fdocs.blastapi.io%2Fblast-documentation%2Ftutorials-and-guides%2Fusing-blast-to-get-a-blockchain-endpoint%20for%20more%20information%5C%22%2529%7D%2529%2529%22%0A-jsonPayload.fields.error%3D~%22insufficient%20funds%22%0A-jsonPayload.fields.text%3D%22%7B%5C%22jsonrpc%5C%22:%5C%222.0%5C%22,%5C%22result%5C%22:%7B%5C%22code%5C%22:429,%5C%22message%5C%22:%5C%22Total%20number%20of%20requests%20exceeded%5C%22%7D,%5C%22id%5C%22:%5C%22%5C%22%7D%22%0A-jsonPayload.fields.text%3D~%22data%20not%20instance%20of%20model%22;summaryFields=jsonPayload%252Ffields%252Ftext,jsonPayload%252Fspan%252Fmethod,jsonPayload%252Ffields%252Ferror:false:32:beginning;timeRange=2023-02-16T06:56:50.422Z%2F2023-03-23T20:56:50.422Z?project=abacus-labs-dev) Since that will eventually rot I am also listing my query here: ``` severity=WARNING jsonPayload.target=~"hyperlane_ethereum::(retrying\|fallback)" (jsonPayload.span.method=~".eth_." OR jsonPayload.fields.method=~".eth_.") -jsonPayload.fields.text="{\"message\":\"no Route matched with those values\"}" -jsonPayload.fields.error=~".connection closed before message completed." -jsonPayload.fields.error=~".tcp connect error." -jsonPayload.fields.text="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n</body>\r\n</html>\r\n" -jsonPayload.fields.error="(code: 429, message: Your app has exceeded its compute units per second capacity. If you have retries enabled, you can safely ignore this message. If not, check out https://docs.alchemy.com/reference/throughput, data: None)" -jsonPayload.fields.text="default backend - 404" -jsonPayload.fields.text="Bad Gateway" -jsonPayload.fields.error=~".Connection reset by peer (os error 104)" -jsonPayload.fields.text="API call rejected because chain is not done bootstrapping" -jsonPayload.fields.error="(code: -32601, message: the method eth_feeHistory does not exist/is not available, data: None)" -jsonPayload.target="hyperlane_ethereum::fallback" -jsonPayload.fields.error=~"operation timed out" -jsonPayload.fields.error="(code: -32000, message: 404 Not Found: {\"jsonrpc\":\"2.0\",\"error\":{\"code\":404,\"message\":\"arb1-sequencer rate limit hit. Try again 1 minute\"}}, data: None)" -jsonPayload.fields.error="(code: -32000, message: 429 Too Many Requests: {\"jsonrpc\":\"2.0\",\"error\":{\"code\":429,\"message\":\"Public RPC Rate Limit Hit, limit will reset in 60 seconds\"}}, data: None)" -jsonPayload.fields.error=~"unable to get local issuer certificate" -jsonPayload.fields.error="EOF while parsing a value at line 1 column 0" -jsonPayload.fields.text="internal service failure\n" -jsonPayload.fields.text="<html>\r\n<head><title>503 Service Temporarily Unavailable</title></head>\r\n<body>\r\n<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n" -jsonPayload.fields.text="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n" -jsonPayload.fields.error=~"error trying to connect: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1543:SSL alert number 40" -jsonPayload.fields.text=~"The gateway cannot get a response, please try again or contact the administrator" -jsonPayload.fields.error=~"Connection reset by peer \(os error 104\)" -jsonPayload.fields.text="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>cloudflare</center>\r\n</body>\r\n</html>\r\n" -jsonPayload.fields.text="{\"jsonrpc\":\"2.0\",\"error\":{\"code\":0,\"message\":\"we can't execute this request\"},\"id\":null}" -jsonPayload.fields.error="(code: -32000, message: transaction underpriced, data: None)" -jsonPayload.fields.error="(code: -32000, message: replacement transaction underpriced, data: None)" -jsonPayload.fields.text="{\"jsonrpc\":\"2.0\",\"result\":{\"code\":429,\"message\":\"Total number of requests exceeded. Want higher rate limit? Contact us at sales@gateway.fm\"},\"id\":\"\"}" -jsonPayload.fields.text="<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.20.2</center>\r\n</body>\r\n</html>\r\n" -jsonPayload.fields.text=~"504 ERROR" -jsonPayload.fields.text="Gateway Timeout" -jsonPayload.fields.text="404 page not found\n" -jsonPayload.fields.error=~"error trying to connect: unexpected EOF" -jsonPayload.fields.error=~"execution reverted: No router enrolled for domain. Did you specify the right domain ID?" -jsonPayload.fields.error=~"VM Exception while processing transaction: revert No router enrolled for domain. Did you specify the right domain ID?" -jsonPayload.fields.error="(code: -32000, message: invalid transaction: nonce too low, data: None)" -jsonPayload.fields.error="(code: -32000, message: already known, data: None)" -jsonPayload.fields.error="(code: -32010, message: AlreadyKnown, data: None)" -jsonPayload.fields.error=~"message: nonce too low" -jsonPayload.fields.error=~"message: execution reverted: delivered" -jsonPayload.fields.error="(code: -32603, message: already known, data: None)" -jsonPayload.fields.error="(code: -32000, message: ALREADY_EXISTS: already known, data: None)" -jsonPayload.fields.text="{\n \"message\":\"An invalid response was received from the upstream server\"\n}" -jsonPayload.fields.error="(code: -32601, message: Method not found, data: Some(Object {\"method\": String(\"'eth_getLogs' is not available on our public API. Head over to https://docs.blastapi.io/blast-documentation/tutorials-and-guides/using-blast-to-get-a-blockchain-endpoint for more information\")}))" -jsonPayload.fields.error=~"insufficient funds" -jsonPayload.fields.text="{\"jsonrpc\":\"2.0\",\"result\":{\"code\":429,\"message\":\"Total number of requests exceeded\"},\"id\":\"\"}" -jsonPayload.fields.text=~"data not instance of model" ``` Which managed to categorize errors over a large time range and I used to verify that my logic is sound. ### Drive-by changes - Moves code logic to avoid duplication ### Related issues - Fixes #1944 - Fixes hyperlane-xyz/issues#412 - Fixes hyperlane-xyz/issues#414 ### Backward compatibility _Are these changes backward compatible?_ Yes _Are there any infrastructure implications, e.g. changes that would prohibit deploying older commits using this infra tooling?_ None ### Testing _What kind of testing have these changes undergone?_ Manual		2 years ago
..
abis	Scaffolding for multiple ISM types in the relayer (#2028 )	2 years ago
src	RPC Error Handling (#2021 )	2 years ago
tests	Regenerate test fixtures and unskip hardhat tests (#2035 )	2 years ago
.gitignore	Finalize rename from abacus->hyperlane (#1312 )	2 years ago
Cargo.toml	Better rust package management (#1975 )	2 years ago
build.rs	Fuel Validators (#1474 )	2 years ago