Fix key issues and remove storing triggered state

I removed triggered state because how it was functioning before would cause potentially some conflicts with manual pagerduty resolutions. For example, if a node broke, it would save true into triggered, and then only trigger again if it was set to false. It could only be set to false if the node came back up, which, when the script ran again, false would be saved into triggered. However, if the alert was accidentally resolved and the node was still broken, no alert would come up as triggered would still have true. The save state was mainly to prevent multiple posts, but as there is the dedup key and the program doesn't stall if there is no alert to resolve, there's no real reason to have it.

scripts/pagerduty.sh

@@ -1,14 +1,14 @@
 #!/usr/bin/env bash
 
-# Pagerduty constants
+### Log variables
 LOGFILE="/home/tmp_log/log-20190628.153354/*.log"
 LOG=$(tail -n 1000 $LOGFILE)
-SHARD_NUM=$(echo "$LOG" | grep -oE -m 1 "Shard\":.?" | rev | cut -c 1)
+
+### POST constants
 IP=$(dig -4 @resolver1.opendns.com ANY myip.opendns.com +short)
-KEY="5d11f6173899423689b6fc98691930de"
+SHARD_NUM=$(echo "$LOG" | grep -oE -m 1 "Shard\":.?" | rev | cut -c 1)
 TRIGGER="\
 {\
-  \"routing_key\": \"$KEY\",\
   \"event_action\": \"trigger\",\
   \"dedup_key\": \"$IP\",\
   \"payload\": {\
@@ -19,39 +19,25 @@ TRIGGER="\
 }"
 RESOLVE="\
 {\
-  \"routing_key\": \"$KEY\",\
   \"event_action\": \"resolve\",\
   \"dedup_key\": \"$IP\"\
 }"
-JSON_HEADER="Content-Type: application/json"
+KEY="${HMY_PAGERDUTY_KEY}"
+HEADER="X-Routing-Key: $KEY"
 POSTURL="https://events.pagerduty.com/v2/enqueue"
 
-# Validation check constants
+### Validation check variables
-if [[ -f "triggered" ]]
-  then
-    triggered=$(< triggered)
-  else
-    triggered=false
-fi
 validation=$(echo "$LOG" | tac | grep -ai -m 1 "bingo\|hooray")
 timestamp=$(echo $validation | cut -f 3 -d 't' | cut -c 4-22 | tr T \ )
 latest=$(date -d "$timestamp" +%s)
 curtime=$(date +%s)
 delay=60
 
-# Pagerduty alert
+### Pagerduty
-if [[ $(($curtime - $latest)) -gt $delay ]]
+if [[ $(($curtime - $latest)) -gt $delay ]]; then
-  then
+    # Trigger an alert
-    cat "$TRIGGER"
+    trigger=curl -X POST -H "$HEADER" -d "$TRIGGER" "$POSTURL"
-    curl -X POST -H "$JSON_HEADER" -d "$TRIGGER" "$POSTURL"
+else
-    echo
+    # Resolve an alert
-    triggered=true
+    resolve=curl -X POST -H "$HEADER" -d "$RESOLVE" "$POSTURL"
-  else
-    if [[ $triggered = true ]]
-      then
-        curl -X POST -H "$JSON_HEADER" -d "$RESOLVE" "$POSTURL"
-	      echo
-       triggered=false
-    fi
 fi
-echo "$triggered" > triggered