I just wanted to follow up here and give the promised details.
For a period of ~39 minutes (18:35 - 19:14 PDT), most webhooks weren't firing due to RAM being exhausted on one of our machines. We were able to fix the problem last night and restore services pretty quickly, but we will be increasing the RAM for that process in the very near future.
Posted Sep 15, 2016 - 09:24 PDT
Resolved
We're closing this incident out after monitoring the system for a few hours. Webhooks are and have been firing properly for some time.
Posted Sep 14, 2016 - 22:41 PDT
Monitoring
We've rebooted a machine and we see web hooks firing again. We'll continue to monitor the situation.
Posted Sep 14, 2016 - 19:27 PDT
Identified
Our system is not sending webhooks and hasn't been for the last 15-20 minutes or so (exact details to follow). We've identified the problem and we're working on an interim fix to happen shortly.