Cellular Connectivity issue
Incident Report for Particle
Postmortem

We would like to share the postmortem summary concerning the 3 separate service incidents on December 17, 18 and 22.

Partial cellular disruption on December 17

There was a partial cellular service disruption beginning at approximately 23:03 UTC. During the service disruption, devices negotiating a new connection were unable to establish service with the Particle cloud. The issue was determined to be an outage on a server provided by our telephony partner. Devices resumed normal service at approximately 02:39 UTC. We are working with our telephony partner to better mitigate the risk through earlier detection and building more resilient service.

Webhooks service disruption on December 18

There was a service disruption of integration and webhook requests beginning 19:33 UTC. During the disruption, some webhook message traffic could not be delivered to integration endpoints. The issue was the result of network availability with AWS services and webhook integration service was restored at approximately 23:38 UTC. We are working with AWS to better understand the root cause of the service outage and will share further information when they become available.

Electron 2G/3G cellular disruption on December 22

There was a partial cellular service disruption beginning at approximately 04:17 UTC. During the disruption, Electron devices using 2G/3G cellular services were disconnected. The disruption was caused by a database outage and services were restored at approximately 06:30 UTC. We are actively working with our telephony partner to mitigate the risks and partner with them on providing earlier detection and delivering more reliable service.

Posted Dec 23, 2020 - 13:15 PST

Resolved
This incident has been resolved.
Posted Dec 21, 2020 - 23:31 PST
Monitoring
Cellular connectivity is recovering and Electrons are coming online. We shall continue to monitor for another hour.
Posted Dec 21, 2020 - 22:30 PST
Update
One of our telephony provider's 3G / 2G service has degraded. We are working with them on resolving this
Posted Dec 21, 2020 - 22:05 PST
Identified
We have identified that the impact is limited to certain devices. We are continuing to investigate this issue.
Posted Dec 21, 2020 - 21:09 PST
Investigating
We are experiencing a cellular connectivity issue; we will provide an update shortly.
Posted Dec 21, 2020 - 20:41 PST
This incident affected: Cellular Connectivity and Telephony Providers (Electron 2G (G350), Electron 3G (U260/U270), or E Series 2G/3G (E310)).