We have restored all major services and resolved all major device connectivity issues. We are still assessing and recovering data during the database outage, and will provide further updates and as they become available. A post-mortem will be provided soon following the conclusion of this incident.
Posted Jul 15, 2020 - 18:13 PDT
Monitoring
Webhook queue delays are expected to be resolved and are expected to be back to standard response times. Photon and P1s devices are no longer experiencing degraded connectivity. All other intermittent degradation is expected to be resolved. Devices making use of System.sleep() may require additional time to recover (estimated ~25 minutes). The Web IDE continues to be unavailable; we will work with the intention of restoring Web IDE functionality in the next business day.
Posted Jul 15, 2020 - 14:07 PDT
Update
Webhooks are continuing to process, but are expected to be delayed by approximately 1.5 hours as of this time. Photons and P1 devices are currently experiencing degraded connectivity. All other devices remain impacted with intermittent failures (e.g. OTAs, connectivity). Functions/Variables appear to be working as expected for devices that are currently connected.
Posted Jul 15, 2020 - 13:03 PDT
Update
Message processing has resumed but will have a considerable backlog. Database migration is still in process and will continue to result in degraded connectivity.
Posted Jul 15, 2020 - 11:59 PDT
Update
Deployment is ongoing, and we will update once it has completed. During this time events will be received, but not processed until the process is complete.
Posted Jul 15, 2020 - 10:58 PDT
Update
During the next few minutes, a subset of devices are expected to experience connectivity issues as we execute a deployment.
Posted Jul 15, 2020 - 10:24 PDT
Update
We have partial recovery of services and are working to restore the remaining services. During this time, you may still continue to experience connectivity issues while we work to restore all services.
What is restored? API service Console access Billing services
What is still to be recovered? Webhooks Web IDE Data from the most recent backup
Posted Jul 15, 2020 - 10:00 PDT
Update
Devices are continuing to come online, but service continues to be degraded across the platform.
Posted Jul 15, 2020 - 09:15 PDT
Update
The database has been restored, but some devices are struggling to connect, we are continuing to investigate. Some services have resumed partial operation.
Posted Jul 15, 2020 - 08:59 PDT
Update
We are actively restoring our database now, and our API is gradually resuming functionality. Devices are coming online gradually. The restore will be missing data (~15 hours of data from July 14th 17:00 PST), and we will backfill as soon as our provider is back online.
Posted Jul 15, 2020 - 07:55 PDT
Update
We are continuing to work to restore services. There are no further updates at this time.
Posted Jul 15, 2020 - 06:58 PDT
Identified
Our provider has indicated a rough 1-2 hour window for the restoration of services.
Posted Jul 15, 2020 - 05:58 PDT
Update
We are continuing to investigate this issue.
Posted Jul 15, 2020 - 05:16 PDT
Update
We have identified the root cause of the outage with a third party service and we are waiting for a timeline from our provider. We are monitoring for updates on their end and are investigating a potential alternative strategy. We will provide updates regularly via the status page, and are committed to doing so every hour until the issue has been resolved.
Posted Jul 15, 2020 - 05:15 PDT
Update
Our database provider is suffering an outage, we are working with the provider to restore service as soon as possible.
Posted Jul 15, 2020 - 04:48 PDT
Investigating
We are investigating degraded performance for the Console and API.
Posted Jul 15, 2020 - 04:39 PDT
This incident affected: Cellular Connectivity, REST API, Integrations, Management Console and Customer Resources (build.particle.io).