Customer Impact: Web Security cloud customers might have observed a loss of connectivity for a brief period while traffic was re-routed to an alternate data center. During the failover period, localized web-content may not have been available.
Current Status & Next Steps: We have implemented the necessary remediation to restore the health of impacted services and traffic has been reintroduced to Sao Paulo.
We are currently monitoring our environment for stability and will provide a final update once we have confirmation on the same.
Next update time: 22:00 UTC
Posted Sep 29, 2024 - 21:13 UTC
Update
Customer Impact: Customer Impact: Web Security Cloud customers may have observed a loss of connectivity when connecting to our Sao Paulo (Cluster P) datacenter until we completed the failover. Customers that have a redundant IPSec/GRE connection would have automatically connected to an alternate datacenter and should not be impacted by this incident.
Current Status & Next Steps: We have identified the problem to be related with a primary infrastructure component in the datacenter. We continue to actively work towards a resolution and will provide more updates shortly.
Next update time: 22:00 UTC
Posted Sep 29, 2024 - 21:00 UTC
Update
Customer Impact: Customer Impact: Web Security Cloud customers may have observed a loss of connectivity when connecting to our Sao Paulo (Cluster P) datacenter until we completed the failover. Customers that have a redundant IPSec/GRE connection would have automatically connected to an alternate datacenter and should not be impacted by this incident.
Current Status & Next Steps: We have identified that the main network hardware experienced an issue, and we are currently working towards remediation of service.
Next update time: 21:00 UTC
Posted Sep 29, 2024 - 19:55 UTC
Identified
Customer Impact: Customer Impact: Web Security Cloud customers may observe a loss of connectivity when connecting to our Sao Paulo (Cluster P) datacenter. Customers that have a redundant IPSec/GRE connection would have automatically connected to an alternate datacenter and should not be impacted by this incident.
Current Status & Next Steps: We have completed the failover process and customers connectivity should be restored via an alternate datacenter. We have identified that the main network hardware experienced an issue, and we are currently working towards remediation of service.
Next update time: 20:00 UTC
Posted Sep 29, 2024 - 19:00 UTC
Investigating
Customer Impact: Web Security Cloud customers may observe a loss of connectivity when connecting to our Sao Paulo (Cluster P) datacenter. Customers that have a redundant IPSec/GRE connection would have automatically connected to an alternate datacenter and should not be impacted by this incident.
Description: We are currently experiencing unavailability of service in Sao Paulo (Cluster P). We are initiating failover services to an alternate datacenter as the automatic failover did not occur. We are investigating the problem and will provide further update as make progress.
Next update time: 19:30 UTC
Posted Sep 29, 2024 - 18:38 UTC
This incident affected: Supporting Service (IPsec Advanced Connectivity, GRE Connectivity), Services (Web Security Cloud), and Data Centers and PoP (São Paulo (P)).