Call Failures in Europe, United States and Australia

Dear Customer,

The incident has now been fully resolved in all regions.
On 3 December, starting around 21:00 GMT, a network loop occurred in our USA region. This loop escalated into a service-impacting event that was brought under control around midnight. The initial loop created technical issues (“hanging threads”) in several parts of our network, which later caused additional loops and memory overload.

Because of this, users experienced service problems at different times and with varying severity, depending on their region and when they were making calls. On average, about 10% of total traffic was impacted, though some customers did see higher impact at some part of the day.

Impact Windows by Region (GMT)
Australia: 3 Dec, 21:00 – 4 Dec, 03:00
United States: 3 Dec, 21:30 – 4 Dec, 00:00
Germany: 4 Dec, 08:00 – 14:00
Netherlands: 4 Dec, 08:00 – 14:00
Poland: 4 Dec, 14:00 – 15:00

Additional Australia Event (Unrelated)

Separately, on 4 December from 08:00 to 13:00, we detected a VPLS connection degradation in Australia. This issue affected audio on already-established calls. Our regional VPLS providers confirmed intermittent performance drops during this time. This issue was not related to the main incident.

We have completed all corrective procedures to prevent further network loops and are continuing to closely monitor VPLS performance.
A full root cause analysis will begin tomorrow after confirming 12 hours of service stability.

We sincerely apologize for the inconvenience caused.

If you have any follow-up questions, please reach out to your usual points of contact at 1GLOBAL.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact on voice services.

We have implemented additional corrective actions on our network, and we can see that the situation has improved dramatically in Australia.

We continue to exhaustively test call scenarios and correct some edge case scenarios where we can see some spontaneous failures are still occurring very intermittently.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact on voice services.

We continue to see great improvements overall.

Australia is still experiencing issues where users can experience calls without audio intermittently.
We are engaging our local partner to troubleshoot this behaviour with the highest priority.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact on voice services.

Mitigation actions have been deployed, and we see a significant improvement across all metrics, across all impacted geographies.

Services are expected to start recovering gradually; nevertheless, this incident is still ongoing, and some intermittent failures still occur.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact on voice services.

We have identified the root cause of the instability on our network, and we are working together with our vendor to mitigate the issue.

From our metrics and proactive tests, remotely and on the ground, we can see with a high confidence level that the failures are now predominantly in Australia, Hong Kong, and Germany.

The impacts remain; customers may perceive dropped calls and/or be unable to make or receive calls.
We strongly suggest not changing any device configurations since the service should recover as we recover service on our network.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

The 1GLOBAL Major Incident Team, along with our vendor, continues to execute preventive actions within our network and we can already see a significant positive impact in calls and registration events in the last 15 minutes.

We are continuing to monitor network activity by the minute.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

The 1GLOBAL Major Incident Team is currently applying mitigation actions with our vendor.

Impacts that you may experience include being unable to make or receive calls temporarily and dropping calls.

All of these reported behaviours are intermittent and not constant. The normal behaviour from your device should be to connect the call via 2G/3G if calls via Voice over LTE and Voice over WiFi are not available, thus limiting the impact.

In geographies where there’s no 2G/3G availability, which is the case in Australia, the impact is more prominent.

Since the root cause of this behaviour is on our network, we do not recommend changing settings on your device.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

Following up on this voice incident, we can see that this impact is predominantly in Voice over WiFi and Voice over LTE technologies.

We have engaged our vendor to join our incident room and do a joint troubleshooting session.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

Following up on this voice incident, we can see again some call failures happening in our network despite our attempts to mitigate the impact.

The 1GLOBAL Major Incident Team continues to troubleshoot and assess new actions to solve this incident.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

Investigation continues with our vendor on the impact on our network that affects voice service in Australia.

The extension of impact has decreased over the last hours, but there are still some failed calls, hence we will continue to have this incident open.

We will share more information as soon as there are new developments.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact on our network that affects voice service.

We can confirm that all geographies, except Australia, have been fully recovered and are stable for the last hour.

To troubleshoot our infrastructure in Australia, we have engaged our vendor, who is currently troubleshooting with the team.

As soon as we have any progress, we will update this incident.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact of voice calls.

The 1GLOBAL Major Incident Team has implemented several mitigation actions and protective measures across our impacted nodes, and we can already see major improvements across all geographies.

We continue to follow up on this and expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact of voice calls.

We can confirm that this is impacting Australia as well, and the behaviour is predominant in Voice over WiFi and Voice over LTE calls.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are continuing to follow up on the impact of voice calls.

The 1GLOBAL Major Incident Team has identified the root cause, and it is currently implementing mitigation actions to solve this disruption.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team

Dear Customer,

We are currently experiencing issues in our network, causing voice call failures.

Customers may experience dropped calls and difficulties in making and/or receiving calls.

The 1GLOBAL major incident team is currently troubleshooting this behaviour and working towards stabilising these services as soon as possible. The root cause points to a loop in our Voice core network, leading to capacity exhaustion.

We expect to provide an update as soon as we have relevant information to share.

Thank you,
1GLOBAL Major Incident Team