Vodafone - what the hell is going on? [last update 09.01.2017]

What the hell...

I'm running smokeping and MRTG to monitor ping and traffic in the jumpNet. For some time I'm experiencing ping spikes to all member networks in the jumpnet. At first I didn't bother to find out what's happening, I just wondered about it. Time passed, I added some hosts in the public internet to watch for and smokeping collected more and more data I could evaluate.

Yesterday I looked into the graphs and I was really suprised: The ping spikes are very regular: Every evening (UTC +1) between 7pm and 12pm, ping rises from around 20ms to more than 100ms in average. Along with the high ping, bandwidth drops from 50MBit/s (VDSL) to ~5MBit/s. Packet loss increases and temporarily hits 5%.

Graphs

All graphs show smokeping results to "denic.de".

The first graph shows a single ping spike. As you can see, it's starting at 7pm (UTC +1) and ends at ~12pm.

Single ping spike

The second graph shows the last 10 days. It's evident that this is a regular phenomenon:

Last 10 days

The last graph shows the last year. Unfortunatelly you can see, that there is a trend: Ping goes up, so does packet loss. You also see, that there must have been a change at the end of February: Ping rose and packet loss begun to occur on a regular basis. Funny thing along the way: Have a look at the interval from August to September: Almost no packet loss and low ping. Why? Summer holidays in Germany - less internet usage ;-)

Last 365 days

tracepath

ruben@jn:~$ tracepath denic.de
 1?: [LOCALHOST]                                         pmtu 1500
 1:  localhost                                             5.105ms
 1:  localhost                                             0.411ms
 2:  localhost                                             0.968ms
 3:  dslb-088-066-040-003.088.066.pools.vodafone-ip.de     0.872ms pmtu 1492
 3:  dslb-088-066-032-001.088.066.pools.vodafone-ip.de    21.464ms
 4:  188.111.169.14                                       21.083ms
 5:  88.79.16.4                                           20.800ms
 6:  92.79.212.217                                        22.412ms asymm  4
 7:  145.254.2.179                                        26.645ms asymm  4
 8:  145.254.2.179                                        25.759ms asymm  4
 9:  ae0-405.fra10.core-backbone.com                      24.713ms asymm  5
10:  ae1-2031.muc10.core-backbone.com                     32.091ms asymm  8
11:  cisco-m-xxxii-te0-0-1-2-406.space.net                34.241ms asymm  5
12:  no reply
13:  cisco-f-x-te0-0-2-2.space.net                        31.578ms asymm  5
14:  denic-router.denic.de                                31.446ms !H
     Resume: pmtu 1492
ruben@jn:~$ tracepath -n denic.de
 1?: [LOCALHOST]                                         pmtu 1500
 1:  10.10.1.1                                             0.339ms
 1:  10.10.1.1                                             0.320ms
 2:  172.16.1.1                                            0.859ms
 3:  88.66.40.3                                            0.785ms pmtu 1492
 3:  88.66.32.1                                          100.774ms
 4:  188.111.169.14                                      327.497ms
 5:  88.79.16.12                                          21.707ms
 6:  92.79.212.217                                        22.421ms asymm  4
 7:  145.254.2.195                                        26.093ms asymm  4
 8:  145.254.2.195                                        25.799ms asymm  4
 9:  80.255.14.221                                        25.259ms asymm  5
10:  80.255.15.42                                         31.129ms asymm  8
11:  81.95.2.234                                          36.258ms asymm  5
12:  no reply
13:  195.30.3.202                                         36.067ms asymm  5
14:  195.30.32.91                                        169.661ms !H
     Resume: pmtu 1492

Okay, what's the cause?

As this is a weird phenomenon, I had a look at my router: Nothing irregular to see in the logs. Hm. Okay, let's try to google..."vodafone hoher ping abends" which translates to "vodafone high ping evening".

Okay, I see. Around 2700 results on Google. Nice one. My favourites:

Conclusion

Let's recap:

  • it's regular
  • you clearly see human behavior in the graphs: high ping and packet loss every evening, on holidays it's less severe
  • it's trending to higher pings and higher packet loss

My conlusion from the observations: The Vodafone network is overbooked and/or has a bottleneck. In some forums users report that they've been told by Vodafone that the interconnect (peering?) to the "Deutsche Telekom" is underdesigned and "Deutsche Telekom" refuses to increase bandwidth.

I think I'll have a call with Vodafone tomorrow...

Semi official annoucement from Vodafone

A Vodafone employer tells the background story:

https://forum.vodafone.de/t5/St%C3%B6rungsmeldungen-Internet-TV/Hilfe-bei-VDSL-Performanceproblemen/td-p/1341687

Calls with vodafone

Today (04.01.2017) I called Vodafone's support hotline 0800 / 172 1212. The women was kind but not helpful at all. Although, the problem I described her is the same problem many Vodafone customers have, she insited on doing speed checks when the problem occurs so Vodafone has ticket numbers they can refer to.

I tried explaining to her that doing these speed checks is a complete waste of time because the problem is known and must be looked for in Vodafone's backbone network or peering point. She repeled my explanation and continued insisting on doing speed checks.

Eventually, she gave me two different phone numbers to call, maybe there I would have more luck with my technical reasoning she added.

0800 / 50 54 484 (number to call after I did the required speed checks)
0800 / 172 12 01 (technical hotline)

Okay, I'll give it a try.

Is it fixed?

For now, it seems the problem has been fixed by Vodafone:

last days latency

Go back