Vodafone - what the hell is going on? [last update 09.01.2017]
What the hell...
I'm running smokeping and MRTG to monitor ping and traffic in the jumpNet. For some time I'm experiencing ping spikes to all member networks in the jumpnet. At first I didn't bother to find out what's happening, I just wondered about it. Time passed, I added some hosts in the public internet to watch for and smokeping collected more and more data I could evaluate.
Yesterday I looked into the graphs and I was really suprised: The ping spikes are very regular: Every evening (UTC +1) between 7pm and 12pm, ping rises from around 20ms to more than 100ms in average. Along with the high ping, bandwidth drops from 50MBit/s (VDSL) to ~5MBit/s. Packet loss increases and temporarily hits 5%.
All graphs show smokeping results to "denic.de".
The first graph shows a single ping spike. As you can see, it's starting at 7pm (UTC +1) and ends at ~12pm.
The second graph shows the last 10 days. It's evident that this is a regular phenomenon:
The last graph shows the last year. Unfortunatelly you can see, that there is a trend: Ping goes up, so does packet loss. You also see, that there must have been a change at the end of February: Ping rose and packet loss begun to occur on a regular basis. Funny thing along the way: Have a look at the interval from August to September: Almost no packet loss and low ping. Why? Summer holidays in Germany - less internet usage ;-)
ruben@jn:~$ tracepath denic.de 1?: [LOCALHOST] pmtu 1500 1: localhost 5.105ms 1: localhost 0.411ms 2: localhost 0.968ms 3: dslb-088-066-040-003.088.066.pools.vodafone-ip.de 0.872ms pmtu 1492 3: dslb-088-066-032-001.088.066.pools.vodafone-ip.de 21.464ms 4: 126.96.36.199 21.083ms 5: 188.8.131.52 20.800ms 6: 184.108.40.206 22.412ms asymm 4 7: 220.127.116.11 26.645ms asymm 4 8: 18.104.22.168 25.759ms asymm 4 9: ae0-405.fra10.core-backbone.com 24.713ms asymm 5 10: ae1-2031.muc10.core-backbone.com 32.091ms asymm 8 11: cisco-m-xxxii-te0-0-1-2-406.space.net 34.241ms asymm 5 12: no reply 13: cisco-f-x-te0-0-2-2.space.net 31.578ms asymm 5 14: denic-router.denic.de 31.446ms !H Resume: pmtu 1492
ruben@jn:~$ tracepath -n denic.de 1?: [LOCALHOST] pmtu 1500 1: 10.10.1.1 0.339ms 1: 10.10.1.1 0.320ms 2: 172.16.1.1 0.859ms 3: 22.214.171.124 0.785ms pmtu 1492 3: 126.96.36.199 100.774ms 4: 188.8.131.52 327.497ms 5: 184.108.40.206 21.707ms 6: 220.127.116.11 22.421ms asymm 4 7: 18.104.22.168 26.093ms asymm 4 8: 22.214.171.124 25.799ms asymm 4 9: 126.96.36.199 25.259ms asymm 5 10: 188.8.131.52 31.129ms asymm 8 11: 184.108.40.206 36.258ms asymm 5 12: no reply 13: 220.127.116.11 36.067ms asymm 5 14: 18.104.22.168 169.661ms !H Resume: pmtu 1492
Okay, what's the cause?
As this is a weird phenomenon, I had a look at my router: Nothing irregular to see in the logs. Hm. Okay, let's try to google..."vodafone hoher ping abends" which translates to "vodafone high ping evening".
Okay, I see. Around 2700 results on Google. Nice one. My favourites:
- Vodafone customer support forum: https://forum.vodafone.de/t5/Internet-Phone/Hoher-Ping-und-schlechte-Downrate-jeden-abend/td-p/1266915
- Vodafone cable customers are also affected: https://forum.vodafone.de/t5/St%C3%B6rungsmeldungen-Internet-TV/Seit-3-Wochen-hoher-Ping-zu-Sto%C3%9Fzeiten/td-p/1351244
- chip.de forum: http://forum.chip.de/dsl-gsm-umts-lte/abends-hohen-ping-vodafone-1843005.html
- teltarif.de https://www.teltarif.de/vodafone-lahmes-dsl-hoher-ping-abends/news/66079.html
- it's regular
- you clearly see human behavior in the graphs: high ping and packet loss every evening, on holidays it's less severe
- it's trending to higher pings and higher packet loss
My conlusion from the observations: The Vodafone network is overbooked and/or has a bottleneck. In some forums users report that they've been told by Vodafone that the interconnect (peering?) to the "Deutsche Telekom" is underdesigned and "Deutsche Telekom" refuses to increase bandwidth.
I think I'll have a call with Vodafone tomorrow...
Semi official annoucement from Vodafone
A Vodafone employer tells the background story:
Calls with vodafone
Today (04.01.2017) I called Vodafone's support hotline 0800 / 172 1212. The women was kind but not helpful at all. Although, the problem I described her is the same problem many Vodafone customers have, she insisted on doing speed checks when the problem occurs so Vodafone has ticket numbers they can refer to.
I tried explaining to her that doing these speed checks is a complete waste of time because the problem is known and must be looked for in Vodafone's backbone network or peering point. She repealed my explanation and continued insisting on doing speed checks.
Eventually, she gave me two different phone numbers to call, maybe there I would have more luck with my technical reasoning she added.
0800 / 50 54 484 (number to call after I did the required speed checks)
0800 / 172 12 01 (technical hotline)
Okay, I'll give it a try.
Is it fixed?
For now, it seems the problem has been fixed by Vodafone: