Hetzner being blocked by some US ISPs? | PlexGuide.com

Hetzner being blocked by some US ISPs?

  • Stop using Chrome! Download the Brave Browser via >>> [Brave.com]
    It's a forked version of Chrome with native ad-blockers and Google's spyware stripped out! Download for Mac, Windows, Android, and Linux!
Welcome to the PlexGuide.com
Serving the Community since 2016!
Register Now

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Are routes to Hetzner being blocked by ISPs in the US?

First, I am a comcast Xfinty internet customer.

Starting about 4 days ago, Plex started getting choppy and would no longer play from home through Comcast. If I use a VPN, then I have no issues.

Using different tracert websites, I am seeing many locations with 100% loss to Hetzner.

Anyway, here is some details:


This is the route to Hetzner's website:

ROUTE on COMCAST from SEATTLE
Tracing route to dedihetznernew.your-server.de [78.47.166.55]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.0.1
2 14 ms 19 ms 16 ms [REMOVED}
3 56 ms 57 ms 54 ms [REMOVED}
4 67 ms 71 ms 64 ms be-103-ar01.seattle.wa.seattle.comcast.net [69.139.164.77]
5 69 ms 73 ms 75 ms lag-39.ear2.seattle1.level3.net [4.68.37.129]
6 * * * Request timed out.

7 148 ms 154 ms 148 ms ae0-3356.lon10.core-backbone.com [212.113.8.42]
8 207 ms 211 ms 216 ms ae18-2025.fra20.core-backbone.com [5.56.17.25]
9 234 ms 238 ms 238 ms core-backbone.hetzner.com [80.255.15.122]
10 234 ms 235 ms 244 ms core11.nbg1.hetzner.com [213.239.224.237]
11 181 ms 194 ms 184 ms ddos-mitigation.dc1.nbg1.hetzner.com [88.198.247.253]
12 182 ms 184 ms 222 ms ex9k2.dc1.nbg1.hetzner.com [213.239.203.214]
13 183 ms 226 ms 225 ms dedihetznernew.your-server.de [78.47.166.55]


Then I used: tools.keycdn.com/tracert

Route From Miami:
Start: 2020-01-08T16:55:08+0000
Loss Snt Last Avg Best Wrst StDev
1.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
2.|-- 108.61.249.33 0.0% 4 0.9 2.2 0.6 6.4 2.8
3.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
4.|-- 129.250.207.169 0.0% 4 1.1 2.7 0.5 8.8 4.0
5.|-- 129.250.3.40 0.0% 4 109.1 108.8 108.5 109.1 0.3
6.|-- 129.250.3.149 0.0% 4 0.8 1.1 0.7 1.9 0.6
7.|-- 129.250.2.86 0.0% 4 30.6 34.2 30.6 42.1 5.4
8.|-- 129.250.3.84 50.0% 4 30.7 31.5 30.7 32.4 1.2
9.|-- 129.250.4.97 0.0% 4 111.9 112.1 111.8 113.0 0.6
10.|-- 129.250.2.101 0.0% 4 116.1 115.9 115.7 116.1 0.2
11.|-- 213.198.82.130 0.0% 4 112.2 112.2 112.1 112.4 0.1
12.|-- 213.239.252.26 0.0% 4 115.3 115.4 115.0 116.4 0.6
13.|-- 88.198.247.253 0.0% 4 123.9 124.3 123.7 125.9 1.0
14.|-- 213.239.203.214 0.0% 4 123.8 124.1 123.8 124.8 0.5
15.|-- 78.47.166.55 0.0% 4 123.8 123.8 123.8 123.8 0.0


From DALLAS:

Start: 2020-01-08T16:55:08+0000
Loss Snt Last Avg Best Wrst StDev
1.|-- 45.79.12.201 0.0% 4 0.7 1.1 0.7 2.0 0.6
2.|-- 45.79.12.4 0.0% 4 0.6 2.9 0.6 9.4 4.3
3.|-- 62.115.172.134 0.0% 4 1.0 1.2 1.0 1.9 0.5
4.|-- 62.115.120.112 25.0% 4 19.5 22.7 19.3 29.3 5.7
5.|-- 62.115.125.190 50.0% 4 117.1 116.9 116.8 117.1 0.2
6.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
7.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0

8.|-- 62.115.114.91 0.0% 4 116.8 116.8 116.7 117.0 0.1
9.|-- 213.248.70.3 0.0% 4 122.3 121.1 120.5 122.3 0.9
10.|-- 213.239.224.233 0.0% 4 123.5 123.5 123.3 123.9 0.3
11.|-- 88.198.247.253 0.0% 4 123.6 123.6 123.6 123.9 0.1
12.|-- 213.239.203.214 0.0% 4 123.3 123.7 123.3 124.6 0.6
13.|-- 78.47.166.55 0.0% 4 122.6 122.5 122.5 122.6 0.0


From Seattle:

Start: 2020-01-08T16:55:08+0000
Loss Snt Last Avg Best Wrst StDev
1.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
2.|-- 45.63.33.225 0.0% 4 6.6 18.4 6.6 24.8 8.1
3.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
4.|-- 129.250.207.201 0.0% 4 0.5 0.5 0.3 0.8 0.2
5.|-- 129.250.5.133 0.0% 4 142.0 142.0 141.9 142.0 0.0
6.|-- 129.250.2.44 0.0% 4 1.3 1.4 0.5 3.1 1.2
7.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0
8.|-- ??? 100.0 4 0.0 0.0 0.0 0.0 0.0

9.|-- 129.250.6.55 0.0% 4 140.4 142.5 140.3 148.9 4.3
10.|-- 129.250.2.85 0.0% 4 137.1 137.1 137.1 137.2 0.0
11.|-- 213.198.82.130 0.0% 4 139.6 140.6 139.4 143.6 2.0
12.|-- 213.239.252.26 0.0% 4 144.8 144.8 144.8 144.8 0.0
13.|-- 88.198.247.253 0.0% 4 147.2 146.2 144.8 147.8 1.6
14.|-- 213.239.203.214 0.0% 4 144.8 145.8 144.8 147.5 1.2
15.|-- 78.47.166.55 0.0% 4 144.8 144.9 144.8 144.9 0.1


Is anyone else experiencing the same thing?


Thanks
 
  • Thinking
Reactions: 1 user

UncleBuck

Governer
Staff
Dec 21, 2018
253
82
The only problems I have seen as of late point to google api errors as the cause. I can't imagine an ISP blocking hetzner but you could be experiencing a peering issue. Many in the US have bad peering to hetzner servers and I think it's especially bad in the south west. Are you using CloudFlare's CDN?
 
  • Like
Reactions: 1 user

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
I didn't think an ISP could block them either, but after looking at the data from tracerts around the globe, I think they are being blocked by multiple locations. Its weird that packets drop 100% at multiple locations.....or somehow they are added to some type of bad ip list and systems are dropping that ip block.
 

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Oh.....well I guess ICMP can be dropped at the router too.......
 

Admin9705

Administrator
Project Manager
Donor
Jan 17, 2018
5,151
2,120
Ya never had that one but good insight
 

mackattack57

Experienced
Apr 14, 2019
63
15
I have experienced the same thing. Being new to this, I had not identified the issue causing the choppy playback. However, I thought I was getting a subtitle error so I checked the logs. I changed subtitle settings. It started playing again fine at first. Then, I started getting choppy playback. Next, I changed CDN to DNS only. That didn't help but when I enabled proxy again, playback was fine. Tonight, I started with issues again and came here to research. I believe @Admin9705 is correct with API issues but I don't know how to test that theory.

I am on if anyone knows what I need to do to test and can provide feedback.
 

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Who is your ISP? Can you do a tracert to www.hetzner.de and post the results?

If you go to https://tools.keycdn.com/traceroute and enter: hetzner.de you will get a tracert from 14 locations.

On about half of them, you will see 100% packet loss somewhere. This just started happening a few days ago.

To get my video working again, I had to setup a local Plex server on my home network.

If anyone else is having this issue, please do a tracert and post the results.


Thanks
 

mackattack57

Experienced
Apr 14, 2019
63
15
New York had one with 100% loss. Miami had three with 100% loss. Seattle had three at 100% and one at 25%. Dallas had one at 100%, one at 50% and one at 25%.

That is pretty bad.

I live out in the middle of nowhere but somehow right next to a cell tower so I have cellular data running my house. It has not been an issue on my hetzner server until a few days ago too.

Now, I get random times where my connections are terrible.

I already had a local server setup on automation for 720p local content when I don't want to eat up bandwidth. Hetzner is for hitting the road or delivery of stuff at night.

Interesting find on the traceroute to see what is happening. I may put in a ticket with Hetzner and see if they know what is going on here.
 

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Please let us know if you do put in a ticket to Hetzner!
 

Edrock200

MVP
Staff
Nov 17, 2019
761
270
Theres a few things you can try to narrow the issue. Not sure if you answered UncleBuck's question, but it's an important one, are you using cloudflare's CDN for plex? Try toggling it off or on in the clouflare dash and test playback.

I don't think it's any of these things since you mentioned VPN seems to resolve the issue, so you are probably on the right track with routing priority changes at the ISP level. I doubt they are blocking/filtering/throttling, but have a peering point issue that they either haven't realized yet or haven't fixed yet. But here's some things you can try:

API issues could cause choppy playback as well if you've got some settings on and are generating too many like thumbnail generation, deep file analysis in plex/radarr/sonarr or embedded subtitle checks turned on in bazarr, etc.

Open netdata in a browser and ssh into your box. Copy a large media file (~1GB+) from your gdrive mount to the local file sytem (i.e. /home/username). While the file is copying watch the inbound data, which is essentially from gdrive to your box. It will probably fluctuate somewhat but should remain fairly steady until the copy is done with occassional spikes and dips, but it should drop to zero for a prolonged period of time, if it does, it could be an API issue. Your google developer console will also give you API hit stats.

Next grab winscp or some ssh file transfer utility that shows you realtime metrics. Now pull the file you just copied to the local HD on your PG box down to your local USA PC and note netdata's outbound meter and keep an eye on it for the same things you did above.

Last, install the firefox container in PG, go to the firefox.yourdomain.com, and in that firefox go to http://plex:32400/web, then play something. It won't be pretty because it's basically screenscraping through VNC, but you should be able to tell if it's actually pausing during playback or if it's playing continuosly "locally" at the dedi (meaning it's not an API or bandwidth issue to Google itself.)

One other thing to consider is primetime shows have finally started up after the recent break. Make sure you have appropriately throttled your rclone upload bandwidth in the settings, because if you are watching shows in the evening while PG is grabbing new stuff, that upload bandwidth is competing with your streaming traffic. Same goes for downloads, make sure sabnzbd/syncthing/torrent clients have bandwidth throttling enabled, I generally set mine to 50% of line rate, because those downloads are competing with your PG box pulling data from google drive, not to mention the disk i/o & CPU overhead as well.
 
Last edited:
  • Love
  • Like
Reactions: 1 users

deva5610

Experienced
Donor
May 1, 2019
53
52
If your packet is getting to the end destination then you're not getting 100% packet loss. Routers can be set to not respond to ICMP traffic, so on those single hops you will see 100% loss, but your packet still continues on. That isn't and won't present an issue, it just means the router is busy routing instead of answering pings.

It's the same as if an intermediate hop shows some packet loss. As long as it isn't carrying through the traceroute from that point, and the final host shows no loss everything is fine. Again - It just means the router is routing instead of allocating resources to answering pings.

The Miami trace for example - Hop 3 shows some packets not being answered, which is just that the router was too busy to bother answering some traffic, but then you see hops 4-7 show no problem. Hop 8 is a router configured to not respond to ICMP requests. Again though hops 9 to the all important destination have no issues.

Your Comcast traceroute is the same. Now I'm not saying there isn't an issue with Comcast and Hetzner, but those traceroutes aren't it, they're perfectly normal.
 
  • Like
Reactions: 1 user

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
A little follow up:

First, thanks for all the responses and great ideas, but like I said in the first post, if I connect via a VPN, then there is no issue. So from that, there is no issue with the server, or google, or cloudflare etc.

I contacted comcast customer support, which kept sending me to someone else.
I contacted Hetzner's customer support who were extremely helpful, and answered every reply almost instantly. They are amazing. But the problem is Comcast peering to level3 and I have no idea how to get to the right technical people to look into the issue.


The original tracert's were first with ICMP, then TCP, then UDP.

I was finally able to see the 6th hop, to london1.level3.net.

Look at the response times. Its broken.

3.|-- 96.110.249.185 10.0% 10 14.3 14.7 10.1 19.9 2.9
4.|-- be-103-ar01.seattle.wa.seattle.comcast.net 0.0% 10 14.8 16.8 12.4 22.6 3.2
5.|-- lag-39.ear2.seattle1.level3.net 80.0% 10 14.0 15.7 14.0 17.3 2.4
6.|-- ae-1-3113.ear2.london1.level3.net 90.0% 10 7500. 7500. 7500. 7500. 0.0
7.|-- ae0-3356.lon10.core-backbone.com 0.0% 10 153.4 154.2 150.3 160.6 2.9
8.|-- ae18-2025.fra20.core-backbone.com 0.0% 10 170.3 168.9 163.3 177.3 4.5
9.|-- core-backbone.hetzner.com 0.0% 10 174.3 181.0 173.1 228.7 17.0
10.|-- core11.nbg1.hetzner.com 0.0% 10 197.6 183.3 176.6 197.6 6.8
11.|-- ddos-mitigation.dc1.nbg1.hetzner.com 0.0% 10 182.7 179.8 176.2 183.8 2.4
12.|-- ex9k2.dc1.nbg1.hetzner.com 0.0% 10 178.0 179.8 174.9 188.4 4.0

I did do a google search on the level3 routes above and see people have had issues with this for over 2 years.

www.reddit.com/r/networking/comments/664g9l/level_3_support_options_comcast_hand_off_packet/
https://github.com/pbek/QOwnNotes/issues/1245
https://www.nwnravenloft.com/forum/index.php?topic=50563.0

I also found this:
https://www.abuseipdb.com/check/78.47.166.55

This really is a pain in the a**!


OPTIONS:

1. Keep using my home Plex to watch everything, when away from home, use the remote one.
2. Setup a VPN on my router, for everything at home.
3. Dump monopoly Comcast. I wish I could, but no other high speed internet here.
4. Get a Comcast engineer to fix the problem. ahahahahah right!

If anyone has worked with Comcast on technical routing issues and knows who to contact, PLEASE let me know.

Thanks
Dave
 

Edrock200

MVP
Staff
Nov 17, 2019
761
270
If your using cloudflare and you toggle proxy on in cloudflare for your Plex entry (make sure to disable caching in your page rules) it will use cloudflares cdn for routing and may use an alternate route around Comcast.

That said, given the tests you've done, I agree with you, seems like a peering issue.
 
  • Like
Reactions: 1 user

deva5610

Experienced
Donor
May 1, 2019
53
52
Again, it's not broken. Look at your last hop, that's the all important one. It doesn't matter if intermediate routers take longer to respond to the request sent to them, as long as your final hop is fine.

If you ping your destination hop and see packet loss or massive ping spikes then there is an issue. If you see intermediate loss in a traceroute but it doesn't carry on from there - it's not an issue.

The first couple of responses in that Reddit thread say the exact same thing.


Specifically pages around 35/36.

Again - I'm not saying there isn't a Comcast issue, and your VPN tests show that there probably is - Those traceroutes don't though.
 
  • Like
Reactions: 1 user

DeadPool

Elite
Staff
May 2, 2018
227
78
people..
i just wanted to say what a BEAUTIFUL exchange of ideas and camaraderie for the common good in this thread.
huge thumbs up.
dP

(edit: apologies though, for being unable to contribute here)
 
Last edited:
  • Like
Reactions: 1 user

Edrock200

MVP
Staff
Nov 17, 2019
761
270
I agree, I love this community and PG! I went on a shootout of all the media boxes out there (quickbox, openflixr, etc) and PG is hands down my favorite, and the community is part of that analysis. That's not to put down those other efforts by any means, just my 2 cents. Sorry for the mini thread-jack!
 

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Sure, but if you look at the london.level3.net, the packet return times are over 7500....
Post automatically merged:

Well, had to quit wasting time on it. I tried about everything possible.

There is something wrong with the route.

I am now using a VPN to route all plex from my home network to my server. With it, I have no issues.
Post automatically merged:

More people having the same problem:
https://forums.xfinity.com/t5/Email...connecting-to-Hetzner-robot-your/td-p/3297473
https://forums.xfinity.com/t5/Your-Home-Network/Accessing-some-sites-in-Germany-is-slow/td-p/3287229
https://hostedtalk.net/t/hetzner-falkenstein-packet-loss-to-texas-usa/3451
 
Last edited:

deva5610

Experienced
Donor
May 1, 2019
53
52
That's true, but then from that point on everything is normal. That's exactly how routers can be configured to operate. They deprioritise responding to pings/traceroutes to actually route packets. So while it looks like an issue in isolation, it actually isn't.

If every other hop from that point on also had that added latency then there's definitely an issue involving that router, but because everything is clear afterwards it's just the router doing normal task prioritisation.

The fact when you connect via a VPN you have no issues shows that there's definitely something sub optimal happening between Comcast and Hetzner (or Comcast/Cloudflare if proxying). From those traceroutes though there's no real info to pinpoint where the weakness is.

If you aren't proxying Plex through Cloudflare then I definitely suggest giving it a go, the guide is here. When proxying Plex the most important things are making sure you get the Page Rules correct (Bypassing the Cache) and following the steps in 3A.

In Aus the difference for me between not proxying and proxying on my 100Mbps connection is 15Mbps versus 70Mbps when doing a speedtest.

If you are already proxying - try switching it off. The issue could be between Cloudflare and Comcast.
 
  • Like
Reactions: 1 user

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
Thanks, I will try the Cloudflare Proxy route and see if that works!
 

zzdave

Citizen
Original poster
Donor
Dec 14, 2018
13
2
I finally had the time to get back to this, and thanks everyone for pushing me to use Cloudflare as the proxy/cache. This did work.

So many other people have posted issues with Comcast and other Piers connecting to Hetzner. If I do not use Cloudflare, I get 500-100% packet loss and stuff does not work. If I use a VPN, everything works fine.

There is definitely something happening through Comcast and its piers getting to Hetzner. No idea why or what. Maybe a net neutrality issue and they are offloading the traffic, or someone stealing the bandwidth to download their private porn collection...... :)

Anyway, for me, Cloudflare is the solution, since the traffic now goes through Cloudflare and they are handling the routes. I am sure, if they were seeing the same drops as me, it would be easy for them to re-route their traffic.....
 

vinceira

Citizen
Mar 29, 2020
11
2
Hi zzdave,

I activated Cloudflare but I don't have the impression that it does not improve the speed. I download at 1 Mega ...

I have a server at hetzner and it worked properly for months and for 2 months nothing ...
I did a WinMTR test and apparently I have peering when I arrive in France, that's why I wanted to go through Cloudflare.

Sorry for my rusty English, I'm French.
 

Recommend NewsGroups

      Up To a 58% Discount!

Trending