Discussion Finally added PG Blitz - Should PG Move still be rate limited?

matt314159

Senior Member
Original poster
Donor
Local time
8:35 PM
Nov 29, 2018
114
21
PG Version
8.7.5
Mount Type
Blitz Unencrypted
Server Type
Remote - Dedicated Server
After happily using PG Move since February on my Hetzner EX-42-NVME I finally bit the bullet last night and set up a team drive, then deployed PG Blitz. I was super intimidated, but it turned out to be very simple. Hat's off to Admin for what appears to be a tremendous amount of work making it so easy creating the service accounts and all that. Wow!

it's all working fine, PGblitz uploads into the tdrive as quickly as I can download. Paused Queues look like they should now be a thing of the past. One thing I noticed though, is that whenever it does load into the gdrive it still rate limits it to 10MB/s. Is that normal, or is that a vestige of my old transport setup that I can change? I figured rclone would just know not to upload more than 750GB per account.

Second question: it says in the setup that 6 service accounts is realistic for most people, but would my setup benefit at all from upping that to 10 or more? I can't see a situation where I'd ever want to upload more than 5TB in a day but maybe to have maximum headroom just in case? Or would that cause issues? I figure my EX-42 is probably more beefy than many others (i7-7700, 64GB, 2x512GB NVME SSDs) so maybe optimal settings are not quite the same as the general settings for most?

Anyway thanks again for all the work! I'm super impressed.
 

Nigel

Senior Member
Staff
Donor
Local time
3:35 AM
Jan 20, 2018
135
40
The Netherlands
The ex-42 is used by alot of others. That is why is alot out of stock. Alot of pgblitz people use it. I would defenitely do a higher number for headspace.
 

matt314159

Senior Member
Original poster
Donor
Local time
8:35 PM
Nov 29, 2018
114
21
Sorry I should have come back to update this OP--when I first deployed it Blitz seemed like when rclone was transferring into the tdrive it was full speed (90+MB/s) and then when moving into the gdrive it was 10MB/s---but I believe I was mistaken, or I happened to see it before the rate limiter kicked in. Now, regardless of whether it goes into the tdrive or gdrive, it's about 56MB/s--and that 56MB/s makes sense, because with 6 service accounts + 1 gdrive stream giving me about 5TB/day, that's the rate at which you could upload 24/7 and achieve 5TB.

Now, I've been working on getting a hefty library of 4k remuxes built now that I have a way to play those extremely well. I'm queueing up 2-5TB at a time and letting nzbget just chew through it.

Is there a point at which account abuses with my gsuite become a threat for just dumping multiple TB's per day into it? I don't think it should take me much longer than another week or two to get my UHD library built out and then the activity pattern will be down to normal levels where it's pretty average but right now I'm adding like 30-50 movies a day.
 

matt314159

Senior Member
Original poster
Donor
Local time
8:35 PM
Nov 29, 2018
114
21
I think this reply will get merged up into my other one so I'll tag you @Nigel --I added a few more service accounts today and redeployed.

Does Blitz still throttle to 60MB/s total no matter what? It sure seems to be doing that. I recognize I need to avoid saturating my upload to allow my users some bandwidth, but even if it were like 80MB/s I feel like that would leave plenty of room for my users' streams. It seems like Blitz is uploading 1 file at about 30MB/s, 2 at 15MB/s each, etc, always keeping that overall around 58-60MB/s.

If that's normal, then cool. Maybe that's just the fastest my server can upload to google. But I think I'd like to hit 80-85MB/s total if possible.
 

Nigel

Senior Member
Staff
Donor
Local time
3:35 AM
Jan 20, 2018
135
40
The Netherlands
I don't know for sure. But I heard/read that Google itself limits to around 50-60 MB/S per file. That was the case when I used it. Because uploading to other services where much higher.
 

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.


Maintenance Donations

Recommend NewsGroups

      Up To a 58% Discount!