Discussion Switching to encrypted mode

Assists Greatly with Development Costs

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
Hi

I'm currently using the unencrypted version with move -
but from time to time i get a bit nervous about this setup

So i would like to switch to an encrypted version - whats the best approach to that?

I have currently about 50TB of data in a google drive (business account)

I'm not a coder and have basically no clue but i thought about the following approach:

Make a backup of all my programms (plex, sonarr, ....)
Install a clean Ubuntu on my dedicated server (just to get rid of any old stuff)
Do a clean install of Plexguide (with the benefit of having a more recent install than my current 6.019)
setup a new teamdrive for encryption
Setup encrypted XXX - Which version works best at the moment?

until here its rather standard
but in order to get the unencrypted content to the encrypted teamdrive i have no clue how to approach this -
a script that moves data from the unencrpyted gdrive to /unionfs/ (where it should be picked up by the "normal" process like the data came from sonarr/radarr)

Is there a better way?
I assume i'm not the only one who has made that move so there might be already a skript/batch whatever out there

Or is there a way to run the encryption on gdrive directly which would make the download/upload unneccessary

All the best
Stammbraeu
 

clinch123

The Newb
Local time
2:15 PM
Apr 7, 2018
63
30
Look into renting a google cloud VPS for the Unencrypted to Encrypted transfer. You should be able to bypass the transfer limit. Set up Plexguide Encrypted on the VPS. Then use the good old "mv" command to move from /mnt/gdrive/{tv | movies}/* to /mnt/unionfs/{tv | movies}. Plexguide should catch everything and encrypt it.

As far as your physical server, you should be okay just using the plexguide setup to change from move to move-en. Id backup sonarr, radarr and plex just in case though. Nothing should have to be changed pointer wise as it will all be done during the setup.
 

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
Hi

Thanks for your feedback

Where is the advantage in using a vps for the encryption?
shouldnt i be able to use the same approach on the dedicated server with more cpu power and ram (not sure what is needed for the encyrption part)

meaning a new setup
add gdrive unencrypted, add gdrive encrypted and move from gdrive unencrypted to unionfs where it gets picked up by the encryption & move process?

all the best
Stammbraeu
 

clinch123

The Newb
Local time
2:15 PM
Apr 7, 2018
63
30
There isnt any benefit per say besides the fact that if you keep the transfer within Google's network, there is no traffic limit within reason. Additionally you would reduce any load on your actual server if you still have the $300 credit that I saw Google was handing out a little bit back.
 

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
i skipped the "google" in your suggestion - makes sense - i somehow thought you meant a hetzner cloud server

i will try that approach

thanks

---- Automatically Merged Double Post ----

google cloud is setup
plexguide with encrypted move is setup
i set the bw limit to 100 (which would mean 7,5TB instead of 750GB)
i created a test.txt in unionfs and it showed up in encrypted gdrive (encrypted)
so far it works

now i tried a test move:
mv /mnt/gdrive/movies/2012\ \(2009\)/ /mnt/unionfs/movies/

with the following result: 1
mv: inter-device move failed: '/mnt/gdrive/movies/2012 (2009)/' to '/mnt/unionfs/movies/2012 (2009)'; unable to remove target: Directory not empty


Then i googled a bit and tried a "cp" instead of move
sudo cp -r /mnt/gdrive/movies/2012\ \(2009\)/ /mnt/unionfs/movies/

which resulted in
cp: cannot create regular file '/mnt/unionfs/movies/2012 (2009)/2012 (2009) - Bluray-1080p - SCiENTOLOGY.mkv': No space left on device

cp: cannot create regular file '/mnt/unionfs/movies/2012 (2009)/2012 (2009) - Bluray-1080p - SCiENTOLOGY.en.srt': No space left on device

cp: cannot create regular file '/mnt/unionfs/movies/2012 (2009)/2012 (2009) - Bluray-1080p - SCiENTOLOGY.de.srt': No space left on device

i checked my space and i have currently 3,1 TB free on the servers ssds


Any idea how to proceed?

All the best
Stammbraeu
 
Last edited:

clinch123

The Newb
Local time
2:15 PM
Apr 7, 2018
63
30
I did "mv /mnt/gdrive/tv/* /mnt/unionfs/tv". I wouldnt know why you would be getting the space errors if you have space. I guess make sure that everything is mounted properly and mnt is actually using one of the TB storage locations
 

kamos69

Senior Member
Donor
Local time
10:15 PM
Apr 22, 2018
126
67
I've done multiple transfers between various cloud providers using Rclone. Two things:
  1. You don't need to mount the drives.
  2. Use rclone sync when transferring. It's faster. The revamped sync option has --no-traverse built in, which is great! So your transfer command would look something like rclone sync gdrive:tv/ gcrypt:tv/ --bwlimit 100M -v
 
  • Like
Reactions: loa92

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
i spent last night trying out the google cloud server solution

it basically worked but i had the issue that i was using the google cloud console and it timed out after 60 mins - solved that issue by setting a ssh flag

but the speed was very disappointing - i had somewhere between 10MB to max 25 MB transferspeed - not sure whether i have setup a wrong machine (not enough CPU, not enough RAM, or something entirely different)

Based on that speed i could have redownloaded everything from usenet again and would have been at least factor 4 faster

---------------------------------

@kamos69: you mentioned the sync to gcrypt:tv/ - i thought i had to use unionfs as there the file would be picked up, encrpyted and then sent to gcrypt - or is this a missunderstanding of the concept on my side and gcrypt is the place to go (would solve the issue with replacing existing files)

Using the rsync option and the bwlimit - wouldnt i run into a API ban by google?

and final question: how do you mean not mount the drives?
the gdrive and crypt drive (with PW and salt) are created by the mounting process
and how would i set the PW and salt for the encryption in your scenario


Really big thanks for your support

All the best
Stammbraeu
 
  • Like
Reactions: Qlogic

kamos69

Senior Member
Donor
Local time
10:15 PM
Apr 22, 2018
126
67
@kamos69: you mentioned the sync to gcrypt:tv/ - i thought i had to use unionfs as there the file would be picked up, encrpyted and then sent to gcrypt - or is this a missunderstanding of the concept on my side and gcrypt is the place to go (would solve the issue with replacing existing files)
The encryption is done by Rclone itself, e.g. if you copy a file to an encrypted remote drive (unmounted), Rclone will encrypt that file before uploading it to the remote. The only reason to use unionfs is to for the automated 'move' functionality. But you can do that manually with the command example I mentioned previously.
Using the rsync option and the bwlimit - wouldnt i run into a API ban by google?
Yes.
and final question: how do you mean not mount the drives?
the gdrive and crypt drive (with PW and salt) are created by the mounting process
and how would i set the PW and salt for the encryption in your scenario
I guess you're referring to PG Blitz. I don't think there is a way around it for encrypted drives yet. PG Blitz only works for unencrypted. Unless you can get PG blitz working for your encrypted remote, you'll have to work within the limits, i.e. 5TB Download and 1TB Upload per day.
 

Escondido

Senior Member
Local time
2:15 PM
Feb 16, 2018
137
26
America Baby
i spent last night trying out the google cloud server solution

it basically worked but i had the issue that i was using the google cloud console and it timed out after 60 mins - solved that issue by setting a ssh flag

but the speed was very disappointing - i had somewhere between 10MB to max 25 MB transferspeed - not sure whether i have setup a wrong machine (not enough CPU, not enough RAM, or something entirely different)

Based on that speed i could have redownloaded everything from usenet again and would have been at least factor 4 faster

---------------------------------

@kamos69: you mentioned the sync to gcrypt:tv/ - i thought i had to use unionfs as there the file would be picked up, encrpyted and then sent to gcrypt - or is this a missunderstanding of the concept on my side and gcrypt is the place to go (would solve the issue with replacing existing files)

Using the rsync option and the bwlimit - wouldnt i run into a API ban by google?

and final question: how do you mean not mount the drives?
the gdrive and crypt drive (with PW and salt) are created by the mounting process
and how would i set the PW and salt for the encryption in your scenario


Really big thanks for your support

All the best
Stammbraeu
How did this work out for you? I'm in a similar boat and would love to hear if you were able to successfully migrate everything over and not have to worry about the daily Gban going over 750GB. Were you able to figure out how to increase the speed?
 

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
Sorry - i skipped the migration as it took way too long and i got hit by the api ban as well
my basic calculation was:

i have 80 TB of data - if i divide those by 750GB per day it would take me aprox 3.5 months to transfer the data

hoping that sooner or later we will have encrypted blitz and try than again
but if you find a solution that would work faster i'm happy to try it out

All the best
Stammbraeu
 
  • Like
Reactions: Admin9705

Admin9705

Administrator
Project Manager
Local time
3:15 PM
Jan 17, 2018
4,760
1,812
if there is, you would have to redownload it all and upload. your fine. i'm 4x your amount unencrypted. just don't openly share your drive and your ok.
 

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
going to revamp the thread as pgblitz encrypted is here - YEAH

so basically i want to move all my files from my "normal" unencrpyted share to the new encrypted team share

pgblitz is setup, new files show up in the correct teamdrive, plex can play those files - all fine

it tried a "mv" from /mnt/unionfs/ to /mnt/move/ - worked as well, although i end up with files in old share and in team drive
the transfer will obviously take quite a while but thats ok

now back to my question:
i used for my tests a simple mv command but as i wont be able to move all files in one huge mv i will have to do it in multiple runs

What is the best way to achieve that - due to the fact that all files that i move to /mnt/move/ will be gone after the upload to teamdrive i cant just simply use "mv" but skip files that are already there as by the time i run the next "mv" the files that are already on the teamshare wont be in /mnt/move/

So basically my idea was to compare /mnt/gdrive/ with /mnt/tcrpyt/ (based on filename and not on timestamp) and then move all files from /mnt/gdrive/ that are not in /mnt/tcrypt/

i tried the rsync option with "--compare-dest=" but thats not working out for me

Additionally i see the problem in a longer run that my download from the old share will be faster than the upload to the new teamshare which will end up in running into full HDs on the server


Any ideas on how to tackle those issues in an effective way?

All the best
Stammbraeu
 

timekills

Legendary Member
Staff
Donor
Local time
11:15 PM
Sep 12, 2018
353
170
I recommend rclone sync.

Example: rclone -v -P --transfers 1 sync gdrive:"/Your/Movie/Folder/" tdrive:"/Your/encrypted/movie/folder/"

The -v i-P s for verbose output (if you want it to output the status as it's moving - which you will.)

--transfers 1 limits it to one file transfer instead of parallel. If you don't include that, by default it will do four transfers at once. That actually often is faster, but YMMV.

Rclone sync synchronizes is actually a mirror. It mirrors all the files from the first location listed to the second.
It also works with folders - you don't have to use rclone locations (i.e. eclone sync /mnt/gdrive/movies /mnt/tdrive/movies)
 
  • Like
Reactions: stammbraeu

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
would your approach use the "mutliple upload users"-aspect from pgblitz?

and i can skip /mnt/move/ totally - i thought that there the magic of encryption and the upload to the teamdrive happens


EDIT:
just tried a single folder with your approach - unfortunately the folder gets transferred but not encrypted
 
Last edited:

timekills

Legendary Member
Staff
Donor
Local time
11:15 PM
Sep 12, 2018
353
170
would your approach use the "mutliple upload users"-aspect from pgblitz?

and i can skip /mnt/move/ totally - i thought that there the magic of encryption and the upload to the teamdrive happens
If you want to ensure it uses PGBilitz, the use "/mnt/unionfs/YourEncryptedMovieFolder" for the second folder name.

i.e. rclone -v -P --transfers 1 sync gdrive:"/Your/Movie/Folder/ "/mnt/unionfs/YourEncryptedMovieFolder"

It will encrypt them as it transfers them using PGBlitz, just like any downloaded files.
 
  • Like
Reactions: stammbraeu

stammbraeu

Respected Member
Original poster
Donor
Local time
9:15 PM
Mar 21, 2018
61
16
i adapted your command a bit by using move instead of sync

rclone -v -P --transfers 1 move gdrive:"/Your/Movie/Folder/ "/mnt/unionfs/YourEncryptedMovieFolder"

this way all the content thats already on the teamdrive gets removed from the old gdrive and if i have to cancel the command i can start a new round and dont have to worry about copying the same files twice

Thanks for your help

enjoy your evening
all the best
Stammbraeu
 
  • Like
Reactions: Admin9705

bodgeup

Respected Member
Donor
Local time
8:15 PM
Aug 12, 2018
76
23
Just tried this on my PGMove to PGBlitz (but unencrypted to unencrypted) it binned all my gdrive /tv files but nothing got uploaded to my teamdrive via PGBlitz which remained idle. Im sure i got the PGBlitz setup correct, the keys backed up to my gdrive ok.

The PG.log shows PGBlitz has nothing to upload, any suggestions anyone?
 
Last edited:

Create an account or login to comment

You must be a member in order to leave a comment

Create account

Create an account on our community. It's easy!

Log in

Already have an account? Log in here.

Similar threads


Maintenance Donations

Recommend NewsGroups

      Up To a 58% Discount!

Trending