Discussion - Too many folders in /movies/ | PlexGuide.com

Discussion Too many folders in /movies/

  • Stop using Chrome! Download the Brave Browser via >>> [Brave.com]
    It's a forked version of Chrome with native ad-blockers and Google's spyware stripped out! Download for Mac, Windows, Android, and Linux!
Welcome to the PlexGuide.com
Serving the Community since 2016!
Register Now

indochild

Experienced
Original poster
Mar 13, 2018
57
23
Hi,

i'm unable to navigate /mnt/unionfs/movies/ anymore. i think i might have too many directories. is there a limit? i'm around 29k folders inside.

Edit:

i'm on 8.7.5 and just started experiencing this issue yesterday. /tv/ and /music/ works fine.

Not sure what the error is. because there are a lot of files in tv. is google enforcing file limits on Team Drives?
 
Last edited:

Datamonkeh

Data Hoarding Primate
Project Manager
Donor
Donor
Jan 20, 2018
888
400
400,000 file limit, you'll know when you hit it.
 

Datamonkeh

Data Hoarding Primate
Project Manager
Donor
Donor
Jan 20, 2018
888
400
Well as both are likely in the same team drive, nothing in the google side. Do they load in the google drive web UI? If so then how exactly are you trying to navigate them?
 

indochild

Experienced
Original poster
Mar 13, 2018
57
23
Well as both are likely in the same team drive, nothing in the google side. Do they load in the google drive web UI? If so then how exactly are you trying to navigate them?
I’m trying to access them through cloudcmd, radarr, & Plex. Perhaps it’s a permissions issue on that directory?
 

Datamonkeh

Data Hoarding Primate
Project Manager
Donor
Donor
Jan 20, 2018
888
400
So again, have you checked the directory via google drive web UI in your browser to make sure you have data? It's pointless guessing at other issues till you establish the data exists.
 

indochild

Experienced
Original poster
Mar 13, 2018
57
23
So again, have you checked the directory via google drive web UI in your browser to make sure you have data? It's pointless guessing at other issues till you establish the data exists.
Sorry, yeah I can access it on google drive webui. It started acting weird on the server on Saturday night. Blitz can still upload files to it, but nothing can be read from it
 

Datamonkeh

Data Hoarding Primate
Project Manager
Donor
Donor
Jan 20, 2018
888
400
Well that’s a positive, nothing worse than chasing a fault at a lower level only to find you had accidentally deleted the content. So you have 29k of folders, I’ve seen examples where various apps will fail to pull a directory listing, the best example is where I have added a directory for Plex to scan and then try to add another directory, leaving it for a minute or two or cancelling the initial scan and adding the directory again generally resolves it for me.
 

user456

Noobz
Sep 9, 2018
2
0
I too am experiencing this same issue. After I reboot my Hetzner server. All folders on the team drive are accessible from cloudcmd except for the movie folder. I have to wait about a day or so before cloudcmd, Radarr and plex are able to read the folder. Tried using new/different keys while redeploying blitz but to no avail. This happens after every server reboot. Very weird.
 

indochild

Experienced
Original poster
Mar 13, 2018
57
23
I too am experiencing this same issue. After I reboot my Hetzner server. All folders on the team drive are accessible from cloudcmd except for the movie folder. I have to wait about a day or so before cloudcmd, Radarr and plex are able to read the folder. Tried using new/different keys while redeploying blitz but to no avail. This happens after every server reboot. Very weird.
Glad it’s not just me, unfortunately for you hahaha

I’m currently migrating it all from one folder to another in hopes that it changes because my tv folder has way more files. Maybe it’s a size issue?
Post automatically merged:

Never realized that i hadn't reset my server until shortly before i started noticing the issue. also didn't bother waiting 2 days for it to fix itself.
Post automatically merged:

I too am experiencing this same issue. After I reboot my Hetzner server. All folders on the team drive are accessible from cloudcmd except for the movie folder. I have to wait about a day or so before cloudcmd, Radarr and plex are able to read the folder. Tried using new/different keys while redeploying blitz but to no avail. This happens after every server reboot. Very weird.
Would Plex crash if you tried to play a movie in those first two days?
 
Last edited:

Edrock200

MVP
Staff
Nov 17, 2019
724
259
Go to sudo pg then pg clone. Go to rclone options. Set your dir cache to the max 7200 i think. By default i think its set to 5m. You will still get dir updates since google supports push update notifications.

I suspect it takes more than 1m worth of api hits (100) to enumerate 29k+ dirs, so you are getting throttled, then rclone resets the dir cache starting all over again.

One thing i do to improve scanning and drive performance is have a shell script with
ls -alR /mnt/unionfs
Set as a cronjob with schedule @reboot, since rclone cache doesnt persist through a remount. It takes a few hours for a full team drive but seeing as i hardly ever reboot my server, it makes scans and directory browsing super fast.

Also check pgui since it will display any errors recorded in the rclone logs.
 
  • Like
Reactions: 2 users

indochild

Experienced
Original poster
Mar 13, 2018
57
23
Go to sudo pg then pg clone. Go to rclone options. Set your dir cache to the max 7200 i think. By default i think its set to 5m. You will still get dir updates since google supports push update notifications.

I suspect it takes more than 1m worth of api hits (100) to enumerate 29k+ dirs, so you are getting throttled, then rclone resets the dir cache starting all over again.

One thing i do to improve scanning and drive performance is have a shell script with
ls -alR /mnt/unionfs
Set as a cronjob with schedule @reboot, since rclone cache doesnt persist through a remount. It takes a few hours for a full team drive but seeing as i hardly ever reboot my server, it makes scans and directory browsing super fast.

Also check pgui since it will display any errors recorded in the rclone logs.
YOU ARE A SAINT!

just did that and it seems to be working. i'll set up the chronjob after i do a bit of research on how to do that.
 
  • Like
Reactions: 1 user

Edrock200

MVP
Staff
Nov 17, 2019
724
259
YOU ARE A SAINT!

just did that and it seems to be working. i'll set up the chronjob after i do a bit of research on how to do that.
Great, glad to hear it.

for cronjob
Code:
sudo nano /opt/dircache.sh
in the file put
Code:
ls -alR /mnt/unionfs
save the file then do
Code:
sudo chmod +x /opt/dircache.sh
then
Code:
sudo crontab -e
if it's your first time running it, it will ask you to pick an editor, pick nano
scroll to the bottom of the file to a new empty line and add
Code:
@reboot /bin/bash -c "/opt/dircache.sh"
That's it. It will cache the file system at every reboot. If you want it to start caching for this one just type
Code:
/opt/dircache.sh
Don't close your terminal while it's running. Depending on the amount of content you have it can take hours.
-Ed
 
  • Like
Reactions: 1 users

doob

Administrator
Project Manager
Jun 7, 2020
970
508
Code:
apt-get install screen

screen -S dircache

bash /opt/dircache.sh

CTRG  + A + D
_____

Code:
#!/bin/bash

if pidof -o %PPID -x "$0"; then
    exit 1
else
    ls -alR /mnt/unionfs
fi
exit 1
 
Last edited:
  • Like
Reactions: 1 users

Edrock200

MVP
Staff
Nov 17, 2019
724
259
I was being too lazy to walk through screen too. :) Thanks Doob. I should probably also add the proper headers to my shell scripts :censored:

If you (indochild) go the screen route (it's the better route) you can change the crontab line to
Code:
@reboot screen -S dircache /opt/dircache.sh
This way after a reboot you can always check on the progress of the dir cache by doing
Code:
sudo screen -r dircache
and disconnect as Doob said above, CTRL-A, then D. If it won't attach to the screen, then it has finished.
 
Last edited:

Edrock200

MVP
Staff
Nov 17, 2019
724
259
Code:
#!/bin/bash

if pidof -o %PPID -x "$0"; then
    exit 1
else
    ls -alR /mnt/unionfs
fi
exit 1
Show off! ;-) JK. I was about to ask you what the extra line does then decided to stop being lazy and google it. From what I can tell, it prevents it running more than one instance simultaneously, is that correct?
 
  • Like
Reactions: 1 user

doob

Administrator
Project Manager
Jun 7, 2020
970
508
Show off! ;-) JK. I was about to ask you what the extra line does then decided to stop being lazy and google it. From what I can tell, it prevents it running more than one instance simultaneously, is that correct?
Ya its correct

So you can run it as one command and don't as multiple runs
 
  • Like
Reactions: 1 user

urgodfather

Citizen
Donor
Donor
Nov 15, 2019
11
19
Code:
apt-get install screen

screen -S dircache

bash /opt/dircache.sh

CTRG  + A + D
_____

Code:
#!/bin/bash

if pidof -o %PPID -x "$0"; then
    exit 1
else
    ls -alR /mnt/unionfs
fi
exit 1

Code:
cat >/opt/dircache.sh <<EOF
#!/bin/bash

if pidof -o %PPID -x "$0"; then
    exit 1
else
    ls -alR /mnt/unionfs
fi
exit 1
EOF 
apt update & apt install screen -y ; chmod a + x /opt/dircache.sh ; screen -S dircache -d -m /bin/bash /opt/dircache.sh
this might suffice as a nice little copy and paste. #untested
 
  • Like
Reactions: 1 users

Recommend NewsGroups

      Up To a 58% Discount!

Trending