Performance moving files to NAS

Discussions, questions, comments and suggestions regarding Capture One PRO, Capture One PRO For Sony / Fujifilm, Capture One DB and Capture One Express For Sony / For Fujifilm 12.x

Performance moving files to NAS

Postby Kasmir » Mon Oct 14, 2019 8:14 pm

Wondering if anyone knows of any settings or anything that could help speed up C1's moving of files from my local machine's drive over to a NAS that I use for long-term storage (all files are in a single library, but moving them around locally is lightning fast, so it's not a library size problem that I can tell). It's really only the moving over to there that seems to be slow. Once they're there, exporting, browsing, editing, everything else is nice and fast, but that initial move is ... very slow. I can manually move these files at ~60MB/s, but C1 seems to be moving them at about 1.5MB/s (a recent move this morning of 160 raw files, around 25MB each, took about 42 minutes. Not a huge deal, but mildly irritating, and I don't really see any reason for it. None of my machine's resources are being taxed at all, it's a quad-core i7 with 32GB memory, and everything on the local machine is on a 1TB NVME SSD drive.

Thanks!
Kasmir
 
Posts: 5
Joined: Fri May 10, 2019 1:33 am

Re: Performance moving files to NAS

Postby SFA » Tue Oct 15, 2019 12:01 am

Which action are you using to move the files?

Catalogue or session?

Which NAS?

How is the NAS configured?
SFA
 
Posts: 7269
Joined: Tue Dec 20, 2011 9:32 pm

Re: Performance moving files to NAS

Postby Kasmir » Tue Oct 15, 2019 12:12 am

I select them all in the browser, and then grab one of them and drag it into the folder I want them to go to in the Library Tree panel.

This is all done in a single Catalog.

The NAS (though I'd think irrelevant), is a Q-NAP TS-451.

"How is the NAS configured?" is far too broad a question to answer in any kind of useful way, and I suspect also mostly irrelevant, unless C1 is somehow doing some special handling & detection for specific NAS configurations, but this seems highly unlikely. This is definitely not a problem with the network connections or the NAS. Everything else works very quickly and consistently.
Kasmir
 
Posts: 5
Joined: Fri May 10, 2019 1:33 am

Re: Performance moving files to NAS

Postby SFA » Tue Oct 15, 2019 12:22 am

I asked about the NAS on the basis that I have one and could possibly attempt to replicate your actions to see what results I get.

However mine is Synology so the test would probably be pointless.

As for the configuration - I was thinking more about what might be happening in the NAS system management than in C1. However as I know nothing about your NAS I agree that there is probably little point in diggiing into the details of the set-up.


Grant
SFA
 
Posts: 7269
Joined: Tue Dec 20, 2011 9:32 pm

Re: Performance moving files to NAS

Postby Kasmir » Tue Oct 15, 2019 12:37 am

Ah I see. Well, if you can drag & drop files across to your NAS without any issues, than that would be useful information to have. I don't think the brand would make much difference. But yeah, there's nothing specific in my setup that would make the connection slow, I have other backups that write to similar shares on the NAS and they run as-fast or faster than general windows explorer copying.

As an extra note, I just had a look at task manager while the copy was going again, and popped up the network monitor, and noticed that the transfers are going in spikes. The spikes are short & fast, but then there's a long period of several seconds where there's nothing noticeable happening.
Kasmir
 
Posts: 5
Joined: Fri May 10, 2019 1:33 am

Re: Performance moving files to NAS

Postby Bobtographer » Tue Oct 15, 2019 1:32 am

10Gb LAN?
Bobtographer
 
Posts: 538
Joined: Tue Feb 12, 2008 1:01 am
Location: England

Re: Performance moving files to NAS

Postby Kasmir » Tue Oct 15, 2019 2:08 am

Bobtographer wrote:10Gb LAN?


No, the NAS has dual gigabit ports, both are plugged in, the rest of the network is all gigabit connected.
Kasmir
 
Posts: 5
Joined: Fri May 10, 2019 1:33 am

Re: Performance moving files to NAS

Postby SFA » Tue Oct 15, 2019 4:09 am

I'm not a Catalog user (other than the occasional test and experiment) so my experience is with Sessions.

For those I typically use the NAS as a mirrored backup system rather than for live editing in the first instance. However if I need to go back to an old session that is no longer on a local drive I will edit directly on the NAS.

I and WiFi connected rather than cable simply because the NAS has only one Network cable connection port and I use that to connect to the router via a WiFi extender as running a cable would be awkward and I decided the WIFi performance was adequate for my needs. I looked at using a Power Cable Network connection but that gave very poor results, probably due to the mains circuit topography of the house.

Interestingly the transfer speeds achieved via WiFi seem broadly the same as I can achieve using USB3 direct connection. Not as fast as hoped and expected but liveable.

The challenge, however, is that the transfer speeds for bulk transfers are more effective for large files but a C1 session has a lot of small files to transfer Using just windows copy facilities (Win 7 in my case) the large files move relatively rapidly on the fasts network connection available but things slow down with the smaller ones in terms of MBs transferred per minute. That said something like 30GB will copy in around an hour is things go well and my other activities on the notebook are not constraining available resources.

When you are transferring your files via C1 (the correct thing to do when using a catalog so that it keeps track of the originals) are you sending them to a backup of the catalog or just a storage folder for original files on the NAS? I'm assuming that your working catalog is always using Referenced files rather than Managed files. Is that a safe assumption?

In effect your action is to move a group of referenced files from your internal drive to a folder on your NAS and maintain the referenced file link within the C1 catalog database. The expectation would be that only the original file is moved to the NSA disk and all of the edit information, previews, thumbnails, etc. stay exactly where they are.

Does that sum things up or have I missed something that needs to be tested for?


Grant
SFA
 
Posts: 7269
Joined: Tue Dec 20, 2011 9:32 pm

Re: Performance moving files to NAS

Postby Kasmir » Tue Oct 15, 2019 4:25 pm

SFA wrote:I'm not a Catalog user (other than the occasional test and experiment) so my experience is with Sessions.

For those I typically use the NAS as a mirrored backup system rather than for live editing in the first instance. However if I need to go back to an old session that is no longer on a local drive I will edit directly on the NAS.

I and WiFi connected rather than cable simply because the NAS has only one Network cable connection port and I use that to connect to the router via a WiFi extender as running a cable would be awkward and I decided the WIFi performance was adequate for my needs. I looked at using a Power Cable Network connection but that gave very poor results, probably due to the mains circuit topography of the house.

Interestingly the transfer speeds achieved via WiFi seem broadly the same as I can achieve using USB3 direct connection. Not as fast as hoped and expected but liveable.

The challenge, however, is that the transfer speeds for bulk transfers are more effective for large files but a C1 session has a lot of small files to transfer Using just windows copy facilities (Win 7 in my case) the large files move relatively rapidly on the fasts network connection available but things slow down with the smaller ones in terms of MBs transferred per minute. That said something like 30GB will copy in around an hour is things go well and my other activities on the notebook are not constraining available resources.

When you are transferring your files via C1 (the correct thing to do when using a catalog so that it keeps track of the originals) are you sending them to a backup of the catalog or just a storage folder for original files on the NAS? I'm assuming that your working catalog is always using Referenced files rather than Managed files. Is that a safe assumption?

In effect your action is to move a group of referenced files from your internal drive to a folder on your NAS and maintain the referenced file link within the C1 catalog database. The expectation would be that only the original file is moved to the NSA disk and all of the edit information, previews, thumbnails, etc. stay exactly where they are.

Does that sum things up or have I missed something that needs to be tested for?


Grant


I'd say that sums things up fairly well. I'm using a laptop, and usually it's wire-connected while I'm doing these transfers (as I'm fortunate enough to have ethernet wired throughout my house), but I agree that the wireless AC1750 connection it has is adequate for most things, and I don't notice any real difference between the two as far as this action goes.

My catalog setup is just a single one, there's no backup one that it's moving to. I use sessions occasionally when I'm tethered, but then I import those into the main catalog first, and then move them to the NAS storage, with the same results. I have the weekly catalog backups going to a folder on the NAS, but that's it, so all of the information about edits, layers, etc, are all local on the laptop. The NAS is contacted via an SMB share using the regular UNC path formatting: \\NAS_Name\PhotoStorage\. and within that I have things broken out by year and then month subfolders, so I'm just dragging files into one of those subfolders as needed.
Kasmir
 
Posts: 5
Joined: Fri May 10, 2019 1:33 am


Return to Capture One 12.x Software



Who is online

Users browsing this forum: No registered users and 6 guests