Exploding RAM Usage

Discussions, questions, comments and suggestions regarding Capture One PRO, Capture One PRO For Sony / Fujifilm, Capture One DB and Capture One Express For Sony / For Fujifilm 12.x for Windows

Exploding RAM Usage

Postby Tom-D » Sat Jan 19, 2019 1:34 pm

Hi,

i'm sitting in front of my work, Report Photography, and i do Ajust my images with 7 Layers, 5 of them are Luminance Layers.

While Copy all the Layers to all other images (570 pcs) my Ram (64GB) usage is Exploding and Goes Beyond, iv'e raised my Virtual Memory to 280 GB just to copy the Layers to all images.

If i don't raise the Virtual Memory Space i did get a out of Memory warning and Windows did stop responding.

With the Wedding season coming, i would ned almost a Terabyte of RAM or Virtual memory for Export and Copy my adjustments to other images.

I did Open a Case, but reaction is slow, so i'm waiting (5th day).

Did anyone else experience this behavor?
Last edited by Tom-D on Sat Jan 19, 2019 3:11 pm, edited 3 times in total.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby SFA » Sat Jan 19, 2019 2:04 pm

The free Technical Support service may not be immediately responsive over a weekend.

5 Luma layers and 7 layers in total copied to 570 images (if I understood correctly) seems like a extreme process decision.

Even more extreme if you think of that as your normal requirement.

Can I ask what all of the layers are doing?

I would be surprised if some adjustments are not working against other adjustments.

I would be even more surprised if the same values were useful for 570 images unless you have a very special requirement indeed.

However I like surprises so please share what you are doing with all of the layers as a standard adjustment for all images.

Grant
SFA
 
Posts: 6456
Joined: Tue Dec 20, 2011 9:32 pm

Re: Exploding RAM Usage

Postby Tom-D » Sat Jan 19, 2019 2:27 pm

SFA wrote:The free Technical Support service may not be immediately responsive over a weekend.

5 Luma layers and 7 layers in total copied to 570 images (if I understood correctly) seems like a extreme process decision.

Even more extreme if you think of that as your normal requirement.

Can I ask what all of the layers are doing?

I would be surprised if some adjustments are not working against other adjustments.

I would be even more surprised if the same values were useful for 570 images unless you have a very special requirement indeed.

However I like surprises so please share what you are doing with all of the layers as a standard adjustment for all images.

Grant


I can't tell you 100%, its a company secret.

Its a seperation Process.

It was a long way until CO 12 and its possibilities.
Last edited by Tom-D on Sun Jan 20, 2019 7:46 pm, edited 1 time in total.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby Tom-D » Sat Jan 19, 2019 4:12 pm

I didn't expect that the Luminance Masking has such a big impact to my work.

I would wish that CO did use the GPU far more, cause GPUs are much more powerful.

But it's GPGPU Processing and the GPU does only the stuff what can be Paralellised.

When my Problem itsn't going to get fixed, cause it would slow down the whole software, i would get a 2 TB NVME Drive just for Windows Virtual Memory.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 9:27 am

Update:

Ram Usage does not Explode while Exporting, remains at 14 Gig.

Only while Copying Layers to a Batch of photos (100+) with some LumaMask Layers.

Will add this to my Case.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby C-M-B » Sun Jan 20, 2019 10:11 am

Please see viewtopic.php?f=76&t=29665

Copy/Pasting Luma Masks doesn't just copy the settings, it has to calculate the specific luma mask for each and every image. When using multiple Luma Masks that means it will take an extremely long amount of time.

I'd suggest to only apply luma masks at the very end to the images that are selected for publication.
C-M-B
 
Posts: 371
Joined: Thu Feb 25, 2010 4:57 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 10:43 am

C-M-B wrote:Please see viewtopic.php?f=76&t=29665

Copy/Pasting Luma Masks doesn't just copy the settings, it has to calculate the specific luma mask for each and every image. When using multiple Luma Masks that means it will take an extremely long amount of time.

I'd suggest to only apply luma masks at the very end to the images that are selected for publication.


"Publication"

If you photograph a wedding, the light changes constantly, with the Luma masks it is possible to create a selection over the luma rage for areas which are covered by a shadow and actually have no drawing and to create an own processing especially for this area.

It is also possible to restore clipped areas to which the HDR tool doesn't react correctly and gives them their own look.

The same applies to the sharpening, if you have a style that runs through the whole job then it is possible with the Luma rage to sharpen areas like faces and or to de-noise.

And the best thing is that it is scalable, due to the opacity.

Now you create a style at the beginning that you drive and copy it over all your photos, so if you call up the pictures all the tools you need are already there, this saves time.

Yea, copies all settings. The Setting folder of the actual Project has 100 GB, thats not a Problem.
A Wedding will have 300 GB or more, also no Problem.

A Problem is the Processor Load and the Load on Memory while copy all the Layers to other images.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 10:52 am

@ C-M-B

Your Problem is the Same.

Techgage did Bench Capture One an had the same Issue with a 32 Core Threadripper CPU, the 32 Core did have the Same Performance as a 9900K 8 core 5 ghz Cpu.

So they have to dig for a bug.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby SFA » Sun Jan 20, 2019 11:49 am

Tom,

Without giving away any business secrets, do you find that doing all this work before presenting the results to the clients is likely to add commercial advantage to the deal that justifies the effort and investment of time and in hardware?

I suspect that if you are not dealing with an art director who mainly wants someone to create the look they have in their mind, most clients make most of their selections based on some instinctive criteria.

On a few occasions I have presented a set of images to people where there might be 2 almost identical shots, one somewhat out of focus (but acceptable at a distance) with minimal processing and one completely sharp and with some additional effort in the processing to reinforce that advantage. Very often the less sharp image is the one selected for some entirely obscure reason.

This was just as true before the mobile phone image became the standard of reference.

Grant
SFA
 
Posts: 6456
Joined: Tue Dec 20, 2011 9:32 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 12:43 pm

@SFA

Our clients don't have to choose pictures from a reportage, we do that ourselves.

Only this week we had a customer who changed from his already booked photographers to us because our style is not muddy and our pictures look rather natural.

The other photographer had a muddy instragram style.

Even last year we had customers who are photographers themselves, who are enthusiastic about our work and ask us how we do it.

Since this year, with capture one 12, we've put even more on top because we can now edit the images the way we want.

Of course it would also be possible via Photoshop, but it takes 5x as long, even with buggy Load.

_________

i wish i could show you the pictures i worked on last and under which circumstances (light conditions) they were created.

But in europe the GPDR is against that.

You wouldn't get out of the amazement.

I didn't become a photographer so that I could take the same pictures as others.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 1:05 pm

I forgot to tell you it's not just technology.

Of course, it also depends on how you handle the Camera, the Angle you take pictures from, the Posture of the people (all natural ones, not models).
Last edited by Tom-D on Sun Jan 20, 2019 7:47 pm, edited 1 time in total.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby SFA » Sun Jan 20, 2019 1:25 pm

Tom,

You have answered my question and it does indeed seem that there is a commercial advantage that makes the effort worthwhile.

At the same time you have reduced the effort required by a factor of 5 compared with previous processing routes.

Three thoughts come to mind.

Firstly and something that you maybe have checked out already, what happens within memory management if you break the process into smaller batches that are sized to avoid the use of virtual RAM?

Secondly, would other hardware - for example the new Phase One systems - offer similar scope directly and reduce post processing complexity?

Thirdly - it seems to be a very specific requirement that is not really mainstream at this time (and if it becomes mainstream your commercial advantage disappears). Have you considered discussing the technical aspects of the process with Phase in order to understand where limitations exist and how they might be overcome?


I would see points 2 and 3 as being good conversations to have for all who might be involved.

I enjoyed looking at your sample image - thanks for sharing it.

You said:

"I didn't become a photographer so that I could take the same pictures as others."

I can fully understand and agree with that statement.
SFA
 
Posts: 6456
Joined: Tue Dec 20, 2011 9:32 pm

Re: Exploding RAM Usage

Postby Tom-D » Sun Jan 20, 2019 1:51 pm

SFA wrote:Tom,

You have answered my question and it does indeed seem that there is a commercial advantage that makes the effort worthwhile.

At the same time you have reduced the effort required by a factor of 5 compared with previous processing routes.


This differ by image, First you have to Process them in a Raw Converter, then you have Process them in Photoshop. <-- and that takes a lot of time.


SFA wrote:Three thoughts come to mind.

Firstly and something that you maybe have checked out already, what happens within memory management if you break the process into smaller batches that are sized to avoid the use of virtual RAM?


That really didn't work, Ram is not cleared after the Process. You have to Restart the Software every time.

SFA wrote:Secondly, would other hardware - for example the new Phase One systems - offer similar scope directly and reduce post processing complexity?


We need fast Cameras, Fast Shutter, Fast Focus (in every Lightcondition), Sturdy housing, Build in Battery Grip (Battery Last for 2500+ images), shaped like a cuboid so you can turn them fast, that can do all the stuff we photograph, we use Canon 1Dx series. During weddings we dance with the puplicum in the crowd while we take pictures.

The xf system would extend the editing possibilities, because it works with 16 bit and not 14 bit. This would mean that it would be more complicated. But at the same time slower because it is not intended for reportage.

SFA wrote:Thirdly - it seems to be a very specific requirement that is not really mainstream at this time (and if it becomes mainstream your commercial advantage disappears). Have you considered discussing the technical aspects of the process with Phase in order to understand where limitations exist and how they might be overcome?


I don't think there's a limit, the way the software is now, it didn't exist before. So there weren't the problems that are occurring now.
With Capture One 10 there was a similar problem while exporting, that one was fixed.

SFA wrote:I would see points 2 and 3 as being good conversations to have for all who might be involved.

I enjoyed looking at your sample image - thanks for sharing it.

You said:

"I didn't become a photographer so that I could take the same pictures as others."

I can fully understand and agree with that statement.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby Tom-D » Tue Jan 22, 2019 12:17 pm

My Case didn't get a update from supporter since 7 Days.

Did post some updates.

I guess they forgot about me.
Win 10 x64, Intel 7820X, 64 GB Ram 2666mhz, Quadro RTX 5000, M.2 2x 1 TB OS an now used Images, 4x HDDs / Backup.
Tom-D
 
Posts: 68
Joined: Wed Jan 13, 2010 1:44 pm

Re: Exploding RAM Usage

Postby SFA » Tue Jan 22, 2019 12:49 pm

Tom-D wrote:My Case didn't get a update from supporter since 7 Days.

Did post some updates.

I guess they forgot about me.


I doubt it. Maybe just not much they can add at the moment.

In previous years there has been a maintenance update around this time (although that is no guarantee of the same thing happening this year.)

If you are processing via the batch queue have you tried clearing the queue? And maybe ensuring, via windows, that all related temporary files are being successfully deleted when the process is complete.

Yes, I know they should be but that is not always the case even at the OS level.

It could in any case be interesting diagnosis.


Grant
SFA
 
Posts: 6456
Joined: Tue Dec 20, 2011 9:32 pm

Next

Return to Capture One 12.x Software for Windows



Who is online

Users browsing this forum: No registered users and 1 guest