Performance

[mattcohen]

Is there a recommended setup to get C1 to perform well with decent sized catalogs? I still mainly use Aperture, but C1 for finishing some client work because I like the RAW rendering better. I've tried several times over several years to switch to C1 completely for simplicity's sake, but it's just way too slow and beach balls are everywhere. Recently I imported just one Aperture library with about 30k pictures. I moved the originals inside C1 catalog, and all of this was on a very very fast multi-SSD RAID. It's a 2016 MBP with plenty of memory and whatever the top GPU was at the time. It's still basically unusable for searching/filtering.

Is there any way to deal with large catalogs that I missed in my searches? Is this something the C1 team cares about, or is it designed to be low volume software and I'm just never going to be able to use it how I need to? I don't need it to be Photo Mechanic fast, but it's still vastly slower than Aperture which has been dead for 5 years...

[tenmangu81]

Hi Matt,

I have a MBP Retina 15" late 2013, and I get an answer from a filter request in about 3 to 5 seconds on all my 22k images.
Cheers,
Robert

[Paul_Steunebrink]

It seems you catalog is on an external system.

An alternative could be to have a catalog with referenced images (=stored outside catalog). Place you catalog on internal drive, and images on external drive.

[mattcohen]

It seems you catalog is on an external system.

An alternative could be to have a catalog with referenced images (=stored outside catalog). Place you catalog on internal drive, and images on external drive.

I tried this first.

[Eric Nepean]

I have found that complex Metada in a large collection of Images can slow the Filter tool by a huge amount.

There was time when I was storing the original file name for each image in one of the IPTC Metadata fields. This results in a different value for every image file.

Removing Filter Tool from the Tool tab, or removing the Metadata, resulted in a significant speedup.

Filter tool indexes every Metadata Field, those that are shown, and those that are not. Indexing more than 10000 images, with a unique value for each images is not only of no value to the user, but takes a very long time. The number of string comparison operations theoretically grows as N*log(N), where N is the number of images. For 10000 images this is theoretically 6.7x as long as for 1000 images, and for 30,000 images 22x as long.

If the developer takes a straightforward approach, then the number of comparison operations grows as N squared (10,000 is 100X worse than 1000). For practical SW faced with data storage and CPU utilisation limits, the scaling is probably somewhere between these two values.

I observed that when Filter tool is on your active (visible) tool tab, it performs this indexing action just about every time you click a key or make a mouse click. On my newer iMac it was a 20s long beachball everytime I made a mouseclick in my All Images Collection of 16000 images. On the older iMac it was over a minute. That was pretty much unuseable and I came close to dumping Capture One at that point.

I ended up making a custom tool tab just for Filter Tool (and removed it from the Library Tool Tab), and this helped a lot. Since then I have decided that I don't need that Metadata, and scrubbed it.

[mattcohen]

I have found that complex Metada in a large collection of Images can slow the Filter tool by a huge amount.

There was time when I was storing the original file name for each image in one of the IPTC Metadata fields. This results in a different value for every image file.

Removing Filter Tool from the Tool tab, or removing the Metadata, resulted in a significant speedup.

Filter tool indexes every Metadata Field, those that are shown, and those that are not. Indexing more than 10000 images, with a unique value for each images is not only of no value to the user, but takes a very long time. The number of string comparison operations theoretically grows as N*log(N), where N is the number of images. For 10000 images this is theoretically 6.7x as long as for 1000 images, and for 30,000 images 22x as long.

If the developer takes a straightforward approach, then the number of comparison operations grows as N squared (10,000 is 100X worse than 1000). For practical SW faced with data storage and CPU utilisation limits, the scaling is probably somewhere between these two values.

I observed that when Filter tool is on your active (visible) tool tab, it performs this indexing action just about every time you click a key or make a mouse click. On my newer iMac it was a 20s long beachball everytime I made a mouseclick in my All Images Collection of 16000 images. On the older iMac it was over a minute. That was pretty much unuseable and I came close to dumping Capture One at that point.

I ended up making a custom tool tab just for Filter Tool (and removed it from the Library Tool Tab), and this helped a lot. Since then I have decided that I don't need that Metadata, and scrubbed it.

This is a very helpful answer, I appreciate it. Unfortunately I have very extensive metadata, and getting rid of it would defeat the purpose (actually being able to search and export the relevant pictures) of a DAM.

I'd be embarrassed if I made something that was more limited (in any way) than five-year-old dead software.

[SFA]

I'd be embarrassed if I made something that was more limited (in any way) than five-year-old dead software.

I get your point but is that not what Apple did when replacing its now dead software?

If the mighty Apple can do that to its users why on earth would you expect comparatively little Phase One to fill the gap left by Apple's development that used Apple's own development tools and, presumably, was optimised for them?

And if they did, would they have to develop on a completely different set of products for the Windows market?


Grant

[mattcohen]

I'd be embarrassed if I made something that was more limited (in any way) than five-year-old dead software.

I get your point but is that not what Apple did when replacing its now dead software?

If the mighty Apple can do that to its users why on earth would you expect comparatively little Phase One to fill the gap left by Apple's development that used Apple's own development tools and, presumably, was optimised for them?

And if they did, would they have to develop on a completely different set of products for the Windows market?


Grant

Maybe you misunderstood, maybe you didn't. Today I can search through an Aperture library of 120,000 pictures with multiple keywords and full IPTC with zero lag. With the same setup, C1 chokes on 30,000, I wouldn't be surprised if it choked on 10,000. Aperture hasn't been developed really since 2013, and C1 has had ~5 versions since. Yes, I absolutely expect that problem to have been solved completely in the intervening years. This is basic DAM functionality.

[Nature Isme]

FileMaker Pro—which is also an Apple product (FM is a wholly owned Apple subsidiary)—is also almost instantaneous with large databases (it's expressly a database program after all). It's been Mac and Windows for something like 20-25 years, since version 2 (they're now on v17). I suspect their internal approach (as well as Aperture's) differs significantly from C1. If I remember correctly, FMP maintains internal indices that track occurrences of words with pointers the records that contain them, so when one searches for items the database only has to parse the appropriate index(es) rather than every field in each record. OTOH, I get the feeling that C1 doesn't do this sort of thing, instead parsing each field in each "record" in the current collection. That means that whereas FMP (and, I assume Aperture) only have to parse relatively small tables, C1 is left parsing the entire metadata table—a MUCH more laborious process as pointed out above.

I saw the tip about moving the Filter tool to its own tab several versions ago (v9 or 10). That sped navigating among images up hugely.

When searching for images, C1's filters start working as soon as you start typing. The "found results" are narrowed with each successive typed character. Generally I only need to type the first 3-4 letters in a word I'm looking for and C1's got a reasonable list. The bad news is that often the list of found images includes a lot that make me scratch my head in wonder—wondering why in the world they showed up in the found items. C1's usually accurate for small numbers of found items in a large collection, but for me falls down when there are a lot of found items.

For example (fictional examples for illustration only), I photograph a lot of birds in the wild. When I search for "description contains bald eagle", C1 will find bald eagles AND others that have descriptions like "golden eagle"—and some that have neither "bald" nor "eagle" in the Description field. Or "description contains great blue heron", C1 will return the desired images PLUS birds that have descriptions like "house finch", "egret"—and don't contain the words "great", "blue", or "heron" at all. I have no idea why this happens, but it's something I live with.

I wish there was a way to tell C1 not to start filtering until the user hits the Return key or something like that. But there isn't. Developing databases is hard work and probably pretty labor intensive (and maybe boring). I can understand why Phase One wants to concentrate on features related to processing, but it sure would be nice if they'd give a bit more love to the DAM aspect.

[mattcohen]

When searching for images, C1's filters start working as soon as you start typing. The "found results" are narrowed with each successive typed character. Generally I only need to type the first 3-4 letters in a word I'm looking for and C1's got a reasonable list. The bad news is that often the list of found images includes a lot that make me scratch my head in wonder—wondering why in the world they showed up in the found items. C1's usually accurate for small numbers of found items in a large collection, but for me falls down when there are a lot of found items.

For example (fictional examples for illustration only), I photograph a lot of birds in the wild. When I search for "description contains bald eagle", C1 will find bald eagles AND others that have descriptions like "golden eagle"—and some that have neither "bald" nor "eagle" in the Description field. Or "description contains great blue heron", C1 will return the desired images PLUS birds that have descriptions like "house finch", "egret"—and don't contain the words "great", "blue", or "heron" at all. I have no idea why this happens, but it's something I live with.

I have noticed this as well. And it's not something I can live with.

[SFA]

I'd be embarrassed if I made something that was more limited (in any way) than five-year-old dead software.

I get your point but is that not what Apple did when replacing its now dead software?

If the mighty Apple can do that to its users why on earth would you expect comparatively little Phase One to fill the gap left by Apple's development that used Apple's own development tools and, presumably, was optimised for them?

And if they did, would they have to develop on a completely different set of products for the Windows market?


Grant

Maybe you misunderstood, maybe you didn't. Today I can search through an Aperture library of 120,000 pictures with multiple keywords and full IPTC with zero lag. With the same setup, C1 chokes on 30,000, I wouldn't be surprised if it choked on 10,000. Aperture hasn't been developed really since 2013, and C1 has had ~5 versions since. Yes, I absolutely expect that problem to have been solved completely in the intervening years. This is basic DAM functionality.

If I understand things correctly Apple replaced Aperture with Photos for OS X.

How does Photos compare to Aperture?

Remember that until recently Phase also offered Media Pro.

So if Apple don't see a long term need for an Aperture type product (or don't wish to compete with other applications available) and Phase don't see a future for Media Pro ( a comprehensive DAM product in its own right as I understand it) then maybe the DAM market is either too small or too specialised or too well populated with free application or something to make it worth the development investment?

Affinity seems to have become quite popular as an alternative to PhotoShop. No DAM functionality that I can find in Affinity. Nor Keywords. Nor much additional metadata.

Lightroom forced the use of a catalogue (at version one - maybe still?) for whatever reasons they had at the time.

I didn't like the catalogue approach at the time.

What else does Adobe offer for cataloguing other than LightRoom?

Canon always provided separate products for RAW conversion and cataloguing. I didn't get into either of them very much.

Other products may well be trying to offer both catalogues and RAW conversion/editing in a single integrated application and UI. Can they do it without performance issues on one side of the application or the other?

The point about the Apple in house development is that they only had to create a product for Apple users, could make use of all of their internal knowledge of their self developed tools and totally integrate to the OS or other useful features from their suites of products in way that third party developers may never be able to do.

The same goes for MS and Windows.

In terms of business strategy they can also to exit a particular market at any time they choose to and both of them do frequently. Some Microsoft applications have extremely short lifespans an no realistic end of life portability.

So if a major developer does create a super performing product based on their own architecture you would expect it to be good. And marketable. To drop it rather suddenly suggests they don't really see a future for that market in that form - or at least not a market they are interested in .

Alternatives development tools and market options may be more compromised.



Grant

[Emile]

Then again, Grant, if the DAM side of things is not really important, then why do so many people use Lightroom? I think we mostly agree on this forum that C1 offers superior RAW tools and conversions, so that can’t be it... And a lot of relative newcomers on the market are busy developing and integrating a DAM solution in their RAW development products.

Personally, the lack of a professional cataloguing feature and Phase completely ignoring complaints about it are my big frustrations with C1. I like the product enough to work around it, with a multitude of 5k to 7k image catalogs, but it’s 2019. They could do a lot better.

[mattcohen]

searching large groups of pictures is a solved problem, i'm not asking for something exotic like compression that gives me negative file sizes. just search on one year's pictures (hopefully more) not leading to beachballs. many apps can do this with no lag. as mentioned, other photo apps are moving toward DAMs, Luminar most recently, Photo Mechanic and other to come. yet C1 updates and charges every single year and still provides DAM functionality that isn't adequate for any professional size catalog. they just re-skinned the whole thing, while functionality is sacrificed year after year.

[SFA]

Reading some comments on here from time to time there do seem to be some users who are happy with the performance.

But then reading comments about LightRoom (not specifically about catalogues) some find it really fast and some find it really slow.

It's difficult to know what to say.

As for LightRoom and catalogues - does it have an option?

Do so many people use it because it has a catalogue and they know they really want to use a catalogue and make full use of the search facilities ....?

Or because they have been told the LightRoom is the way to go and they find PhotoShop and its derivatives unfathomable for their needs?

I do wonder if we are in some sort of Catch 22 with databases.

Some business software I use is, for high volume users with very large data sets, somewhat compromised on performance for desktop use but, if installed with an MS Server configuration as its workhorse database, much faster.

Also much more expensive and requiring more significant attention to manage the systems on which that sort of application runs.

The developers tell me that, when fine tuning the settings for the system, they have to make a balanced choice between UI level responsiveness so that users do not perceive long waits to do anything on screen, and then handling large numbers of records when performing analysis instructions. They could make both extremely fast by tweaking the settings - but not at the same time.

The application will allow for manipulation and analysis of millions of records reasonably quickly but not extremely fast and the UI is very responsive so long as the working data set does not have more than about 30k - 40k records. That's the point at which the setup parameters have been tuned for on the basis that most users will tend to work interactively with smaller datasets. If they are running a larger task they expect it to take a while and plan their time accordingly.

Better still they persuade their employer that the automated options in the higher versions of the product - running on a server - are great productivity enhancers. That's the sort of option that would not be available or practical for most of us.


Grant



I don't know the answer but it's a question that might be interesting to have answered.


Grant

[Wesley]

Then again, Grant, if the DAM side of things is not really important, then why do so many people use Lightroom? I think we mostly agree on this forum that C1 offers superior RAW tools and conversions, so that can’t be it... And a lot of relative newcomers on the market are busy developing and integrating a DAM solution in their RAW development products.

Personally, the lack of a professional cataloguing feature and Phase completely ignoring complaints about it are my big frustrations with C1. I like the product enough to work around it, with a multitude of 5k to 7k image catalogs, but it’s 2019. They could do a lot better.

Lightroom is what's placed on people's lap so they just use it, also a Mac. People think this is the right way to do photography since they are given it or listened to someone and thus see everyone else do it out in the wild.

I took a Photo 101 class in college and we were taught to use Lightroom and all the computers in the room were iMac. I must have been the only one in class that ditched Lightroom for Bridge afterward than I found Capture One.

Catalog is a tacked on feature put in the middle of C1's current lifespan. It's evident that catalog is like Phase One's middle child.