A script to find duplicate image files

Discussion regarding all scripting related questions
Please DO NOT post to this thread anything that is not directly related to scripting of Capture One.

A script to find duplicate image files

Postby Eric Nepean » Wed Apr 12, 2017 8:39 am

This script searches the "All Images" collection to find duplicate image files.

It ignores RAW-JPG pairs.

It also detects and ignores burst sequences.

It doesn't do any checking for the Capture One version, if Capture One is running, if a catalog is open.

It's a little slow, but a user shouldn't have to run it often.

It works by first sorting the images in order of date, and then it walks through looking for sequences of images with the same date-time stamp. When it finds a sequence, it categorises the images according to image type.
Files with the same image type and the same date-time are reported as duplicates.

It detects burst sequences by extracting the numeric digits and the non-numeric characters separately from the image name. When it finds a set of images with the same date-time, the same image type, an unchanging non-numeric part, and a numeric part that increments by 1, that is a burst sequence which is not reported as duplicate.

The part that is too slow is probably the subroutine that separates file types and detects burst sequences; it runs much faster if I replace that with something simple.
Code: Select all
global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug, image_count, dupe_image_list, name_list

set debug to 1

tell application "Capture One 10"
   set date_list to EXIF capture date of every image
   set name_list to name of every image
end tell
set image_count to count of date_list

sort_date_list()

if debug ≥ 2 then pointer_check()

if debug ≥ 1 then log "Initialising Dupe Search"
set list_length to image_count
set next_pointer to start_pointer_up
set next_date to item next_pointer of date_list
set dupe_found to false
set first_date to 0
set dupe_image_list to {}
repeat with counter from 1 to list_length - 1
   set prev_date to next_date
   set prev_pointer to next_pointer
   set next_pointer to item next_pointer of up_pointer_list
   set next_date to item next_pointer of date_list
   if next_date = prev_date then
      if dupe_found = false then -- start of a series of duplicates -  found two
         if debug ≥ 3 then log "Found Dupe"
         set dupe_found to true
         set first_date to prev_date
         add_to_dupe_list(item prev_pointer of name_list)
         add_to_dupe_list(item next_pointer of name_list)
      else -- adding another duplicates
         add_to_dupe_list(item next_pointer of name_list)
      end if
   else if dupe_found = true then -- end of a series of duplicates
      if debug ≥ 4 then log dupe_image_list
      evaluate_display_dupe_list()
      set dupe_found to false
      set dupe_image_list to {}
      set first_date to 0
   end if
end repeat

################# Handlers

on add_to_dupe_list(image_name)
   global debug, dupe_image_list, name_list
   --set image_name to item image_index of name_list
   set image_type to text ((offset of "." in image_name) - (length of image_name)) thru -1 of image_name -- extract image type
   if (count of dupe_image_list) = 0 then -- if the list is empty
      set end of dupe_image_list to {image_type} -- add image type to the list
      set sublist_index to 1
   else -- is the image type in the list
      set type_found to false
      repeat with sublist_index from 1 to count of dupe_image_list
         if item 1 of item sublist_index of dupe_image_list = image_type then
            set type_found to true
            exit repeat
         end if
      end repeat
      if not type_found then --else add image type to the list
         set end of dupe_image_list to {image_type}
         set sublist_index to count of dupe_image_list
      end if
   end if
   set end of item sublist_index of dupe_image_list to image_name --add image name to the sublist
end add_to_dupe_list


on evaluate_display_dupe_list()
   global debug, dupe_image_list
   if debug ≥ 4 then log dupe_image_list
   
   repeat with sublist_index from 1 to count of dupe_image_list --For each sublist
      
      if (count of item sublist_index of dupe_image_list) > 2 then -- if there is more than one image name then determine if this could be a burst
         --log (item sublist_index of dupe_image_list)
         set is_burst to true
         set image_seq_list to {}
         repeat with sublist_counter from 2 to count of item sublist_index of dupe_image_list -- the first entry in the sublist is the extension
            set image_name_ext to item sublist_counter of item sublist_index of dupe_image_list
            set image_name to (text 1 thru ((offset of "." in image_name_ext) - 1) of image_name_ext)
            
            set image_name_s to quoted form of image_name
            do shell script "sed s/[^0-9]//g <<< " & image_name_s
            set image_num_list to the result
            if (count of image_num_list) = 0 then -- if there are no numbers in the image name, this is not a burst
               set is_burst to false
               exit repeat
            else
               set image_num to image_num_list as integer
               set end of image_seq_list to image_num
            end if
            
            do shell script "sed s/[0-9]//g <<< " & image_name_s
            set image_char_list to the result
            
            if sublist_counter = 2 then
               if (count of image_char_list) > 0 then
                  set has_image_char to true
                  set image_char_ref to image_char_list
               else
                  set has_image_char to false
               end if
               
               set image_seq_min to image_num
               
            else -- sublist counter >2
               
               if has_image_char then -- check that the image characters match the first image
                  if image_char_list ≠ image_char_ref then
                     set is_burst to false
                     exit repeat
                  end if
               else -- first image name has no characters
                  if (count of image_char_list) > 0 then
                     set is_burst to false
                     exit repeat
                  end if
               end if
               
               if image_num < image_seq_min then set image_seq_min to image_num -- find the starting sequence number
            end if
            if debug ≥ 4 then log image_num
            if debug ≥ 4 then log image_char_list
         end repeat
         
         if is_burst then -- check that the image numbers are sequential with increments of 1
            
            set image_seq_count to count of image_seq_list
            set image_sorted_seq_list to {image_seq_min}
            
            repeat with incr_counter from 2 to count of image_seq_count
               set seq_found to false
               set last_seq to end of image_sorted_seq_list
               repeat with seq_counter from 1 to image_seq_count
                  if item seq_counter of image_seq_list = last_seq + 1 then
                     set end of image_sorted_seq_list to (item seq_counter of image_seq_list)
                     set seq_found to true
                     exit repeat
                  end if
               end repeat
               if seq_found = false then
                  set is_burst to false
                  exit repeat
               end if
            end repeat
         end if
         
         if not is_burst then
            log (items 2 through (count of item sublist_index of dupe_image_list) of item sublist_index of dupe_image_list)
         end if
      end if
   end repeat
end evaluate_display_dupe_list

on sort_date_list()
   global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug, image_count
   
   set list_end to image_count + 1
   set list_begin to 0 as integer
   
   set list_length to 1 as integer
   set start_pointer_up to list_begin + 1
   set start_pointer_down to list_begin + 1
   set up_pointer_list to {list_end}
   set down_pointer_list to {list_begin}
   set image_index to 0
   
   if debug ≥ 1 then
      log {"Up Pointer List", start_pointer_up, "--", up_pointer_list}
      log {"Down Pointer List", start_pointer_down, "--", down_pointer_list}
      log (items 1 thru 2 of date_list)
      log {"Starting List", (items 1 thru list_length of date_list)}
   end if
   
   repeat while image_index < image_count
      set image_index to list_length + 1
      set image_date to item image_index of date_list
      if debug ≥ 2 then log {"Start Image ", image_index, image_date}
      
      set compare_index to image_index - 1
      set compare_date to item compare_index of date_list
      
      if image_date < compare_date then
         search_down(compare_index, compare_date)
      else
         search_up(compare_index, compare_date)
      end if
      if debug ≥ 2 then
         log {"Done", "up:", start_pointer_up, "-", up_pointer_list, "down:", start_pointer_down, "-", down_pointer_list}
         log "  "
      end if
      if image_index > 10 and debug ≥ 4 then exit repeat
   end repeat
end sort_date_list

on search_up(prev_candidate_image_index, prev_candidate_image_date)
   global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug
   if debug ≥ 3 then log {"Search_Up", prev_candidate_image_index, prev_candidate_image_date}
   set found to false
   repeat while not found
      set next_candidate_image_index to item prev_candidate_image_index of up_pointer_list
      if next_candidate_image_index = list_end then -- we have reached the end of the list
         if debug ≥ 3 then log "reached the start of the list"
         insert_image_at(next_candidate_image_index, prev_candidate_image_index) --insert the new item at the list end
         set found to true
      else --search for the first image which is newer than this image
         set next_candidate_image_date to item next_candidate_image_index of date_list
         if image_date > next_candidate_image_date then --search further up
            set prev_candidate_image_index to next_candidate_image_index
         else
            insert_image_at(next_candidate_image_index, prev_candidate_image_index)
            set found to true
         end if
      end if
   end repeat
end search_up

on search_down(prev_candidate_image_index, prev_candidate_image_date)
   global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug
   if debug ≥ 3 then log {"Search_Down", prev_candidate_image_index, prev_candidate_image_date}
   set found to false
   repeat while not found
      set next_candidate_image_index to item prev_candidate_image_index of down_pointer_list
      if next_candidate_image_index = list_begin then --we have reached the start of the list
         if debug ≥ 3 then log "reached the start of the list"
         insert_image_at(prev_candidate_image_index, next_candidate_image_index) --insert the new item at the list start
         set found to true
      else --search for the first image which is older than this image
         set next_candidate_image_date to item next_candidate_image_index of date_list
         if image_date < next_candidate_image_date then -- search further down
            set prev_candidate_image_index to next_candidate_image_index
         else
            insert_image_at(prev_candidate_image_index, next_candidate_image_index)
            set found to true
         end if
      end if
   end repeat
end search_down

on insert_image_at(higher_image_index, lower_image_index)
   global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug
   if debug ≥ 3 then log {"Insertion", higher_image_index, lower_image_index}
   set end of up_pointer_list to higher_image_index
   set end of down_pointer_list to lower_image_index
   
   if higher_image_index = list_end then
      if debug ≥ 3 then log {"End Insertion"}
      set start_pointer_down to image_index
   else
      set item higher_image_index of down_pointer_list to image_index
   end if
   
   if lower_image_index = list_begin then
      set start_pointer_up to image_index
      if debug ≥ 3 then log {"Begin Insertion"}
   else
      set item lower_image_index of up_pointer_list to image_index
   end if
   
   set list_length to list_length + 1
end insert_image_at

on pointer_check()
   global date_list, list_begin, list_length, list_end, up_pointer_list, down_pointer_list, start_pointer_up, start_pointer_down, image_date, image_index, debug
   
   if debug ≥ 4 then
      log (items 1 thru list_length of date_list)
      set next_pointer to start_pointer_up
      set sorted_date_list to {item next_pointer of date_list}
      set sorted_image_list to {next_pointer}
      
      repeat with counter from 1 to list_length - 1
         set next_pointer to item next_pointer of up_pointer_list
         set end of sorted_date_list to item next_pointer of date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log sorted_date_list
      log sorted_image_list
      
      set next_pointer to start_pointer_down
      set sorted_date_list to {item next_pointer of date_list}
      set sorted_image_list to {next_pointer}
      repeat with counter from 1 to list_length - 1
         set next_pointer to item next_pointer of down_pointer_list
         set end of sorted_date_list to item next_pointer of date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log sorted_date_list
      log sorted_image_list
      
   end if
   
   if debug ≥ 1 then
      set next_pointer to start_pointer_up
      set this_date to item next_pointer of date_list
      set list_good to true
      repeat with counter from 1 to list_length - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of up_pointer_list
         set this_date to item next_pointer of date_list
         if this_date < prev_date then
            set list_good to false
            log {"Up pointer error between items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of up_pointer_list ≠ list_end then
         log {"error in last up pointer", next_pointer, (item next_pointer of up_pointer_list)}
      end if
      
      if list_good = true then log "Up Pointers are OK"
      
      set next_pointer to start_pointer_down
      set this_date to item next_pointer of date_list
      set list_good to true
      repeat with counter from 1 to list_length - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of down_pointer_list
         set this_date to item next_pointer of date_list
         if this_date > prev_date then
            set list_good to false
            log {"Down pointer error between items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of down_pointer_list ≠ list_begin then
         log {"error in last down pointer", next_pointer, (item next_pointer of down_pointer_list)}
      end if
      
      if list_good = true then log "Down Pointers are OK"
      
   end if
   
end pointer_check

Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Mon Apr 17, 2017 3:02 am

Thanks Eric

A really, really basic request: how do we use AppleScript in C1?

Thanks in advance.
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby Eric Nepean » Mon Apr 17, 2017 5:09 am

ShaneB wrote:Thanks Eric

A really, really basic request: how do we use AppleScript in C1?

Thanks in advance.

Hi Shane
Applescript is a SW tool and language that can talk to many applications in OSX, like Finder, Calendar, TextEditor, CaptureOne and Adobe Photoshop ... and many more.

So you don't use Applescript from "inside" C1, you run Applescript separately from C1, and it asks C1 (and other bits and pieces of SW) for information, and can tell C1 to do things.

In order to run an AppleScript script like I just provided, first run Capture One and open the catalog you are using. If Capture One isn't running, then Applescript can't ask it stuff like "give me the dates of all the images in collection 1" or "give me the name of the open document"

Then, while Capture One is busy opening, use finder to navigate over to the /Applications/Utilities folder, and find the Application called "Script Editor" and start it (it used to be called just "Applescript")

Once Script Editor starts it will ask to open a document, and you won't have one just yet, so just click "new document" and a blank one will open, called "Untitled.scpt"

The next thing to do is to make a place to save the blank document (script), and save it there with a sensible name, like "FindDupes.scpt"
So click on save, and the usual menu will come up allowing you to rename the document and create a new folder to save it in. Don't save it to iCloud. I have made a folder /Users/EV/Documents/Scripts where I save all my scripts.

The next thing you want to do is click on the "window" menu at the top of the screen and click "log history". This opens the log history window, which is where all the output comes. In the log history window, select the tab "messages". It will of course be empty, you haven't run anything yet, so no messages.

Now, formalities are complete. Copy the script from the Phase one website (remember to "select all" - there's a lot of code in that little window!) and paste it into the blank script document. The script will be all purple text, meaning that Script editor hasn't done anything with it yet.

Now click on the little "hammer and tongs" symbol at the top of the window (this is the compile icon), and (assuming no errors) very quickly the script will be reformatted, and various bits of it will turn different colors according to its purpose. If you get any errors my first guess might be that you didn't copy all of the script.

Now save the document again (but now it's been compiled).

At this point, just click the triangular symbol to the left of the compile icon, this will run the script. There are a couple of indicators on document window that indicate that the script is running, the "run" icon is greyed out, a little turning wheel, and the word "running.." somewhere.

Click on the log history window and watch for stuff to appear. Keep in mind that Capture One has to runing through out all of this.

Once the script is done, you can copy any useful information nfrom the log history window, and save it in a text editor document for future reference.
Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby Eric Nepean » Mon Apr 17, 2017 5:36 am

Here is the updated Find Duplicates Script.

A number of things have been fixed and optimised to make it run faster.

I've tested it on catalogs with 4000 and 15000 referenced images, catalogs and images stored on an external SSD.

The original version took quite a long time (i.e. overnight) to chew throught the 15000 image catalog, this version does it in a couple of minutes.

For large catalogs, it helps if you open the "all images" catalog and sort by date before running this script.

I'm using Capture One Pro 10.0.2 running on OSX 10.11.6 on a late 2015 27" imac with 24GB of RAM and a 4 GHz i7.
Code: Select all
-- Applescript to search a COP 10 Catalog for Duplicate Images
-- Version 1.0 !! NO SUPPORT !!  Eric Valk, Ottawa, Canada

-- ***To Initialise
-- Start Script Editor, open a new (blank) file, copy and paste all of this code into the file, compile (hammer symbol) and save.
-- Best if you make "Scripts" folder somewhere handy in your Douments or Desktop

-- *** Operation
-- Start Capture One Pro with the desired catalog
-- Open  the compiled and saved file in Script Editor
-- Open the Script Editor log window, and select the messages tab
-- Run the script
-- Results appear first in the AppleScript Log when the search  is completed
-- This script does not write or delete any information in the COP Catalog or Session or the image file

global image_date_list, debug, heavy_debug_image_limit, image_count, image_name_list, no_date_info_D

set debug to 1
set heavy_debug_image_limit to 10

set COPDocName to ""

set minCOPversion to 10.0 as real


tell application "System Events"
   set COPProcList to every process whose name contains "Capture One" and background only is false
end tell

if (count of COPProcList) = 0 then error "COP is not running"
if (count of COPProcList) ≠ 1 then error "Unexpected: >1 COP instances"
set theAppRef to item 1 of COPProcList

tell application "System Events" to set theAppName to ((get name of theAppRef) as text)

using terms from application "Capture One 10"
   tell application theAppName to set copVersion to (get app version)
end using terms from

tell application "System Events" to set copDetailedVersion to get version of my application theAppName

if debug ≥ 2 then
   --properties of application "Capture One 10"
   tell application "System Events"
      log (get every process whose background only is false)
   end tell
   log "theAppName: " & theAppName
   log "COP Version: " & copVersion
   log "COP Detailed Version: " & copDetailedVersion
   log "Max Search Level: " & maxSearchLevel
   log "Measured Search Time Per Variant: " & searchTimePerVariant
   log "maxSearchTime: " & maxSearchTime
end if

if the theAppName ≠ "Capture One 10" then
   display notification "Wrong COP Application"
   error "Found COP Application " & theAppName & " The only supported COP application is Capture One 10"
end if

set numCOPversion to (item 1 of (extract_version_strings(".", (word -1 of copVersion)))) as integer
if numCOPversion < minCOPversion then
   display notification "COP Version is too low"
   error "This COP Version is " & numCOPversion & " - the minimum supported COP version is " & minCOPversion
end if

tell application "System Events" to set frontmost of process theAppName to true

set everyDocName to name of every document of my application theAppName
if (count of everyDocName) = 0 then error theAppName & " has no open documents"

-- searching for the window with AXMain = true can choose the Activities, Live View, Events or Viewer windows if they are open
-- so we search for windows with the same name as open documents

tell application "System Events"
   tell process theAppName
      set WinTitles_L to get title of every window
   end tell
end tell

if debug ≥ 2 then log "Windows: "
if debug ≥ 2 then log WinTitles_L

set WinTitlesCount to count of WinTitles_L
set COPDocName to ""

if WinTitlesCount > 0 then
   repeat with WinTitlesCtr from 1 to WinTitlesCount --find the first window which is a catalog
      set thisWinTitle to item WinTitlesCtr of WinTitles_L
      if everyDocName contains thisWinTitle then -- we find the document's main window
         if debug ≥ 2 then log "found: " & thisWinTitle
         set COPDocName to thisWinTitle
         exit repeat
      end if
   end repeat
end if

if COPDocName = "" then
   display notification "Document not found " & everyDocName & " :" & WinTitlesCount & ": " & WinTitles_L
   error "No open documents found"
end if

log "Capture One Document: " & COPDocName

using terms from application "Capture One 10" -- now check if this is a catalog
   tell application theAppName to set thisDocKind to (get kind of document thisWinTitle) as text
end using terms from
log "Capture One " & copDetailedVersion & " " & thisDocKind

if thisDocKind ≠ "catalog" then
   display notification "This Applescript only works with Catalogs" & return & COPDocName & " is a " & thisDocKind
   error "This Applescript only works with Catalogs:   " & COPDocName & " is a " & thisDocKind
end if

------ Main Application starts here

set no_date_info_D to (date "Thursday, January 1, 1970 at 12:00:00 AM")

if debug ≥ 1 then log "Getting Data"
tell application "Capture One 10"
   set image_date_list to EXIF capture date of every image
end tell
set image_count to count of image_date_list

tell application "System Events" to set frontmost of process "Script Editor" to true

log "Retrieved " & image_count & " images"

sort_imagedates()

if debug ≥ 2 then pointer_check()

cleanup_sort()

tell application "Capture One 10" to set image_name_list to name of every image

Duplicate_Search()

cleanup()

log "Done"

################# Handlers
on cleanup()
   global image_date_list, image_name_list, pointer_dates_L_Len, up_pointer_dates_L, up_start_pointer_dates
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying all  Variables" --  Avoids stack overflow problems
   set image_date_list to {}
   set image_name_list to {}
   set up_pointer_dates_L to {}
   set pointer_dates_L_Len to {}
   set up_start_pointer_dates to {}
end cleanup

on cleanup_sort()
   global down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying Variables used only for Sorting" --  Avoids stack overflow problems
   set down_pointer_dates_L to {}
   set down_start_pointer_dates to {}
   set ymd_list to {}
   set yearmonth_list to {}
   set year_list to {}
   set ymd_start_pointers to {}
end cleanup_sort

on Duplicate_Search()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, up_start_pointer_dates, debug, heavy_debug_image_limit, image_count, duplicate_image_list, image_name_list
   if debug ≥ 1 then log "Starting Duplicate Search"
   set pointer_dates_L_Len to count of up_pointer_dates_L
   set this_pointer to up_start_pointer_dates
   set this_date to item this_pointer of image_date_list
   set dupe_found to false
   set first_dup_date to 0
   set duplicate_image_list to {}
   repeat with counter from 1 to pointer_dates_L_Len - 1
      set prev_date to this_date
      set prev_pointer to this_pointer
      set this_pointer to item this_pointer of up_pointer_dates_L
      set this_date to item this_pointer of image_date_list
      if this_date = prev_date then
         if dupe_found = false then -- start of a series of duplicates -  found first two
            if debug ≥ 3 then log "Found Dupe"
            set dupe_found to true
            set first_dup_date to prev_date
            add_to_dupe_list(item prev_pointer of image_name_list)
            add_to_dupe_list(item this_pointer of image_name_list)
         else -- adding other duplicates
            add_to_dupe_list(item this_pointer of image_name_list)
         end if
      else if dupe_found = true then -- end of a series of duplicates
         evaluate_display_dupe_list(prev_date)
         set dupe_found to false
         set duplicate_image_list to {}
         set first_dup_date to 0
      end if
      if debug ≥ 4 and counter > heavy_debug_image_limit then error
   end repeat
end Duplicate_Search

on add_to_dupe_list(image_name)
   global debug, duplicate_image_list, image_name_list
   
   set image_type to text -1 of image_name
   set image_type_found to false
   repeat with char_counter from 2 to 5
      set this_char to item (-char_counter) of image_name
      if this_char = "." then
         set image_type_found to true
         exit repeat
      else
         set image_type to this_char & image_type
      end if
   end repeat
   if not image_type_found then error "Image type not found in: " & image_name
   
   set image_type_found to false
   repeat with sublist_index from 1 to count of duplicate_image_list
      if item 1 of item sublist_index of duplicate_image_list = image_type then
         set image_type_found to true
         exit repeat
      end if
   end repeat
   if not image_type_found then --add image type to the list
      set end of duplicate_image_list to {image_type}
      set sublist_index to count of duplicate_image_list
   end if
   
   set end of item sublist_index of duplicate_image_list to image_name --add image name to the sublist
   
end add_to_dupe_list


on evaluate_display_dupe_list(dupe_date)
   global debug, duplicate_image_list, no_date_info_D
   -- the intent of this handler is to eleminate obvious non-duplicates
   ----Images with the same date but different image types (likely a RAW-JPG pair, but this is not checked)
   ----Image groups which are part of a (high speed) burst sequence
   -- Images which are reported are likely duplicates, but are NOT guaranteed duplicates
   
   if debug ≥ 3 then log "Evaluating  Duplicate"
   if debug ≥ 4 then log duplicate_image_list
   
   if dupe_date ≠ no_date_info_D then
      repeat with sublist_index from 1 to count of duplicate_image_list --For each sublist
         
         if (count of item sublist_index of duplicate_image_list) > 2 then -- if there is more than one image name then determine if this could be a burst
            if debug ≥ 3 then log (item sublist_index of duplicate_image_list)
            set is_burst to true -- start by assuming its a burst, and then test it
            set image_seq_list to {}
            set image_name_ext to item 1 of item sublist_index of duplicate_image_list
            repeat with sublist_counter from 2 to length of item sublist_index of duplicate_image_list -- the first entry in the sublist is the extension
               
               set image_name to item sublist_counter of item sublist_index of duplicate_image_list
               set image_num_list to ""
               set image_char_list to ""
               repeat with char_counter from 1 to count of image_name
                  set this_char to item char_counter of image_name
                  if this_char = "." then
                     exit repeat
                  else if "0123456789" contains this_char then
                     set image_num_list to image_num_list & this_char
                  else
                     set image_char_list to image_char_list & this_char
                  end if
               end repeat
               
               set image_num_count to count of image_num_list
               if image_num_count = 0 then -- if there are no numbers in the image name, this is not a burst
                  set is_burst to false
                  exit repeat
               end if
               set image_num to image_num_list as integer
               set end of image_seq_list to image_num
               
               if sublist_counter = 2 then
                  if (count of image_char_list) > 0 then
                     set has_image_char to true
                     set image_char_ref to image_char_list
                  else
                     set has_image_char to false
                  end if
                  set image_num_count_ref to image_num_count
                  set image_seq_min to image_num
               else -- case: sublist counter >2
                  
                  if has_image_char then -- check that the image characters match the first image
                     if image_char_list ≠ image_char_ref then
                        set is_burst to false
                        exit repeat
                     end if
                  else -- first image name has no characters
                     if (count of image_char_list) > 0 then
                        set is_burst to false
                        exit repeat
                     end if
                  end if
                  
                  if image_num_count ≠ image_num_count_ref then -- now check the numbers
                     set is_burst to false
                     exit repeat
                  end if
                  if image_num < image_seq_min then set image_seq_min to image_num -- find the starting sequence number
               end if
               if debug ≥ 4 then log image_num
               if debug ≥ 4 then log image_char_list
            end repeat
            
            if is_burst then -- check that the image numbers are sequential with increments of 1
               set image_seq_count to count of image_seq_list
               set last_seq to image_seq_min
               repeat with incr_counter from 2 to image_seq_count
                  set this_seq_found to false
                  repeat with seq_counter from 1 to image_seq_count
                     if item seq_counter of image_seq_list = last_seq + 1 then
                        set last_seq to (item seq_counter of image_seq_list)
                        set this_seq_found to true
                        exit repeat
                     end if
                  end repeat
                  if this_seq_found = false then
                     set is_burst to false
                     exit repeat
                  end if
               end repeat
            end if
            if not is_burst then
               log {dupe_date, "Possible-->", (items 2 through (count of item sublist_index of duplicate_image_list) of item sublist_index of duplicate_image_list)}
            end if
         end if
      end repeat
   else
      log {"Images with no date info: ", dupe_date, duplicate_image_list}
      
   end if
end evaluate_display_dupe_list

on sort_imagedates()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Starting Date Sort"
   set list_end to image_count + 1
   set list_begin to 0 as integer
   set list_start to list_begin + 1
   set pointer_dates_L_Len to 1 as integer
   set up_start_pointer_dates to list_start
   set down_start_pointer_dates to list_start
   set up_pointer_dates_L to {list_end}
   set down_pointer_dates_L to {list_begin}
   set iteration_counter to 0 as integer
   set this_image_index to list_start
   set this_image_date to item this_image_index of image_date_list
   set {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day} to this_image_date
   
   set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set year_list to {{year:this_image_date_year, up_ptr:list_end, down_ptr:list_begin, ym_ptr:list_start}}
   set yearmonth_list to {{year:this_image_date_year, month:this_image_date_month, up_ptr:list_end, down_ptr:list_begin, ymd_ptr:list_start}}
   set ymd_list to {{year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:list_end, down_ptr:list_begin, dates_ptr:list_start}}
   
   if debug ≥ 2 then log_status()
   
   repeat while this_image_index < image_count
      set compare_index to this_image_index
      set compare_date to this_image_date
      set this_image_index to this_image_index + 1
      set this_image_date to item this_image_index of image_date_list
      if debug ≥ 2 then log {"Now processing image ", this_image_index, this_image_date, "with Compare Image ", compare_index, compare_date}
      
      
      if (get {year, month, day} of this_image_date) ≠ (get {year, month, day} of compare_date) then set {compare_index, compare_date} to update_compare_index()
      
      if this_image_date < compare_date then
         search_down(compare_index, compare_date)
      else
         search_up(compare_index, compare_date)
      end if
      
      if debug ≥ 2 then
         log {"Done", "up:", up_start_pointer_dates, "-", up_pointer_dates_L, "down:", down_start_pointer_dates, "-", down_pointer_dates_L}
         log "  "
         if debug ≥ 4 then
            log_status()
            --if this_image_index > 10 then error
            if this_image_index > heavy_debug_image_limit and debug ≥ 4 then exit repeat
         end if
      end if
   end repeat
   if debug ≥ 1 then log {"number of Iterations: ", iteration_counter}
   if debug ≥ 2 then log_status()
end sort_imagedates

on log_status()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates, this_image_date, this_image_index, debug, image_count
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   log "*****"
   log "Status"
   log {"Up Pointer List", up_start_pointer_dates, "--", up_pointer_dates_L}
   log {"Down Pointer List", down_start_pointer_dates, "--", down_pointer_dates_L}
   log {"Image Date List:", (items 1 thru pointer_dates_L_Len of image_date_list)}
   log {"ymd_start_pointers", ymd_start_pointers}
   log {"year_list", year_list}
   log {"yearmonth_list", yearmonth_list}
   log {"ymd_list", ymd_list}
   log "*****"
end log_status

on update_compare_index()
   -- This subroutine maintains a list of pointers to years, months and days used to increase the search speed
   -- It returns a pointer to an already sorted image in the nearest day (same day if there is one)
   -- there is a lot of code, but most of it does not execute often
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   
   set {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day} to this_image_date
   set indexing_entry_missing to false
   
   -- Search for Year entry
   -- Reference: set year_list to {{year:this_image_date_year, up_ptr:list_end, down_ptr:list_begin, ym_ptr:list_start}}
   -- Reference: set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set year_index_found to false
   set prev_year_index to list_begin
   set compare_year_index to year_up of ymd_start_pointers
   repeat
      set iteration_counter to iteration_counter + 1
      if compare_year_index = list_end then exit repeat -- hit the end of year_list and didn't find the entry
      if (get year of (item compare_year_index of year_list)) = this_image_date_year then
         set this_year_index to compare_year_index
         set year_index_found to true
         if debug ≥ 3 then log "year found"
         set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
         exit repeat
      else if (year of (item compare_year_index of year_list)) < this_image_date_year then
         set prev_year_index to compare_year_index
         set compare_year_index to up_ptr of item prev_year_index of year_list
      else -- (year of (item compare_year_index of year_list)) > this_image_date_year
         exit repeat -- didn't find the entry between already existing entries
      end if
   end repeat
   
   if not year_index_found then -- add index entry for the year
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding Year", this_image_date, "this_image_index", this_image_index}
      
      set end of year_list to {year:this_image_date_year, up_ptr:compare_year_index, down_ptr:prev_year_index, ym_ptr:(1 + (length of yearmonth_list))}
      set this_year_index to length of year_list
      
      if compare_year_index = list_end then
         set year_down of ymd_start_pointers to this_year_index
         set compare_yearmonth_index to list_end
         set prev_yearmonth_index to yearmonth_down of ymd_start_pointers
      else
         set down_ptr of item compare_year_index of year_list to this_year_index
         set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
         set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      end if
      
      if prev_year_index = list_begin then
         set year_up of ymd_start_pointers to this_year_index
      else
         set up_ptr of item prev_year_index of year_list to this_year_index
      end if
      
      if debug ≥ 3 then log {"Added this_year_index", this_year_index, "compare_year_index", compare_year_index, "prev_year_index", prev_year_index}
      if debug ≥ 4 then log {year_list}
      if debug ≥ 4 then log {ymd_start_pointers}
   end if
   
   -- Search for YearMonth entry
   -- Reference: set yearmonth_list to {{year:this_image_date_year, month:this_image_date_month, up_ptr:list_end, down_ptr:list_begin, ymd_ptr:list_start}}
   -- Reference: set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set yearmonth_index_found to false
   if not indexing_entry_missing then -- search for the Year:month
      set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_yearmonth_index = list_end then exit repeat -- hit the end of yearmonth_list and didn't find the entry
         if (get {year, month} of (item compare_yearmonth_index of yearmonth_list)) = {this_image_date_year, this_image_date_month} then
            set this_yearmonth_index to compare_yearmonth_index
            set yearmonth_index_found to true
            if debug ≥ 3 then log "yearmonth found"
            set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
            exit repeat
         else if (((get month of (item compare_yearmonth_index of yearmonth_list)) < this_image_date_month) and ((get year of (item compare_yearmonth_index of yearmonth_list)) = this_image_date_year)) then
            set prev_yearmonth_index to compare_yearmonth_index
            set compare_yearmonth_index to up_ptr of item prev_yearmonth_index of yearmonth_list
         else -- (month of (item compare_yearmonth_index of yearmonth_list)) > this_image_date_month  or (year of (item compare_yearmonth_index of yearmonth_list)) ≠ this_image_date_year)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not yearmonth_index_found then -- add index entry for the yearmonth
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonth", this_image_date, "this_image_index", this_image_index}
      
      set end of yearmonth_list to {year:this_image_date_year, month:this_image_date_month, up_ptr:compare_yearmonth_index, down_ptr:prev_yearmonth_index, ymd_ptr:(1 + (length of ymd_list))}
      set this_yearmonth_index to length of yearmonth_list
      
      if compare_yearmonth_index = list_end then
         set yearmonth_down of ymd_start_pointers to this_yearmonth_index
         set compare_ymd_index to list_end
         set prev_ymd_index to ymd_down of ymd_start_pointers
      else
         set down_ptr of item compare_yearmonth_index of yearmonth_list to this_yearmonth_index
         set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
         set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      end if
      if prev_yearmonth_index = list_begin then
         set yearmonth_up of ymd_start_pointers to this_yearmonth_index
      else
         set up_ptr of item prev_yearmonth_index of yearmonth_list to this_yearmonth_index
      end if
      
      if year_index_found then -- if year already existed then update its yearmonth pointer if this is the first yearmonth of the year
         if prev_yearmonth_index = list_begin then
            set ym_ptr of item this_year_index of year_list to this_yearmonth_index
         else
            if (year of item prev_yearmonth_index of yearmonth_list) ≠ this_image_date_year then
               set ym_ptr of item this_year_index of year_list to this_yearmonth_index
            end if
         end if
      end if
      
      if debug ≥ 3 then log {"Added this_yearmonth_index", this_yearmonth_index, "compare_yearmonth_index", compare_yearmonth_index, "prev_yearmonth_index", prev_yearmonth_index}
      if debug ≥ 4 then log {yearmonth_list}
      if debug ≥ 4 then log {ymd_start_pointers}
   end if
   
   -- Search for YMD entry
   -- Reference: set ymd_list to {{year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:list_end, down_ptr:list_begin, dates_ptr:list_start}}
   -- Reference: set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set ymd_index_found to false
   if not indexing_entry_missing then -- search for Year:month:day
      set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_ymd_index = list_end then exit repeat -- hit the end of ymd_list and didn't find the entry
         if (get {year, month, day} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month, this_image_date_day} then
            set this_ymd_index to compare_ymd_index
            set ymd_index_found to true
            if debug ≥ 3 then log "ymd found"
            exit repeat
         else if (((get day of (item compare_ymd_index of ymd_list)) < this_image_date_day) and ((get {year, month} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month})) then
            set prev_ymd_index to compare_ymd_index
            set compare_ymd_index to up_ptr of item prev_ymd_index of ymd_list
         else -- (day of (item compare_ymd_index of ymd_list)) > this_image_date_month  or (month of (item compare_ymd_index of ymd_list)) ≠ this_image_date_month)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not ymd_index_found then -- add index entry for the yearmonthday
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonthDay", this_image_date, "this_image_index", this_image_index}
      
      set end of ymd_list to {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:compare_ymd_index, down_ptr:prev_ymd_index, dates_ptr:this_image_index}
      set this_ymd_index to length of ymd_list
      
      if compare_ymd_index = list_end then
         set ymd_down of ymd_start_pointers to this_ymd_index
      else
         set down_ptr of item compare_ymd_index of ymd_list to this_ymd_index
      end if
      if prev_ymd_index = list_begin then
         set ymd_up of ymd_start_pointers to this_ymd_index
      else
         set up_ptr of item prev_ymd_index of ymd_list to this_ymd_index
      end if
      
      if yearmonth_index_found then -- if yearmonth already existed then update its ymd pointer if this is the first ymd of the month
         if prev_ymd_index = list_begin then
            set ymd_ptr of item this_yearmonth_index of yearmonth_list to this_ymd_index
         else
            if (month of item prev_ymd_index of ymd_list) ≠ this_image_date_month then
               set ymd_ptr of item this_yearmonth_index of yearmonth_list to this_ymd_index
            end if
         end if
      end if
      
      if debug ≥ 3 then log {"Added this_ymd_index", this_ymd_index, "compare_ymd_index", compare_ymd_index, "prev_ymd_index", prev_ymd_index}
      if debug ≥ 4 then log {ymd_list}
      if debug ≥ 4 then log {ymd_start_pointers}
   end if
   
   if indexing_entry_missing then
      if compare_ymd_index = list_end then
         set updated_compare_date_ptr to down_start_pointer_dates -- the pointer to the image with highest date already sorted
      else
         set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- the pointer to the first image for the next higher date
      end if
   else -- indexing entry was not missing
      set updated_compare_date_ptr to dates_ptr of (item this_ymd_index of ymd_list) -- the pointer to the first image for this date
      --set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- the pointer to the first image for the next higher date
   end if
   
   set updated_compare_date to item updated_compare_date_ptr of image_date_list
   return {updated_compare_date_ptr, updated_compare_date}
   
end update_compare_index

on search_up(prev_candidate_image_index)
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, up_start_pointer_dates, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Up", prev_candidate_image_index}
   set found to false
   repeat while not found
      set next_candidate_image_index to item prev_candidate_image_index of up_pointer_dates_L
      if next_candidate_image_index = list_end then -- we have reached the end of the list
         if debug ≥ 3 then log "reached the start of the list"
         insert_image_at(next_candidate_image_index, prev_candidate_image_index) --insert the new item at the list end
         set found to true
      else --search for the first image which is newer than this image
         set next_candidate_image_date to item next_candidate_image_index of image_date_list
         if this_image_date > next_candidate_image_date then --search further up
            set prev_candidate_image_index to next_candidate_image_index
         else
            insert_image_at(next_candidate_image_index, prev_candidate_image_index)
            set found to true
         end if
      end if
      set iteration_counter to iteration_counter + 1
   end repeat
end search_up

on search_down(prev_candidate_image_index)
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, down_pointer_dates_L, down_start_pointer_dates, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Down", prev_candidate_image_index}
   set found to false
   repeat while not found
      set next_candidate_image_index to item prev_candidate_image_index of down_pointer_dates_L
      if next_candidate_image_index = list_begin then --we have reached the start of the list
         if debug ≥ 3 then log "reached the start of the list"
         insert_image_at(prev_candidate_image_index, next_candidate_image_index) --insert the new item at the list start
         set found to true
      else --search for the first image which is older than this image
         set next_candidate_image_date to item next_candidate_image_index of image_date_list
         if this_image_date < next_candidate_image_date then -- search further down
            set prev_candidate_image_index to next_candidate_image_index
         else
            insert_image_at(prev_candidate_image_index, next_candidate_image_index)
            set found to true
         end if
      end if
      set iteration_counter to iteration_counter + 1
   end repeat
end search_down

on insert_image_at(higher_image_index, lower_image_index)
   global list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates, this_image_date, this_image_index, debug
   if debug ≥ 3 then log {"Insertion", higher_image_index, lower_image_index}
   set end of up_pointer_dates_L to higher_image_index
   set end of down_pointer_dates_L to lower_image_index
   
   if higher_image_index = list_end then
      if debug ≥ 3 then log {"End Insertion"}
      set down_start_pointer_dates to this_image_index
   else
      set item higher_image_index of down_pointer_dates_L to this_image_index
   end if
   
   if lower_image_index = list_begin then
      set up_start_pointer_dates to this_image_index
      if debug ≥ 3 then log {"Begin Insertion"}
   else
      set item lower_image_index of up_pointer_dates_L to this_image_index
   end if
   
   set pointer_dates_L_Len to pointer_dates_L_Len + 1
end insert_image_at

on pointer_check()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, up_start_pointer_dates, down_start_pointer_dates, this_image_date, this_image_index, debug
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   
   if debug ≥ 4 then --  really long detailed listing  Only do this with the short list in debug level; 4.
      log "Listing Dates List Pointers"
      log (items 1 thru pointer_dates_L_Len of image_date_list)
      set next_pointer to up_start_pointer_dates
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of up_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log sorted_image_date_list
      log sorted_image_list
      
      set next_pointer to down_start_pointer_dates
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of down_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log sorted_image_date_list
      log sorted_image_list
      
   end if
   
   if debug ≥ 2 then
      log "Checking Dates List Pointers"
      set next_pointer to up_start_pointer_dates
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of up_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date < prev_date then
            set list_good to false
            log {"Up pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of up_pointer_dates_L ≠ list_end then
         log {"error in last Dates List up pointer", next_pointer, (item next_pointer of up_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Up Pointers for Dates List are OK"
      
      set next_pointer to down_start_pointer_dates
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of down_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date > prev_date then
            set list_good to false
            log {"Down pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of down_pointer_dates_L ≠ list_begin then
         log {"error in last Dates List down pointer", next_pointer, (item next_pointer of down_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Down Pointers for Dates List are OK"
      
      
   end if
   
end pointer_check

on extract_version_strings(d, t)
   set astid to AppleScript's text item delimiters
   try
      set AppleScript's text item delimiters to d
      set version_strings to text items of t
   on error
      set AppleScript's text item delimiters to astid
   end try
   set AppleScript's text item delimiters to astid
   
   return version_strings
end extract_version_strings

Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Tue Apr 18, 2017 1:23 am

Gidday Eric

Firstly, thank you for your very detailed response to my message. Much appreciated!

Secondly, with your revised script, I'm getting an error message ("Invalid date and time date") from the compiler on the line:

set no_date_info_D to (date "Thursday, January 1, 1970 at 12:00:00 AM")

Changing it to: "set no_date_info_D to (date "Thursday, 1 January 1970 at 12:00:00")" solved it for me.

Regards
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby ShaneB » Tue Apr 18, 2017 1:39 am

Hi Eric

Running the script with the revised date format produced the following error message:

Result:
error "System Events got an error: Script Editor is not allowed assistive access." number -1728 from every window of process "Capture One 10"

All the best
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby Eric Nepean » Tue Apr 18, 2017 8:44 am

ShaneB wrote:Hi Eric

Running the script with the revised date format produced the following error message:

Result:
error "System Events got an error: Script Editor is not allowed assistive access." number -1728 from every window of process "Capture One 10"

All the best


Hi Shane
I made some changes to simplify the logic, and also to remove the assistive access problem, and to set up the NoDateTime in such away that it doesn't get bollixed up by the user's default time settings (apparently yours and mine are different)

Here's version 1.1

Code: Select all
-- Applescript to search a COP 11 Catalog for Duplicate Images
-- Version 1.1 !! No Support !!  Eric Valk, Ottawa, Canada

-- ***To Initialise
-- Start Script Editor, open a new (blank) file, copy and paste all of this code into the file, compile (hammer symbol) and save.
-- Best if you make "Scripts" folder somewhere handy in your Douments or Desktop

-- *** Operation
-- Start Capture One Pro with the desired catalog
-- Open  the compiled and saved file in Script Editor
-- Open the Script Editor log window, and select the messages tab
-- Run the script
-- Results appear first in the AppleScript Log when the search  is completed
-- This script does not write or delete any information in the COP Catalog or Session or the image file

global debug, heavy_debug_image_limit
global image_date_list, image_count, image_name_list, no_date_info_D, start_date_D, end_date_D, image_count0, list_start, list_finish

log "Find Dupes Version 1.1"
set debug to 1
set heavy_debug_image_limit to 10

validateCOP()

## Set up date limits
## the date assigned if there is no date
set no_date_string to "01/01/70"
set no_date_info_D to (date no_date_string)
## The starting date for sorting (one year before the date assigned if there is no date)
copy no_date_info_D to start_date_D
set year of start_date_D to (year of start_date_D) - 1
## # The ending date for sorting (one year after the current date)
set end_date_D to current date
set year of end_date_D to (year of end_date_D)
set time of end_date_D to 0

if debug ≥ 1 then log "Getting image dates"
tell application "Capture One 10"
   set image_date_list to EXIF capture date of every image
end tell
set image_count0 to length of image_date_list
log "Retrieved " & image_count0 & " images"
if debug ≥ 2 then log {items 1 through 2 of image_date_list, class of image_date_list, "end", (item image_count0 of image_date_list)}

## Add the starting and ending  dates at the beginning of the date list
## This provide search endpoints so that the search routines will not run off the ends of the list
## All the pointers to image lists are now 2 higher than they would otherwise be
tell image_date_list
   set beginning of image_date_list to end_date_D
   set beginning of image_date_list to start_date_D
end tell
set image_count to length of image_date_list
set list_start to 1
set list_finish to 2

tell application "System Events" to set frontmost of process "Script Editor" to true

sort_imagedates()

if debug ≥ 2 then pointer_check()

cleanup_sort()

Duplicate_Search()

cleanup_dupe_search()

if debug ≥ 1 then log "Getting image names"
tell application "Capture One 10" to set image_name_list to name of every image
tell image_date_list -- to make the items in the name list sync with the items in the date list
   set beginning of image_name_list to "EndofList Dummy Image.bad"
   set beginning of image_name_list to "StartofList Dummy Image.bad"
end tell

Evaluate_display_dupe_list()

final_cleanup()

log "Done"

################# Handlers

on validateCOP()
   set COPDocName to ""
   
   set minCOPversion to 10.0 as real
   
   tell application "System Events"
      set COPProcList to every process whose name contains "Capture One" and background only is false
      if debug ≥ 2 then log COPProcList
   end tell
   if (count of COPProcList) = 0 then error "COP is not running"
   if (count of COPProcList) ≠ 1 then error "Unexpected: >1 COP instances"
   set theAppRef to item 1 of COPProcList
   
   tell application "System Events" to set theAppName to ((get name of theAppRef) as text)
   
   using terms from application "Capture One 10"
      tell application theAppName to set copVersion to (get app version)
   end using terms from
   
   tell application "System Events" to set copDetailedVersion to get version of my application theAppName
   
   if debug ≥ 2 then
      --properties of application "Capture One 10"
      tell application "System Events"
         log (get every process whose background only is false)
      end tell
      log "theAppName: " & theAppName
      log "COP Version: " & copVersion
      log "COP Detailed Version: " & copDetailedVersion
   end if
   
   if the theAppName ≠ "Capture One 10" then
      display notification "Wrong COP Application"
      error "Found COP Application " & theAppName & " The only supported COP application is Capture One 10"
   end if
   
   set numCOPversion to (item 1 of (extract_version_strings(".", (word -1 of copVersion)))) as integer
   if numCOPversion < minCOPversion then
      display notification "COP Version is too low"
      error "This COP Version is " & numCOPversion & " - the minimum supported COP version is " & minCOPversion
   end if
   
   tell application "System Events" to set frontmost of process theAppName to true
   
   set COPeveryDoc_L to name of every document of my application theAppName
   if debug ≥ 2 then log COPeveryDoc_L
   if (count of COPeveryDoc_L) = 0 then error theAppName & " has no open documents"
   
   set is_catalog_found to false
   repeat with COP_doc in COPeveryDoc_L -- now check find the catalog
      using terms from application "Capture One 10"
         tell application theAppName to set thisDocKind to (get kind of document COP_doc) as text
      end using terms from
      if thisDocKind = "catalog" then
         set is_catalog_found to true
         exit repeat
      end if
   end repeat
   
   if not is_catalog_found then
      display notification ("This Applescript only works with Catalogs - Only found " & COPeveryDoc_L)
      error "No open catalog found"
   end if
   
   log "Capture One " & thisDocKind & ": " & COPDocName
   
   log "Capture One Version" & copDetailedVersion
   
end validateCOP

on extract_version_strings(d, t)
   set astid to AppleScript's text item delimiters
   try
      set AppleScript's text item delimiters to d
      set version_strings to text items of t
   on error
      set AppleScript's text item delimiters to astid
   end try
   set AppleScript's text item delimiters to astid
   
   return version_strings
end extract_version_strings

on sort_imagedates()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Starting Date Sort"
   
   set list_end to image_count + 1
   set list_begin to 0 as integer
   
   set pointer_dates_L_Len to 2 as integer
   set up_pointer_dates_L to {list_finish, list_end}
   set down_pointer_dates_L to {list_begin, list_start}
   
   set iteration_counter to 0 as integer
   set this_image_index to list_finish
   set this_image_date to item this_image_index of image_date_list
   
   ## Initialisation for the update_compare_index() handler
   set {year:start_image_date_year, month:start_image_date_month, day:start_image_date_day} to (item list_start of image_date_list)
   set {year:finish_image_date_year, month:finish_image_date_month, day:finish_image_date_day} to (item list_finish of image_date_list)
   set year_list to {{year:start_image_date_year, up_ptr:list_finish, down_ptr:list_begin, ym_ptr:list_start}}
   set yearmonth_list to {{year:start_image_date_year, month:start_image_date_month, up_ptr:list_finish, down_ptr:list_begin, ymd_ptr:list_start}}
   set ymd_list to {{year:start_image_date_year, month:start_image_date_month, day:start_image_date_day, up_ptr:list_finish, down_ptr:list_begin, dates_ptr:list_start}}
   set end of year_list to {year:finish_image_date_year, up_ptr:list_end, down_ptr:list_start, ym_ptr:list_finish}
   set end of yearmonth_list to {year:finish_image_date_year, month:finish_image_date_month, up_ptr:list_end, down_ptr:list_start, ymd_ptr:list_finish}
   set end of ymd_list to {year:finish_image_date_year, month:finish_image_date_month, day:finish_image_date_day, up_ptr:list_end, down_ptr:list_start, dates_ptr:list_finish}
   
   if debug ≥ 2 then log_status()
   set sort_start_time to current date
   repeat while this_image_index < image_count
      set compare_index to this_image_index
      set compare_date to this_image_date
      set this_image_index to this_image_index + 1
      set this_image_date to item this_image_index of image_date_list
      if debug ≥ 2 then log {"Now processing image ", this_image_index, this_image_date, "with Compare Image ", compare_index, compare_date}
      
      ## Get a better search start if the previous image was on a different day --  speeds up searching by up to 4x
      if (get {year, month, day} of this_image_date) ≠ (get {year, month, day} of compare_date) then set {compare_index, compare_date} to update_compare_index()
      
      if this_image_date < compare_date then
         search_down(compare_index, compare_date)
      else
         search_up(compare_index, compare_date)
      end if
      
      if debug ≥ 3 then
         log {"Done", "up:", up_pointer_dates_L, "down:", down_pointer_dates_L}
         log "  "
         if debug ≥ 4 then
            log_status()
            --if this_image_index > 10 then error
            if this_image_index > heavy_debug_image_limit and debug ≥ 4 then exit repeat
         end if
      end if
   end repeat
   set sort_stop_time to current date
   if debug ≥ 1 then log {"number of Iterations: ", iteration_counter, "  Sort Duration: ", (sort_stop_time - sort_start_time), " Seconds"}
   --if debug ≥ 2 then log_status()
end sort_imagedates

on log_status()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, image_count, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   log "*****"
   log "Status"
   log {"Up Pointer List", up_pointer_dates_L}
   log {"Down Pointer List", down_pointer_dates_L}
   log {"Image Date List:", (items 1 thru pointer_dates_L_Len of image_date_list)}
   log {"year_list", year_list}
   log {"yearmonth_list", yearmonth_list}
   log {"ymd_list", ymd_list}
   log "*****"
end log_status

on update_compare_index()
   -- This subroutine maintains a list of pointers to years, months and days used to increase the search speed
   -- It returns a pointer to an already sorted image in the nearest day (same day if there is one)
   -- there is a lot of code, but most of it does not execute often
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   
   set {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day} to this_image_date
   set indexing_entry_missing to false
   
   -- Search for Year entry
   -- Reference: set year_list to {{year:this_image_date_year, up_ptr:list_end, down_ptr:list_begin, ym_ptr:list_start}}
   -- Reference: set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set year_index_found to false
   set prev_year_index to list_start
   set compare_year_index to up_ptr of item prev_year_index of year_list
   repeat
      set iteration_counter to iteration_counter + 1
      if compare_year_index = list_finish then exit repeat -- hit the end of year_list and didn't find the entry
      if (get year of (item compare_year_index of year_list)) = this_image_date_year then
         if debug ≥ 3 then log "year found"
         set this_year_index to compare_year_index
         set year_index_found to true
         set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
         exit repeat
      else if (year of (item compare_year_index of year_list)) < this_image_date_year then
         set prev_year_index to compare_year_index
         set compare_year_index to up_ptr of item prev_year_index of year_list
      else -- (year of (item compare_year_index of year_list)) > this_image_date_year
         exit repeat -- didn't find the entry between already existing entries
      end if
   end repeat
   
   if not year_index_found then -- add index entry for the year
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding Year", this_image_date, "this_image_index", this_image_index}
      
      set end of year_list to {year:this_image_date_year, up_ptr:compare_year_index, down_ptr:prev_year_index, ym_ptr:(1 + (length of yearmonth_list))}
      set this_year_index to length of year_list
      set down_ptr of item compare_year_index of year_list to this_year_index
      set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
      set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      set up_ptr of item prev_year_index of year_list to this_year_index
      if debug ≥ 3 then log {"Added this_year_index", this_year_index, "compare_year_index", compare_year_index, "prev_year_index", prev_year_index}
      if debug ≥ 4 then log {year_list}
   end if
   
   -- Search for YearMonth entry
   -- Reference: set yearmonth_list to {{year:this_image_date_year, month:this_image_date_month, up_ptr:list_end, down_ptr:list_begin, ymd_ptr:list_start}}
   
   set yearmonth_index_found to false
   if not indexing_entry_missing then -- search for the Year:month
      set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_yearmonth_index = list_finish then exit repeat -- hit the end of yearmonth_list and didn't find the entry
         if (get {year, month} of (item compare_yearmonth_index of yearmonth_list)) = {this_image_date_year, this_image_date_month} then
            set this_yearmonth_index to compare_yearmonth_index
            if debug ≥ 3 then log "yearmonth found"
            set yearmonth_index_found to true
            set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
            exit repeat
         else if (((get month of (item compare_yearmonth_index of yearmonth_list)) < this_image_date_month) and ((get year of (item compare_yearmonth_index of yearmonth_list)) = this_image_date_year)) then
            set prev_yearmonth_index to compare_yearmonth_index
            set compare_yearmonth_index to up_ptr of item prev_yearmonth_index of yearmonth_list
         else -- (month of (item compare_yearmonth_index of yearmonth_list)) > this_image_date_month  or (year of (item compare_yearmonth_index of yearmonth_list)) ≠ this_image_date_year)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not yearmonth_index_found then -- add index entry for the yearmonth
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonth", this_image_date, "this_image_index", this_image_index}
      
      set end of yearmonth_list to {year:this_image_date_year, month:this_image_date_month, up_ptr:compare_yearmonth_index, down_ptr:prev_yearmonth_index, ymd_ptr:(1 + (length of ymd_list))}
      set this_yearmonth_index to length of yearmonth_list
      set down_ptr of item compare_yearmonth_index of yearmonth_list to this_yearmonth_index
      set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
      set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      set up_ptr of item prev_yearmonth_index of yearmonth_list to this_yearmonth_index
      
      if year_index_found then -- if year already existed then update its yearmonth pointer if this is the first yearmonth of the year
         if (year of item prev_yearmonth_index of yearmonth_list) ≠ this_image_date_year then set ym_ptr of item this_year_index of year_list to this_yearmonth_index
      end if
      
      if debug ≥ 3 then log {"Added this_yearmonth_index", this_yearmonth_index, "compare_yearmonth_index", compare_yearmonth_index, "prev_yearmonth_index", prev_yearmonth_index}
      if debug ≥ 4 then log {yearmonth_list}
   end if
   
   -- Search for YMD entry
   -- Reference: set ymd_list to {{year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:list_end, down_ptr:list_begin, dates_ptr:list_start}}
   
   set ymd_index_found to false
   if not indexing_entry_missing then -- search for Year:month:day
      set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_ymd_index = list_finish then exit repeat -- hit the end of ymd_list and didn't find the entry
         if (get {year, month, day} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month, this_image_date_day} then
            if debug ≥ 3 then log "ymd found"
            set this_ymd_index to compare_ymd_index
            set ymd_index_found to true
            exit repeat
         else if (((get day of (item compare_ymd_index of ymd_list)) < this_image_date_day) and ((get {year, month} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month})) then
            set prev_ymd_index to compare_ymd_index
            set compare_ymd_index to up_ptr of item prev_ymd_index of ymd_list
         else -- (day of (item compare_ymd_index of ymd_list)) > this_image_date_month  or (month of (item compare_ymd_index of ymd_list)) ≠ this_image_date_month)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not ymd_index_found then -- add index entry for the yearmonthday
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonthDay", this_image_date, "this_image_index", this_image_index}
      
      set end of ymd_list to {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:compare_ymd_index, down_ptr:prev_ymd_index, dates_ptr:this_image_index}
      set this_ymd_index to length of ymd_list
      set down_ptr of item compare_ymd_index of ymd_list to this_ymd_index
      set up_ptr of item prev_ymd_index of ymd_list to this_ymd_index
      
      if yearmonth_index_found then -- if yearmonth already existed then update its ymd pointer if this is the first ymd of the month
         if (month of item prev_ymd_index of ymd_list) ≠ this_image_date_month then
            set ymd_ptr of item this_yearmonth_index of yearmonth_list to this_ymd_index
         end if
      end if
      
      if debug ≥ 3 then log {"Added this_ymd_index", this_ymd_index, "compare_ymd_index", compare_ymd_index, "prev_ymd_index", prev_ymd_index}
      if debug ≥ 4 then log {ymd_list}
   end if
   
   if indexing_entry_missing then
      set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- the pointer to the first image for the next higher date
   else -- indexing entry was not missing
      set updated_compare_date_ptr to dates_ptr of (item this_ymd_index of ymd_list) -- the pointer to the first image for this date
      --set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- (alternate) the pointer to the first image for the next higher date
   end if
   
   set updated_compare_date to item updated_compare_date_ptr of image_date_list
   return {updated_compare_date_ptr, updated_compare_date}
   
end update_compare_index

on search_up(prev_candidate_image_index)
   global image_date_list, pointer_dates_L_Len, up_pointer_dates_L, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Up", prev_candidate_image_index}
   set found to false
   repeat while not found --search for the first image which is newer than this image
      set iteration_counter to iteration_counter + 1
      set next_candidate_image_index to item prev_candidate_image_index of up_pointer_dates_L
      set next_candidate_image_date to item next_candidate_image_index of image_date_list
      if this_image_date > next_candidate_image_date then --search further up
         set prev_candidate_image_index to next_candidate_image_index
      else
         insert_image_at(next_candidate_image_index, prev_candidate_image_index)
         set found to true
      end if
   end repeat
end search_up

on search_down(prev_candidate_image_index)
   global image_date_list, pointer_dates_L_Len, down_pointer_dates_L, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Down", prev_candidate_image_index}
   set found to false
   repeat while not found --search for the first image which is older than this image
      set iteration_counter to iteration_counter + 1
      set next_candidate_image_index to item prev_candidate_image_index of down_pointer_dates_L
      set next_candidate_image_date to item next_candidate_image_index of image_date_list
      if this_image_date < next_candidate_image_date then -- search further down
         set prev_candidate_image_index to next_candidate_image_index
      else
         insert_image_at(prev_candidate_image_index, next_candidate_image_index)
         set found to true
      end if
   end repeat
end search_down

on insert_image_at(higher_image_index, lower_image_index)
   global pointer_dates_L_Len, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug
   if debug ≥ 3 then log {"Insertion", this_image_index, "between", higher_image_index, lower_image_index}
   set end of up_pointer_dates_L to higher_image_index
   set item lower_image_index of up_pointer_dates_L to this_image_index
   set end of down_pointer_dates_L to lower_image_index
   set item higher_image_index of down_pointer_dates_L to this_image_index
   set pointer_dates_L_Len to pointer_dates_L_Len + 1
end insert_image_at

on pointer_check()
   global image_date_list, pointer_dates_L_Len, list_begin, list_end, up_pointer_dates_L, down_pointer_dates_L, debug, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   log {"list_start", list_start, "list_finish", list_finish, "up_pointer_dates_L", (items 1 thru heavy_debug_image_limit of up_pointer_dates_L), "image_date_list", (items 1 thru heavy_debug_image_limit of image_date_list), "image_name_list"}
   
   if debug ≥ 4 then --  really long detailed listing  Only do this with the short list in debug level; 4.
      log "Listing Dates List Pointers"
      log {"unsorted: ", (items 1 thru pointer_dates_L_Len of image_date_list)}
      set next_pointer to list_start
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of up_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log {"Sorted up: ", sorted_image_date_list}
      log {"Sorted up: ", sorted_image_list}
      
      set next_pointer to list_finish
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of down_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log {"Sorted down: ", sorted_image_date_list}
      log {"Sorted down: ", sorted_image_list}
   end if
   
   if debug ≥ 2 then
      log "Checking Dates List Pointers"
      set next_pointer to list_start
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of up_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date < prev_date then
            set list_good to false
            log {"Up pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of up_pointer_dates_L ≠ list_end then
         log {"error in last Dates List up pointer", next_pointer, (item next_pointer of up_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Up Pointers for Dates List are OK"
      
      set next_pointer to list_finish
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of down_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date > prev_date then
            set list_good to false
            log {"Down pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of down_pointer_dates_L ≠ list_begin then
         log {"error in last Dates List down pointer", next_pointer, (item next_pointer of down_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Down Pointers for Dates List are OK"
   end if
   
end pointer_check

on cleanup_sort()
   global down_pointer_dates_L
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying Variables used only for Sorting" --  Avoids stack overflow problems
   set down_pointer_dates_L to {}
   set ymd_list to {}
   set yearmonth_list to {}
   set year_list to {}
   set ymd_start_pointers to {}
end cleanup_sort

on Duplicate_Search()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, debug, heavy_debug_image_limit, image_count, duplicate_image_lists_L, list_start, list_finish
   if debug ≥ 1 then log "Starting Duplicate Search"
   set this_pointer to item list_start of up_pointer_dates_L
   set this_date to item this_pointer of image_date_list
   set first_dup_date to {}
   set dupe_found to false
   set duplicate_image_sublist to {}
   set duplicate_image_lists_L to {}
   repeat with counter from 1 to pointer_dates_L_Len - 2
      set prev_date to this_date
      set prev_pointer to this_pointer
      set this_pointer to item prev_pointer of up_pointer_dates_L
      set this_date to item this_pointer of image_date_list
      if this_date < prev_date then log {"error prev_pointer", prev_pointer, "prev_date", prev_date, "this_pointer", this_pointer, "this date", this_date, "counter", counter}
      if this_date = prev_date then
         if dupe_found = false then -- start of a series of duplicates -  found first two
            if debug ≥ 3 then log "Found Dupe"
            if debug ≥ 4 then log {"prev_pointer", prev_pointer, "prev_date", prev_date, "this_pointer", this_pointer, "this date", this_date, "counter", counter}
            set dupe_found to true
            set first_dup_date to prev_date
            set duplicate_image_sublist to {first_dup_date, prev_pointer, this_pointer}
         else -- adding other duplicates
            set end of duplicate_image_sublist to this_pointer
         end if
      else if dupe_found = true then -- end of a series of duplicates
         set dupe_found to false
         set end of duplicate_image_lists_L to duplicate_image_sublist
         set duplicate_image_sublist to {}
         set first_dup_date to {}
      end if
      if debug ≥ 4 and counter > heavy_debug_image_limit then error
   end repeat
end Duplicate_Search

on cleanup_dupe_search()
   global up_pointer_dates_L, image_date_list
   if debug ≥ 1 then log "Emptying Variables used only for Duplicate Searching" --  Avoids stack overflow problems
   set image_date_list to {}
   set up_pointer_dates_L to {}
end cleanup_dupe_search

on Evaluate_display_dupe_list()
   global debug, duplicate_image_lists_L, no_date_info_D
   -- the intent of this handler is to eleminate obvious non-duplicates
   ----Images with the same date but different image types (likely a RAW-JPG pair, but this is not checked)
   ----Image groups which are part of a (high speed) burst sequence
   -- Images which are reported are likely duplicates, but are NOT guaranteed duplicates
   
   if debug ≥ 3 then log "Evaluating  Duplicates"
   if debug ≥ 4 then log duplicate_image_lists_L
   
   
   
   repeat with sublist_index from 1 to (length of duplicate_image_lists_L)
      
      set {dupe_date, duplicate_image_list} to make_dupe_name_list(item sublist_index of duplicate_image_lists_L)
      
      if dupe_date ≠ no_date_info_D then
         repeat with sublist_index from 1 to count of duplicate_image_list --For each sublist
            
            if (count of item sublist_index of duplicate_image_list) > 2 then -- if there is more than one image name then determine if this could be a burst
               if debug ≥ 3 then log (item sublist_index of duplicate_image_list)
               set is_burst to true -- start by assuming its a burst, and then test it
               set image_seq_list to {}
               set image_name_ext to item 1 of item sublist_index of duplicate_image_list
               repeat with sublist_counter from 2 to length of item sublist_index of duplicate_image_list -- the first entry in the sublist is the extension
                  
                  set image_name to item sublist_counter of item sublist_index of duplicate_image_list
                  set image_num_list to ""
                  set image_char_list to ""
                  repeat with char_counter from 1 to count of image_name
                     set this_char to item char_counter of image_name
                     if this_char = "." then
                        exit repeat
                     else if "0123456789" contains this_char then
                        set image_num_list to image_num_list & this_char
                     else
                        set image_char_list to image_char_list & this_char
                     end if
                  end repeat
                  
                  set image_num_count to count of image_num_list
                  if image_num_count = 0 then -- if there are no numbers in the image name, this is not a burst
                     set is_burst to false
                     exit repeat
                  end if
                  set image_num to image_num_list as integer
                  set end of image_seq_list to image_num
                  
                  if sublist_counter = 2 then -- first item
                     if (count of image_char_list) > 0 then
                        set has_image_char to true
                        set image_char_ref to image_char_list
                     else
                        set has_image_char to false
                     end if
                     set image_num_count_ref to image_num_count
                     set image_seq_min to image_num
                  else -- case: sublist counter >2  (additional items)
                     
                     if has_image_char then -- check that the image characters match the first image
                        if image_char_list ≠ image_char_ref then
                           set is_burst to false
                           exit repeat
                        end if
                     else -- first image name has no characters
                        if (count of image_char_list) > 0 then
                           set is_burst to false
                           exit repeat
                        end if
                     end if
                     
                     if image_num_count ≠ image_num_count_ref then -- now check the numbers
                        set is_burst to false
                        exit repeat
                     end if
                     if image_num < image_seq_min then set image_seq_min to image_num -- find the starting sequence number
                  end if
                  if debug ≥ 4 then log image_num
                  if debug ≥ 4 then log image_char_list
               end repeat
               
               if is_burst then -- check that the image numbers are sequential with increments of 1
                  set image_seq_count to count of image_seq_list
                  set last_seq to image_seq_min
                  repeat with incr_counter from 2 to image_seq_count
                     set this_seq_found to false
                     repeat with seq_counter from 1 to image_seq_count
                        if item seq_counter of image_seq_list = last_seq + 1 then
                           set last_seq to (item seq_counter of image_seq_list)
                           set this_seq_found to true
                           exit repeat
                        end if
                     end repeat
                     if this_seq_found = false then
                        set is_burst to false
                        exit repeat
                     end if
                  end repeat
               end if
               if not is_burst then
                  log {"Possible Duplicates: ", (items 2 through (count of item sublist_index of duplicate_image_list) of item sublist_index of duplicate_image_list), "on", dupe_date}
               end if
            end if
         end repeat
      else
         log {"Images with no date info: ", dupe_date, duplicate_image_list}
         
      end if
      
   end repeat
end Evaluate_display_dupe_list

on make_dupe_name_list(date_ptr_sublist)
   global debug, image_name_list
   
   set dupe_date to item 1 of date_ptr_sublist --  the date and time of these duplicates
   
   set dupe_name_list to {}
   repeat with pointer_counter from 2 to length of date_ptr_sublist
      set image_name to item (item pointer_counter of date_ptr_sublist) of image_name_list -- get the name of the image
      
      ## extract the type of the image
      set image_type_found to false
      set image_type to text -1 of image_name
      repeat with char_counter from 2 to 5
         set this_char to item (-char_counter) of image_name
         if this_char = "." then
            set image_type_found to true
            exit repeat
         else
            set image_type to this_char & image_type
         end if
      end repeat
      
      if not image_type_found then
         log "WTF?? Image type not found in Image Name: " & image_name
         exit repeat
      end if
      
      ##  search for a sublist with this type of image already in the dupe_name_list
      set image_type_in_list to false
      repeat with sublist_index from 1 to count of dupe_name_list
         if item 1 of item sublist_index of dupe_name_list = image_type then
            set image_type_in_list to true
            exit repeat
         end if
      end repeat
      
      ## if not already in dupe_name_list add a sublist for this image type
      if not image_type_in_list then
         set end of dupe_name_list to {image_type}
         set sublist_index to count of dupe_name_list
      end if
      
      set end of item sublist_index of dupe_name_list to image_name --add the image name to the sublist
      
   end repeat
   return {dupe_date, dupe_name_list}
end make_dupe_name_list

on final_cleanup()
   global image_date_list, image_name_list, pointer_dates_L_Len, up_pointer_dates_L, up_start_pointer_dates
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying all  Variables" --  Avoids stack overflow problems
   set image_date_list to {}
   set image_name_list to {}
   set up_pointer_dates_L to {}
   set pointer_dates_L_Len to {}
   set up_start_pointer_dates to {}
end final_cleanup
Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Wed Apr 19, 2017 1:40 am

Gidday Eric

I'm back!

I ran the new script on a catalogue with ~37k images and it ran for a bit and returned:

Result:
error "Capture One 10 got an error: AppleEvent timed out." number -1712


:(

I like to think Im being helpful here, as a beta tester or something, but maybe I'm just being a pain!

And on that note ... I also ran the script on a catalogue of 1,100 images, almost all of which are scanned. It resulted in the quite a few "missing value" messages and then this:

end tell
(*Retrieved 1049 images*)
tell application "System Events"
set frontmost of process "Script Editor" to true
end tell
(*Starting Date Sort*)
tell current application
current date
--> date "Wednesday, 19 April 2017 at 08:36:45"
Result:
error "Can’t get year of missing value." number -1728 from year of missing value


Maybe at least some of the scanned images have dodgy dates - if I read the script output correctly?

Regards
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby Eric Nepean » Wed Apr 19, 2017 3:08 am

Hi Shane, it's nice to hear back from you. You are being helpful.
ShaneB wrote:Gidday Eric

I'm back!

I ran the new script on a catalogue with ~37k images and it ran for a bit and returned:

Result:
error "Capture One 10 got an error: AppleEvent timed out." number -1712


:(

I like to think Im being helpful here, as a beta tester or something, but maybe I'm just being a pain!

More Like an Alpha tester than a Beta tester. However I can't always respond with an update as I have a day job too :D

This is suggesting a few things to me.

1) It sounds like the dates or names of 37K images is too large for COP to send Applescript to handle in 1 chunk. Instead of trying to digest every image in the All Images folder, I think I'm better to have the user select a set of images, and limit that to about 15000, which I can test.
2) I've put quite a bit of work into making the date sort work. But not surprisingly, COP has a more robust date sort. Paid staff and all that. Perhaps I should just ask the user to set the COP browser to sort by dates before running the Applescript, and then have the script check that the dates are in order and not garbage.

And on that note ... I also ran the script on a catalogue of 1,100 images, almost all of which are scanned. It resulted in the quite a few "missing value" messages and then this:

end tell
(*Retrieved 1049 images*)
tell application "System Events"
set frontmost of process "Script Editor" to true
end tell
(*Starting Date Sort*)
tell current application
current date
--> date "Wednesday, 19 April 2017 at 08:36:45"
Result:
error "Can’t get year of missing value." number -1728 from year of missing value


Maybe at least some of the scanned images have dodgy dates - if I read the script output correctly?

Regards

3) I don't make any test for dodgy dates - sounds like I should be doing that.
Could you send me some more dignostic info?

In about the 4 line of the applescript (not including comment lines starting with "--") there is a statement "Set debug to 1" - change that to "Set debug to 2" (0 provide no debug - 4 provides a whole lot)
Then in the log hisrtory window instead of selecting the "messages" tab, could you select the "replies" tab, and run the script again on your catalog with 1049 images - the listing will be quite long - too long to post in it's entirety.
Then copy me any interesting looking snippets with "error" and "missing value" and a few lines on both sides.

You could use the "code" feature on this website to contrrol the length of the posting - clicking the Code button generates the sequence {code][/code} , and anything you paste between those two sequences shows up in the scrolling window.
Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Thu Apr 20, 2017 2:49 am

Hi Eric

Well, you're a glutton for punishment - but I for one appreciate it.

This is some of the output from that catalogue with the date issues, with chunks of "missing value" scattered through it:

Code: Select all
date "Wednesday, 9 April 2003 at 17:35:14", date "Sunday, 31 July 2005 at 18:22:42", date "Wednesday, 10 January 2007 at 19:39:52", missing value, missing value, date "Sunday, 16 November 2008 at 17:40:40", date "Sunday, 16 November 2008 at 17:40:40", date "Sunday, 16 November 2008 at 21:08:44", date "Sunday, 16 November 2008 at 17:42:09", date "Sunday, 10 April 2011 at 17:22:23", missing value, missing value, date "Monday, 19 September 2011 at 12:19:54", date "Monday, 19 September 2011 at 12:41:28", date "Saturday, 3 July 2010 at 17:15:09", date "Tuesday, 4 January 2005 at 05:19:25", date "Wednesday, 21 April 2004 at 16:15:38", date "Tuesday, 27 June 2006 at 11:27:14", date "Saturday, 31 May 2008 at 17:00:11", date "Sunday, 7 January 2007 at 12:58:38", date "Friday, 25 March 2005 at 18:42:30", date "Saturday, 31 May 2008 at 17:00:37", date "Wednesday, 20 August 2003 at 20:03:26", date "Friday, 4 April 2003 at 23:11:13", date "Friday, 4 April 2003 at 23:11:13", date "Saturday, 18 September 2004 at 15:13:47", date "Saturday, 3 October 2015 at 14:00:14", missing value, missing value, date "Sunday, 31 December 2006 at 23:48:53", date "Tuesday, 29 July 2003 at 15:53:42", date "Saturday, 31 May 2008 at 16:59:16", date "Saturday, 31 May 2008 at 17:16:39", date "Tuesday, 29 July 2003 at 15:53:23", date "Friday, 25 March 2005 at 20:43:28",


The end result was "Result:
error "Can’t get year of missing value." number -1728 from year of missing value
" .

I'm not sure it's of much value, but what do I know?

Regards
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby Eric Nepean » Thu Apr 20, 2017 6:34 am

Yeah, I think I see what is happening. Some of the images have no date, and so the value retreived is "missing value" which is special Applescript value meaning "nothing here, move along"
And then my script tries to find the year of "missing value", and Applescript is not robust enough to say that this shall also be "missing value", and throws an error.

Here is an updated version that converts all dodgy dates - missing values, values that aren't dates, dates before January 1 1970 and dates more than 1 year into a no date value.

Have a go at this, and let me know how it goes.
Code: Select all
-- Applescript to search a COP 11 Catalog for Duplicate Images
-- Version 1.3 !! Best Effort Support !!  Eric Valk, Ottawa, Canada

-- ***To Initialise
-- Start Script Editor, open a new (blank) file, copy and paste all of this code into the file, compile (hammer symbol) and save.
-- Best if you make "Scripts" folder somewhere handy in your Douments or Desktop

-- *** Operation
-- Start Capture One Pro with the desired catalog
-- Open  the compiled and saved file in Script Editor
-- Open the Script Editor log window, and select the messages tab
-- Run the script
-- Results appear first in the AppleScript Log when the search  is completed
-- This script does not write or delete any information in the COP Catalog or Session or the image file

global debug, heavy_debug_image_limit
global image_date_list, image_count, image_name_list, no_date_info_D, start_date_D, end_date_D, image_count0, list_start, list_finish

log "Find Dupes Version 1.3"
set debug to 1
set heavy_debug_image_limit to 10

validateCOP()

## Set up date limits
## the date assigned if there is no date
set no_date_string to "01/01/70"
set no_date_info_D to (date no_date_string)
## The starting date for sorting (one year before the date assigned if there is no date)
copy no_date_info_D to start_date_D
set year of start_date_D to (year of start_date_D) - 1
## # The ending date for sorting (one year after the current date)
set end_date_D to current date
set year of end_date_D to (year of end_date_D)
set time of end_date_D to 0

if debug ≥ 1 then log "Getting image dates"
tell application "Capture One 10"
   set image_date_list to EXIF capture date of every image
end tell
set image_count0 to length of image_date_list
log "Retrieved " & image_count0 & " images"
if debug ≥ 2 then log {items 1 through 2 of image_date_list, class of image_date_list, "end", (item image_count0 of image_date_list)}

## Add the starting and ending  dates at the beginning of the date list
## This provide search endpoints so that the search routines will not run off the ends of the list
## All the pointers to image lists are now 2 higher than they would otherwise be
tell image_date_list
   set beginning of image_date_list to end_date_D
   set beginning of image_date_list to start_date_D
end tell
set image_count to length of image_date_list
set list_start to 1
set list_finish to 2

tell application "System Events" to set frontmost of process "Script Editor" to true

sort_imagedates()

if debug ≥ 2 then pointer_check()

cleanup_sort()

Duplicate_Search()

cleanup_dupe_search()

if debug ≥ 1 then log "Getting image names"
tell application "Capture One 10" to set image_name_list to name of every image
tell image_date_list -- to make the items in the name list sync with the items in the date list
   set beginning of image_name_list to "EndofList Dummy Image.bad"
   set beginning of image_name_list to "StartofList Dummy Image.bad"
end tell

Evaluate_display_dupe_list()

final_cleanup()

log "Done"

################# Handlers

on validateCOP()
   set minCOPversion to 10.0 as real
   
   tell application "System Events"
      set COPProcList to every process whose name contains "Capture One" and background only is false
      if debug ≥ 2 then log COPProcList
   end tell
   if (count of COPProcList) = 0 then error "COP is not running"
   if (count of COPProcList) ≠ 1 then error "Unexpected: >1 COP instances"
   set theAppRef to item 1 of COPProcList
   
   tell application "System Events" to set theAppName to ((get name of theAppRef) as text)
   
   using terms from application "Capture One 10"
      tell application theAppName to set copVersion to (get app version)
   end using terms from
   
   tell application "System Events" to set copDetailedVersion to get version of my application theAppName
   
   if debug ≥ 2 then
      --properties of application "Capture One 10"
      tell application "System Events"
         log (get every process whose background only is false)
      end tell
      log "theAppName: " & theAppName
      log "COP Version: " & copVersion
      log "COP Detailed Version: " & copDetailedVersion
   end if
   
   if the theAppName ≠ "Capture One 10" then
      display notification "Wrong COP Application"
      error "Found COP Application " & theAppName & " The only supported COP application is Capture One 10"
   end if
   
   set numCOPversion to (item 1 of (extract_version_strings(".", (word -1 of copVersion)))) as integer
   if numCOPversion < minCOPversion then
      display notification "COP Version is too low"
      error "This COP Version is " & numCOPversion & " - the minimum supported COP version is " & minCOPversion
   end if
   
   tell application "System Events" to set frontmost of process theAppName to true
   
   set COPeveryDoc_L to name of every document of my application theAppName
   if debug ≥ 2 then log {length of COPeveryDoc_L, "Documents", COPeveryDoc_L}
   if (count of COPeveryDoc_L) = 0 then error theAppName & " has no open documents"
   
   set is_catalog_found to false
   repeat with COP_doc in COPeveryDoc_L -- now find the catalog
      using terms from application "Capture One 10"
         tell application theAppName to set thisDocKind to (get kind of document COP_doc) as text
      end using terms from
      if thisDocKind = "catalog" then
         set is_catalog_found to true
         exit repeat
      end if
   end repeat
   
   if not is_catalog_found then
      display notification ("This Applescript only works with Catalogs - Only found " & COPeveryDoc_L)
      error "No open catalog found"
   end if
   
   log "Capture One " & thisDocKind & ": " & COP_doc
   
   log "Capture One Version" & copDetailedVersion
   
end validateCOP

on extract_version_strings(d, t)
   set astid to AppleScript's text item delimiters
   try
      set AppleScript's text item delimiters to d
      set version_strings to text items of t
   on error
      set AppleScript's text item delimiters to astid
   end try
   set AppleScript's text item delimiters to astid
   
   return version_strings
end extract_version_strings

on sort_imagedates()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter, list_start, list_finish, start_date_D, end_date_D, no_date_info_D
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Starting Date Sort"
   
   set list_end to image_count + 1
   set list_begin to 0 as integer
   
   set pointer_dates_L_Len to 2 as integer
   set up_pointer_dates_L to {list_finish, list_end}
   set down_pointer_dates_L to {list_begin, list_start}
   
   set iteration_counter to 0 as integer
   set this_image_index to list_finish
   set this_image_date to item this_image_index of image_date_list
   
   ## Initialisation for the update_compare_index() handler
   set {year:start_image_date_year, month:start_image_date_month, day:start_image_date_day} to (item list_start of image_date_list)
   set {year:finish_image_date_year, month:finish_image_date_month, day:finish_image_date_day} to (item list_finish of image_date_list)
   set year_list to {{year:start_image_date_year, up_ptr:list_finish, down_ptr:list_begin, ym_ptr:list_start}}
   set yearmonth_list to {{year:start_image_date_year, month:start_image_date_month, up_ptr:list_finish, down_ptr:list_begin, ymd_ptr:list_start}}
   set ymd_list to {{year:start_image_date_year, month:start_image_date_month, day:start_image_date_day, up_ptr:list_finish, down_ptr:list_begin, dates_ptr:list_start}}
   set end of year_list to {year:finish_image_date_year, up_ptr:list_end, down_ptr:list_start, ym_ptr:list_finish}
   set end of yearmonth_list to {year:finish_image_date_year, month:finish_image_date_month, up_ptr:list_end, down_ptr:list_start, ymd_ptr:list_finish}
   set end of ymd_list to {year:finish_image_date_year, month:finish_image_date_month, day:finish_image_date_day, up_ptr:list_end, down_ptr:list_start, dates_ptr:list_finish}
   
   if debug ≥ 2 then log_status()
   set sort_start_time to current date
   repeat while this_image_index < image_count
      set compare_index to this_image_index
      set compare_date to this_image_date
      set this_image_index to this_image_index + 1
      set this_image_date to item this_image_index of image_date_list
      
      if debug ≥ 2 then log {"Now processing image ", this_image_index, this_image_date, "with Compare Image ", compare_index, compare_date}
      
      ## if the date info is dodgy, missing or out of range, then assign the date "no_date_info_D"
      if not (this_image_date > no_date_info_D and this_image_date < end_date_D and (class of this_image_date) = date) then -- this expression also catches the case image_date = missing value
         if debug ≥ 2 then log {"Date Error", this_image_index, this_image_date, (class of this_image_date)}
         set this_image_date to no_date_info_D
      end if
      
      ## Get a better search start if the previous image was on a different day --  speeds up searching by up to 4x
      if (get {year, month, day} of this_image_date) ≠ (get {year, month, day} of compare_date) then set {compare_index, compare_date} to update_compare_index()
      
      if this_image_date < compare_date then
         search_down(compare_index, compare_date)
      else
         search_up(compare_index, compare_date)
      end if
      
      if debug ≥ 3 then
         log {"Done", "up:", up_pointer_dates_L, "down:", down_pointer_dates_L}
         log "  "
         if debug ≥ 4 then
            log_status()
            --if this_image_index > 10 then error
            if this_image_index > heavy_debug_image_limit and debug ≥ 4 then exit repeat
         end if
      end if
   end repeat
   set sort_stop_time to current date
   if debug ≥ 1 then log {"number of Iterations: ", iteration_counter, "  Sort Duration: ", (sort_stop_time - sort_start_time), " Seconds"}
   --if debug ≥ 2 then log_status()
end sort_imagedates

on log_status()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, image_count, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   log "*****"
   log "Status"
   log {"Up Pointer List", up_pointer_dates_L}
   log {"Down Pointer List", down_pointer_dates_L}
   log {"Image Date List:", (items 1 thru pointer_dates_L_Len of image_date_list)}
   log {"year_list", year_list}
   log {"yearmonth_list", yearmonth_list}
   log {"ymd_list", ymd_list}
   log "*****"
end log_status

on update_compare_index()
   -- This subroutine maintains a list of pointers to years, months and days used to increase the search speed
   -- It returns a pointer to an already sorted image in the nearest day (same day if there is one)
   -- there is a lot of code, but most of it does not execute often
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug, heavy_debug_image_limit, image_count, iteration_counter, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   
   set {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day} to this_image_date
   set indexing_entry_missing to false
   
   -- Search for Year entry
   -- Reference: set year_list to {{year:this_image_date_year, up_ptr:list_end, down_ptr:list_begin, ym_ptr:list_start}}
   -- Reference: set ymd_start_pointers to {year_up:list_start, yearmonth_up:list_start, ymd_up:list_start, year_down:list_start, yearmonth_down:list_start, ymd_down:list_start}
   set year_index_found to false
   set prev_year_index to list_start
   set compare_year_index to up_ptr of item prev_year_index of year_list
   repeat
      set iteration_counter to iteration_counter + 1
      if compare_year_index = list_finish then exit repeat -- hit the end of year_list and didn't find the entry
      if (get year of (item compare_year_index of year_list)) = this_image_date_year then
         if debug ≥ 3 then log "year found"
         set this_year_index to compare_year_index
         set year_index_found to true
         set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
         exit repeat
      else if (year of (item compare_year_index of year_list)) < this_image_date_year then
         set prev_year_index to compare_year_index
         set compare_year_index to up_ptr of item prev_year_index of year_list
      else -- (year of (item compare_year_index of year_list)) > this_image_date_year
         exit repeat -- didn't find the entry between already existing entries
      end if
   end repeat
   
   if not year_index_found then -- add index entry for the year
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding Year", this_image_date, "this_image_index", this_image_index}
      
      set end of year_list to {year:this_image_date_year, up_ptr:compare_year_index, down_ptr:prev_year_index, ym_ptr:(1 + (length of yearmonth_list))}
      set this_year_index to length of year_list
      set down_ptr of item compare_year_index of year_list to this_year_index
      set compare_yearmonth_index to ym_ptr of item compare_year_index of year_list
      set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      set up_ptr of item prev_year_index of year_list to this_year_index
      if debug ≥ 3 then log {"Added this_year_index", this_year_index, "compare_year_index", compare_year_index, "prev_year_index", prev_year_index}
      if debug ≥ 4 then log {year_list}
   end if
   
   -- Search for YearMonth entry
   -- Reference: set yearmonth_list to {{year:this_image_date_year, month:this_image_date_month, up_ptr:list_end, down_ptr:list_begin, ymd_ptr:list_start}}
   
   set yearmonth_index_found to false
   if not indexing_entry_missing then -- search for the Year:month
      set prev_yearmonth_index to down_ptr of item compare_yearmonth_index of yearmonth_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_yearmonth_index = list_finish then exit repeat -- hit the end of yearmonth_list and didn't find the entry
         if (get {year, month} of (item compare_yearmonth_index of yearmonth_list)) = {this_image_date_year, this_image_date_month} then
            set this_yearmonth_index to compare_yearmonth_index
            if debug ≥ 3 then log "yearmonth found"
            set yearmonth_index_found to true
            set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
            exit repeat
         else if (((get month of (item compare_yearmonth_index of yearmonth_list)) < this_image_date_month) and ((get year of (item compare_yearmonth_index of yearmonth_list)) = this_image_date_year)) then
            set prev_yearmonth_index to compare_yearmonth_index
            set compare_yearmonth_index to up_ptr of item prev_yearmonth_index of yearmonth_list
         else -- (month of (item compare_yearmonth_index of yearmonth_list)) > this_image_date_month  or (year of (item compare_yearmonth_index of yearmonth_list)) ≠ this_image_date_year)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not yearmonth_index_found then -- add index entry for the yearmonth
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonth", this_image_date, "this_image_index", this_image_index}
      
      set end of yearmonth_list to {year:this_image_date_year, month:this_image_date_month, up_ptr:compare_yearmonth_index, down_ptr:prev_yearmonth_index, ymd_ptr:(1 + (length of ymd_list))}
      set this_yearmonth_index to length of yearmonth_list
      set down_ptr of item compare_yearmonth_index of yearmonth_list to this_yearmonth_index
      set compare_ymd_index to ymd_ptr of item compare_yearmonth_index of yearmonth_list
      set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      set up_ptr of item prev_yearmonth_index of yearmonth_list to this_yearmonth_index
      
      if year_index_found then -- if year already existed then update its yearmonth pointer if this is the first yearmonth of the year
         if (year of item prev_yearmonth_index of yearmonth_list) ≠ this_image_date_year then set ym_ptr of item this_year_index of year_list to this_yearmonth_index
      end if
      
      if debug ≥ 3 then log {"Added this_yearmonth_index", this_yearmonth_index, "compare_yearmonth_index", compare_yearmonth_index, "prev_yearmonth_index", prev_yearmonth_index}
      if debug ≥ 4 then log {yearmonth_list}
   end if
   
   -- Search for YMD entry
   -- Reference: set ymd_list to {{year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:list_end, down_ptr:list_begin, dates_ptr:list_start}}
   
   set ymd_index_found to false
   if not indexing_entry_missing then -- search for Year:month:day
      set prev_ymd_index to down_ptr of item compare_ymd_index of ymd_list
      repeat
         set iteration_counter to iteration_counter + 1
         if compare_ymd_index = list_finish then exit repeat -- hit the end of ymd_list and didn't find the entry
         if (get {year, month, day} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month, this_image_date_day} then
            if debug ≥ 3 then log "ymd found"
            set this_ymd_index to compare_ymd_index
            set ymd_index_found to true
            exit repeat
         else if (((get day of (item compare_ymd_index of ymd_list)) < this_image_date_day) and ((get {year, month} of (item compare_ymd_index of ymd_list)) = {this_image_date_year, this_image_date_month})) then
            set prev_ymd_index to compare_ymd_index
            set compare_ymd_index to up_ptr of item prev_ymd_index of ymd_list
         else -- (day of (item compare_ymd_index of ymd_list)) > this_image_date_month  or (month of (item compare_ymd_index of ymd_list)) ≠ this_image_date_month)
            exit repeat -- didn't find the entry between already existing entries
         end if
      end repeat
   end if
   
   if not ymd_index_found then -- add index entry for the yearmonthday
      set indexing_entry_missing to true
      if debug ≥ 2 then log "***"
      if debug ≥ 2 then log {"adding YearMonthDay", this_image_date, "this_image_index", this_image_index}
      
      set end of ymd_list to {year:this_image_date_year, month:this_image_date_month, day:this_image_date_day, up_ptr:compare_ymd_index, down_ptr:prev_ymd_index, dates_ptr:this_image_index}
      set this_ymd_index to length of ymd_list
      set down_ptr of item compare_ymd_index of ymd_list to this_ymd_index
      set up_ptr of item prev_ymd_index of ymd_list to this_ymd_index
      
      if yearmonth_index_found then -- if yearmonth already existed then update its ymd pointer if this is the first ymd of the month
         if (month of item prev_ymd_index of ymd_list) ≠ this_image_date_month then
            set ymd_ptr of item this_yearmonth_index of yearmonth_list to this_ymd_index
         end if
      end if
      
      if debug ≥ 3 then log {"Added this_ymd_index", this_ymd_index, "compare_ymd_index", compare_ymd_index, "prev_ymd_index", prev_ymd_index}
      if debug ≥ 4 then log {ymd_list}
   end if
   
   if indexing_entry_missing then
      set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- the pointer to the first image for the next higher date
   else -- indexing entry was not missing
      set updated_compare_date_ptr to dates_ptr of (item this_ymd_index of ymd_list) -- the pointer to the first image for this date
      --set updated_compare_date_ptr to dates_ptr of (item compare_ymd_index of ymd_list) -- (alternate) the pointer to the first image for the next higher date
   end if
   
   set updated_compare_date to item updated_compare_date_ptr of image_date_list
   return {updated_compare_date_ptr, updated_compare_date}
   
end update_compare_index

on search_up(prev_candidate_image_index)
   global image_date_list, pointer_dates_L_Len, up_pointer_dates_L, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Up", prev_candidate_image_index}
   set found to false
   repeat while not found --search for the first image which is newer than this image
      set iteration_counter to iteration_counter + 1
      set next_candidate_image_index to item prev_candidate_image_index of up_pointer_dates_L
      set next_candidate_image_date to item next_candidate_image_index of image_date_list
      if this_image_date > next_candidate_image_date then --search further up
         set prev_candidate_image_index to next_candidate_image_index
      else
         insert_image_at(next_candidate_image_index, prev_candidate_image_index)
         set found to true
      end if
   end repeat
end search_up

on search_down(prev_candidate_image_index)
   global image_date_list, pointer_dates_L_Len, down_pointer_dates_L, this_image_date, this_image_index, debug, iteration_counter
   if debug ≥ 3 then log "***"
   if debug ≥ 3 then log {"Search_Down", prev_candidate_image_index}
   set found to false
   repeat while not found --search for the first image which is older than this image
      set iteration_counter to iteration_counter + 1
      set next_candidate_image_index to item prev_candidate_image_index of down_pointer_dates_L
      set next_candidate_image_date to item next_candidate_image_index of image_date_list
      if this_image_date < next_candidate_image_date then -- search further down
         set prev_candidate_image_index to next_candidate_image_index
      else
         insert_image_at(prev_candidate_image_index, next_candidate_image_index)
         set found to true
      end if
   end repeat
end search_down

on insert_image_at(higher_image_index, lower_image_index)
   global pointer_dates_L_Len, up_pointer_dates_L, down_pointer_dates_L, this_image_date, this_image_index, debug
   if debug ≥ 3 then log {"Insertion", this_image_index, "between", higher_image_index, lower_image_index}
   set end of up_pointer_dates_L to higher_image_index
   set item lower_image_index of up_pointer_dates_L to this_image_index
   set end of down_pointer_dates_L to lower_image_index
   set item higher_image_index of down_pointer_dates_L to this_image_index
   set pointer_dates_L_Len to pointer_dates_L_Len + 1
end insert_image_at

on pointer_check()
   global image_date_list, pointer_dates_L_Len, list_begin, list_end, up_pointer_dates_L, down_pointer_dates_L, debug, list_start, list_finish
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   log {"list_start", list_start, "list_finish", list_finish, "up_pointer_dates_L", (items 1 thru heavy_debug_image_limit of up_pointer_dates_L), "image_date_list", (items 1 thru heavy_debug_image_limit of image_date_list), "image_name_list"}
   
   if debug ≥ 4 then --  really long detailed listing  Only do this with the short list in debug level; 4.
      log "Listing Dates List Pointers"
      log {"unsorted: ", (items 1 thru pointer_dates_L_Len of image_date_list)}
      set next_pointer to list_start
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of up_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log {"Sorted up: ", sorted_image_date_list}
      log {"Sorted up: ", sorted_image_list}
      
      set next_pointer to list_finish
      set sorted_image_date_list to {item next_pointer of image_date_list}
      set sorted_image_list to {next_pointer}
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set next_pointer to item next_pointer of down_pointer_dates_L
         set end of sorted_image_date_list to item next_pointer of image_date_list
         set end of sorted_image_list to next_pointer
      end repeat
      log {"Sorted down: ", sorted_image_date_list}
      log {"Sorted down: ", sorted_image_list}
   end if
   
   if debug ≥ 2 then
      log "Checking Dates List Pointers"
      set next_pointer to list_start
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of up_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date < prev_date then
            set list_good to false
            log {"Up pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of up_pointer_dates_L ≠ list_end then
         log {"error in last Dates List up pointer", next_pointer, (item next_pointer of up_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Up Pointers for Dates List are OK"
      
      set next_pointer to list_finish
      set this_date to item next_pointer of image_date_list
      set list_good to true
      repeat with counter from 1 to pointer_dates_L_Len - 1
         set prev_date to this_date
         set prev_pointer to next_pointer
         set next_pointer to item next_pointer of down_pointer_dates_L
         set this_date to item next_pointer of image_date_list
         if this_date > prev_date then
            set list_good to false
            log {"Down pointer error between Dates List items ", next_pointer, this_date, "and", prev_pointer, prev_date}
         end if
      end repeat
      
      if item next_pointer of down_pointer_dates_L ≠ list_begin then
         log {"error in last Dates List down pointer", next_pointer, (item next_pointer of down_pointer_dates_L)}
         set list_good to false
      end if
      
      if list_good = true then log "Down Pointers for Dates List are OK"
   end if
   
end pointer_check

on cleanup_sort()
   global down_pointer_dates_L
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying Variables used only for Sorting" --  Avoids stack overflow problems
   set down_pointer_dates_L to {}
   set ymd_list to {}
   set yearmonth_list to {}
   set year_list to {}
   set ymd_start_pointers to {}
end cleanup_sort

on Duplicate_Search()
   global image_date_list, list_begin, pointer_dates_L_Len, list_end, up_pointer_dates_L, debug, heavy_debug_image_limit, image_count, duplicate_image_lists_L, list_start, list_finish
   if debug ≥ 1 then log "Starting Duplicate Search"
   set this_pointer to item list_start of up_pointer_dates_L
   set this_date to item this_pointer of image_date_list
   set first_dup_date to {}
   set dupe_found to false
   set duplicate_image_sublist to {}
   set duplicate_image_lists_L to {}
   repeat with counter from 1 to pointer_dates_L_Len - 2
      set prev_date to this_date
      set prev_pointer to this_pointer
      set this_pointer to item prev_pointer of up_pointer_dates_L
      set this_date to item this_pointer of image_date_list
      if this_date < prev_date then log {"error prev_pointer", prev_pointer, "prev_date", prev_date, "this_pointer", this_pointer, "this date", this_date, "counter", counter}
      if this_date = prev_date then
         if dupe_found = false then -- start of a series of duplicates -  found first two
            if debug ≥ 3 then log "Found Dupe"
            if debug ≥ 4 then log {"prev_pointer", prev_pointer, "prev_date", prev_date, "this_pointer", this_pointer, "this date", this_date, "counter", counter}
            set dupe_found to true
            set first_dup_date to prev_date
            set duplicate_image_sublist to {first_dup_date, prev_pointer, this_pointer}
         else -- adding other duplicates
            set end of duplicate_image_sublist to this_pointer
         end if
      else if dupe_found = true then -- end of a series of duplicates
         set dupe_found to false
         set end of duplicate_image_lists_L to duplicate_image_sublist
         set duplicate_image_sublist to {}
         set first_dup_date to {}
      end if
      if debug ≥ 4 and counter > heavy_debug_image_limit then error
   end repeat
end Duplicate_Search

on cleanup_dupe_search()
   global up_pointer_dates_L, image_date_list
   if debug ≥ 1 then log "Emptying Variables used only for Duplicate Searching" --  Avoids stack overflow problems
   set image_date_list to {}
   set up_pointer_dates_L to {}
end cleanup_dupe_search

on Evaluate_display_dupe_list()
   global debug, duplicate_image_lists_L, no_date_info_D
   -- the intent of this handler is to eleminate obvious non-duplicates
   ----Images with the same date but different image types (likely a RAW-JPG pair, but this is not checked)
   ----Image groups which are part of a (high speed) burst sequence
   -- Images which are reported are likely duplicates, but are NOT guaranteed duplicates
   
   if debug ≥ 3 then log "Evaluating  Duplicates"
   if debug ≥ 4 then log duplicate_image_lists_L
   
   
   
   repeat with sublist_index from 1 to (length of duplicate_image_lists_L)
      
      set {dupe_date, duplicate_image_list} to make_dupe_name_list(item sublist_index of duplicate_image_lists_L)
      
      if dupe_date ≠ no_date_info_D then
         repeat with sublist_index from 1 to count of duplicate_image_list --For each sublist
            
            if (count of item sublist_index of duplicate_image_list) > 2 then -- if there is more than one image name then determine if this could be a burst
               if debug ≥ 3 then log (item sublist_index of duplicate_image_list)
               set is_burst to true -- start by assuming its a burst, and then test it
               set image_seq_list to {}
               set image_name_ext to item 1 of item sublist_index of duplicate_image_list
               repeat with sublist_counter from 2 to length of item sublist_index of duplicate_image_list -- the first entry in the sublist is the extension
                  
                  set image_name to item sublist_counter of item sublist_index of duplicate_image_list
                  set image_num_list to ""
                  set image_char_list to ""
                  repeat with char_counter from 1 to count of image_name
                     set this_char to item char_counter of image_name
                     if this_char = "." then
                        exit repeat
                     else if "0123456789" contains this_char then
                        set image_num_list to image_num_list & this_char
                     else
                        set image_char_list to image_char_list & this_char
                     end if
                  end repeat
                  
                  set image_num_count to count of image_num_list
                  if image_num_count = 0 then -- if there are no numbers in the image name, this is not a burst
                     set is_burst to false
                     exit repeat
                  end if
                  set image_num to image_num_list as integer
                  set end of image_seq_list to image_num
                  
                  if sublist_counter = 2 then -- first item
                     if (count of image_char_list) > 0 then
                        set has_image_char to true
                        set image_char_ref to image_char_list
                     else
                        set has_image_char to false
                     end if
                     set image_num_count_ref to image_num_count
                     set image_seq_min to image_num
                  else -- case: sublist counter >2  (additional items)
                     
                     if has_image_char then -- check that the image characters match the first image
                        if image_char_list ≠ image_char_ref then
                           set is_burst to false
                           exit repeat
                        end if
                     else -- first image name has no characters
                        if (count of image_char_list) > 0 then
                           set is_burst to false
                           exit repeat
                        end if
                     end if
                     
                     if image_num_count ≠ image_num_count_ref then -- now check the numbers
                        set is_burst to false
                        exit repeat
                     end if
                     if image_num < image_seq_min then set image_seq_min to image_num -- find the starting sequence number
                  end if
                  if debug ≥ 4 then log image_num
                  if debug ≥ 4 then log image_char_list
               end repeat
               
               if is_burst then -- check that the image numbers are sequential with increments of 1
                  set image_seq_count to count of image_seq_list
                  set last_seq to image_seq_min
                  repeat with incr_counter from 2 to image_seq_count
                     set this_seq_found to false
                     repeat with seq_counter from 1 to image_seq_count
                        if item seq_counter of image_seq_list = last_seq + 1 then
                           set last_seq to (item seq_counter of image_seq_list)
                           set this_seq_found to true
                           exit repeat
                        end if
                     end repeat
                     if this_seq_found = false then
                        set is_burst to false
                        exit repeat
                     end if
                  end repeat
               end if
               if not is_burst then
                  log {"Possible Duplicates: ", (items 2 through (count of item sublist_index of duplicate_image_list) of item sublist_index of duplicate_image_list), "on", dupe_date}
               end if
            end if
         end repeat
      else
         log {"Images with no date info: ", dupe_date, duplicate_image_list}
         
      end if
      
   end repeat
end Evaluate_display_dupe_list

on make_dupe_name_list(date_ptr_sublist)
   global debug, image_name_list
   
   set dupe_date to item 1 of date_ptr_sublist --  the date and time of these duplicates
   
   set dupe_name_list to {}
   repeat with pointer_counter from 2 to length of date_ptr_sublist
      set image_name to item (item pointer_counter of date_ptr_sublist) of image_name_list -- get the name of the image
      
      ## extract the type of the image
      set image_type_found to false
      set image_type to text -1 of image_name
      repeat with char_counter from 2 to 5
         set this_char to item (-char_counter) of image_name
         if this_char = "." then
            set image_type_found to true
            exit repeat
         else
            set image_type to this_char & image_type
         end if
      end repeat
      
      if not image_type_found then
         log "WTF?? Image type not found in Image Name: " & image_name
         exit repeat
      end if
      
      ##  search for a sublist with this type of image already in the dupe_name_list
      set image_type_in_list to false
      repeat with sublist_index from 1 to count of dupe_name_list
         if item 1 of item sublist_index of dupe_name_list = image_type then
            set image_type_in_list to true
            exit repeat
         end if
      end repeat
      
      ## if not already in dupe_name_list add a sublist for this image type
      if not image_type_in_list then
         set end of dupe_name_list to {image_type}
         set sublist_index to count of dupe_name_list
      end if
      
      set end of item sublist_index of dupe_name_list to image_name --add the image name to the sublist
      
   end repeat
   return {dupe_date, dupe_name_list}
end make_dupe_name_list

on final_cleanup()
   global image_date_list, image_name_list, pointer_dates_L_Len, up_pointer_dates_L, up_start_pointer_dates
   global ymd_start_pointers, year_list, yearmonth_list, ymd_list
   if debug ≥ 1 then log "Emptying all  Variables" --  Avoids stack overflow problems
   set image_date_list to {}
   set image_name_list to {}
   set up_pointer_dates_L to {}
   set pointer_dates_L_Len to {}
   set up_start_pointer_dates to {}
end final_cleanup
Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Fri Apr 21, 2017 2:58 am

Well done, young man!

That script worked very well both on the catalogue with the date issues and also the big (37k) catalogue.

The bad news is that I have a LOT of time ahead cleaning out rubbish!

BTW, an off-topic question: why is a file recognised as having no date, when COP shows it has one (of sorts) - see here: https://drive.google.com/open?id=0B_WBq7y8HMviVmsyWHRYVW5iOEE.

Regards - and thanks again
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia

Re: A script to find duplicate image files

Postby Eric Nepean » Fri Apr 21, 2017 7:56 am

ShaneB wrote:Well done, young man!

That script worked very well both on the catalogue with the date issues and also the big (37k) catalogue.

Great news. Good to hear that it works for 37k images.
The bad news is that I have a LOT of time ahead cleaning out rubbish!

BTW, an off-topic question: why is a file recognised as having no date, when COP shows it has one (of sorts) - see here: https://drive.google.com/open?id=0B_WBq7y8HMviVmsyWHRYVW5iOEE.

Regards - and thanks again


OSX isn't actually OSX from the ground up. OSX is actually a graphic user interface (i.e. windows and menus used with a mouse) built on top of the old and very well developed UNIX operating system.

In the UNIX operating system, the date and time are defined as the number of seconds since midnight January 1, 1970. This is the same year that the first offical UNIX operating system ran on a PDP 11, the first Macs and IBM PCs were introduced over 10 years later.

So in many OSX SW applications, like Aperture and Preview (but not Applescript), when the date for an image or some other object is not available, the number 0 is used for the date. And so the date of the object becomes January 1, 1970.

So, within my script, I've defined the value used for date of an image which comes with no date information to be midnight January 1, 1970. This includes those images which come from COP with a date of January 1, 1970, and those that come with a date of "missing value" - for the purpose of sorting and finding duplicates, its almost the same.

Because I'm short of time, I've also included any images which have a date earlier than January 1, 1970, any images having a date later than 1 year into the future, and any images with corrupted date values. They all get assigned a date of January 1, 1970, since the image date information is likely bogus anyway, and there are likely very few or no images that fall into these categories.
Cheers
Eric
(OSX 10.12, iMac and MacBook Air, Panasonic GX7,GM5,G5, Olympus E-M1)
Eric Nepean
 
Posts: 374
Joined: Sat Jun 28, 2014 8:54 pm
Location: Ontario, Canada

Re: A script to find duplicate image files

Postby ShaneB » Sun Apr 23, 2017 1:19 am

Thanks Eric.
Shane Baker
Perth, Western Australia
ShaneB
 
Posts: 68
Joined: Sun Aug 02, 2015 3:20 pm
Location: Perth, Western Australia


Return to Scripting



Who is online

Users browsing this forum: No registered users and 1 guest