Author Topic: DxO Network Based Processing  (Read 365 times)

johnmac

  • Newbie
  • *
  • Posts: 1
DxO Network Based Processing
« on: August 03, 2017, 09:14:23 pm »
Hi all,

Lets start with a little background, as a company we have been using DxO for nearly 7-8 year to process large batches of imagery. Generally is worked without too many issue, a little slow at time, some randomly failing/freezing and the odd frozen screen but nothing a good cup of coffee and patience at time doesn't sort out.

Our quarterly turn over of imagery has reached the 450,000 image or (35-38TB), for the last year with 2 computer ( Intel Core i7 6900K, 512GB SSD (OS and TEMP), 64GB RAM, and GTX1070, windows 7)  running 24/7 processing and queued batches of 5,000-10,000 images, processing all over 1GbE dedicated network to a 100TB HPE StoreEasy with 4 paired 1GbE connections.

Is there anyone else out there who is processing over dedicated local area network to and from a NAS or other network based storage?

In recent time we have notice a slow down in the process and can't put our finger on it at present? Is there anyone else experiencing any issue with the latest updates in terms of processing to a device which is not local?

Have you had any success in performance moving form 1GbE to 10GbE or sfp+, Pair, etc...? Are you using SATA or SAS drive in your NAS? What type of NAS/storage are you using?

Have you found any setups that just don't work?

Have you found any setups/situations that works very well?

And before we get into local v's NAS, the practicality of moving 3-4 TB data packet to and from a local drive to a network location would be very changeling its something we have tried and resulted in increased staff time, so it a last resort for now. As larger SSD come on the market this may become an option once again, in a process of exchangeable processing caddies.

Cheers in advance, John


wyliec2

  • Jr. Member
  • **
  • Posts: 46
Re: DxO Network Based Processing
« Reply #1 on: August 16, 2017, 03:54:23 am »
I have done significant benchmarking with DxO primarily comparing CPU and direct attached HD and SSD.  For input/output files, I haven't seen DxO be especially sensitive to HD vs SSD given the processing overhead and reading/writing 20-40MB image files is not taxing - likely fits into most HD cache and is written before the cache fills up.

I did a comparison of 12 24MB RAW images processing using HIGH noise reduction and got the following results (note using same device for input and output):
* 41 seconds on NVMe SSD
* 41 seconds on SATA 3 HD
* 49 seconds on USB 3.0 flash drive
* 71 seconds on Gigabit Network location

Other info:
- The network connection and devices had virtually no other competing traffic.
- Using PRIME NR would reduce the percentage difference since there is longer process time with unchanged read/write.
- Test done on I7-5960x at 4.3 Ghz; 32 GB RAM; 2 concurrent images.

I have looked at 10Gb Ethernet - I would expect you could equip a NAS, a few PCs with 10Gig Ethernet adapters and get a 10Gb switch for $1000 - $1500.

Given the overhead of network processing of relatively small files and application-direct read/writing I don't know that 10Gb Ethernet would get you from (in my example) from 71 seconds down to 41 seconds??

I think it boils down to a time vs money decision for a 30% or so reduction in throughput time.
Win 10/64, I7-5960x 4.3 GHz,  32Gb 2666 RAM, Sapphire R9-390 GPU, Dell 25" 2560x1440 monitor 117DPI, m.2 NVMe OS

Bencsi

  • Hero Member
  • *****
  • Posts: 1458
Re: DxO Network Based Processing
« Reply #2 on: August 16, 2017, 07:28:43 am »
Based om wyliec2 test I made same - or similar - time comparison. In my case a network NAS ( through gigabit connection ) give me similar processing time as per files location in SSD or a HDD. By the way, in the past I tested a RAMDisk application to speed up the image processing. Unfortunately it had no advantage at all to SSD / HDD. The RAMDisk throughput was >5000 Mb/s compare to SSD 345 Mb/s and 140 Mb/s of HDD.

If such time lag existing, let me recommend to cut the DoP processing 2 parts:
1. make the image processing to a local quick drive
2. make an automatic image copy from local drive to network drive independently. The task should separate the transfer from image processing. Such task is a typical backup job. I use Cobian Backup Utility for daily save. After copy of all images, a simple batch file can delete the TEMP folder.
« Last Edit: August 16, 2017, 07:44:33 am by Bencsi »
Win7/64 PC, i7-3770, 3.9GHz, 24G RAM, Intel HD-4000 GPU, 27" calibrated LG monitor 1920*1080 px res. 82 DPI

wyliec2

  • Jr. Member
  • **
  • Posts: 46
Re: DxO Network Based Processing
« Reply #3 on: August 20, 2017, 07:14:43 pm »
After reading Benci's post I checked and the network storage I was accessing was running 100Mbps - so the 71 second time does not represent Gigabit network.

Original results:
I did a comparison of 12 24MB RAW images processing using HIGH noise reduction and got the following results (note using same device for input and output):
* 41 seconds on NVMe SSD
* 41 seconds on SATA 3 HD
* 49 seconds on USB 3.0 flash drive
* 71 seconds on Gigabit Network location

I reran using Gigabit and got the following times:
* 41 seconds with input/output on local SATA 3 HD (same as above)
* 47 seconds with input/output on network HD
* 45 seconds with input/output on network SSD
* 44 seconds with input on network SSD and output to local SSD

There does not appear to be a significant penalty exporting images with input and output on a network device.
Win 10/64, I7-5960x 4.3 GHz,  32Gb 2666 RAM, Sapphire R9-390 GPU, Dell 25" 2560x1440 monitor 117DPI, m.2 NVMe OS

 

photography