How to sanely compress Fuji RAF files

If you’re a Fuji DSLR shooter like myself, particularly a Fuji S5 shooter, you’ve probably discovered that if you shoot RAW, space on your hard drives disappears rather quickly. Unless you’ve got a mighty speedy workflow and can get your RAFs off the disk in a hurry, a space crunch is practically inevitable.

Without actually moving them to another media, there are basically three sane tacks:

  1. Compress the drive or directory the RAW files live on.
  2. Convert the files to DNG format.
  3. Use Fuji’s HS-V3 to convert to the files to compressed RAF.

#1 doesn’t yield a huge amount of compression as it’s geared to linear data rather than photos. #2 works great, but leaves your images limited to what RAW converters can handle them. In particular Fuji’s HS-V3 (HyperUtility Software version 3) can’t currently handle the DNGs, which is important to me since arguably HS-V3 does the best conversion job (even if it’s too painful in my opinion for regular use).

So, that leaves option #3. The problem with option #3 is you have to use HS-V3 to do the conversion. Not only is HS-V3 klunky in this regards, it can be extremely slow because the “RAW FILE COMPRESSOR” continually communicates with HS-V3’s management interface to redraw the RAFs as they are compressed. This makes the compressing take even longer and consumes more cycles on the PC, making it difficult to multi-task.

The answer is actually fairly simple. It turns out the “RAW FILE COMPRESSOR” is a standalone application. You can easily add it to your desktop or “Start” menu by creating a shortcut (note of course, this is Windows – if you’re Mac, well, I’m sorry you’ll have to figure this out yourself).

Creating a “Start” menu shortcut is a bit beyond the scope of this document, however creating a desktop shortcut is simple and if you do want a “Start” menu shortcut, you can always drag the desktop shortcut to your “Start” menu.

So here’s what you do:

  • Right click the desktop.
  • Choose “New / Shortcut”
  • When the “Shortcut” dialog pops up select “Browse”.
  • Navigate in the “Browse For Folder” to “My Computer / Local Disk (C:) / HyperUtility / HELPERS” and select “RAFCOMP.EXE”. Then click “OK” at the bottom. Note “Local Disk (C:)” may vary depending on how you installed Windows (it is also of course possible you didn’t put HyperUtility in the default directory).
  • You will drop back to the “Create Shortcut” dialog and it will have the “RAFCOMP.EXE” path inserted into the “location of the item”. Click “Next” on the bottom to continue.
  • In “Type a name for this shortcut” replace “RAFCOMP.EXE” with a reasonable name. I put “Fuji RAW File Compressor” as my value.
  • Select “Finish” at the bottom of the dialog.

Ok, so this creates a shortcut for the RAW File Compressor on your desktop (graphic is a vise), now what do you do?

Well, you run it. Double click to start it up. You’ll see that it brings up the same dialog as the compressor would if you ran it out of HS-V3. The thing is, what Fuji doesn’t tell you is you can drag and drop to this tool. So, open an “Windows Explorer” and find the RAF files you want to compress. Select them (using some combination of <CTRL> or <SHIFT> keys) and drag and drop them onto the compressor. They automatically get added to the compress list.

You can drag and drop as many as you want from various directories, even putting different directories into the same batch. Once you click “Convert” the compressor smartly handles these correctly and creates a “COMPRESS_1” subdirectory under each directory where the RAF files were stored. This “COMPRESS_1” directory contains the compressed RAF files. Usually I just drag the completed files right over the originals, letting it overwrite. Compression is roughly 50%.

By the way, you can even add files while it’s compressing, and it will handle those files correctly as well. Unlike running it out of HS-V3 it doesn’t endlessly try to redraw screens, it’s just a utility compressor in this state, just as it should be (well, at least in my opinion).

It actually turns out that it’s fairly easy to add to your “Send To” menu as well, but that seems to be limited to about 35 items per “Send To”, so I didn’t find it worth the effort.

One last note, just as many converters don’t handle Fuji DNGs, many also don’t handle compressed RAF, so there’s a bit of a trade off here. Capture One doesn’t, nor does DxO. My two favorite converters for Fuji do – Lightroom and HS-V3. Lightroom can also convert the “compressed RAF” to a DNG, which may give you a few more options (in my tests the conversion of a “compressed RAF” to DNG was the exact equivalent of a vanilla RAF to DNG – the compression is redone in DNG’s format). Capture One says they’ll handle Fuji DNGs some day, but it hasn’t shown up in a few revs already.

UPDATE:

As “dlbogdan” pointed out on DPReview, you can get similar compression levels out of WinZip or 7-Zip. Here’s some result testing with 27 RAF files with both S&R pixels (ie: 24mb RAFs):

  • 638 mb total for the originals (uncompressed)
  • 329 mb total for the Fuji Compressed RAF format
  • 421 mb when zipped into a ZIP archive using WinZip in default WinZip 2.0 compatibility mode
  • 323 mb when zipped into a “7z” archive using 7-Zip
  • 324 mb when zipped into a ZIP archive using WinZip with compression set to “Optimize for best compression”

The issue for me is that neither Lightroom nor HS-V3 can see into any of these other archive formats (ie: “.ZIP” or “.7z”). That means you have to unpack and re-pack to work with them. Supposedly HS-V2 could actually browse the WinZip 2.0 compatible archives, but unfortunately HS-V2 can’t handle Fuji S5 RAW files (HS-V2 only handles Fuji S2 RAF files plus a few older Fuji bridge cameras).

However, given that you preserve the original uncompressed RAF format in these archives formats, it may be worth the trade off. Using WinZip with say “Optimize for best compression” means that the RAW files will be usable by all converters that understand Fuji S5 format for the forseeable future.

UPDATE 2:

One question that also came up on DPReview from “acrystalball” (aka Crystal) is, “Is the Fuji RAW compression ‘destructive’?” The answer is, I don’t know – the HS-V3 help files don’t indicate one way or another. My personal guess is “no”. Certainly I have seen no degredation and given that there is in fact only 12 megapixels of RAW data, I would expect a much smaller RAW file by default than the 24 mb the S5 outputs. Thus, I think there is plenty of room to non-destructively compress.

UPDATE 3:

It also looks like SilkyPix handles compressed RAFs, which apparently a “free” version also exist:

http://www.isl.co.jp/SILKYPIX/english/download

However it’s pretty severely limited over the for-pay version.

UPDATE 4:

itt” (as in “Cousin”) at DPReview made some other notes based on this post:

  • Files can be dragged from Lightroom and Adobe Bridge directly into the Fuji compressor.
  • BreezeBrowser does show thumbnails and can export the embedded JPEG from compressed RAFs (to note, from what I can tell any app that can handle uncompressed RAFs can actually see the thumbnail and embedded JPEG in compressed RAFs, though they may not be able to actually open the RAF and/or process it).
  • It’s much easier to create a shortcut by dragging the “RAFCOMP.EXE” binary while holding the right mouse button to the desktop. This automatically creates a shortcut. Note make sure that you’re holding the right mouse button otherwise you will drag the actual binary to your desktop, removing it from the location it would normally run out of (and potentially breaking things!)!
  • Earlier he also pointed out that “s7raw“, a free Fuji converter, also handles compressed RAFs.

Handy Lightroom to Flickr plugin

If you use Lightroom and Flickr you ought to check out this plugin:

http://regex.info/blog/photo-tech/lightroom-flickr

It is a very handy tool that allows you to upload from Lightroom directly to Flickr via the Lightroom “Export” command. Not only can it get the images to Flickr, it can add them to sets, export the EXIF data into Flickr fields, and can set browsing permissions.

Other than an occasional bug that is the fault of the Lightroom Export SDK, the plugin works flawlessly and is very simple to use. It is also updated regularly.

If you use other services like Picasa Web, SmugMug, etc. the developer has created other plugins for them as well. See the links on this page:

http://regex.info/blog/photo-tech

Definitely worth checking out. The author was an official “Beta-Tester” for the Lightroom SDK, so he seems to know what he’s doing.

Very cool antique/old photo site

If you haven’t checked out this site, you ought to:

http://www.shorpy.com

This is a great collection of antique photos from all sorts of genres. In particular some of the large format Kodachromes are amazing.

I’m hoping to contribute some myself when I get a chance.

Looking for caps?

Looking for caps, eyepieces, covers, and all the other junk that seems to consistently and mysteriously fall off your Nikon/Fuji DSLR body? Here’s a source that has done a great job putting it all into one place:

http://bocaphoto.com/products/new/nikon/nikonacc.htm

Personally I’ve never used them, but just to have them all layed out is a huge help!

Digital Focus Hell

One of the biggest new issues with digital photography are problems with auto focus. It’s not exactly clear why, but digital cameras (DSLRs) seem to suffer from focusing issues far more prevalently than film cameras. It might be that we tend to look at our images more closely than film, with 100% pixel peeping. It might be the sensors are just more sensitive to focusing errors.

In any case it is prevalent, with some cameras like the Nikon D200 being notorious for it.

Focusing issues are generally grouped into two categories:

  • Front focusing.
  • Back focusing.

“front focusing” is where the focus is in front of the the object being focused on. “back focusing” is where the focus is behind the the object being focused on. So, for instance in the case of “back focusing” where you focused on the eyes, the ears might be in focus instead of the eyes. With front focus in the same example, the nose might be in focus instead of the eyes.

It doesn’t take much back or front focus to ruin a photo, particularly with portraits, macros, and images with low depth of field (DOF). Stopping down more can help, but DOF isn’t a panacea for accurate focus, and usually it’s evident when focus is off, even with sufficient DOF. It’s sharp, but not exactly the same.

Not all cameras suffer from the issue and often the effect is different depending on the lens being used. For instance on my Fuji S5 (based on the D200 body) my 85mm f1.8 was dead on, while my 60mm f2.8 was about .5 cm back focused. My recently purchased 35-70mm f2.8 was a full 2 cm back focussed!

I say “was” because after sending the camera in to Fuji for warranty repair and the lens to Nikon, it appears to have actually become slightly front focused, but much, much better. Similarly other lenses, like the 60mm look dead on, though I haven’t had time yet to do actual focus tests.

Speaking of focus tests, if you suspect you have front or back focusing issues, there are a number of sites with focusing charts, including:

Usually the tests involve shooting a chart at a 45 degree angle, however lining up a set of 5 AA batteries on a diagonal can be equally as effective, if perhaps not as accurate at telling you the exact distance of back/front focus.

There is also a lot of argument about of whether shooting a flat object for a test is accurate. In fact if you see this post:

http://photography-on-the.net/forum/showthread.php?t=10526

It shows an actual example from a Canon repair center where the focus point is perpendicular to the flat test chart, essentially creating a 3D object to focus on.

My personal experience with the tests seemed to show that the using a flat chart exaggerates the focusing errors. This isn’t a particularly bad thing because it makes the issue more obvious, but it also can convince you the issue is severe when it isn’t. In the end what I’ve done is placed batteries on a diagonal on the test chart lined up with the marks, effectively mixing the two, but primarily using the batteries as my indicator.

So, why do these focusing errors happen? There are a number of reasons including possible misalignment of the mirror box, but according to this post (it’s pretty technical but definitely worth a read):

http://forums.dpreview.com/forums/read.asp?forum=1030&message=24839477

most of it comes from an interaction of the focus sensor position and the design of the lens.

There is hope though, with adjustment things can be improved.

Now you might be tempted to use the D.I.Y. (Do It Yourself) technique as suggested by sites like these:

Personally I would avoid it because as pointed out in this post:

http://www.nikonians.org/forums/dcboard.php?az=show_topic&forum=147&topic_id=37618

at the manufacturer it isn’t as simple as a mirror box adjustment, which is what these are doing. Yes, for minor focusing adjustments it may help, but in many cases if you break it you bought it and it isn’t really the way to do it. The way to do it involves computers and special equipment.

This is consistent with my conversations with both Fuji and Nikon regarding the issue. Both technicians I talked to indicated that it isn’t simply moving the entire focus back or forth (which is what happens with the mirror box adjustments). In that case, say if you were fixing a back focus issue, you’d be moving the general focus to the front. Thus if you happen to have a lens that unlike your others is tack on, it’s going to become front focused with this adjustment. Not exactly optimal.

However with their computerized methodology, and both vendors indicated this, the lenses that were back focused come into alignment, while the lenses that are currently on, stay on! In fact Nikon even suggests sending your “standard” (most used) offending lens in with the body and they claim, when done, that the camera lens combination will be calibrated together, with the already “on” lenses staying on. Fuji on the other hand indicated that usually only the body is needed. Incidentally the term for calibrating the lens and body together is called “collimation“.

Because computers are involved, though I have no proof of this, what I suspect may be going on is (among other adjustment) is they’re actually programming the camera with lens information. On the D300 you can actually do this yourself through menus, which frankly is one of the more compelling features of that camera.

Sometimes yes, it is an issue with the lens itself, however mostly not. If the elements are misaligned it is possible it could throw focus off, but in most cases where it effects more than one lens, it’s the body.

Interestingly enough, according to Nikon technicians, AF-S lenses can have focus adjustments made to them. This I suspect is possible because they have some intelligence built into them, whereas D and pre-D lenses are more simplistic and cannot have the same firmware adjustments. However, again, if they have a real mechanical issue they can be repaired, which seems to be the case with my 35-70mm. However in general I wouldn’t count on it.

If you make these tests and do find an issue, a repair is likely to be around $120 in the examples I see. That to me seems a deal for the peace of mind knowing the work is being done by a professional who knows what they’re doing.

Well that’s it. Hope that helps.

Sources of Crystal Archive Ultra-Glossy Prints

These look like two good sources of Fuji Crystal Archive Ultra-Glossy prints. This paper is supposed to look like Cibachrome/Ilfochrome with the Melinex coating:

Denver Digital Imaging pricing seems to be a little less expensive, but West Coat Imaging’s web site looks easier to navigate as far as uploading images. Both are using Chromira printers, which unfortunately look to be only 8 bit and only have the sRGB color space, which means the gammut can’t be as wide as one would hope.

Are both S and R pixels used in Lightroom with the Fuji S5/S3 Pro?

Yes, according to Thomas Knoll:

Internally, the S&R images are first merged into a single HDR image. The S pixels are used for lower part of the tone range, and the R pixel are used top of the range, with a smooth transition between the two. This merging is completely independent of all the camera raw controls. The exposure and recovery sliders adjust the white clipping point in this merged HDR image. The other tone controls adjust the rendering of the values between zero and the white clipping point.

In the same thread it was also asked if the S&R pixels were handled under Compressed RAF and DNG formats, to which Thomas replied:

Yes and Yes (both).

For the full thread, see this Adobe thread.