Scanning negative film with Vuescan, part 3: using the whole contrast range

0. Introduction - What's the problem?

The procedures explained in parts 1 & 2 all work with only a part of the contrast range of the negative film. They all lead to the loss of detail in either highlights or shadows of the image.

Color negative film can handle a very broad range of object contrast (up to 14 f-stops). However, it will reduce this to a fairly small range on the actual film. This results in a low maximum density, which (as opposed to slide film) even fairly cheap flatbed scanners with their transparency units can handle.

Vuescan allows you to easily scan the whole contrast range of your film. The resulting images appear dull because the whole contrast range of the object is squeezed together to accommodate the low contrast range of your screen.

Using suitable adjustment curves or other procedures, you can increase the contrast that the result will again look natural. This will, on the other hand, also amplify any color cast the film might show, especially in very bright or very dark regions. So, if you want to use the full contrast range, you will also have to correct colors over the full range.

Simple IT8 profiling won't do. There are no IT8 targets on negative film. The IT8 format is furthermore not designed to provide high contrast images. One possible way out of this might be exposure bracketing, as explained in part 1 of this tutorial.

This third part shall show you how to
- find suitable Vuescan settings to really capture the full contrast range of your film,
- use the lcms Measurement Tool to analyze your photographed targets,
- correlate the resulting data with the nominal values that come with the target, and
- calculate a correction curve for use with Photoshop or another imaging software.

This third part is not intended to
- replace the user's manual of Vuescan or the lcms measurement tool
- explain color management in general, or
- be used by complete newbies to photography or scanning

0.1 Note

While parts 1 & 2 of the tutorial as well as section 2 of this part are basically derived from the Vuescan user's manual and looking at scan results and the way the lcms profiler works, part 3 section 3 is largely my own work.

I am, therefore, not absolutely convinced about being right. I hope that color management pros will excuse the occasional mis-use of terms and that they will tell me where I am way off.

0.2 Acknowledgments

My thoughts were, of course, based on the work of other people:
- Jacek Zagaja came up with the idea of building accurate look-up tables using Excel: http://tme.szczecin.pl/~jacek/lut.html
- Timo Autiokari provided the VBA routines to write the curve file. They are also copyrighted by him: http://www.aim-dtp.net/
- Charles Poynton's FAQ-pages have greatly helped me understanding color management: http://www.poynton.com/notes/colour_and_gamma/
- Wolf Faust produces excellent IT8 targets. I have shown him an outline of this tutorial, and he thinks it is "basically right".

1. Prerequisites

To work your way through this part additionally to the requisites used for parts 1 and 2, you need
- to have read and understood parts 1 & 2,
- Excel or a similar spreadsheet and some knowledge about how to use it, as well as
- Photoshop

2. Scanning without Clipping.

Clipping describes the loss of image information below or above a specified brightness level. In general, there are three reasons for clipping
- limited exposure range of the sensor (usually a CCD)
- implied (unintentional) clipping through the software used
- subsequent intentional clipping in order to enhance contrast

2.1 Optimal usage of CCD exposure range

When using CN-films, there are seldom CCD underexposure problems. Maximum density of the film material is very low, so even common cheap scanners work as expected as far as exposure of the highlights is acceptable.

CCD exposure ('RGB exposure') should be as high as possible to minimize the noise of the CCD which mostly occurs in areas of low exposure.

Vuescan therefore sets the exposure a little bit higher, resulting in slight clipping. Changing the value for 'Exposure Clipping (%)' located at the 'Input' tab controls the amount of clipped pixels in relation to the whole picture. Setting this value to 0.0 should prevent any clipping as a result of CCD exposure.

Before this feature had been introduced in an earlier release of Vuescan, I used a different procedure which I still think is the safer way, because a second source of clipping is eliminated: 'film base color'.

Vuescan stores the film base color in three values where each value is usually below 1.0. You can modify these values in the 'Color' tab if 'lock film base color' is selected at the 'Input' tab.

Values below 1.0 for film base color will also result in clipping. You can visualize this effect using a high contrast slide with these values set to 1.0. As a result you will see more details in the lights. For a true subtraction of film base color with optimized ccd exposure, at least one of these values should be 1.0.

On a scanner with arbitrary control of CCD exposure, you can check how film base color changes when you change the CCD exposure. You need an unexposed part of film, e.g. the beginning of the film roll. First scan this part without checking 'Lock exposure' or 'Lock film base color' on the 'Input' tab.

After checking 'Lock exposure', Vuescan shows you the value for 'RGB exposure'. Now, do exposure bracketing from about half the indicated value up to this value. For a first test, you can use 5 steps, but 10 steps will make it more obvious. Using my LS-40, automatic exposure gave me a value of about 3.9. So I started at 1.8 with increments of 0.2 (that means 1.8, 2.0, 2.2 and so on.)

For each step, select the corresponding 'RGB exposure' value, deselect 'Lock film base color', do a 'Preview', then re-select 'Lock film base color'. Copy the values from the 'Color' tab into a spreadsheet along with the corresponding exposure. You can now easily visualize film base color values against RGB exposure in a diagram.

This diagram should show three linearly increasing lines up to a color value of 1. The ratio of these values (at specific exposure) should stay constant in the 'good' exposure area.

If these lines don't increase linearly (i.e. they are curved), your scanner does not react linearly to RGB exposure control. This will be no problem as long as the values stay proportional (see below).

If you normalize the values by setting the highest value of each triple to 1 and adjusting the other two proportionally, you should get nearly identical values for all the 'good' RGB exposures.

This moralized film base color (averaged with optimal ccd exposure over several films if you want) should avoid clipping caused by too low values and will at the same time effectively neutralise film base color, leading to nice, neutral blacks later.

The point where the lines begin to deviate from linearity marks the start of overexposure. It is the beginning of exposure saturation of the CCD for the corresponding color. For perfect optimization, you can create a trend line through each point inside the linear range, output the formula from these points and calculate the junction with film base color 1. The lowest of the three ccd exposure values is the one we want to use.

It's interesting to note that sometimes these lines "jump" even in the linear range and with essentially the same ratios between the different colors. It seems that Vuescan does some distinction through the comparison of differences at some point and sets very low ccd exposures to a higher value.
 

2.2 Avoid artificial Clipping

Basically Vuescan provides all information to be able to control clipping. Unfortunately this is made more difficult by somehow unorthodox programming of the Software.

Using the 'Color Balance - None' mode definitely no information will get lost. But the images can come up with such a low contrast, that there is hardly anything to recognize.

With 'Color balance - Neutral' the areas with no information at both ends of the histogram will be discarded. The fields "Black point %" and "White point %" determine the relative number of pixels of the image which will be clipped. Setting both to 0.0 no image information will be lost, but the contrast of the image will be spread over the whole available range.

In all cases you can see the absolute values for the black and white points when you check 'Lock image color' on the 'Input' tab. These values are between 0.0 and 1.0 and allow only limited conclusions about the RGB values within the image (as it does e.g. Photoshop in the 'Levels' dialog box). Unfortunately, Vuescan performs level correction (which results from the gamma of the color space as well as the values for brightness) already before the black and white point adjustments. Depending on the brightness settings the adjustment of black and white points has a different effect.

Therefore I recommend to work with 'Color - Balance Neutral' or 'Color - Balance None' and later do adjustments of black and white point in the image editing software. For our purpose - scanning the complete contrast range - only the 'Color Balance - None' mode can be applied.
 

 3. Analysis

With the help of the optimized scanning procedure the corresponding type and extent of color shift has to be determined and respective correction curves have to be created. Here, I have limited myself to color information alone and don't create curves for correcting brightness (thus not analyzing the gradation curve of the emulsion). The corrections are limited to the errors in the gray scale ignoring all other color fields in the target. See appendix for further details.

The following steps are necessary:
- identification of the color values for the gray scale in every picture of the exposure series
- transformation of the original data of the IT8-targets into RGB
- adjustment of brightness (simulation of different exposures)
- comparison of the data
- generation of the curve for further use in photoshop

With the exception of the first item, all steps can be executed within the Excel worksheet CN_Curve.xls, which can be downloaded from http://www.erik-krause.de/tutorial/CN_Curve.zip (350kB)

3.1 Scanning the targets

All images of the exposure bracketing series of the camera IT8-target originating from part 1 have to be re-scanned, using the optimum CCD exposure and normalized film base color as determined in above procedure.

Check 'Color balance - none' and export as TIFF, now using Adobe RGB space (I'll come back to that again later) and convert to PNG for the lcms measurement tool.

3.2 Measurement and calculation

As in part 2, the image data is loaded into the measurement tool and files with the measured results are generated.

In contrast to part 2, the generated data is loaded into a separate Excel worksheet for analysis. These files transferred to Excel by drag & drop should each generate a new worksheet with separated columns. If this this does not work for some reason, the file can be loaded manually and the separation has to be made by hand.

The four columns SAMPLE_ID, RGB_R, RGB_G and RGB_B have to be selected and copied to the respective worksheets ('scan_00' through 'scan_05') in CN_Curve.xls. We additionally need the original IT8 data of the target. They are treated equally and the columns SAMPLE_ID, XYZ_X, XYZ_Y and XYZ_Z are copied to 'IT8 Sheet'.

3.2.1 Why do we do this?

The target's IT8 data are given in the xyz-format (and perhaps in lab) and refer to a white point of D50 (daylight 5000K). For further comparison with the lcms derived values this data must first be converted into a defined RGB color space.

For this purpose I have used AdobeRGB. It's available both within Photoshop as well as Vuescan and is much wider than sRGB. On http://www.brucelindbloom.com you can find appropriate conversion matrices from D50 to D65 (which is the white point of AdobeRGB) as well as from XYZ to AdobeRGB.

This conversion is performed in the worksheet 'IT8Sheet'. Here also a calculation of the respective brightness values derived from RGB is done. The result is needed for accommodating the brightness between the differently exposed targets (brightness as a weighted average according to http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html).

Sheets 'scan_00' to 'scan_05' will also calculate the brightness from the RGB_R, RGB_G and RGB_B ('luminance'). These values together with the corresponding values from the 'IT8Sheet' are used to calculate a gamma value which in turn is used for calculating target values (Des_R, Des_G, Des_B) from the original IT8 data. These are the values which the image should have when keeping brightness constant.

The gray scale values (bottom line - DMIN, GS1 to GS22, DMAX) from 'scan_00' to 'scan_05' can be found again as aliases on worksheet 'Curves'. In columns Norm_R, Norm_G and Norm_B the target values are expanded to the interval 0 to 255 with the minimum represented by 0, the maximum by 255. All values in between are assigned proportionally. This procedure ensures an optimal usage of the available range from 0 to 255.

Finally, in the 'Curves' worksheet target and real values are brought into relation (beginning in line 146) within a diagram. For this purpose Excel derives a polynomial trend curve of 6th order (4th order might be probably good enough).

There's a menu item called 'Create and save curve'. The sub items 'linear luminance curve map' and 'sine luminance curve map' will create a set of correction curves with either linear brightness or with brightness adapted to a sinus curve.

The latter will result in an improved contrast in the mid tone range with a compressed contrast at the dark and light ends.

Additionally, the effect of the sine curve can be controlled by the 'sine-factor' field. The default value of 0.07 results in a curve which just avoids any clipping. Higher values will create a steeper curve in the middle part and lead to greater losses at the bright and dark end.

The macros behind the menu items transfer the formulae for the trend lines into columns R_calc, G_calc, B_calc on sheet 'AMP'. For generating the curves the columns R, G, B with a limited data range of 0 to 255 will be used. A tiny VBA program written by Timo Autiokari writes these values into an .AMP file which can be loaded into photoshop's gradation curve dialogue as an 'effect-curve'.

If you want to change contrast by other means like e.g. with my contrast masking action (see below) the use of the linear curve is mandatory. Otherwise, just try which one fits better. You can anyway create as many .AMP files as you want and check the result already with the preview within Photoshop.

3.3 Application

All the efforts result in quite an easy application:
- scan the negatives with Vuescan with optimized CCD exposure, 'Color Balance, None' and maximum bit depth and save it to a 48 bit TIFF using AdobeRGB color space.
- open the images in photoshop.
- activate the gradation curve dialogue and load the curve (chose .AMP for the type).
- set white and black points and increase the contrast if necessary, e.g. using the methods mentioned at http://www.erik-krause.de/kontrast

4 Discussion

As I said earlier, paragraph 3 is entirely based on my own ideas. You may well ask "Why not do it differently?".

4.1 Why not use raw data?

Vuescan can save raw scanner data in two different formats. If you select 'Raw output with Scan' on the 'Output' tab, Vuescan will save raw image data as it comes from the scanner into a TIFF file. If, on the other hand, you select 'Raw output with Save', Vuescan applies the 'Infrared clean' and 'Grain reduction' filters if you have selected them on the Filter tab.

Vuescan does not apply gamma correction before it saves raw data, so it seems to be better to use raw files to generate the curves. If you select 'Image type Color negative' Vuescan applies a logarithmic curve (P = log10(I) ^ 2.2/G according to Ed Hamrick), by the way.

I was, however, not able to transform this raw data to a defined color space. The IT8 target data comes in XYZ format which has to be converted to RGB for the comparison. But to do this conversion, we need a defined color space.

4.2 Why only the gray scale is used?

If you made the film profiling according to part 2 of this tutorial using lcms, you could enter target definition ('Target') and measured values ('Measurement') along with your profile in the ProfileChecker. The CIE diagram would then show you the differences between theoretical and measured colors, the color shift, as lines between target color and real (measured) color.

Alarmingly, this color shift has completely different directions for the various colors. If you wanted to correct those, you would have to use much more complex algorithms than our simple curve interpolation.

5 Sources for errors

Apart from the issues mentioned in part 1 and 2, there is of course the possibility that by accident this whole procedure just worked for me only. I would therefore like to hear about your experiences with it, whether they are positive or negative!