How Digital Camera Sensors Work

In future posts, I intend to discuss several aspects of raw capture and processing that I have found to cause confusion. Before getting into workflow, however, it’s very worthwhile to have a good understanding of how camera sensors capture light and how this data is converted into full-colour images. A modern full-frame DSLR has a sensor chip behind its shutter curtain, and this chip has roughly the same surface area as a small matchbox. Cropped-sensor cameras, like Nikon’s DX range, have smaller sensors that nevertheless have the same aspect ratio, or proportions of width and height.

These sensors are covered in a vast grid of cells called photosites, which I like to think of as microscopic buckets designed to catch photons, or particles of light. These photosites capture the data that we refer to as pixels (the term is short for picture elements). Thus, my Nikon D810, which is a 36-megapixel camera, has 36 million photosites on its full-frame sensor. When an exposure is made, the shutter opens and exposes the sensor to light from the lens. The photosites fill with photons and, once the shutter closes, the fullness of each ‘bucket’ is interpreted by the camera as a number. An empty bucket has a level of zero, which translates as black, whereas a full bucket has a correspondingly higher number, and translates as white. Many modern cameras shoot 12-bit raw files, in which the digital level of white is 4095. (I will discuss how bit depth works in a future article.) In between these extremes we have many shades of grey, from dark to light. Here is a representation of a fragment of a sensor, showing only 20 pixels. Following the exposure, the levels have been calculated and I’ve shown their numbers on some pixels, along with the corresponding shades of grey:

I’m sure you can appreciate how 36 million of these will form a greyscale image. The intrigue occurs when colour is introduced; after all, raw capture is usually in colour in DSLRs. Solving the colour problem involves placing a coloured filter over each and every photosite, something referred to as a Coloured Filter Array or CFA. The technique that most sensors use today is known as the Bayer pattern, after Kodak engineer Bryce Bayer, who invented it in the mid-1970s. (How ironic that Kodak pioneered digital capture, then sat on it in order to safeguard its film business.) Let’s see how the Bayer pattern works.

The primary colours of light are red, green and blue; they can be mixed together in varying quantities to make any other colour you like. The Bayer pattern places a filter over each photosite, such that it only catches red, green or blue light. Here is that same sensor fragment with the CFA overlaid:

Now some sites catch red light, some green and some blue. The pattern alternates red and green filters on the first row, then green and blue on the second. This is then repeated all over the sensor. Notice that there are twice as many green sites as red or blue: this is by careful design. The human eye is most sensitive to light in the middle of the visible spectrum, and this light is green-yellow. For this reason, we often make day-glow jackets a green-yellow colour. The Bayer pattern captures more green light to simulate this sensitivity, so that our pictures will have colouring comparable to what our eyes see.

Above is the fragment after image capture; the numeric levels of some sites are shown, and I’ve indicated the colours these translate to. For example, the top-left photosite has a red filter, and it captured a large amount of light, converted as level 4000; the next sensor along has captured a reasonable amount of green light, converted as level 2800. The blue sites are quite dark, since not much blue light struck this part of the sensor. In a nutshell, this data is what a raw file contains. It’s an array of numeric values giving the amount of light in each photosite, along with the primary colour that site represents. Each camera manufacturer has its own proprietary format for raw files, but this is the general pattern of the data they contain.

The trouble now is that each photosite, or pixel, is only one colour. What about pixels that are pink, or sea-blue, or yellow? The camera doesn’t care. The job of working out the full colour of each pixel is performed by your raw conversion software—Adobe’s Camera Raw in my case. In yours, it could be Lightroom, or some other proprietary package. The following diagram will allow us to explore how the software does this:

The procedure is the same for each pixel, so I’ve chosen one at random. As you can see, it’s a red pixel that has captured a pretty high amount of red light. Let’s suppose the photographed scene is orange at this location (a sunset, perhaps), and this pixel should therefore be a shade of orange. The raw converter averages the values of all the neighbouring pixels, then adds and emphasizes the pixel in question’s value to form a sort of weighted average. Thus, the surrounding pixels supply the missing green and blue information. When all three colours are blended together, we end up with the real colour of the pixel: orange. So, no pixel captures its real colour in the camera, but rather relies on its immediate friends to supply the information it lacks. As our images attest, the process is remarkably accurate.

This process of working out the full colour is called interpolation, which is a mathematician’s way of saying ‘educated guess’. The process is applied to every pixel in the image and the entire operation we call demosaicing (an ugly word). As you might imagine, the pixels at the extreme edges don’t have as many neighbours and won’t be as accurately interpolated—they are often simply discarded, since losing a single pixel border is virtually unnoticeable.

There are some sensors, such as Sony’s Foveon range, that capture the full colour of each and every pixel. They achieve this by placing three filters over each photosite, and take advantage of the fact that the wavelengths of the different colours penetrate the photosite to different depths. But they are expensive and have not become mainstream. Nevertheless, they don’t require demosaicing and therefore avoid some of the problems caused by this process. One such problem is a subtle lessening of sharpness caused by interpolation. In my next article, I will discuss this problem and show how to use raw pre-sharpening to compensate for it.

To finish, I want to answer the question, ‘if the camera doesn’t work out the full colour of the pixels, why do I see a preview on the back?’ As you’re no doubt aware, DLSRs can also shoot to the JPEG format. This is a compressed format that produces files which require less disk space. The trade-off is quality: the JPEG conversion process throws away some colour data it figures your eyes won’t be able to notice. The raw data comes from the sensor and is immediately converted into a JPEG file. As part of this process, some settings are applied which you would ordinarily apply yourself in your raw conversion software: things like boosting contrast, extra saturation, a little sharpening. You can set these values on the camera, but they are applied in the same way to every JPEG you shoot. Capturing raw files instead preserves every bit of data from the sensor and applies no processing; that’s up to you to perform later.

However, when you shoot raw, the camera performs a quick-and-dirty JPEG conversion anyway, and embeds this ‘preview’ JPEG inside the raw file. When you see the image on the camera’s LCD, or view histograms and wotnot, you’re seeing the data from the JPEG, not the raw file. For this reason, the JPEG is never one-hundred percent accurate.

Leave a Reply

Your email address will not be published. Required fields are marked *