 |
I've had several ah ha moments recently about the Lytro camera, the technology behind it and what all that means to the future. The first big aha moment was when I figured out that it uses a standard CCD which is the technology at the heart of all digital cameras. For those who don't know, a CCD is a piece of silicon that translates the light that comes into a camera into digital values that can be stored and processed by computers. CCD stands for "Charge Coupled Device" and if you click on the first occurrence of CCD above you can learn all sorts of interesting things about how they work.
When Lytro talks about their first camera they prefer to refer to "magarays", or the number of light rays they capture. There are multiple reasons for this. First this is a very different camera from anything we've seen before and the terminology really does need to change to make understanding the differences easier to explain. The other reason is that the actual 2D product that comes out at the end of the process isn't very impressive right now; more on that topic later.
The eleven megaray number comes from the fact that Lytro is using an eleven megapixel CCD. Most smart phones have between five and eight megapixel CCD's so from that perspective the Lytro camera is actually a bit ahead of the game. High end SLR style digital cameras have much higher resolutions but they have both more expensive CCD's and higher quality optics to support that increased resolution. The higher quality optics are needed because ever smaller imperfections in the lens will lead to noticeable distortions in the image captured as pixel density increases. With the exception of some very impressive additional hardware and software the Lytro camera is essentially an eleven megapixel camera but it can't cheat as many low price digital cameras do by using optics that are not as good enough to support the reported pixel densities.
The Lytro camera has a rather odd aspect ratio of 1:1. In other words it takes square pictures. The native format of Lytro images is proprietary but you can export a jpeg image. When you do this you get a 1080x1080 picture which is a bit less that 1.2 megapixels. This is an interesting number when you take into account the fact that the camera captures 11 megarays of light. What this implies is that it currently takes around nine light rays for each pixel in the 2D image you can export from the Lytro software. (11 megarays/1.2 megapixel final picture = 9.2 rays per pixel). I'm not an expert in CCD's but a bit of research indicates that the high end of commercially available CCD's is in the 36 megapixel range. That would translate into about 3.9 megapixel 2d images if such a part were substituted for the CCD currently in the Lytro camera. As noted above though you would probably need better optics as well and there would be other implications such as much larger file sizes which would necessitate more storage and faster processing. All of which could greatly increase the cost of the camera.
Cost is currently one of the things people note as a limiting factor in the Lytro cameras appeal. Because of that ~ 9.2 to one ratio of rays to pixels in the final image you need higher quality components throughout the camera to get a particular level of quality. To be fair though this isn't a traditional camera and the comparison I'm making here is essentially comparing the Lytro's biggest weakness with one of the traditional digital camera's greatest strengths. Still, cost is likely going to be a constant challenge for Lytro in regards to their products both because they are a small company and because their technology appears to require higher quality components. This would be particularly true for high end products that had pixel densities sufficient to satisfy professionals.
I can think of at least one way to lower costs on future models though. It should be possible to use multiple lower price/resolution CCD's to capture higher light ray counts. If you look at the image below of the internals of a Lytro camera I believe the CCD is just behind the small light blue rectangle about 2/3rds of the way to the right.
If I'm understanding things correctly the light field sensor is taking the incoming light and breaking it up into a series of regions that are projected onto the CCD and in turn saved. Later on software is used to generate an image from these stored light rays.
You could in theory combine several CCD's into one mega CCD. This wouldn't work in traditional digital cameras as there would be gaps in the image where the various CCD's joined. This wouldn't have to be a problem for the Lytro since the sensor could in theory be tuned to accommodate such an arrangement by directing light rays only onto CCD surfaces and ignoring the borders.
Another advantage of such an approach would be the fact that you would have N paths to storage where N is the number of CCD's. Depending on the architecture this could mean lower costs as the communications bus between the image capture engine and storage could run at a lower clock rate if it were able to write to memory in parallel.
I'm getting a bit esoteric and am out of my depth here but suffice it to say that Lytro has some challenges in the area of creating higher quality traditional images. Future improvements in their software may yield a better than 9.2 to one ratio of rays to pixels. That would essentially be a free upgrade for those of us who bought the first generation product.
Significant improvements are probably going to require new hardware though. This isn't to say that the current camera is bad, it's not. It is very useful as is and based on conversations with Lytro employees there is a firm belief that a lot can be done to extend the functionality of the current model. I've spent most of my time talking about hardware in this entry and while the hardware is set in stone the software is not. When you take a picture with a Lytro camera you are capturing information about the scene beyond what is currently being processed and displayed. Support for three dimensional images is one enhancement that has been mentioned several times and there will likely be others.
I'll close here with one last observation. The Lytro camera is very durable. I accidentally dropped mine last night onto a hard wood floor from several feet up. Other than a very small amount of distortion in the case on one corner of the lens end it is fine.
Image via Wikipedia
I was just browsing through Lytro CEO Ren Ng's dissertation when I came across the following passage which invalidates my concern about CCD densities.
ReplyDelete"Digital image sensor resolution is growing exponentially, and today it is not uncommon to see commodity cameras with ten megapixels (mp) of image resolution [Askey 2006].
Growth has outstripped our needs, however. here is a growing consensus that raw sensor
resolution is starting to exceed the resolving power of lenses and the output resolution of displays and printers. For example, for the most common photographic application of printing 4”×6” prints, more than 2 mp provides little perceptible improvement [Keelan 2002].
What the rapid growth hides is an even larger surplus in resolution that could be produced, but is currently not. Simple calculations show that photosensor resolutions in excess
of 100 mp are well within today’s level of silicon technology. For example, if one were to
use the designs for the smallest pixels present in low-end cameras (1.9 micron pitch) on the
large sensor die sizes in high-end commodity cameras (24 mm×36 mm) [Askey 2006], one
would be able to print a sensor with resolution approaching 250 mp. here are at least two
reasons that such high resolution sensors are not currently implemented. First, it is an implicit acknowledgment that we do not need that much resolution in output images. Second,
decreasing pixel size reduces the number of photons collected by each pixel, resulting in
lower dynamic range and signal-to-noise ratio (snr). his trade-of is unacceptable at the
high-end of the market, but it is used at the low-end to reduce sensor size and miniaturize the overall camera. he main point in highlighting these trends is that a compelling application for sensors with a very large number of small pixels will not be limited by what can actually be printed in silicon. However, this is not to say that implementing such high-resolution chips would be easy. We will still have to overcome signiicant challenges in reading so many pixels of the chip efficiently and storing them."
This is from Chapter 1, page 5.
What this implies is that smart phones could indeed come with this technology in the future. I need to read some more to get a better understanding of the optics requirements. I'm starting to suspect that the real reason behind the 11 megaray resolution of Lytro's first camera is limitations in computation. Processing larger images may not be feasible at this point. If it were then a higher resolution CCD would apparently be minimally problematic.
i think you're making a big assumption about their production methods when you assume that they could use multiple CCD's
ReplyDeletethanks for the diagram though, now just to figure out how this "microlense" works, I recon the real magic is in the software, if I can find out if they have to calibrate each ccd for this "lense" type thingy, that sort of might reveal the magic, I can't imagine any sort of production process where you'd need to line up things so precise it would be madness
Thank you for the response Rich. It turns out they aren't using CCD's, but rather a similar technology known as a "CMOS censor", which is also know as an "Active Pixel Censor". I don't know a lot about it but apparently its a newer technology that is evolving rapidly.
ReplyDeleteI think its fair to say that the bulk of the magic is in the software but the micro lens array seems kind of magic to me as well. I'm not sure how big a deal alignment between the micro lens array and the censor is though.