[FoRK] coffee drone

Stephen Williams sdw at lig.net
Sun Jun 24 12:12:05 PDT 2012

On Sun Jun 24 10:25:07 2012, Damien Morton wrote:
> On Sun, Jun 24, 2012 at 12:14 PM, Aaron Burt <aaron at bavariati.org> wrote:
>> On Sat, Jun 23, 2012 at 08:14:32AM -0400, Damien Morton wrote:
>>> Or, if you felt your drone had enough energy budget to carry something
>> like
>>> a beaglebone around with it, you might be able to do the mapping
>> on-board.

You're going to use a production cell phone with the right kind of camera. The computational power and bandwidth capabilities 
along with power efficiency are expanding to very impressive levels over the next 2 years. Nothing else will come close: I 
expect greater than 100-150 giga-ops peak with 8-20 GBs memory bandwidth and very fast peripheral paths.

>> I think one of those drones demo'd was carrying an x86 chip.

Intel low power mobile processors may catch up soon, however right now ARM and associated GPUs and DSPs are king.  Although even 
the most modern ARM chips don't have Pentium level efficiency tricks, they are impressive at their raw simple power and power 
efficiency. And they are in shipping and soon to be shipping devices that have a great compliment of hardware and operating system.

> Actually a BeagleBone or Gumstix might be able to do it on-board. These
> boards are in the <40g range, and require 2W of power - a battery weighing
> <10g should be enough to power them for a single flight.

For any serious computational work and/or high bandwidth, you'll want to scale up to something that can carry a cell phone. And 
that gives you fast wireless network / Internet access also, along with very sophisticated and dynamic power management.

Already, they are being used for tiny satellites:

>>> What we really need is an inexpensive low-power per-pixel depth sensor.
>> You cited working implementations of two of the most effective techniques
>> used
>> in nature. (I count optical flow as structure from motion.)
> I was actually giving Stephen a chance to perk up about the company he is
> working at - they are working on plenoptic imaging chips for mobile phones,
> and these chips are capable of per-pixel depth measuring at low to moderate
> z-resolution. Far less resolution than Lidar, but a lot cheaper, and
> sufficient for obstacle avoidance.

That's Pelican Imaging ( http://pelicanimaging.com/ ); we're still semi-stealth. I've been the director of SW eng at PI for a 
little while. You can see on the website that it is a thin array camera meant for mobile device use and that it provides depth 
information for each pixel. It is a "computational camera" in the sense that the hardware is relatively simple and inexpensive 
while there is significant computational load to achieve the best results.  It's technically not a plenoptic camera like Lytro 
and Raytrix, although it can provide a similar focus-later capability.  I would call it a special class of array camera.  In 
most respects, it is much better than a plenoptic camera.

We're still looking for people with a good subset of these skills:

  * Very good, creative, accomplished architect-level developer in C++, plus preferably Java
  * Very active learner, able to ramp quickly on new things, passion for solving problems and finding new ones.
  * Aggressive optimization experience, especially in imaging, graphics, multimedia or similar
  * Assembly experience, preferably ARM NEON in hard optimization situation
  * GPU experience: shaders, shaders used for general purpose programming, OpenCL (need), CUDA (similar enough to OpenCL), plus
    preferably ARM mobile GPUs.
  * DSP experience, ideally modern mobile DSPs doing performance and power sensitive development, but any DSP would be good.
  * Imaging, 3D, machine vision, AI, and related experience would be good.

Here are interesting clips from press releases and articles linked to from the web site:

> Pelican Imaging has developed a computational camera array architecture and fundamental intellectual property with 12 pending 
> patent applications in array optics, sensors and image reconstruction algorithms. Pelican’s camera improves upon image and 
> video quality while allowing for thinner smartphones. New applications are also enabled by introducing features such as 3-D 
> depth, gesture control, and the ability for users to interact with the image before and after capturing the shot.
> “What Pelican has developed represents a paradigm shift in imaging and video that has the potential to overcome many of the 
> inherent limitations of mobile cameras,” said Professor Shree Nayar of Columbia University. “Pelican’s expertise in optics, 
> architecture and software algorithms uniquely positions the company to bring computational imaging applications to the mass 
> market.
> ”Professor Marc Levoy, of Stanford University commented, “Pelican’s technology has the potential to upset the traditional 
> tradeoff between the sensitivity and resolution of a camera and its thickness. It also brings new capabilities to cameras, 
> including post-capture focusing, foveal imaging and programmable frame rates.  We have been investigating these aspects of 
> computational photography in our laboratory at Stanford for a number of years, through the Stanford Multi-Camera Array, which 
> is big, slow and expensive. Pelican’s solution is small, fast and inexpensive – which makes it a very exciting technology.

> California startup Pelican Imaging wants to put slimmer, higher-quality cameras into cellphones with its new camera-array 
> technology. Instead of using a single lens and sensor, the camera uses an array of smaller modules and uses computation to 
> combine them into a single image. The resulting hardware is thin, but that is probably the least interesting thing about it.
> The Pelican array uses “light-field photography”, and aside from just stitching small pictures into a big one, it does some 
> things a regular camera can’t. For instance, you can diddle with the picture after it is taken, blurring a background, say. 
> And in principle at least, you could use a kind of 3D “healing brush” to paint out distractions behind and in front of your 
> subject.
> Pelican’s camera also promises to give high resolution images with low-noise results in low-light situations, and could enable 
> gesture-controls on tablets.

> Pelican Imaging wasfounded in 2008 <http://gigaom.com/2011/02/09/pelican-imaging/>and has received $17 million in venture 
> funding to date from investors like Globespan Capital Partners, Granite Ventures, InterWest Partners, and IQT,according 
> <http://www.crunchbase.com/company/pelican-imaging>to CrunchBase.
> For Levoy, Pelican's prototype is exciting because it miniaturizes a type of technology already proven on the larger scale: 
> "We have been investigating these aspects of computational photography in our laboratory at Stanford for a number of years, 
> through the Stanford Multi-Camera Array, which is big, slow and expensive," hesaid 
> <http://www.pelicanimaging.com/pelican-unveils.htm>in a release. "Pelican's solution is small, fast and inexpensive - which 
> makes it a very exciting technology." Nayar goes so far as to call Pelican's technology a "paradigm shift in imaging and 
> video" likely to "bring computational imaging applications to the mass market."
> But it's not just people on Pelican's payroll who are shilling for the novel protoype. Om Malik at GigaOm thinks that the new 
> array camera helps explode what hecalls <http://gigaom.com/2011/02/09/pelican-imaging/>"the megapixel myth"--the notion that 
> "the more megapixels we have on our mobile phone camera, the better our photos will be." A higher number of smaller lenses may 
> be the real path forward--making Pelican's new camera more than the sum of its parts.

> Pelican Imaging Secures Strategic Investment form IQT
> July 28, 2010 - Pelican Imaging Corporation, a pioneer in computational cameras for the mobile market, today announced a 
> strategic investment and technology development agreement with In-Q-Tel (IQT). IQT is the independent, strategic investment 
> firm that identifies innovative technology solutions to support the mission of the U.S. Intelligence Community.
> Squeezed by industrial design constraints (thinner and smaller cameras) and the “megapixel myth” (the mistaken belief that 
> more pixels mean better quality), Smartphone manufacturers and consumers face two critical problems: poor image quality and a 
> stagnating feature set. For traditional camera architectures, the push towards more megapixels in small form factors shrinks 
> the pixel size and compromises its light-gathering capabilities. This has a direct negative impact on overall imaging performance.
> Pelican solves these problems by introducing a novel camera architecture coupled with patent-pending software algorithms. The 
> result is a powerful and disruptive technology that provides simultaneous advantages in critical camera performance 
> parameters: low-light sensitivity, dynamic range, white balance, anti-shake, near focus, and low-power photography – while 
> continuing to enable high resolution, ultra-thin cameras.
> “Pelican Imaging’s high-resolution, wafer-scale cameras will tap into a highly-compelling consumer market and enable better 
> camera phones,” said William Strecker, Executive Vice President of Architecture & Engineering and CTO at IQT. “In a multimedia 
> rich age, this technology has the potential to significantly impact the future of digital imaging, video, and photography.”

>> About the only way to improve on that is with active techniques, which
>> pollute the EM environment and give away position. But I guess you could
>> build something like a CCD array with a PLL and phase detector at each
>> pixel, coupled with a modulated LED emitter. That, or a threshold
>> detector, a latch and a cap at each pixel coupled with a picosecond laser
>> emitter. Sounds silly, though.
> Its called Flash Lidar. Not silly at all, per pixel time-of-flight sensors.
> Expensive though.
> https://www.youtube.com/watch?v=RN7WkRaOCuU

$250K per camera for a huge, power hungry beast. Other than the fact that it can see in the dark, not so useful.

There is a better way, at least for some range of use and resilient algorithms.


More information about the FoRK mailing list