Paired cameras for stereoscopic enhancement

I'm surprised it's not been prominent if someone has done it (even if it entails using two RPIs, or multiplexing in hardware), because it seems such a good use for these cams. I planned to try two lower-res cams for machine vision (so no need for HD in this case) on the BBB, but I had some issues with the amount of memory currently allocated for the PRUs.

But seeing it done in HD would be superb.

Indeed, the new opportunities for machine vision are quite huge, possibly unlimited --- price affects everything.

Even good ol' human vision may get a boost when low pricing allows it.  For example, I was initially pondering applications in microscopy because of our earlier discussion about it, and that's why I started thinking about using a pair of cameras so that I wouldn't to lose the stereoscopic benefits of my current binocular microscope.  CastAR + dual cameras would preserve the current benefits and probably improve on them.

But by decoupling the camera site from the projection site, this automatically gives us remote 3D viewing, and that means remote surgery will benefit for starters.  Seeing exactly what you're doing with remote manipulators can be pretty important in that application, as in many others.

And if you can do surgery remotely, then soldering or reworking SMD boards remotely must be a walk in the park.

At the other end of the size scale, how about hooking together views from cameras on opposite sides of the Earth?  Blink imaging is already hugely powerful for detecting things that change in the night sky, but long-baseline parallax with the help of CastAR would add even more capability for near-Earth object detection.  And at the low price of the Pi camera, everyone with a telescope could be providing image feeds for use by CastAR-equiped viewers.  Some may even cool their cameras for low noise with Peltiers or dry ice or even liquid nitrogen if the sensor survives --- at this price, one can afford to experiment.  I expect a lot of such applications to appear.

And to bring this full circle back to your point, now imagine this parallax-aided visual data being used by machines.  This could be big, and not only for citizen science.  Pi-niche pricing changes everything.

Did you see this ?

http://www.raspberrypi.org/archives/3376

as noted, likely the biggest problem will be a lack of synchronisation capability. 

I think you're spot on that the low price changes things and opens up use cases you couldn't otherwise consider.  However, if you spend even five mins in the RPF forums camera subsection it's quickly clear that the lack of control over what the GPU really does is a problem as you can't guarantee the two cameras will do the same thing when looking at the same image.   There are lots of people with interesting ideas that the RPF had never thought of, which is exactly as you'd expect, but the lack of a certain level of control is making some things difficult.

Some posts in another thread from 'gsh' suggests that there may be the possibility of getting direct access to the CSI from the Arm if Broadcom will let them document it. Presumably at that point you'll have much more control, even if it does become a case of simply dumping the data for processing later.

Thanks for the link.  Cool, so there is a  bit of activity ongoing with dual cameras, Gordon's MicroDB and the work done by Maximum PC magazine.  There's sure to be much more soon I think.

Regarding lack of control over the CSI and the GPU, what software control do users have currently over the camera module?

PS. I couldn't perceive any extra depth  on that stereo video using red/green glasses, nor any feeling of nausea.  I suspect it's just due to the terribly shaky capture though, which I find extremely annoying as I can't focus on anything.  I look forward to watching some smoothly filmed 3D content.

Morgaine Dinova wrote:

 

Regarding lack of control over the CSI and the GPU, what software control do users have currently over the camera module?

Not as much as they'd like is probably a fair assesment.

The parameters for the raspistill prog are as follows

Image parameter commands

 

-sh, --sharpness        : Set image sharpness (-100 to 100)

-co, --contrast : Set image contrast (-100 to 100)

-br, --brightness       : Set image brightness (0 to 100)

-sa, --saturation       : Set image saturation (-100 to 100)

-ISO, --ISO     : Set capture ISO

-vs, --vstab    : Turn on video stablisation

-ev, --ev       : Set EV compensation

-ex, --exposure : Set exposure mode (see Notes)

-awb, --awb     : Set AWB mode (see Notes)

-ifx, --imxfx   : Set image effect (see Notes)

-cfx, --colfx   : Set colour effect (U:V)

-mm, --metering : Set metering mode (see Notes)

-rot, --rotation        : Set image rotation (0-359)

-hf, --hflip    : Set horizontal flip

-vf, --vflip    : Set vertical flip

 

 

Notes

 

Exposure mode options :

off,auto,night,nightpreview,backlight,spotlight,sports,snow,beach,verylong,fixedfps,antishake,fireworks

 

AWB mode options :

off,auto,sun,cloud,shade,tungsten,fluorescent,incandescent,flash,horizon

 

Image Effect mode options :

none,negative,solarise,sketch,denoise,emboss,oilpaint,hatch,gpen,pastel,watercolour,film,blur,saturation,colourswap,washedout,posterise,colourpoint,colourbalance,cartoon

 

Metering Mode options :

average,spot,backlit,matrix

plus various stuff like what size, jpg/png, saved image quality etc.

However some of these don't work (iso settings were only recently enabled) and some don't seem to do what's expected. For example there are reports of the GPU selecting low ISO and bright daylight like exposure times for night time images.

Also reports that in timelapse mode the camera locks the ISO setting early and never changes it, or (and the reason I started watching the camera forum) that timelapse images of a bright outdoor scene gradually fade off to black over some minutes, but maybe only if you turn the preview mode off.

Obviously still early days for the camera and things will get fixed, or new features added. However the stumbling block seems to be that jamesh is the only one with access to do any of the GPU side driver work and he doesn't seem to have time/motivation/something to work on it much. That's fair enough as he's just a volunteer doing it in his free time, but I think it's also why some people are interested in getting direct access without the GPU.

Great, thanks for the info!

I've now also found the source repo for the camera utilities on Github, so I'll poke around with those when I get some time.  Time is in very short supply here just at the moment, and I haven't even powered on the model A that I want to run the camera, which is still in its anti-static bag.

While BBB has leapfrogged almost all of the bullet points on Pi's features list, it's nice that Pi still has media centres (in which I'm not interested) and camera capability (in which I am, for microscopy) to enjoy as unrivalled applications.  I hope to put it to good use within a few months.

Addendum:  So, the Pi camera interface is a bit like that for the GPU, just a thin open source but uninformative shim in the kernel which passes messages to the VideoCore which then pokes at the CSI interface?

This is mostly off-topic, except for the fact that it was microscopy that made me interested in the Pi camera.  But perhaps Jeri Ellsworth is always on-topic at E14, so here's a fairly recent video about her refurbushed Scanning Electron Microscope.

I was quite involved in the nanotechnology community in the early years of the topic (before Bush and his vested interest advisers redefined the terms and handed NNI funding on a plate to Big Chem to make sunscreen), and was getting ready to build an Atomic Force Microscope, so this is pretty engrossing for me.

Further on the subject of low-cost video-aided microscopy, Chemical & Engineering News published an interesting article yesterday on "Smart Phones Snap Pictures Of Individual Virus Particles", which describes using a  laser diode and angled lenses in conjunction with a phone camera to provide cheap and portable fluorescence microscopy of particles and viruses down to 100nm in size.

C&EN's article is based on the paper published in ACS Nano 10.1021/nn4037706, 9 Sep 2013, by Qingshan Wei et al, UCLA.  The full paper is available at pdf/10.1021/nn4037706.

The above offers a very interesting area of experimentation with the Pi camera, needing only very inexpensive additional components that are available everywhere --- a laser diode and some lenses.  (Some experimentation with lens coatings could replace the UCLA team's long-pass thin-film interference filter.)  What's more, entirely electronic observation without an eyepiece provides useful first-order protection against eye damage, which is an ever-present danger even with very low power laser devices.  Needless to say, care is still needed, and preferably optical shielding as well.

This kind of project goes far beyond IT engineering, and could encourage interest in materials and life sciences and many others as well.

Addendum:  An offshoot from the UCLA research team, Holomic LLC, provides some very informative videos and product/technology descriptions on its website.

fyi - CastAR is KickStarter closes in 29 hours. the stretch goal is to add gyroscopes:

http://www.kickstarter.com/projects/technicalillusions/castar-the-most-versatile-ar-and-vr-system/posts

Woz is a backer now, too!

Whoops, thanks Drew!  The kickstarter is indeed very good news, and this $1m+ should accelerate work a lot without incurring wholesale slavery to a VC.

With the site in such a mess, I didn't see this update until now, but I'm very glad that castAR's attempt succeeded.  The rewards list for backers has a curious void between $85 and $189, I wonder why that was --- maybe to tempt a few more to join the ranks of those receiving an actual pair of glasses at $189, but I suspect that it had the opposite effect and just cut down the pledge total.

I guess the $189 means that the retail price of a pair of glasses + mat will be $200 + vendor margin.  I'm a bit saddened by that pricing.  Jeri & Co. are going to get only the hardcore gamers on-board in that market slot, given that castAR is an occasional-use peripheral.  Although there are certainly quite a few enthusiasts with large bank balances around, it's not really an attempt to enter the volume market.

With all the troubles we've been having here, I didn't enjoy clicking this castAR link somewhere on the site.  Our periodic lament about the Software Crisis probably needs an update --- I think the downward spiral is accelerating.

Morgaine.