Episode 459: Raspberry Pi 4 VR Conference Call Assistant

If I understand this set-up correctly i.e. you have a single physical conference room which has a robotic mannequin to represent each remote participant joining the call. Each participant in the call then looks at a particular mannequin as opposed to another participant's face.

Doesn't this then create a bit of a problem for conferencing applications where you typically would want to be able to see facial expressions or hand gestures of the person talking to you ? Or do you also get to see the remote participant's live webcam video alongside the mannequin webcam in the VR headset ? but then you would end up looking at another person wearing a VR headset which may be a bit weird.

One issue which may arise is the way that people look around a room. Sometimes you will turn your head and other times you will just move your eyes. Some form of eye-tracking may be required in addition to head tracking.

Hi Dave,

In a conference room, sitting around a table you won't really be able to just move your eyes to see everyone. You'd have to be sat quite far back!

Video of the other people would be awesome. If you all use the website and not the VR headset then you could see everyone and their facial expressions. I'd worry about home bandwidth there with sending and receiving video. But there's lots of future potential with this project!

Hi Lorraine.

Indeed, in real life head movement will vary with seating layout and distance. With VR it will also depend on the field of view of the cameras.

As you say, to see everyone in a room, you would have to move your head, but if you are just looking at the two or three people positioned directly opposite, then at times I suspect that you may only be moving your eyes between people as it takes less effort. If this is the case then in some instances you may see the mannequin move to look at you but other times not.

With the VR setup I suspect you could perhaps limit the horizontal field of view to cover just one person so that you always had to move your head to look at someone else, but that would end up giving you tunnel vision.

This will also likely depend on how you intend to place the mannequins. With the general move to replace physical space with virtual space these days then the conference room may end up having to be scaled down in size. Do you then place the mannequins optimised in a small circle in the corner of an office or do you stick with the more traditional layouts where you can never see the people to the extreme left or right of you ?

Also, how do you start to include presentation content. When your mannequin turns away to look at the presentation screen, then ideally your video feed needs to switch to the on-screen content instead.

(Mixed mode conferences are another issue that are cropping up where you may have 7 participants in the room and 70 remote, or alternatively 70 in the room and 7 remote.

Also there are now requests coming in for multi-site conferencing where you a) either stay at home and remote in b) drive to your nearest local centre to participate in a satellite group, c) travel to the main host venue.

Then beyond that there are the international conferences where the main conference venues are located across separate countries, each with country specific satellite venues and remote participants all wanting to communicate with each other as naturally as possible.)

I think the head tracking makes it a more natural user experience than the desktop virtual joystick control. However perhaps it needs more of a lighter weight smart glasses type solution to reduce the bulk of the VR headset. However the head tracking takes you away from the desktop device so everything has to be accessible through the VR.

(Looking back we had remote camera control on the old Tandberg video conference units back in the 90's. People got tired of it quickly though and ended up sticking with the one wide angle view for the duration.)

Network bandwidth has always been an issue with conferencing as it quickly adds up as the numbers of participants increase. As bandwidth increases, so does expectations of image quality. Minimising latency is another ongoing issue.

Have you looked at WebRTC which is gathering momentum ?

Perhaps another interesting area to investigate might be with the audio channels. If two people are looking at each other, then perhaps there could be an option to duck the main room audio to permit a secondary conversation to take place between the individuals. This could improve on in-person conferences as it removes that irritating problem when you have two people constantly trying to talk across you but it also allows for them not to lose out on the value of a spontaneous quick discussion.

I once did some video conferencing with a "bot" in the middle of the table that had sound sensors and turned to whoever was speaking. It didn't turn 360 as it had multiple cameras and could switch between them.

I've used the likes of the roundtable ones which have the 360 panoramic camera and audio direction detection which then crops the video to centre it on the area it thinks the sound is coming from.

However I was thinking more of the ability to have multiple simultaneous channels, so that two people looking at each other could talk on a separate channel from the person currently speaking.

This would get around the problem of two people quickly commenting on something that the active speaker has just said without interrupting the main conversation.

I'd love the play around with a 360 camera.. it kind of negates the need for a moving camera which I kind of like for the sound and movement! But a 360 camera accessed by a vr headset would be cool

"...I'd love the play around with a 360 camera..."

lorrainbow perhaps speak to Pete Hurst in ISS - he has a Polycom CX5500 that you could play with.

It may be a bit too specific for your experimental purposes - depends how much documentation there is available on the API. It was designed primarily to work with S4B / Lync web conferencing. If used standalone as a USB webcam it behaves as an autotrack camera.

"...I'd love the play around with a 360 camera..."

lorrainbow perhaps speak to Pete Hurst in ISS - he has a Polycom CX5500 that you could play with.

It may be a bit too specific for your experimental purposes - depends how much documentation there is available on the API. It was designed primarily to work with S4B / Lync web conferencing. If used standalone as a USB webcam it behaves as an autotrack camera.