Group Actions
Engagement
  • Author Author: dougw
  • Date Created: Date Created
  • Views 2753 views
  • Likes 9 likes
  • Comments 10 comments
Related
Recommended

PiCassoTizer - Absolute Mouse Position - HID Simulation - Blog 4

dougw

The last PiCassoTizer blog showed how the touch screen was going to work using image processing on Raspberry Pi's to monitor finger position.

This blog shows the other end, where a Raspberry Pi is used to edit screen graphics using external position information to control the process.

This system uses existing Raspberry Pi resources where possible, so the graphics program is any existing graphics program. All of these graphics programs can be manipulated with a mouse, so the external touch screen information can simply be converted to emulate a mouse to provide full functionality.

It sounds simple but there are a number of issues to deal with:

  • The touch digitizer system provides angular data that needs to be translated to Cartesian coordinates
  • The Cartesian coordinates are absolute, where normal mouse data is only relative position data (unfortunately it seems Linux doesn't support absolute mouse positioning)
  • The only Raspberry Pi that can emulate a mouse is a Pi Zero, which isn't part of this challenge

The simplest way I could think of to convert angular position data to USB mouse data was to convert the angular data to analog signals and use an arduino to convert these signals to look like a USB HID mouse. Although the Raspberry Pi's do a lot of heavy lifting - camera dizitization and image processing on one end and graphics editing on the other end, there is still room for a little arduino to complete the system.

I ordered some digital-to-analog converters for the touch digitizers, but they have not arrived, so I am working on the mouse emulation system and just simulating the camera system outputs with potentiometers.

The potentiometer wiring was a confused spaghetti mess, so I designed a frame to hold everything in a more understandable relationship.

Here is a video showing the back end apparatus in operation with a paint program:

The positioning system is pretty close to absolute, so it should be useable once the digitized data is available.

This demo used X and Y position data directly, so there is still firmware needed to translate the angular data.

While waiting for the D-to-A modules, I can start designing the camera frame geometry to ensure the whole screen is in the field of view and the angles provide the best accuracy possible.

I can't complete it though, until the cameras arrive and I figure out their field of view.

 

Relevant links:

PiCasso Design Challenge

 

PiCassoTizer - Introduction - blog 1

PiCassoTizer - Electronic Parts - Blog 2

PiCassoTizer - Finger Detection Image Processing - Blog 3

PiCassoTizer - Absolute Mouse Position - HID Simulation - Blog 4

Parents

  • Hello Doug,

     

    I think that there is an easier way to manage absolute mouse positions on the screen, without adding the weight of a microcontroller (Arduino, as you mention) and pass from digital to analog then to digital, etc. If I have understood correctly. Let me suggest a possible alternative.

    The issue you mention about the relative mouse position is not only Linux-related, but it is a hardware feature, specific of this kind of HID devices. If the hardware tracked position was not relative, it will be impossible to take a mouse in your hands, put it everywhere on the surface and move the cursor from there. The absolute mouse pointing mechanism was with the first optical mouse using a base tablet.

    As you know, there is plenty of applications showing the absolute X, Y coordinates associated with the mouse movement, so IMHO you are seeing the question from the wrong point.  Trying to imagine what should happen accordingly with your explanation in the first paragraph of this post, there are a couple of software methods that probably can solve or be helpfully applied to your case.

     

    When the mechanism starts we should be able to move coherently the cursor on the screen, so probably it is needed a self-calibration. I suppose you will start some sort of graphics application and for example, having the cursor centered on the screen. As the display is based on the X11 server, a few lines of code (C, but not difficult to convert to Python or something else) just position the cursor where you decide.

     

        Display* dpy = XOpenDisplay(0);
    int scr = XDefaultScreen(dpy);
    Window root_window = XRootWindow(dpy, scr);

    // Calculate the current screen size (the absolute coordinate system of the cursor working area
    int height = DisplayHeight(dpy, scr);
    int width  = DisplayWidth(dpy, scr);
    std::cout << "Screen size : " << width << "x" << height << std::endl;

    float m = (float)height/(float)width;

    int j;
    for(int i=0; i<width; i++){
        j = m*i;
        XWarpPointer(dpy, None, root_window, 0, 0, 0, 0, i, j);
        XFlush(dpy);
        usleep(50);
    }

     

    Maybe that digging on google there is more elegant methods than this.

    At this point, as we know where the cursor is and what is the pixel size of our motion area, these values can be considered the offset for every further mouse relative move.

    If for some reason you should check instead what is the current mouse position and get the offset as well, the display class of the Xlib Python library does the job.

     

    fromXlibimport display

def mousepos():
  data = display.Display().screen().root.query_pointer()._data
  return data["root_x"], data["root_y"]

if __name__ =="__main__":
print("screen position : {0}".format(mousepos()))

     

    Btw, I use this method, available from the Python package python-xlib

     

    Hope this can be useful. Enrico

Comment

  • Hello Doug,

     

    I think that there is an easier way to manage absolute mouse positions on the screen, without adding the weight of a microcontroller (Arduino, as you mention) and pass from digital to analog then to digital, etc. If I have understood correctly. Let me suggest a possible alternative.

    The issue you mention about the relative mouse position is not only Linux-related, but it is a hardware feature, specific of this kind of HID devices. If the hardware tracked position was not relative, it will be impossible to take a mouse in your hands, put it everywhere on the surface and move the cursor from there. The absolute mouse pointing mechanism was with the first optical mouse using a base tablet.

    As you know, there is plenty of applications showing the absolute X, Y coordinates associated with the mouse movement, so IMHO you are seeing the question from the wrong point.  Trying to imagine what should happen accordingly with your explanation in the first paragraph of this post, there are a couple of software methods that probably can solve or be helpfully applied to your case.

     

    When the mechanism starts we should be able to move coherently the cursor on the screen, so probably it is needed a self-calibration. I suppose you will start some sort of graphics application and for example, having the cursor centered on the screen. As the display is based on the X11 server, a few lines of code (C, but not difficult to convert to Python or something else) just position the cursor where you decide.

     

        Display* dpy = XOpenDisplay(0);
    int scr = XDefaultScreen(dpy);
    Window root_window = XRootWindow(dpy, scr);

    // Calculate the current screen size (the absolute coordinate system of the cursor working area
    int height = DisplayHeight(dpy, scr);
    int width  = DisplayWidth(dpy, scr);
    std::cout << "Screen size : " << width << "x" << height << std::endl;

    float m = (float)height/(float)width;

    int j;
    for(int i=0; i<width; i++){
        j = m*i;
        XWarpPointer(dpy, None, root_window, 0, 0, 0, 0, i, j);
        XFlush(dpy);
        usleep(50);
    }

     

    Maybe that digging on google there is more elegant methods than this.

    At this point, as we know where the cursor is and what is the pixel size of our motion area, these values can be considered the offset for every further mouse relative move.

    If for some reason you should check instead what is the current mouse position and get the offset as well, the display class of the Xlib Python library does the job.

     

    fromXlibimport display

def mousepos():
  data = display.Display().screen().root.query_pointer()._data
  return data["root_x"], data["root_y"]

if __name__ =="__main__":
print("screen position : {0}".format(mousepos()))

     

    Btw, I use this method, available from the Python package python-xlib

     

    Hope this can be useful. Enrico

Children
  • in reply to balearicdynamics

    If I understand this, that is what I am doing, and there are other things to handle as well.

    When the mouse is directed to go beyond the screen, it won't, so relative position is lost in this scenario.

    Mouse handlers use acceleration - the size of the relative move is proportional to the speed of mouse movement, not just how far it moves.

    These can be handled, but they do require handling.

  • in reply to dougw

    Hi Doug,

     

    My approach was just to point you the attention over some points you have mentioned to be solved via hardware. About your last comment:

    Mouse handlers use acceleration - the size of the relative move is proportional to the speed of mouse movement, not just how far it moves.

    It is not a must. When the mouse is out of range, the cursor just stop. Then as there is always a human between and it is a question of feedback, I think that the acceleration can be managed or not, at least to see if the stuff works. But also the acceleration is easy to detect as - I suppose - the mouse emulator is checked at a certain known frequency, or via an interrupt or something similar. It is sure that solving via software needs handles.  image

     

    Enrico