GNOME Shell Magnifier track focus and caret

Summary:

GNOME Shell Magnifier does not track focus or the caret. As a result, keyboard only users, who also have magnification on, must either regularly move the mouse to see the active area, or use an extra application to cause the area of interest to be displayed in the magnified view. Screen magnifiers for other platforms are able to track both focus and caret independently.

Ideally, GNOME Shell Magnifier should display the focused area and the caret independently of a focus tracker such as Orca. For those users who require both magnification and speech output and/or braille, Orca and GNOME Shell Magnifier should coexist, each doing the jobs they are intended to do without interacting with the other.

While the project is described as adding focus tracking to the magnifier, actually it is made up of two separate tasks. First, implement a focus and caret tracking device that is independent of the magnifier. While the magnifier is likely the first consumer of the focus tracker, there are other applications that would make use of the tracker -- an example is an onscreen keyboard (Caribou). As such, the focus tracker is a separate object independent of any client.

Secondly, modify the magnifier to use the focus tracker in a way that makes sense for magnifier users. This involves providing different tracking settings (e.g., centered vs. proportional), and a user interface so users can choose among the settings.

Details:

Some background: there is a proof of concept standalone python script that shows how to track the focus and caret.

• it uses AT-SPI to detect changes in keyboard focus and changes in caret position
• it uses D-Bus to drive the magnifier in terms of what to display.
• see the "examples" folder of the pyatspi2 project, specifically, "magFocusTracker.py".

1. Implement a FocusCaretTracker object that runs within GNOME Shell

• written in JavaScript and uses introspection for accessing AT-SPI.

• does not use D-Bus to communicate with other GNOME Shell objects; rather, uses the JavaScript emit/connect architecture from GJS 'signal.js' module.

  • In summary:
    • uses AT-SPI register/callback to listen for focus/caret events.
    • relays the events via emit/connect using signals.js.
  • Current Status:

    • using AT-SPI to track focus/caret events works from JavaScript.

    • acquiring information from the 'accessible' member of the AT-SPI event works for out-of-process events.

    • however, if the accessible is within GNOME Shell, the system locks up for about 5-10 seconds when acquiring info from the accessible. When again responsive, the information from the accessible is blank, and not usable. See: Reentrancy problems on gnome-shell.

    • using emit/connect works.

2. Modify GNOME Shell magnifier to use the FocusCaretTracker.

• magnifier connects() to the tracker to receive notification when the focus changes or the caret moves.
• magnifier handles the event by shifting the magnified contents to bring the focussed object into view.
• note that most of the code for shifting contents is in place (since GNOME 3.0).
• what's missing is integrating the connect() with the shift methods.
• Unknown: possible synchronous vs. asynchronous callback issues here since this code is executing within an AT-SPI event processing "thread".

3. (User preferences) Provide support for two kinds of settings.

1. Positioning:
   • Preferences include "centred", "proportional", and "push" positioning.

     • these are specific to the magnifier, and not part of the FocusCaretTracker.

     • Issue: separate preferences for focus tracking and for caret tracking?
     • gsettings-desktop-schemas:
       • focus-tracking { centered, proportional, push (default) }
       • caret-tracking { centered, proportional, push (default) }
     • magnifier: make it sensitive to changes in these settings.
     • gnome-control-center: dialog to allow user to modify these settings.
     • Issue: is this level of detail necessary? Is it not sufficient to support "push" mode only? And, is it necessary to have independent focus and caret tracking modes -- e.g., centered for focus, but push for caret?
2. Linking mouse position:
   • Some users want the position of the mouse to be independent of focus/caret tracking.
     • the experience is that if the mouse is located far from the focus point, which is typical, then if the user moves the mouse, the view abruptly changes from the focus point to what is near the mouse. And, vice versa on a subsequent focus/caret event.
   • Other users prefer that the mouse position is linked to the focus/caret position.
     • here, as the view is modified due to focus/caret changes, the mouse is re-positioned to the focus point.
     • nonetheless, thereafter, the mouse can be moved away from the focus point.
   • gsettings-desktop-schemas: boolean for mouse-follows-focus setting.
   • gnome-control-center: switch or checkbox for mouse-follow-focus.
   • magnifier: sensitive to mouse-follow-focus. Really, only needed when the two are linked.

     • Unknown: Likely cannot move mouse from within GNOME Shell nor JavaScript. Probably need to access this ability through X11, possibly XFixesCursor.