On Screen Keyboard

Contents

On Screen Keyboard

Text entry with a touch screen or pointer.

Goals

Primary text entry method with a touch screen device.
Support common text entry activities.
Match typical keyboard layout for the locale (support for Latin scripts is the primary goal/non-Latin input needs to be considered separately).
Adjust to the type of text being entered and the context of use. Examples: URLs or email addresses, numbers, search.
Allow emojis to be entered, when appropriate.
Universal access? (See discussion below.)

Non-goals

Not intended to support every possible key - modifier keys like ctrl/alt/super/F keys.

Relevant Art

Windows 10

Default layout:

Full layout:

Can be floating or fixed to the bottom screen edge. Can also be resized. And there's a "inking" hand writing mode. You can show a button in the task bar to manually raise the keyboard.

iPad

Regular:

iPad pro has additional keys - a row of numeric keys, tab, caps lock, etc:

Can be shrunk down to a floating mode (done with two-finger pinch):

Chrome OS

Has a floating mode:

Yet to find evidence of there being a way to show more keys than this.

iOS

https://developer.apple.com/library/ios/documentation/StringsTextFonts/Conceptual/TextAndWebiPhoneOS/KeyboardManagement/KeyboardManagement.html

Android

Key to the left of space is determined by type of text being entered:

Email entry: @
URL entry: /

Ubuntu

Ubuntu uses Onboard for its on-screen keyboard. Onboard is intended as an assistive technology, to be used in combination with a pointing device, rather than for use via a touchscreen.

Click helpers: keys are included for middle click, right click, double click, drag click, and hover click. Clicking one of these modifies the behavior of the pointer (most modify the action of the primary/left mouse button).

The visibility of the click helper buttons is controlled by a "toggle click helpers" button (these buttons are not shown by default). The "toggle click helpers" button can be activated by hovering over it - this allows using the pointer without pressing any buttons.

Onboard also allows creating text snippets which can be inserted.

Other

Maliit
Indic Onscreen Keyboard
Florence Virtual Keyboard
Eekboard (Presentation about eekboard and its library)
GOK (GNOME On-screen Keyboard)
Qt Virtual Keyboard

Experimental approaches:

CellWriter (natural handwriting input)
Swype - A touch-based on-screen keyboard system. Basically, you drag a trail across the on screen keyboard to form words. Based on the trail, it predicts which words you could have meant to type. You keep your finger pressed to the screen and do not have to lift it off, and it doesn't require as precise targeting as a traditional on-screen touch keyboard.
Retype
Zeta Type
Dasher (mouse-based)

Old relevant art

Windows 8

1, 2, 3, 4, 5, 6, 7, 8, 9, 10

Windows 8 also has a separate on-screen keyboard for accessibility - http://i.stack.imgur.com/20aW1.png.

Features:

Hover over keys to enter characters.
"Scan through keys" - allows sequentially moving through the keyboard using a joystick, mouse, or a keyboard key.
Word prediction/suggestions.

Discussion

The OSK should integrate with GNOME Shell in appearance and behavior.

What is the relationship to language Input Methods?

Issues to consider:

Non-Latin scripts (Chinese, Japanese, Korean, Indic, etc.)
Word prediction.
Might be interesting to show a the paste buffer and offer a Paste action
Smart web address / email address handling
Different layout for login.
Mode for numeric entry?
Emoji.

Unicode Keyboard Layout definitions - provides layout Android definitions for approximately 50 keyboard layouts. Layouts claim to be compatible with Android 4.4.

Accessibility

Potential accessibility applications of an OSK include:

Users who are unable to operate a physical keyboard, and are only able to interact using a pointing device.
Users who have difficulty operating the buttons on a pointing device. This could include an inability to hold down a button for a length of time, or an inability to double-click. Or it could mean being unable to press any button.

For 1, the standard OSK can be used in conjunction with a pointing device.

For 2, the standard OSK is insufficient. Additional required features:

Hover to click (seen in Windows 8, Onboard, and Florence).
Assistance for double click, middle click, secondary click, drag/move.
Ability to raise the OSK at any time, by pressing/hovering something on screen.

On-screen keyboards that focus on accessibility tend to look like physical keyboards, and allow access to modifier keys like Ctrl, Shift and Alt. This allows any kind of keyboard input, without the need for a physical keyboard. This is probably because some applications require modifier keys for some actions.

Tentative Design

OSK Activation

OSK shows when a text entry is focused by the user (switching windows/workspaces shouldn't activate the OSK).
OSK hides if the hide button is pressed, or if the user interacts with a widget that doesn't allow text entry.
This behavior can be overridden by applications - these can signal that the OSK should not be used, or what type of text is being entered. The calculator is an example of this.
If there is a physical keyboard present, the on screen keyboard shouldn't ever be shown (unless the Universal Access Screen Keyboard option is turned on).

When the OSK is displayed, the view should be shifted in order to keep the text cursor visible.

Layout and Appearance

OSK layout should follow the configured input sources (in the Region & Language settings panel). When an OSK layout isn't available for an input source, we fall back to the closest one that is (eg. show the OSK layout with the same locale as the input source).

Modes and Application Customization

Applications should be able to indicate what type of text is going to be entered. It should also be possible to specify whether emoji are wanted, and what the return key should do.

Text/numeric entry.
Emojis/no-emojis.
Text type: freeform text, URL entry, email address entry.
Return key: return (default), search, done, go, ...