User-Directed Screen Reading

Ka-Ping Yee

This is a short paper for CHI 2003 that suggests we can use screen reading to assist sighted users as well as blind ones. In particular, clicking the mouse button in an area of the screen could tell the computer to read the text in that part of the screen, and bring up a menu of operations to do on the text. Possible operations might be "schedule an appointment for this date", "look up this person's phone number", and so on.


This paper proposes a variation on existing screen-reading technology to help sighted users automate common operations. When the user wants to perform an operation related to some displayed text, the user can direct the window system to read text near the mouse pointer and offer possible actions. This can be considered an extension of the context menu applied to freeform text instead of GUI objects. The proof-of-concept implementation of this technique helps the user make appointments based on dates and times mentioned in e-mail.

The paper was rejected. Here are the reviewers' comments.