Let’s talk (Tts)Spans in Android Accessibility

A great talk from the Google IO 2019 called “Demystifying Android Accessibility Development” mentions that when designing apps, we often miss to account for the users with accessibility needs. Users with accessibility needs won’t interact with the app directly, but instead they will use tools such as the Android Accessibility Suite (includes Talkback and Switch Access). The user will interact with the Accessibility service and then the service will interact with the app.

Accessibility services need information on what the screen has or shows to be able to provide the correct contextual information to the user or to be able to navigate through the app. An example of that information can be provided using Content Descriptions.

In this blog post we’ll talk about Spans in Android and how to enrich Spannables to provide a better UX to users with accessibility needs.

From the official docs:

Spans are powerful markup objects that you can use to style text at a character or paragraph level.

With Spans we can change the text color of a substring or have a link-clickable part within a string, or even different size substrings. Sky is the limit 🚀

In this post, we’ll specifically talk about TtsSpan.

TtsSpan

A TtsSpan can provide metadata for a Spannable. The metadata will be supplied to Text-To-Speech Engines such as Talkback.

This span comes with several builders and each builder helps building metadata for a different type. The types supported by the builders are:

To demonstrate their benefits, we’ll explore the following types:

TtsSpan.TYPE_DATE
TtsSpan.TYPE_MEASURE
TtsSpan.TYPE_TIME
TtsSpan.TYPE_ELECTRONIC

Brief introduction to demo

The demo application has a list of items. Each item is duplicated, one without TtsSpan and one with TtsSpan to highlight the differences.

Demo App Screenshot

When an item is clicked, we pass the Spannable (with or without TtsSpan) to the TextToSpeech service to output the metadata.

Verification

To verify that the metadata are supplied to the TextToSpeech engines correctly, we could do the following:

Supply the spannables to the TextToSpeech.speak method which will output the data
Turn on Talkback and navigate the demo using the service.

The demo videos use the first point + Live Caption to verify and present you the output.

Building the list

The list is built using RecyclerView with TtsItem classes. Each item has a title, a caption and a nullable type of TtsSpan (if null then no TtsSpan is built).

data class TtsItem(
    val title: String,
    val caption: String,
    private val ttsSpanType: String?
) {
  var id: Int = 0
  fun toSpannable(): SpannableString? { ... }
}

TtsItem demo

To produce the different TtsItem, we have a data factory called DummyDataFactory .

object DummyDataFactory {
    fun getListOfTtsItem(): List<TtsItem> = listOf(
        TtsItem("18/04/2020", "Date without TTSSpan", null),
        TtsItem("18/04/2020", "Date with TtsSpan.DateBuilder", TtsSpan.TYPE_DATE),

        TtsItem("5 meter", "Measure without TTSSpan", null),
        TtsItem("5 meter", "Measure with TTSSpan", TtsSpan.TYPE_MEASURE),

        TtsItem("14:00", "Time without TTSSpan", null),
        TtsItem("14:00", "Time with TTSSpan", TtsSpan.TYPE_TIME),

        TtsItem("admin:123456789", "Password without TTSSpan", null),
        TtsItem("admin:123456789", "Password with TTSSpan", TtsSpan.TYPE_ELECTRONIC)
    ).also { list ->
        list.forEachIndexed { index, ttsItem -> ttsItem.id = index }
    }
}

Explore toSpannable() from TtsItem

Disclaimer: Please note that some of the code shown below is for demonstration purposes only and mapping strings to TtsSpan most likely won’t work like that in real life projects.

Note: Have a look at the captions to see the difference with and without TtsSpan.

TtsSpan.TYPE_DATE

val calendar = Calendar.getInstance()
calendar.time = simpleDataFormat.parse(title)
   ?: throw IllegalStateException("Not expected null Date")
TtsSpan.DateBuilder()
   .setWeekday(calendar.get(Calendar.DAY_OF_WEEK))
   .setDay(calendar.get(Calendar.DAY_OF_MONTH))
   .setMonth(calendar.get(Calendar.MONTH))
   .setYear(calendar.get(Calendar.YEAR))

The above code block will take a String date, parse it into a Date object which is then supplied to a Calendar. Then the Calendar object is used to extract different information that would be useful to TtsSpan.DateBuilder().

Caption without TtsSpan: “18 slash 04 slash 2020“
Caption with TtsSpan: “Sunday the 18th of April 2020“

TtsSpan.TYPE_MEASURE

val number = digitsPattern.find(title)?.value // extracts digits
val unit = stringPattern.find(title)?.value // extracts string
TtsSpan.MeasureBuilder()
   .setNumber(number)
   .setUnit(unit)

The above code block will extract the digits from the string which will be treated as the number and then extract the text from the string which will be treated as the Measurement unit. All the extracted data are supplied to the TtsSpan.MeasureBuilder.

Caption without TtsSpan: “5 metre“
Caption with TtsSpan: “5 metres“

As you can see the metadata helps identify whether the measurement is singular or plural.

TtsSpan.TYPE_TIME

val hours = title.split(":")[0]
val minutes = title.split(":")[1]
TtsSpan.TimeBuilder()
   .setHours(hours.toInt())
   .setMinutes(minutes.toInt())

The above code block builds metadata needed for time. It simply extracts hours and minutes from string and supplies them to the TtsSpan.TimeBuilder().

Caption without TtsSpan: “14 colon zero zero“
Caption with TtsSpan: “14 hundred“

TtsSpan.TYPE_ELECTRONIC

This particular type can be used to build several “electronic” metadata. In our example we’ll build metadata for a username and password.

val username = title.split(":")[0]
val password = title.split(":")[1]
TtsSpan.ElectronicBuilder()
   .setPassword(password)
   .setUsername(username)

The above code block uses the TtsSpan.ElectronicBuilder to build the metadata. The first part of the string is treated as the username and the second part as the password.

Caption without TtsSpan: “admin 123 million 456 thousands 7 hundred and 89“
Caption with TtsSpan: “admin passoword 1 2 3 4 5 6 7 8 9“ The above example is my favourite as it demonstrates how powerful the Text-to-Speech engine can be with the correct metadata.

Conclusion

Providing rich UX is important, and we need to make sure that our apps are accessible for all users. We have seen some examples on how to add some metadata in apps so that Text to Speech services provide contextual information.

➡ All the above examples can be found at the sample project on Github.

Feel free to ping me on Twitter.

Till next time! 👋

Giorgos Neokleous