Build Siri experiences across apps, Part 1: Onscreen Awareness

In our previous series, we explored how App Entities make our app's content available to Siri and how App Intents and App Schemas let Siri perform actions with it.

But people don't always describe content by name. They might refer to whatever is on the screen as “this” or “that.” How do we help Siri understand what they mean?

In this two-part series, we'll explore how to build Siri experiences across apps. We'll start with onscreen awareness, then cover content transfer in Part 2.

How to enable on-screen awareness

Someone looking at a message might say, “Summarize this message.” For Siri to act on that, it needs to know which message the view is showing.

We do that by connecting our views to App Entities, which tells Siri what content each view represents.

There are two APIs for annotating on-screen content:

• NSUserActivity annotations: Use when there's one primary item on the screen.
• View annotations: Use when multiple meaningful items are visible at once.

NSUserActivity annotations

Use NSUserActivity annotations when there's one primary item on the screen, such as a message detail view.

In SwiftUI, we use the userActivity modifier and set the activity's appEntityIdentifier to identify the message being displayed:

MessageDetailView(message: message)
    .userActivity(
        "com.example.messages.viewMessage",
        element: message
    ) { message, activity in
        activity.title = "View message"
        activity.appEntityIdentifier = EntityIdentifier(
            for: MessageEntity.self,
            identifier: message.id
        )
    }

In this example, we connect MessageDetailView to a MessageEntity using message.id. The ID must match the one our entity query uses to find the message.

com.example.messages.viewMessage identifies the activity type. appEntityIdentifier identifies the message we're viewing.

View annotations

Use view annotations when multiple meaningful items are visible at once, such as a list of messages. We connect each item's view to its App Entity so Siri can distinguish the content represented by those views.

In SwiftUI, we apply the appEntityIdentifier modifier to each row inside the list:

List {
    ForEach(messages) { message in
        MessageRow(message: message)
            .appEntityIdentifier(
                EntityIdentifier(
                    for: MessageEntity.self,
                    identifier: message.id
                )
            )
    }
}

Each MessageRow carries the identifier of the message it displays, so Siri can tell which row someone is referring to.

You can easily support sarunw.com by checking out this sponsor.

Offline Transcription: Fast, privacy-focus way to transcribe audio, video, and podcast files. No data leaves your Mac.

Try it for Free

Summary

Siri can now resolve “this” to the content we're looking at. We get there by connecting our views to App Entities, and which API we use depends on the screen: NSUserActivity annotations for a single primary item, view annotations when several items matter at once.

On-screen awareness gets more useful alongside content transfer. Siri can identify the photo we're looking at, then pass it to a messaging app to send.

We'll cover content transfer in Part 2: Content Transfer.