Skip to Content
Platform.Ontology

Assistant screen context

Lets the assistant see what is on screen to answer questions.

Works without askingOn-device AIAssistantAndroid and iPhoneWhile you use it

In one line

Can see your screen when you call the assistant, if it is your chosen one

Lets the assistant see what is on screen to answer questions.

The system can capture the screen, or a camera view, and hand it to an assistant. Apps can opt out of capture on Android and iOS.

What it can and can't do

What the user does: You set an assistant as default, or use Circle to Search or Visual Intelligence

It can

  • Look at the screen content to answer a question about it
  • Share a screenshot with ChatGPT on iPhone after you agree
  • Use camera based search on iPhone 16 and newer

It cannot

  • Look at your screen when you have not called the assistant
  • Be used by any app that is not the default assistant on Android
  • Share a screenshot with ChatGPT on iPhone without your consent

How it differs between platforms: Android lets your chosen default assistant see the screen, and Circle to Search is Google's. iPhone uses Visual Intelligence from iOS 18.2.

On Android and iPhone

AndroidiPhone
StatusPartialPartial
First versionAndroid 6.0 (API 23)iOS 18.2
Permission or entitlementAssistant role (ROLE_ASSISTANT); FLAG_SECURE blocks captureNone; NSUserActivity and App Intents expose content
Protection levelrolenone
How access is grantedrolenone
What the user seesDefault assistant selection; Circle to Search gestureVisual Intelligence (camera control) iPhone 16; onscreen awareness in iOS 26
How long it lastsPer usePer use
How to take it backSettings > Apps > Default apps > Digital assistantSettings > Apple Intelligence & Siri
Store policyNone listedNone listed
APIVoiceInteractionService, AssistContent, onProvideAssistContent, FLAG_SECUREApp Intents entities, Visual Intelligence search integration (iOS 26)
NotesAndroid 15 adds assist content structure to AssistContent; Circle to Search is Google owned.Screenshots can be shared with ChatGPT after consent.

Data you get

What comes back: Screen capture
Shape: object
How often: per use

FieldTypeUnitRangePrecision
imagebinaryPNGn/an/a

Technical limits

  • Secure windows blocked

The story

What it is

Lets an assistant see what is on the screen so it can answer questions about it.

For you

You circle a pair of shoes in a video or point the camera at a plant, and the assistant tells you what it is. It sees whatever is on screen at that moment.

For makers

Makers of assistants get rich context without apps adding anything. It is great for search and help, but it can capture private things on screen.

Where you see it

TypeExampleApps
shoppingYou circle a jacket in a video and see where to buy it.Google Lens
educationYou point at a math problem and get a step by step hint.None named
travelYou point your camera at a sign and get a translation.None named
accessibilityThe assistant reads what is on screen aloud.None named
socialYou ask what a meme in a chat means.None named
gamesYou ask the assistant for a tip about a puzzle on screen.None named
bankingA bank app can block screen capture on sensitive pages.None named

Risks

  • Sensitive content may be captured
  • Screenshots may leave the device

Android and iPhone in short

  • Android: Circle to Search is a Google feature. Apps get assist data from Android 15.
  • iPhone: Visual Intelligence needs iPhone 16. iOS 26 adds on screen awareness.

What the platform says

Good uses and abuse

Legitimate uses

  • Search
  • Accessibility

Known abuse

  • Sensitive content captured

Scores

ScoreValueMeaning
Sensitivity5 of 5How revealing the data is
Utility3 of 5How useful it is to honest apps
Friction1 of 5How much the user has to do
Fingerprint bits1Rough identifying entropy, when known

History

  • 2024-01, Android Circle to Search: Launched on Pixel 8 and Galaxy S24
  • 2024-12, iPhone iOS 18.2: Visual Intelligence

Sources

How well checked: Inferred from related behavior.

On this page