Assistant screen context
Lets the assistant see what is on screen to answer questions.
In one line
Can see your screen when you call the assistant, if it is your chosen one
Lets the assistant see what is on screen to answer questions.
The system can capture the screen, or a camera view, and hand it to an assistant. Apps can opt out of capture on Android and iOS.
What it can and can't do
What the user does: You set an assistant as default, or use Circle to Search or Visual Intelligence
It can
- Look at the screen content to answer a question about it
- Share a screenshot with ChatGPT on iPhone after you agree
- Use camera based search on iPhone 16 and newer
It cannot
- Look at your screen when you have not called the assistant
- Be used by any app that is not the default assistant on Android
- Share a screenshot with ChatGPT on iPhone without your consent
How it differs between platforms: Android lets your chosen default assistant see the screen, and Circle to Search is Google's. iPhone uses Visual Intelligence from iOS 18.2.
On Android and iPhone
| Android | iPhone | |
|---|---|---|
| Status | Partial | Partial |
| First version | Android 6.0 (API 23) | iOS 18.2 |
| Permission or entitlement | Assistant role (ROLE_ASSISTANT); FLAG_SECURE blocks capture | None; NSUserActivity and App Intents expose content |
| Protection level | role | none |
| How access is granted | role | none |
| What the user sees | Default assistant selection; Circle to Search gesture | Visual Intelligence (camera control) iPhone 16; onscreen awareness in iOS 26 |
| How long it lasts | Per use | Per use |
| How to take it back | Settings > Apps > Default apps > Digital assistant | Settings > Apple Intelligence & Siri |
| Store policy | None listed | None listed |
| API | VoiceInteractionService, AssistContent, onProvideAssistContent, FLAG_SECURE | App Intents entities, Visual Intelligence search integration (iOS 26) |
| Notes | Android 15 adds assist content structure to AssistContent; Circle to Search is Google owned. | Screenshots can be shared with ChatGPT after consent. |
Data you get
What comes back: Screen capture
Shape: object
How often: per use
| Field | Type | Unit | Range | Precision |
|---|---|---|---|---|
image | binary | PNG | n/a | n/a |
Technical limits
- Secure windows blocked
The story
What it is
Lets an assistant see what is on the screen so it can answer questions about it.
For you
You circle a pair of shoes in a video or point the camera at a plant, and the assistant tells you what it is. It sees whatever is on screen at that moment.
For makers
Makers of assistants get rich context without apps adding anything. It is great for search and help, but it can capture private things on screen.
Where you see it
| Type | Example | Apps |
|---|---|---|
| shopping | You circle a jacket in a video and see where to buy it. | Google Lens |
| education | You point at a math problem and get a step by step hint. | None named |
| travel | You point your camera at a sign and get a translation. | None named |
| accessibility | The assistant reads what is on screen aloud. | None named |
| social | You ask what a meme in a chat means. | None named |
| games | You ask the assistant for a tip about a puzzle on screen. | None named |
| banking | A bank app can block screen capture on sensitive pages. | None named |
Risks
- Sensitive content may be captured
- Screenshots may leave the device
Android and iPhone in short
- Android: Circle to Search is a Google feature. Apps get assist data from Android 15.
- iPhone: Visual Intelligence needs iPhone 16. iOS 26 adds on screen awareness.
What the platform says
- Apple: Visual Intelligence. Apple pitches Visual Intelligence as a way to learn about what you see with the camera or on screen.
- Google: Circle to Search. Google pitches Circle to Search as a way to search anything on screen with a gesture.
Good uses and abuse
Legitimate uses
- Search
- Accessibility
Known abuse
- Sensitive content captured
Scores
| Score | Value | Meaning |
|---|---|---|
| Sensitivity | 5 of 5 | How revealing the data is |
| Utility | 3 of 5 | How useful it is to honest apps |
| Friction | 1 of 5 | How much the user has to do |
| Fingerprint bits | 1 | Rough identifying entropy, when known |
History
- 2024-01, Android Circle to Search: Launched on Pixel 8 and Galaxy S24
- 2024-12, iPhone iOS 18.2: Visual Intelligence
Related capabilities
Related
- App actions for assistants (App Intents, AppFunctions). Can let Siri, Shortcuts or Gemini run actions inside the app, with your confirmation
Sources
How well checked: Inferred from related behavior.