Text recognition and visual look up (Live Text, Visual Look Up, ML Kit)
Reads text in images and scans documents on device.
In one line
Can pick out text in pictures and scan documents on your phone
Reads text in images and scans documents on device.
Apps can recognise text, scan pages and identify objects in photos. Visual Look Up and Live Text work inside the system apps with no code.
What it can and can't do
What the user does: You tap Scan document, point the camera at text, or open a photo
It can
- Pick out the words in a photo or live camera view
- Scan a document without the camera permission on Android, using Google's scanner
- Look up landmarks and plants on iPhone, with Apple's servers if needed
It cannot
- Use the live camera without you allowing the camera
- Look at photos you did not open or share
- Keep working fully offline on iPhone for landmark look up
How it differs between platforms: Both need a camera prompt for live scanning. Android's document scanner uses Google's screen, so it needs no camera permission.
On Android and iPhone
| Android | iPhone | |
|---|---|---|
| Status | Supported | Supported |
| First version | Android 5.0 (API 21) | iOS 13.0 |
| Permission or entitlement | CAMERA for live capture (dangerous) | None for images; camera permission for live |
| Protection level | dangerous | prompt |
| How access is granted | prompt | prompt |
| What the user sees | Camera permission for live; none for static images | Camera prompt for live; none for photos |
| How long it lasts | Until revoked | Until revoked |
| How to take it back | Settings > Privacy > Camera | Settings > Privacy > Camera |
| Store policy | None listed | None listed |
| API | ML Kit Text Recognition v2, ML Kit Document Scanner, Google Lens | VisionKit ImageAnalyzer, DataScannerViewController (iOS 16), VNRecognizeTextRequest |
| Notes | ML Kit Document Scanner needs no camera permission as it uses Play services UI. | Visual Look Up uses Apple models and may call servers for landmarks. |
Data you get
What comes back: Recognised text
Shape: object
How often: per image
| Field | Type | Unit | Range | Precision |
|---|---|---|---|---|
text | string | chars | per language | good |
Technical limits
- Quality varies with language and lighting
The story
What it is
Reads text in pictures and scans documents on the phone.
For you
You point the camera at a page and can copy the words, or tap a plant in a photo to learn its name. It works on the phone, though some lookups ask a server.
For makers
Makers add scanning, receipts and ID capture without building their own system. It also turns photos into searchable data, which brings privacy questions.
Where you see it
| Type | Example | Apps |
|---|---|---|
| banking | A bank app reads a check or card number from a photo. | Chase |
| productivity | A scanner app turns a paper receipt into a text file. | Adobe Scan, Notes |
| travel | A traveler copies a hotel address from a sign. | None named |
| shopping | A shopping app reads a barcode or label. | None named |
| education | A student scans a page into notes. | None named |
| accessibility | A person with low vision has a sign read aloud. | Seeing AI |
| games | An AR game reads text from a real poster to unlock a clue. | None named |
Risks
- Captures documents and IDs
- Some lookups call servers
Android and iPhone in short
- Android: Camera permission for live scans. The ML Kit Document Scanner needs no camera permission.
- iPhone: Camera prompt for live scans. Visual Look Up may call Apple servers for landmarks.
What the platform says
- Apple: Live Text and Visual Look Up. Apple lets you copy text from a photo and learn about objects, plants and landmarks in pictures.
- Google: ML Kit text recognition. Google offers on device text recognition in ML Kit for reading text from images.
Good uses and abuse
Legitimate uses
- Scanning
- Accessibility
Known abuse
- Document capture
Scores
| Score | Value | Meaning |
|---|---|---|
| Sensitivity | 3 of 5 | How revealing the data is |
| Utility | 4 of 5 | How useful it is to honest apps |
| Friction | 1 of 5 | How much the user has to do |
| Fingerprint bits | 1 | Rough identifying entropy, when known |
History
- 2019-09, iPhone iOS 13: Vision text recognition
- 2022-09, iPhone iOS 16: Live Text in apps and DataScanner
Related capabilities
Related
- On-device ML runtime (Core ML, LiteRT). Can run trained AI models on your phone's own chip
Same idea on the web
- Shape Detection: FaceDetector and TextDetector. Can find faces and printed text in a picture, but only with a special setting on
- Shape Detection: BarcodeDetector. Can find barcodes and QR codes in a picture or camera view
Sources
How well checked: Based on official documentation.