You don’t need us to tell you just how smart the smartphone has become: From recognizing our voices to plotting complex routes in seconds, this device is a real box of tricks.
With the right app, they can also help identify what’s in the world around us, whether it’s the breed of the dog that’s just come up to make friends with you, or information about a landmark you’re visiting.
These apps work like visual search tools, using the information coming in through your phone’s camera and adding some artificial intelligence-powered object recognition magic to figure out what’s currently in view.
Visual Lookup and Live Text
If you own an Apple device with the A12 Bionic or a newer chip inside (iPhones launched in 2018 or later, or iPads launched in 2019 or later) and it’s running the latest software, you can take advantage of two features called Visual Lookup and Live Text.
Visual Lookup works on photos you’ve already taken and stored on your phone or tablet, and can identify items including art, landmarks, flowers, books, and pet breeds. Open the image in your Photos app, and you’ll see a special sparkle effect around the info button (an “i” within a circle) at the bottom if your phone can identify something in the picture.
[Related: Organize your catastrophic photo library]
Tap the button, then tap the icon in the center of the image on the next screen. This icon will indicate what has been identified (it’ll be a location pin for a landmark, for example, or a petal for a flower). More information will appear on screen, though its contents will vary depending on what’s in the picture. You might be shown details of the dog breed you’re looking at, or the book cover that’s in the photo, to name a couple potential outcomes.
Live Text works in the Photos app and the Camera app—if you’re using the latter, just point the camera at text on a page or on a sign without actually taking a photo. If your phone identifies text, a Live Text icon (three horizontal lines in a frame) will appear in the lower right-hand corner, and you can tap this to start highlighting text in the image.
Long-press on text to select it, then drag the blue handles to expand the selection. Once you’ve selected some, a pop-up list of options will appear, including the ability to search the web for the selected text (Look Up) or translate it from a foreign language (Translate).
Google Lens
Google Lens is available as a standalone Android app, and as part of the Google app on iOS and iPadOS. Among its skills are the ability to recognize plants, animals, and landmarks, and it works a bit like an image search on the main Google website.
On Android, open the Google Lens app and you’ll be able to look through images already on your phone or tap Search with your camera. Choose the latter option, and point your phone’s camera at anything to see if Google Lens can identify it—we’ve had success with dog breeds, book covers, famous buildings, and US presidents.
The search feature works in a similar way on photos already stored on your phone. Open one of them inside Google Lens, and after a few moments, the bottom of your screen will display any results that Google Lens can find. Tapping through on any result will lead to more information from the web.
In the Google app for iOS or iPadOS, tap the camera icon inside the search box to launch Google Lens, which works just as it does on Android. You’ll also see a Google Lens button (a small camera icon) in Google Photos for Android and iOS when you open an individual image—use this to search using a picture you’ve already taken.
Google Lens can also copy and translate text, shop for items online, and even solve math equations—just use it as described above, and the options that appear once you run a search will change depending on what you’re pointing the camera at (or what’s in the photo you’ve selected).
Other apps
Various other apps are able to recognize objects you’re pointing your phone camera at, or that have already been photographed. Snapchat for Android and iOS can do this, for example, via a feature called Scan: Open the camera screen, then tap and hold on what you’re looking at on the screen (on iOS, you’ll also need to tap Scan).
If Snapchat Scan can find a match for what you’re looking at, you’ll see it pop up at the bottom of the screen, together with any relevant Snapchat content (such as themed lenses and backgrounds). Snapchat says the feature can identify cars, dog breeds, plants and bottles of wine, and solve math problems too—expect more features and capabilities to be added over time.
Depending on what you’re looking at, more specific apps may be available. The Merlin Bird ID app for Android and iOS works through your phone’s microphone rather than its camera, but it can help you identify birds from their songs with impressive accuracy. If you want to take a picture instead, try Picture Bird for Android and iOS.
[Related: How to start birding in any US city]
Keen gardeners will want to take a look at PictureThis for Android and iOS. Open the app, point it toward a plant, and the program will instantly tell you what kind of plant you’re looking at. The app is even smart enough to diagnose plant illnesses by the shape and color of the leaves, and will give you tips about restoring unwell plants.
If you’re heading to the great outdoors, you can use the camera on your smartphone to identify the mountain peaks you come across. Install PeakVisor for Android or iOS, point your phone at a mountain range, and the app will look at your location and the shape of the peaks to tell you what they’re called and how high they are.