Written by Kate Knibbs
As commonly observed by frequent internet users recently, Google’s major AI improvement to its search functionalities began on shaky grounds. Shortly after its inauguration, the newly introduced AI Overviews for search inquiries, the feature was quickly ridiculed for delivering incorrect and bizarre responses, such as suggesting to consume stones or to prepare pizza with glue.
Recent insights from SEO company BrightEdge indicate that there has been a noticeable decline in how frequently Google has been displaying AI Overviews since its launch, with a significant pullback prior to the outburst of criticism. The firm has been monitoring Google’s AI responses in search results from a substantial database of queries since the trial phase of the feature last year.
Following the release of AI Overviews to logged-in US users in English post-Google’s I/O event on May 14, BrightEdge recorded these AI-generated responses in nearly 27 percent of the searches it observed. However, their visibility significantly reduced shortly thereafter, particularly in the days preceding the widespread sharing of error-laden AI Overview screenshots on the web. By the conclusion of the last week, coinciding with Google addressing the shortcomings of its AI feature in a blog post, the feature was seen in just 11 percent of search results, with no significant change at the start of this week.
Jim Yu, the founder and executive chairman of BrightEdge, suggests that Google’s cautious progression with the rollout of AI Overviews indicates the presence of some risks they are managing closely. Yu, however, remains optimistic about Google’s strategy with AI Overviews, perceiving early difficulties as temporary setbacks rather than inherent flaws.
“We’re continually refining our deployment of AI Overviews to maximize their utility, including numerous technical enhancements over the last week to elevate the quality of responses,” stated Google’s spokesperson, Ned Adriance. Although Google has not disclosed any internal data on how often AI Overviews are generated in searches, Adriance mentioned that the figures reported by BrightEdge do not align with Google’s own observations.
The reasons behind Google’s apparent decision to reduce the visibility of AI Overviews shortly after its initial launch remain unknown. Recent blog updates by the company revealed that the extensive usage of this feature by millions provided new insights into its performance and faults. Liz Reid, head of search at Google, acknowledged that over a dozen technical improvements had been implemented to minimize issues such as the undesired appearance of satirical content. These modifications are intended to limit occurrences of AI Overviews, although specific details on how these restrictions impact the frequency of AI-generated results were not provided.
Following the opening of a beta test for AI Overviews to users late last year, BrightEdge has been monitoring its performance across a spectrum of sample queries categorized into areas like ecommerce, insurance, and education. These queries not only cover common searches but also include less frequent ones, with ongoing tests sometimes conducted several times a day. In December 2023, BrightEdge recorded AI Overviews in 84 percent of its searches but noted a decline over time. Adriance clarified that AI Overviews were not triggered in 84 percent of all searches automatically and provided no further details on Google’s internal metrics. After making AI Overviews accessible to all, BrightEdge kept tracking their occurrence, using both accounts from the initial beta group and new users, and found no substantial variance between the two groups in their encounters with AI Overviews.
Google declined to share exactly how much it changed how many AI Overviews it showed the general public versus people enrolled in the beta test, but Adriance said that people who had opted in to the test were shown AI Overviews on a wider range of queries.
BrightEdge’s data also sheds light on the topics where Google believes AI Overviews can be most helpful. AI answers appeared on the majority of health care keyword searches, most recently at a frequency of 63 percent. Sample queries included in BrightEdge’s data included “foot infection,” “bleeding bowel,” and “telehealth urgent care.” In comparison, queries about ecommerce return AI Overviews at around 23 percent, while restaurants or travel very rarely trigger AI overview answers.
By Matt Burgess
By Eric Ravenscraft
By Ryan Wani Tribe
By Dell Cameron
Yu calls those results “surprising,” because health queries can be especially sensitive, and Google has made a concerted push in recent years to refine what it shows people who ask health questions.
Mark Traphagen, an executive at the search-engine-optimization platform seoClarity, has also taken special notice of how Google is handling health-care-related queries. To track how AI Overviews are rolling out, the company is monitoring the responses to a list of thousands of searches over time. For 26 popular health-related keywords, including “how to treat insomnia” and “symptoms of Lyme disease,” Google shows an AI response for around 58 percent.
Like Yu, Traphagen has been surprised by how often AI Overviews appear in response to this type of question. But they say the way Google’s feature sources its responses to health queries, often relying on trusted websites like the Mayo Clinic or the US Centers for Disease Control and Prevention, is encouraging. “They have really turned up the safeguards,” Traphagen says. “They’re all from well-known, credible sources.”
Despite Google’s AI sometimes providing inaccurate responses, including health-related ones, it has been noted that the sources cited for these answers are often of questionable quality. Lily Ray, a search engine optimization consultant, has pointed out through her research that these AI Overviews struggle with less definitive health topics such as aging, building muscle, and skincare, though they perform better with more direct medical questions.
The New York Times highlighted issues regarding the reliability of sources used by Google’s algorithms in providing health information. It reported that questions concerning the health benefits of chocolate were answered by citing sources like an Italian chocolate maker and a seller of home health tests. However, upon a subsequent check, these references were replaced with credible health care organizations, improving the answer’s reliability, although the health benefits of dark it still mentioned as advisable daily, remains a contentious point.
Despite initial challenges, Yu from BrightEdge believes that AI in search is a lasting trend that’s likely to improve significantly. Although it’s still uncertain if future enhancements in AI search will sufficiently overcome its early criticisms.
