Optimizing Token Usage: The Value of Markdown Output in AI Queries

AI-driven search results can consume an astonishing number of tokens, especially when pulling large datasets for analysis. For instance, a simple query like "find coffee shops" may generate extensive nested objects filled with countless metadata and URLs, much of which goes unused. This inefficiency leads to elevated costs since every token used comes with a price tag. Fortunately, some strategies can help minimize token consumption.

One viable approach is adopting Markdown output, introduced by SerpApi. This tool enables users to significantly cut down on token usage; a query for "coffee" in JSON format initially amounted to 24,723 tokens. By switching to a Markdown format, the same query dropped to just 6,435 tokens—a remarkable 74% reduction. Further optimizations in filtering can even reduce this figure to around 1,298 tokens, enhancing the model's efficiency and allowing more information to stay within the critical context window.

Understanding the Markdown vs. JSON Shift

At the heart of this reduction lies how Markdown processes output. SerpApi's Markdown distinctly removes unnecessary components while preserving key information that an AI model requires. Some of the typical elements stripped or transformed include:

• Tracking links and internal URLs: JSON often includes various links for internal navigation within SerpApi, which aren't relevant for model processing. Markdown eliminates this clutter, focusing solely on the necessary user-facing links.
• Icon and thumbnail data: While visual elements may enhance user interfaces, they don’t contribute to the AI’s understanding and can inflate token counts unnecessarily. Markdown automatically omits these.
• Nested metadata and duplicates: JSON maintains intricate structures like pagination and repeated fields. Markdown condenses this information into more streamlined formats, such as tables or YAML.
• Title and link separation: In JSON, titles and links are often separate fields with accompanying metadata. Markdown unifies these into a single line, reducing overhead.

SerpApi hasn't disclosed specific token costs per field, but the comparative figures provide a clear directive: anything coded for UI rendering, rather than model comprehension, constitutes potential token waste.

When JSON Still Reigns Supreme

Despite its benefits, Markdown output isn’t universally applicable. When precise data formats are critical—like numeric values for pricing and structured arrays for further processing—JSON remains the superior choice. It retains the same foundational data but arranges it in a way that suits specific applications, such as analytics or pricing engines.

For example, when retrieving a result from Naver Shopping using JSON, one might find fields for price, old_price, and rating provided in precise formats. However, for applications where a simpler summary suffices, Markdown’s streamlined rows would convey the essential information more concisely.

Implementing SerpApi's Solution

SerpApi facilitates this transition to Markdown output with a simple toggle. Users can request this output format through various means:

• By appending output=md to the existing search query,
• By switching the endpoint from /search to /search.md, or
• By setting the request header to Accept: text/markdown.

This switch is available for all 100+ APIs on SerpApi, at no additional cost. The Markdown format provides structured tables, native inline links, and automatically discards unnecessary internal tracking information. For greater precision, SerpApi also offers a json_restrictor feature that helps limit specific output fields server-side before data is transmitted.

Testing with Your Own Queries

The token savings aren’t just theoretical. Users can validate the performance of Markdown by testing a familiar query, adding output=md, and comparing the token counts from JSON to Markdown output. Experimenting with the json_restrictor can further illustrate specific savings tailored to your workflow.

Should You Make the Switch to Markdown?

While Markdown isn't a cure-all, the way data is structured undeniably influences token costs—a consideration often overlooked. In AI systems where each token adds up quickly, reassessing and reshaping how data is delivered is a smart move. If you’re curious about Markdown's advantages versus JSON, visit SerpApi's feature page for more insights.