When Search Gets Visual

Picture this: You're a market researcher trying to understand competitor product positioning. You have their marketing videos, product screenshots, PDFs of their whitepapers, and audio from their earnings calls. Traditional search? Good luck piecing that together manually. Enter multimodal deep research, where your AI agent doesn't just read the web, it sees, hears, and understands across every media type.

While the world obsessed over text-based DeepSearch in early 2025, we've been quietly building something more ambitious: multimodal deep research agents that can analyze visual content, extract insights from audio, process video frames, and synthesize findings across media types. Think of it as giving your research assistant eyes, ears, and the reasoning power to connect dots across formats.

Why Multimodal Deep Research Matters Now

The shift from single-pass RAG to iterative DeepSearch was just the appetizer. The real game-changer? Cross-modal reasoning.

Consider these scenarios where text-only search fails spectacularly:

• Product Intelligence: Analyzing competitor UIs from screenshots while cross-referencing their technical documentation
• Content Compliance: Scanning video content for brand safety while analyzing accompanying transcripts
• Market Research: Processing social media images, video reviews, and written feedback to understand sentiment trends
• Technical Documentation: Understanding architectural diagrams alongside code repositories and documentation

The Technical Reality Check: Most "multimodal" solutions today are just text search with image OCR bolted on. True multimodal deep research requires:

1. Native multimodal understanding (not just OCR + text search)
2. Cross-modal reasoning (connecting insights across media types)
3. Iterative refinement (the DeepSearch loop applied to multimedia)
4. Context preservation across modalities

Architecture Deep Dive: Building the Multimodal Engine

The Core Loop: Search → See → Hear → Reason

Building on the text-based DeepSearch pattern, our multimodal agent follows an expanded loop:

// Multimodal DeepSearch Core Loop
while (tokenUsage < budget && attempts <= maxAttempts) {
  const currentQuery = getNextQuery(gaps, originalQuestion);
  
  // Multimodal search across content types
  const searchResults = await multimodalSearch({
    text: await textSearch(currentQuery),
    images: await imageSearch(currentQuery), 
    videos: await videoSearch(currentQuery),
    audio: await audioSearch(currentQuery)
  });
  
  // Process results by modality
  const insights = await Promise.all([
    processTextContent(searchResults.text),
    processVisualContent(searchResults.images),
    processVideoContent(searchResults.videos),
    processAudioContent(searchResults.audio)
  ]);
  
  // Cross-modal reasoning
  const synthesis = await crossModalReasoning(insights, context);
  
  if (synthesis.isComplete) break;
  
  // Generate new gap questions based on multimodal analysis
  gaps.push(...synthesis.newQuestions);
}

Modality-Specific Processing Pipelines

1. Visual Content Pipeline

async function processVisualContent(images) {
  const results = [];
  
  for (const image of images) {
    // Multi-stage visual analysis
    const analysis = await Promise.all([
      // Scene understanding
      visionModel.analyzeScene(image.url),
      // Text extraction (OCR)
      extractTextFromImage(image.url),
      // Object detection
      detectObjects(image.url),
      // Facial recognition (if applicable)
      analyzeFaces(image.url)
    ]);
    
    // Combine visual insights
    const insight = await synthesizeVisualInsights(analysis, image.context);
    results.push(insight);
  }
  
  return results;
}

2. Video Content Pipeline

async function processVideoContent(videos) {
  const results = [];
  
  for (const video of videos) {
    // Extract keyframes for analysis
    const keyframes = await extractKeyframes(video.url, { interval: 30 });
    
    // Process audio track
    const audioAnalysis = await processAudioTrack(video.audioUrl);
    
    // Analyze visual progression
    const visualProgression = await analyzeFrameSequence(keyframes);
    
    // Combine temporal insights
    const temporalInsight = await synthesizeTemporalContent(
      visualProgression,
      audioAnalysis,
      video.metadata
    );
    
    results.push(temporalInsight);
  }
  
  return results;
}

3. Audio Content Pipeline

async function processAudioContent(audioFiles) {
  const results = [];
  
  for (const audio of audioFiles) {
    const analysis = await Promise.all([
      // Speech-to-text
      transcribeAudio(audio.url),
      // Speaker identification
      identifySpeakers(audio.url),
      // Sentiment analysis from voice
      analyzeToneAndSentiment(audio.url),
      // Background audio analysis
      analyzeAudioScene(audio.url)
    ]);
    
    const audioInsight = await synthesizeAudioInsights(analysis, audio.context);
    results.push(audioInsight);
  }
  
  return results;
}

The Cross-Modal Reasoning Engine

Here's where the magic happens, and where most implementations fall flat. Cross-modal reasoning isn't just about processing different content types; it's about finding semantic connections across modalities.

Implementation Strategy

async function crossModalReasoning(insights, context) {
  // 1. Extract semantic embeddings for each insight
  const embeddings = await Promise.all(
    insights.map(insight => generateMultimodalEmbedding(insight))
  );
  
  // 2. Find cross-modal connections
  const connections = findSemanticConnections(embeddings, threshold=0.8);
  
  // 3. Build knowledge graph
  const knowledgeGraph = buildCrossModalGraph(insights, connections);
  
  // 4. Reason across the graph
  const reasoning = await reasonAcrossModalities(knowledgeGraph, context);
  
  return {
    synthesis: reasoning.conclusion,
    confidence: reasoning.confidence,
    newQuestions: reasoning.gaps,
    isComplete: reasoning.confidence > 0.85
  };
}

The Semantic Bridge Pattern

One breakthrough we discovered: using semantic bridges to connect insights across modalities. For instance:

• Visual-Text Bridge: Screenshot of a UI element + documentation describing that feature
• Audio-Visual Bridge: Speaker in video + voice sentiment analysis
• Temporal Bridge: Sequence of events across video frames + corresponding audio timeline

Challenges I Faced Along the Way

Challenge 1: The Context Explosion Problem

The Problem: Multimodal content generates massive context. A single video can produce thousands of tokens from transcription, visual analysis, and metadata. Traditional context windows couldn't handle comprehensive multimodal analysis.

The Solution: Hierarchical context compression with modality-aware summarization.

class ModalityAwareContextManager {
  constructor(maxTokens = 200000) {
    this.maxTokens = maxTokens;
    this.compressionRatios = {
      text: 0.3,      // Aggressive text compression
      visual: 0.5,    // Moderate visual compression  
      audio: 0.4,     // Audio needs more context
      temporal: 0.6   // Video sequences need flow preservation
    };
  }
  
  async compressContext(multimodalContext) {
    const compressed = {};
    
    for (const [modality, content] of Object.entries(multimodalContext)) {
      if (this.getTokenCount(content) > this.getModalityLimit(modality)) {
        compressed[modality] = await this.smartCompress(content, modality);
      } else {
        compressed[modality] = content;
      }
    }
    
    return compressed;
  }
}

Challenge 2: Multimodal Model Hallucinations

The Problem: Vision models confidently describing things that aren't there. Audio models inventing conversations. The reliability issues compound when you're reasoning across modalities.

The Solution: Cross-modal validation and confidence scoring.

async function validateCrossModal(insight, supportingEvidence) {
  // Check consistency across modalities
  const consistencyScore = await checkModalityConsistency(
    insight,
    supportingEvidence
  );
  
  // Validate against external sources when possible
  const externalValidation = await validateAgainstKnownFacts(insight);
  
  // Confidence scoring
  const confidence = calculateConfidence(consistencyScore, externalValidation);
  
  return {
    insight,
    confidence,
    validated: confidence > 0.7,
    warnings: insight.confidence < 0.5 ? ["Low confidence insight"] : []
  };
}

Challenge 3: The Modality Bias Problem

The Insight: Different modalities have different "authority" for different types of questions. Text is authoritative for facts, visuals for spatial relationships, audio for emotional context.

The Solution: Modality-weighted reasoning with domain-specific authority.

Practical Implementation Guide

Setting Up Your Multimodal Stack

// Core dependencies
import { OpenAI } from 'openai';
import { GoogleGenerativeAI } from '@google/generative-ai';
import { AssemblyAI } from 'assemblyai';
import { Vision } from '@google-cloud/vision';
import { SpeechClient } from '@google-cloud/speech';

class MultimodalDeepResearchAgent {
  constructor(config) {
    this.textModel = new OpenAI({ apiKey: config.openaiKey });
    this.visionModel = new GoogleGenerativeAI(config.geminiKey);
    this.audioProcessor = new AssemblyAI({ apiKey: config.assemblyKey });
    this.objectDetector = new Vision.ImageAnnotatorClient();
    
    // Mixpeek integration for multimodal search
    this.multimodalSearch = new MixpeekClient({
      apiKey: config.mixpeekKey,
      enableCrossModal: true
    });
  }
  
  async research(query, options = {}) {
    const context = new MultimodalContext();
    const gaps = [query];
    let attempts = 0;
    
    while (gaps.length > 0 && attempts < options.maxAttempts) {
      const currentQuery = gaps.shift();
      
      // Multimodal search
      const results = await this.multimodalSearch.search(currentQuery, {
        modalities: ['text', 'image', 'video', 'audio'],
        crossModal: true
      });
      
      // Process each modality
      const insights = await this.processAllModalities(results);
      
      // Cross-modal reasoning
      const reasoning = await this.reasonAcrossModalities(insights, context);
      
      if (reasoning.isComplete) {
        return this.generateReport(reasoning, context);
      }
      
      gaps.push(...reasoning.newQuestions);
      attempts++;
    }
    
    return this.generatePartialReport(context);
  }
}

Integration with Mixpeek for Multimodal Search

// Leveraging Mixpeek's multimodal capabilities
async function setupMixpeekIntegration() {
  const mixpeek = new MixpeekClient({
    apiKey: process.env.MIXPEEK_API_KEY,
    features: {
      crossModalSearch: true,
      semanticSimilarity: true,
      temporalAnalysis: true
    }
  });
  
  // Index your multimodal content
  await mixpeek.index({
    collection: 'research_corpus',
    sources: [
      { type: 'video', path: 's3://my-bucket/videos/' },
      { type: 'audio', path: 's3://my-bucket/audio/' },
      { type: 'image', path: 's3://my-bucket/images/' },
      { type: 'document', path: 's3://my-bucket/docs/' }
    ],
    processing: {
      extractFrames: true,
      transcribeAudio: true,
      ocrImages: true,
      semanticChunking: true
    }
  });
  
  return mixpeek;
}

Performance Optimizations & Trade-offs

The Parallel Processing Pipeline

Here's where we get into the juicy engineering details.

class OptimizedMultimodalProcessor {
  constructor() {
    this.processingPool = new WorkerPool(8); // Adjust based on your infra
    this.cacheLayer = new RedisCache();
    this.rateLimiter = new TokenBucket({
      capacity: 1000,
      refillRate: 100
    });
  }
  
  async processInParallel(multimodalResults) {
    // Smart batching based on processing requirements
    const batches = this.createOptimalBatches(multimodalResults);
    
    const results = await Promise.allSettled(
      batches.map(batch => this.processBatch(batch))
    );
    
    return this.mergeResults(results);
  }
  
  createOptimalBatches(results) {
    // Group by processing complexity and API rate limits
    const textBatch = results.text.slice(0, 20); // OpenAI batch limit
    const imageBatches = this.chunkArray(results.images, 5); // Vision API limits
    const videoBatches = results.videos.map(v => [v]); // Process individually
    
    return { textBatch, imageBatches, videoBatches };
  }
}

Performance Insights:

• Text processing: ~50ms per document
• Image analysis: ~200ms per image
• Video keyframe extraction: ~2s per minute of video
• Audio transcription: ~1s per minute of audio
• Cross-modal reasoning: ~500ms per insight cluster

Advanced Features: Going Beyond Basic Multimodal

1. Temporal Reasoning Across Video Content

class TemporalReasoningEngine {
  async analyzeVideoProgression(videoUrl, query) {
    const keyframes = await this.extractKeyframes(videoUrl, {
      interval: 15, // Every 15 seconds
      sceneChange: true // Extract on scene changes
    });
    
    const frameAnalyses = await Promise.all(
      keyframes.map(frame => this.analyzeFrame(frame, query))
    );
    
    // Build temporal narrative
    const narrative = await this.buildTemporalNarrative(frameAnalyses);
    
    return {
      timeline: narrative.timeline,
      keyInsights: narrative.insights,
      confidence: narrative.confidence
    };
  }
}

2. Cross-Modal Fact Verification

async function verifyAcrossModalities(claim, evidence) {
  const verificationSources = [];
  
  // Text-based fact checking
  if (evidence.text) {
    verificationSources.push(await factCheckText(claim, evidence.text));
  }
  
  // Visual verification (for claims about visual content)
  if (evidence.images && isVisualClaim(claim)) {
    verificationSources.push(await verifyVisualClaim(claim, evidence.images));
  }
  
  // Audio verification (for claims about spoken content)
  if (evidence.audio && isAudioClaim(claim)) {
    verificationSources.push(await verifyAudioClaim(claim, evidence.audio));
  }
  
  return aggregateVerification(verificationSources);
}

Real-World Use Cases & Results

Case Study 1: Competitive Product Analysis

The Challenge: Analyze competitor's product positioning across their website, demo videos, marketing materials, and user reviews.

Our Approach:

1. Visual Analysis: Screenshot analysis of UI/UX patterns
2. Video Processing: Demo video analysis for feature detection
3. Text Mining: Marketing copy and documentation analysis
4. Audio Analysis: Podcast appearances and earnings calls

Results:

• 90% accuracy in feature detection vs manual analysis
• 75% faster than traditional competitive research
• Discovered 3 unannounced features from demo video analysis

Case Study 2: Content Compliance at Scale

The Problem: Analyzing thousands of hours of video content for brand safety compliance.

The Solution: Multimodal agent processing video frames, audio transcription, and metadata simultaneously.

async function analyzeContentCompliance(videoUrl) {
  const [visualAnalysis, audioAnalysis, metadataCheck] = await Promise.all([
    analyzeVisualContent(videoUrl, brandSafetyRules),
    analyzeAudioContent(videoUrl, speechComplianceRules),
    checkMetadataCompliance(videoUrl, platformGuidelines)
  ]);
  
  const complianceScore = calculateOverallCompliance([
    visualAnalysis,
    audioAnalysis, 
    metadataCheck
  ]);
  
  return {
    compliant: complianceScore > 0.8,
    score: complianceScore,
    violations: extractViolations([visualAnalysis, audioAnalysis, metadataCheck]),
    recommendations: generateRecommendations(complianceScore)
  };
}

Key Takeaways & Engineering Wisdom

What We Learned Building This

1. Modality Authority Matters: Not all modalities are equally reliable for all queries. Build authority hierarchies.
2. Context Compression is Critical: Multimodal content explodes your token usage. Invest in smart compression strategies.
3. Cross-Modal Validation Prevents Hallucinations: Use modalities to validate each other rather than trusting single-source insights.
4. Temporal Reasoning is Undervalued: Most systems treat video as "images + audio." True video understanding requires temporal reasoning.
5. Caching Saves Your API Budget: Multimodal processing is expensive. Cache aggressively.

The Engineering Trade-offs

Accuracy vs Speed: High-accuracy multimodal analysis takes time. Design your system with appropriate SLAs.

Cost vs Coverage: Processing all modalities is expensive. Build smart filtering to focus on high-value content.

Complexity vs Maintainability: Multimodal systems are inherently complex. Invest in good abstraction layers.

Looking Forward: The Multimodal Future

The shift to multimodal deep research isn't just about handling different file types, it's about understanding the world the way humans do: through sight, sound, and context.

What's Next:

• Real-time multimodal analysis for live content streams
• 3D spatial reasoning for architectural and design applications
• Emotion detection across visual and audio modalities
• Interactive multimodal interfaces that respond to gesture, voice, and visual cues

The Bottom Line: While everyone else is still figuring out text-based DeepSearch, teams building multimodal capabilities today will have a significant advantage tomorrow.

Get Started: Your Next Steps

1. Start Small: Begin with text + image analysis before expanding to video/audio
2. Choose Your Stack: Consider Mixpeek for multimodal search infrastructure
3. Design for Scale: Plan your architecture with token limits and API costs in mind
4. Test Ruthlessly: Multimodal systems have more failure modes than text-only systems

Want to dive deeper? Check out our Deep Research Docs for implementation details and code samples.

P.S. - If you made it this far, you're definitely part of the "nerd expert" audience. Drop me a line if you're building something similar, always happy to geek out about multimodal reasoning architectures over coffee (virtual or otherwise).

Also published on dev.to.

The Problem

Cookie deprecation is here. Third-party targeting signals are disappearing, and publishers need alternatives that:

1. Don't rely on user identifiers
2. Don't add latency to header bidding
3. Work with existing SSP/DSP infrastructure

The Mixpeek RTD module solves this by enriching Prebid bid requests with contextual + adjacency signals via standard OpenRTB fields.

What You'll Get

After this 15-minute setup, your bid requests will include:

SSPs and DSPs consume these via ortb2.site.content and ortb2Imp.ext.data, no custom integration required on their end.

Step 1: Install the Module

npm install @mixpeek/prebid

Or add via CDN:

<script src="https://unpkg.com/@mixpeek/prebid@latest/dist/mixpeek-rtd.min.js"></script>

Step 2: Get Your Credentials

1. Sign up at mixpeek.com
2. Generate an API key in your dashboard
3. Create a collection with feature extractors
4. Note your namespace ID (format: ns_xxxxx)

Step 3: Configure Prebid

Add the RTD provider to your Prebid config:

import '@mixpeek/prebid'pbjs.setConfig({  realTimeData: {
    auctionDelay: 250,  // Max wait time (ms)
    dataProviders: [{
      name: 'mixpeek',
      waitForIt: true,
      params: {
        apiKey: 'YOUR_API_KEY',
        collectionId: 'YOUR_COLLECTION_ID',
        namespace: 'YOUR_NAMESPACE'
      }
    }]
  }
})

That's it. Bid requests are now enriched automatically.

Step 4: Verify the Output

Check that contextual data is being injected:

pbjs.onEvent('mixpeekContextReady', function(context) {
  console.log('Category:', context.taxonomy?.label)
  console.log('Keywords:', context.keywords)
  console.log('Brand Safety:', context.brandSafety)
})

Your ortb2 object will now include:

{
  "site": {
    "content": {
      "cat": ["IAB19-11"],
      "cattax": 6,
      "genre": "Technology",
      "keywords": "ai,technology,ml",
      "ext": {
        "data": {
          "mixpeek": {
            "score": 0.94,
            "brandSafety": 0.98,
            "sentiment": "positive"
          }
        }
      }
    }
  }
}

How DSPs Consume the Signals

DSPs can use the enriched data for:

Contextual Targeting

// Target technology content
if (ortb2.site.content.cat.includes('IAB19-11')) {
  // Bid on tech inventory
}

Competitive Separation

// Avoid back-to-back competing ads
const prevCat = ortb2Imp.ext.data.hb_mixpeek_prev_cat
if (prevCat === 'IAB18-1' && currentAd.category === 'IAB18-1') {
  // Skip this impression
}

Brand Safety

// Only bid on safe inventory
if (ortb2.site.content.ext.data.mixpeek.brandSafety > 0.9) {
  // Safe to bid
}

Shadow Mode: Risk-Free Testing

Want to test without affecting live auctions? Use shadow mode:

pbjs.setConfig({
  realTimeData: {
    auctionDelay: 0,  // No delay
    dataProviders: [{
      name: 'mixpeek',
      waitForIt: false,  // Don't wait for response
      params: {
        apiKey: 'YOUR_API_KEY',
        collectionId: 'YOUR_COLLECTION_ID',
        namespace: 'YOUR_NAMESPACE'
      }
    }]
  }
})

This logs contextual data without blocking auctions. Perfect for validating the integration before going

Also published on dev.to.

Where the wrong scene comes from

Fixed windows. Splitting every 10 seconds ignores where the cuts are. A window that holds the last three seconds of a kitchen scene and the first seven of a street scene produces an embedding that is half of each. Ask for "street at night" and you may get the kitchen timestamp.

Frame sampling without boundaries. Sampling one frame a second gives you lots of vectors, many nearly identical. The nearest neighbour is often a frame from a similar-looking shot elsewhere in the video.

One vector per video. Averaging a whole video into one embedding throws away where anything happens. Search can find the right file and still cannot tell you where in it.

How segmentation fixes it

1. Shot boundary detection. Compare consecutive frames. A hard cut spikes a histogram difference. Dissolves and fades spread the change over many frames, so an adaptive detector flags a boundary when the score beats a rolling local mean by two to three standard deviations.
2. Group shots into scenes. A shot is one camera take. A scene can span many shots, like a conversation cutting between two angles.
3. Clean up. Merge segments under about two seconds into a neighbour, and split long ones at audio silence.
4. Embed per scene, keep timestamps. One embedding per scene, stored with its start and end time. Results come back as moments.

This also cuts index size by 10 to 100 times, because you store one vector per scene.

Quick checks on your own pipeline

• Print the boundaries of the scene your query wrongly returned. If a cut falls inside it, segmentation is the problem.
• Check whether dissolves are being missed. A fixed threshold catches hard cuts and misses gradual transitions.
• Check whether on-screen text or speech carries the meaning. If so, a visual embedding alone will keep missing, and you need OCR or transcript features on the same scene.

The full walkthrough, with the algorithms and thresholds: https://mixpeek.com/guides/video-scene-segmentation

Disclosure: I work on Mixpeek.

Also published on dev.to.

What AWS gives you

BDA's standard video output has:

• shots and chapters, with timestamps and a summary for each chapter
• IAB categories per chapter
• a full transcript with speaker labels
• text that appears on screen, with bounding boxes
• logo detection (off by default)
• seven moderation categories, such as violence, alcohol and hate symbols

The IAB list in the BDA docs has 24 tier 1 and 85 tier 2 categories. IAB Content Taxonomy 3.1, the current version, has 704 categories across four tiers, 37 of them at tier 1. If a buyer, SSP or curation deal asks for tier 3 codes or a specific taxonomy version, plan for a mapping step.

AWS also publishes guidance for contextual advertising with generative AI, which builds the same kind of labels from multimodal models you choose.

What Twelve Labs gives you

• Marengo turns video into embeddings, used for search and classification.
• Pegasus turns video into text, for summaries and structured descriptions.

Marengo 2.7 and Pegasus 1.2 became fully managed models in Bedrock in July 2025. Twelve Labs' own API carries newer versions.

Their contextual ad engine tutorial shows what building on these models looks like. Pegasus 1.5 writes structured metadata for each scene. Marengo 3.0 embeddings feed search. The candidate labels are matched against IAB 3.1 rows, and the pipeline picks ad breaks and writes FreeWheel-style key-values. It ships as a reference build with a repo, so the pipeline is yours to run.

How to choose

• You are on AWS and want labels this quarter: start with BDA. Check its category depth against what your partners expect.
• You need to search an archive by what happens on screen, such as every scene at a beach at sunset, or define your own segments: embeddings fit better. That means Marengo, or any multimodal embedding you can run.
• You need audited IAB 3.1 codes at full depth: either route needs a classification step you can measure. Label a few hundred of your own scenes by hand and compare top-1 accuracy before you commit.

A third option

Disclosure: I work on Mixpeek.

Mixpeek indexes video from your own S3 or GCS bucket, scene by scene. It classifies each scene into all 704 IAB Content Taxonomy 3.1 categories through a retriever you can inspect and change. On IAB Tech Lab's open 704-page benchmark, our 3.1 classifier scored 72.7% top-1 and 93.8% top-5. That benchmark is text only, and the write-up includes where it lost. A side-by-side with Twelve Labs is at mixpeek.com/comparisons/mixpeek-vs-twelvelabs.

Also published on dev.to.

1. The platforms' own servers

• Meta Ads MCP. A remote server hosted by Meta, documented under Ads AI Connectors. Its tools cover reporting, campaign and ad creation and editing, catalogs, datasets, A/B tests and lift studies. Apps need the ads_mcp_management permission.
• Amazon Ads MCP server. Open beta for Amazon Ads partners with active API credentials. Its prebuilt tools run multi-step jobs such as creating a campaign with its ad groups and ads from one prompt.
• Google Ads MCP. An open-source server from the Google Ads API team, googleads/google-ads-mcp. Its tools search the account, describe API resources and list accessible customers. It reads data and changes nothing.

Use these when the agent acts inside one platform. Each needs that platform's credentials, and each returns that platform's field names.

2. Cross-channel marketing data

Improvado's MCP server is a good example. It sits on Improvado's connectors and maps fields such as Google's cost and Meta's spend to one metric, so an agent can compare channels in a single query. Servers like this fit reporting and budget questions that span platforms. They need the vendor's platform underneath.

3. Agent-to-agent buying: AdCP

AdCP defines tasks that buyer and seller agents call over MCP or A2A. Examples are get_products and create_media_buy for media, get_signals and activate_signal for audience data, and build_creative and sync_creatives for creative. AgenticAdvertising.org stewards it, and the current release is 3.2.1. If your agent buys from publishers or sells inventory to other agents, implement AdCP so you do not build a custom integration for each counterparty. Some IAB Tech Lab work in this area overlaps; this comparison of AdCP and AAMP covers how they differ.

4. Content understanding

The servers above say what an ad account holds and what inventory costs. None of them says what is inside a piece of video, so they cannot answer whether a CTV episode is safe for a brand or which IAB category a scene belongs to. For that the agent needs a server that has indexed the media.

Disclosure: I work on Mixpeek. Mixpeek's MCP server (https://mcp.mixpeek.com/app/mcp, OAuth, read-only) lets an agent search video, images, audio and documents you have indexed from your own bucket, and run retrievers you define, for example one that classifies each scene into IAB Content Taxonomy 3.1. Setup docs.

A reasonable starting stack

• One platform server for each platform where you buy.
• A cross-channel data server if your reporting spans platforms.
• AdCP if you buy from, or sell to, other agents.
• A content server if any decision depends on what the creative or the inventory shows.

Give the agent read-only tools first, and add write access once its reads have proven correct.

Also published on dev.to.

1. The current version is 3.1, and the bid request should say so

IAB Tech Lab's content taxonomy page lists 3.1 as current and describes the change from 3.0 as a genre update that adds Thriller. In the 3.1 file Thriller is ID 700 under Genres. Every 3.0 ID is also a 3.1 ID, so moving up needs only a new version label.

The part that breaks things is cattax. OpenRTB sends category IDs in cat and says which taxonomy they come from in cattax, using AdCOM's List: Category Taxonomies:

• 2 is Content Taxonomy 2.0
• 7 is Content Taxonomy 3.0
• 9 is Content Taxonomy 3.1

If you label content with 3.1 IDs and send cattax: 7, a strict receiver reads them as 3.0, which holds for every ID except the ones 3.1 added, such as Thriller. Send cattax: 2 and the same numbers mean different 2.0 categories.

2. It is 704 categories, in four tiers

"700+" is close, and it hides the shape. Counting the 3.1 file:

The tier matters more than the total. Some classification services return only the top two tiers: the IAB list in Amazon Bedrock Data Automation's video output docs has 24 tier 1 and 85 tier 2 categories. If a buyer, an SSP or a curated deal asks for tier 3 codes, check the depth your classifier actually returns before you promise it.

3. GARM shut down in 2024, and its categories live on in vendor products

The World Federation of Advertisers discontinued GARM on 9 August 2024, days after X filed an antitrust suit against the WFA, GARM and its members. The WFA named no successor.

What survived is the Brand Safety Floor and Suitability Framework: eleven content categories, such as arms and ammunition, hate speech and terrorism, each graded by risk from floor to low. Platforms and verification vendors still use that vocabulary in their own products, and each now maintains its own version. IAB Tech Lab's human-labeled GARM benchmark was last updated in 2022, before the shutdown.

So "GARM-aligned" today means "uses categories that GARM defined". Nobody certifies it. Ask a vendor which version of the categories its labels follow and when it last changed them.

Quick checks before you ship

• Your labels and your cattax value name the same version.
• You know which tiers your classifier returns, and they match what your partners ask for.
• Brand suitability tiers are written down as your own policy, with the vendor's category definitions attached.

Disclosure: I work on Mixpeek. We classify video, images, audio and pages into all 704 Content Taxonomy 3.1 categories, scene by scene for long video, and return IDs ready for cat with cattax: 9 (how it works). Our GARM explainer has the longer history, and the open-source IAB mapper converts 2.x labels to 3.1 IDs.

Also published on dev.to.

Why search a tree instead of the whole index

Flat vector search ranks every chunk against the query. On a large, varied corpus the top results often include chunks that share vocabulary with the query and come from the wrong topic. An agent that retrieves those chunks has to spend context reading them.

A cluster tree changes three things:

• Bounded context. A summary of a cluster is short. An agent can read the summaries of ten sibling clusters in a few hundred tokens and decide where to look, so the amount of text in its prompt grows with the depth of the tree. The size of the corpus barely moves it.
• Fewer irrelevant chunks. Once the agent has picked a branch, the final search runs against that branch only. A query about "mountain" never competes with chunks from a cooking archive.
• An explainable path. The route from root to leaf (for example Sports > Skateboarding) is a record of why a document was returned. You can log it, show it to a user, and debug it when the answer is wrong.

Building the tree

The tree has three ingredients: embeddings, a clustering algorithm applied recursively, and a label plus summary for every node.

Embed. Use one embedding model for the whole corpus so distances mean the same thing at every level.

Cluster recursively. Run a clustering algorithm on the whole set to get top-level groups, then run it again inside each group that is big enough to split, and keep going until groups are small or you reach a depth limit. K-means works when you know roughly how many groups you want at each level. HDBSCAN works when you do not, and it marks outliers as noise instead of forcing them into a group. In both cases, reducing dimensionality first (UMAP is common) tends to make the splits cleaner.

Label and summarize. For each cluster, sample member documents and ask an LLM for a short label and a one or two sentence summary. These two fields are what the agent reads while navigating, so their quality sets the ceiling on the whole system. A label such as "Species" on a parent that holds both animals and landforms will send queries about mountains down the wrong branch.

The best-known paper on this family of ideas is RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (Sarthi, Abdullah, Tuli, Khanna, Goldie and Manning, January 2024). Its abstract describes recursively embedding, clustering and summarizing chunks of text to build a tree with different levels of summarization, and reports that pairing RAPTOR retrieval with GPT-4 improved the best prior result on the QuALITY benchmark by 20 percentage points of absolute accuracy. RAPTOR retrieves from the tree. The loop below adds an agent that decides which branches to open.

The agent loop

Give the agent three tools:

• list_children(cluster_id) returns the child clusters with their labels and summaries. With no argument it returns the top level.
• read_summary(cluster_id) returns the full summary of one cluster.
• search_within(cluster_id, query) runs vector or keyword search over documents in that cluster and its descendants.

A sketch in Python, with llm_choose and enough_evidence standing in for whatever model calls you use to pick clusters and judge sufficiency:

def agentic_search(query, tree, llm_choose, beam=3, max_depth=4, max_hits=20):
    frontier = [tree.root]            # clusters we are about to open
    leaves = []

    for _ in range(max_depth):
        candidates = []
        for node in frontier:
            kids = tree.list_children(node)
            if not kids:
                leaves.append(node)   # nothing below: search here
                continue
            candidates += kids

        if not candidates:
            break

        # The model sees label + summary for each candidate and
        # returns the `beam` most promising ones.
        frontier = llm_choose(query, candidates, k=beam)

    leaves += frontier
    hits = []
    for leaf in leaves:
        hits += tree.search_within(leaf, query)

    # Stop when the evidence suffices, otherwise widen the beam and retry.
    if enough_evidence(query, hits):
        return hits[:max_hits]
    if beam >= 9:
        return hits[:max_hits]
    return agentic_search(query, tree, llm_choose, beam=beam + 2)

Three details matter in practice.

First, ask the model to score every child, not to name one. Scores let you keep the top beam paths and rank finished paths against each other. Averaging scores with a geometric mean keeps a deep leaf from being penalized for having more decisions behind it.

Second, show a parent cluster to the model as the leaves reachable below it. A parent label is a compromise, and the leaf names tell the model what is actually inside.

Third, give the agent an explicit stop rule. Without one it keeps descending or searching after the answer is already in context.

Costs and failure modes

A wrong branch early. The biggest risk is the first decision. If the agent opens the wrong top-level cluster, everything below it is searching the wrong region, and the final answer can be confidently wrong. Two mitigations help. Keep a beam of k branches at each level, so a near-miss on the first choice is still alive at the second. And let the agent backtrack: if search_within returns weak hits, return to the last level where a sibling scored close and try that one.

Model calls per level. Every level costs a model call. A tree of depth three with a beam of three costs a handful of calls before any retrieval happens. Batch the candidates for all paths at one level into one request, and use a small, fast model for navigation decisions since each one is a short classification.

Stale clusters. Clusters describe the corpus as it was when you ran the clustering. New documents either land in the nearest existing cluster or sit outside the tree, and labels drift out of date as content shifts. Re-cluster on a trigger: a schedule, a document count threshold, or a quality signal such as falling cohesion or a rising share of outliers.

Summary quality. A vague summary hides a cluster from the agent. Spot-check labels and summaries when you build the tree, and keep a set of queries with known answers to run after every rebuild.

What Mixpeek documents

Mixpeek's Agent Search retriever stage has a cluster_navigation strategy. Per the docs, a cluster run writes one centroid document per cluster with its label, summary and parent_cluster_id. The stage starts at the top-level clusters, asks a decision model which child cluster most likely holds documents that answer the query, and repeats until it reaches leaf clusters. It then returns the member documents of the best leaves. The walk is a beam search: beam_width defaults to 3, and max_leaf_clusters sets how many leaves return members. Each returned document carries cluster_path and cluster_path_score, and the stage metadata includes a navigation_trace with the candidates and probabilities at every level. The stage returns the members of the chosen clusters. It does not run a separate vector search inside them, though you can chain a search stage after it. The docs do not describe backtracking.

On the build side, the Clusters docs describe vector clustering with eight algorithms including HDBSCAN and K-Means, optional UMAP preprocessing, and hierarchical sub-clustering through hierarchical: true and max_hierarchy_depth. LLM labeling is configurable per cluster. Each execution reports quality metrics, including a should_recluster recommendation, and clusters can be re-run automatically with cron, interval or event-based triggers. The hierarchical search tutorial walks through the whole sequence from clustering to a retriever that navigates the tree.

Disclosure: I work on Mixpeek. Mixpeek builds clusters over collections, including hierarchical vector clusters with LLM-written labels (Clusters docs), and its Agent Search stage can navigate those clusters top-down with the cluster_navigation strategy (Agent Search docs). The tutorial linked above shows the end-to-end setup.

Also published on dev.to.

Why people move vectors to object storage

A hosted vector database keeps your whole index in memory on machines you rent all month. That is fine at a few million vectors. It gets expensive when most of your namespaces sit idle, which is the normal shape for multi-tenant apps and media archives.

Object storage flips the cost model. The data sits at storage prices, around $0.023 per GB-month on S3 Standard, and compute only runs when a query arrives.

The three options

S3 Vectors. Native to AWS. You create a vector bucket and an index, put vectors with metadata, and query by similarity. Good when you are all-in on AWS and need plain nearest-neighbor search with metadata filters.

LanceDB. An embedded library over the Lance file format. Point it at s3://bucket/path and query from your own code. Good when you want no server at all and your queries run inside one application.

A vector store on object storage. The index lives in your bucket, and a query layer loads what it needs, caches the hot parts, and scales idle namespaces to zero. MVS works this way: you bring your own vectors, run dense, sparse and BM25 hybrid search, and the vectors persist on your own S3-compatible storage. It also imports from Pinecone, Qdrant or Weaviate without re-embedding.

What to check before you switch

• Cold latency. The first query against an idle namespace has to fetch from storage. Measure it on your own data size.
• Hybrid search. If your users type exact product codes or names, pure vector search misses them. You want BM25 alongside dense vectors.
• Filters at query time. Metadata filters should apply inside the search. A store that filters the top results afterwards hands back fewer than you asked for.
• Write path. Check how writes become durable and searchable, and how long that takes.

Full details on how MVS does this, with a cost estimator: https://mixpeek.com/mvs

Disclosure: I work on Mixpeek.

Also published on dev.to.vectordatabaseawss3searchVector search on S3, without a hosted vector databaseYou can keep your vectors in S3 and search them there. Three ways work today. AWS S3 Vectors stores...

Consumer options

Google Lens. Google's help page says you can search with an image by uploading a file, dragging it in, or pasting an image URL. Take a frame from the video, save it as an image, and upload it. The page does not mention video input, so the frame step is yours.

Bing Visual Search. Microsoft's support page describes searching the web with an image from the camera icon on Bing Images. It accepts uploads, pasted images and URLs. Same workflow: export a frame, upload it.

TinEye. TinEye fingerprints the image you submit and compares it with its index, which it builds by crawling the web. Its help center says it finds exact and altered copies, including cropped, color adjusted, resized, heavily edited or slightly rotated ones, and that it does not commonly return similar images of the same subject. Its help pages describe image search only.

InVID-WeVerify. This browser plugin was built for journalists and fact-checkers. Its keyframe tab takes a video URL or an uploaded file, segments it into keyframes, and lets you right-click a keyframe to search it on Google, Yandex, TinEye and Baidu. It automates the manual workflow above, so it is the quickest way to try it.

YouTube Content ID. Content ID does match video, but only for rights holders. Google's help page says owners must hold exclusive rights to a substantial body of original material that is frequently uploaded to YouTube. They supply reference files, and every new upload is scanned against that library. A regular user cannot submit a clip and ask where it appears.

What these tools miss

All four web options search an index of images. None of them has your video in it. That has practical consequences:

• A clip that circulated only as video, never as a thumbnail, screenshot or article image, may have no still in the index to match.
• You choose which frames to search. A frame from a title card or a black transition returns nothing useful, and a single frame says nothing about when in the clip it occurs.
• Copies that were cropped, mirrored, letterboxed, overlaid with captions or re-encoded at low quality produce frames that differ from the originals. TinEye's documentation covers crops and edits, but horizontal flips and heavy overlays are where fingerprint matching usually gives out, so test your own samples.
• Your private footage is not on the web, so no public tool can find it. Searching a library you own requires indexing it.

Build one over your own library

The pipeline has four steps.

1. Sample frames. The simplest approach is a fixed rate such as one frame per second. A more economical one is shot-based: ffmpeg's select filter exposes a scene score between 0 and 1, and the documentation says a threshold between 0.3 and 0.5 is generally a sane choice. Extract one representative frame per shot and store its timestamp.

2. Choose a fingerprint. Perceptual hashes (average, perceptual, difference, wavelet) from the imagehash library produce a short code, and the distance between two codes is a Hamming distance. They are cheap and handle re-encoding, resizing and mild color shifts. They are weak against heavy crops and flips. Embeddings from a vision model such as CLIP are larger and need a vector index, but they tolerate more edits and also support searching by meaning. Many systems run a hash for near-exact copies and embeddings for everything else.

3. Index. Store one row per sampled frame: video ID, timestamp, hash or vector. For hashes, a linear scan over a few million rows is fine. Past that, use a vector database or a BK-tree.

4. Align in time. Matching single frames gives noisy results, because unrelated frames sometimes collide. Real matches agree on an offset. If query frame at 3 s matches library frame at 63 s, and query frame at 4 s matches library frame at 64 s, the offset is 60 s in both cases. Count votes per (video, offset) and return the winner. That also gives you the start timestamp of the clip inside the source file.

Here is a small version using hashes and offset voting. It needs ffmpeg on your PATH, plus pip install imagehash pillow.

import subprocess, tempfile, glob
from collections import Counter
from PIL import Image
import imagehash

def frames(video, fps=1):
    """Return [(timestamp_seconds, phash)] sampled at `fps`."""
    with tempfile.TemporaryDirectory() as d:
        subprocess.run(
            ["ffmpeg", "-loglevel", "error", "-i", video,
             "-vf", f"fps={fps}", f"{d}/f_%06d.png"],
            check=True,
        )
        out = []
        for i, path in enumerate(sorted(glob.glob(f"{d}/f_*.png"))):
            out.append((i / fps, imagehash.phash(Image.open(path))))
        return out

# Build the index: one row per sampled frame.
index = []  # (video_id, timestamp, hash)
for vid in ["a.mp4", "b.mp4"]:
    index += [(vid, t, h) for t, h in frames(vid)]

def search(clip, max_dist=10):
    votes = Counter()
    for tq, hq in frames(clip):
        for vid, ts, h in index:
            if hq - h <= max_dist:          # Hamming distance
                votes[(vid, round(ts - tq))] += 1
    if not votes:
        return None
    (vid, offset), n = votes.most_common(1)[0]
    return vid, offset, n

print(search("query_clip.mp4"))  # e.g. ('a.mp4', 60, 14)

The third number is the vote count, which works as a confidence signal. The distance threshold of 10 on a 64-bit hash is a starting point, so tune it on clips you know are matches and clips you know are not. The first frame of the extracted images is labeled 0 seconds, which is approximate to within one sampling interval.

For production use, replace the linear scan with an index, add shot detection so static scenes do not produce hundreds of near-identical rows, and run a second embedding pass for mirrored or cropped copies.

Disclosure: I work on Mixpeek. Mixpeek indexes video in your own S3 or GCS bucket, splits it into scenes, and embeds frames. A retriever then returns the matching file and timestamp for an uploaded clip or image. We compare ten approaches in Best Reverse Video Search Tools, and the reverse search tutorial walks through the setup.

Also published on dev.to.