Information Gain in SEO (search engine optimisation) measures the net-new factual, structural, and semantic utility a webpage offers relative to Google’s existing web index and a searcher’s active session trajectory.
QUOTE: “Information gain scores state how much more information one source may bring to a person who has seen other sources on the same topic. Pages with higher information gain scores may be ranked higher than pages with lower information gain scores.” Bill Slawski 2020.
Optimising for Information Gain requires separating static pre-computed index signals (calculated during crawling and parsing) from dynamic runtime session decay (evaluated during an active user search path).
1. Theoretical Foundation & Patent Specifications
Codified across Google Patents US20200349181A1 and US11354342B2, Information Gain penalises document redundancy by measuring conditional information entropy and structural entity deltas.
[ Crawl & Parsing Layer ] [ Active Search Session Layer ] ├── Static Entity Extraction (ΔE) ├── User Click Stream (S_t) └── Text Uniqueness (originalContent) └── Dynamic Entropy Decay I(D; S_t)
Dynamic Session-State Entropy (I(D; St))
Grounding the mechanism in Information Theory, Information Gain estimates the conditional entropy delta I(D; St) of a candidate document D relative to the user’s dynamic click-path session state St = {D0, D1, …, Dt-1}:
- H(D): Total information entropy (factual density and cross-modal token distributions) of candidate document D.
- H(D | St): Conditional entropy of document D given the user’s historical session trajectory St.
A document does not hold a single, static “Information Gain Score.”
Instead, its score decays dynamically during a live session.
As a searcher clicks through links (D0, D1), previously consumed information reduces H(D | St).
If a candidate page merely rephrases facts the user already viewed in St, H(D | St) ≈ H(D), causing net Information Gain to collapse to near zero and suppressing the page from subsequent click options.
Index-Time Entity Graph Delta Matrix (ΔE)
To evaluate factual novelty before a search session occurs, retrieval systems analyse structural changes in information during index processing. Extracted entities and relationships are mapped into a Knowledge Graph adjacent space. Let E(D) be the set of entity-attribute-value triplets (ei, r, ej) in document D, and E(U) be the union of triplets across top-SERP documents U. Incorporating edge confidence scores c(τ) ∈ [0, 1] and normalised property divergence:
Where c(τ) denotes the directional edge confidence weight in the Knowledge Graph, and diff(τD, τU) ∈ [0, 1] measures attribute vector divergence (such as updated empirical measurements, fresh temporal timestamps, or fine-grained parameter bounds).
A candidate document scores a high ΔE at crawl time when it establishes verified entity nodes or novel edge relationships absent from E(U), establishing the static baseline required to be surfaced.
2. Patent Specification vs. Production Codebase Reality
QUOTE: “Implementations described herein relate to determining an information gain score for one or more documents of potential interest to the user and presenting information from one or more of those documents that are selected based on their respective information gain scores. Implementations described herein relate to determining an information gain score for one or more documents of potential interest to the user and presenting information from one or more of those documents that are selected based on their respective information gain scores.” Google, 2020
The Google Content Warehouse API leak illustrates how theoretical patent models map directly to production attributes and modules:
| Patent Concept (US20200349181A1) | Production API Attribute | Signal Layer | Production System Function |
|---|---|---|---|
| Static Novelty Baseline | originalContentScore |
Static / Index-Time | Evaluates the ratio of non-duplicated text and structural nodes relative to the web index during crawling. Acts as an entry filter. |
| Perceived Effort & Utility | contentEffort |
Static / Index-Time | Model-based estimation measuring human labour, layout complexity, cross-modal asset density, and depth. |
| Semantic Representation | docEmbeddings |
Static / Index-Time | Maps documents into dense multimodal vector spaces to compute distance (Δ) from existing SERP clusters. |
| Sitewide Quality Footprint | siteAuthority |
Static Aggregation | Aggregates novelty and quality sitewide in QualityNsrData; low uniqueness across URLs triggers domain demotions. |
| Behavioral Validation | NavBoost |
Dynamic / Historical | Tracks 13 months of click interactions (goodClicks, badClicks, lastLongestClicks) to validate whether users found the page valuable. |
Decoupling Predictive Algorithms from User Behaviour
It is critical to separate mathematical information gain from behavioural tracking:
- Predictive Static Signals: Attributes like
originalContentScoreandcontentEffortestimate uniqueness using computational linguistic models before traffic arrives. - Behavioural Feedback Loops: Systems like NavBoost do not calculate real-time differential equations during a query. Instead, they record long-term interaction metrics (
lastLongestClicks,unsquashedClicks). If a page receives initial exposure due to high static novelty but users bounce back to the SERP (badClicks), the ranking boost is systematically neutralised.
3. Multimodal Vector Space & Signal Architecture
In retrieval engines like Mustang, Google maps documents into dense semantic embeddings (docEmbeddings). To identify content that fills conceptual gaps without requiring heavy entity extraction on every pass, the system evaluates Cosine Dissimilarity paired with a Subspace Orthogonal Projection Delta relative to the existing SERP consensus centroid ĈSERP.
To prevent vector magnitude distortion, the composite gain score is formulated as a bounded convex combination with λ ∈ [0, 1]:
- ĈSERP: Baseline SERP centroid derived via Pseudo-Relevance Feedback (PRF).
- P⊥(ĈSERP): Projection operator mapping candidate vectors onto the orthogonal subspace of the current SERP consensus.
- λ ∈ [0, 1]: Weighting factor balancing global thematic dissimilarity against local orthogonal subspace expansion.
[ Vector Subspace Projection ]
V_candidate
/|
/ | <-- Bounded Orthogonal Projection Delta (λ)
/ | (Indicates unique angle/sub-topic)
/___|___________
0 C_SERP (Current SERP Consensus)
Three-Stage Retrieval Architecture
Due to real-time latency constraints, Google executes these checks across three operational phases, leveraging specialised machine learning subsystems to compute vector weights:
- Stage 1: Index Processing (Pre-Query) — ΔE graph deltas, text duplication ratios, and static scores (
originalContentScore,contentEffort) are computed during document ingestion. Dense vector embeddings (docEmbeddings) are generated using TwinBERT (for dual-encoder text matching) and MUM (Multitask Unified Model) to map cross-modal asset density and conceptual themes. - Stage 2: Primary Candidate Retrieval (Query Time) — The primary retrieval engine tier (Mustang) processes the query, while RankBrain calculates dynamic intent vectors. These vectors (ΔGain) are compared against cached SERP centroids (ĈSERP) to generate the initial, high-novelty candidate pool.
- Stage 3: Late-Stage Re-Ranking (Session State) — If a searcher clicks through multiple links in a single session, dynamic entropy decay (I(D; St)) adjusts the remaining candidate links to prevent duplicate information exposure.
4. Operational Implementation & Editorial Transformation
To operationalise Information Gain, editorial teams must shift from rephrasing consensus SERP points to injecting net-new entity-attribute-value triplets (ei, r, ej) and expanding orthogonal vector space.
Practical Workflow Engine
- Integrate First-Party Data (ΔE Boost): Conduct proprietary benchmarks, run empirical experiments, or publish primary survey data. Inorganically inserting verified, unindexed triplets guarantees that index parsing filters flag the page as high novelty.
- Expand Entity Property Precision (diff Attribute): Correct stale metrics, add specific bounds to generic claims, or address obscure edge cases that the top-10 SERP results ignore.
- Deploy Non-Textual Effort Signals (
contentEffort): Replace generic stock photography with custom visual schematics, proprietary UI flows, or interactive calculators. Multimodal encoders incorporate visual structural layouts directly intopageEmbeddings. - Target Subspace Dissimilarity (ΔGain): Introduce structural sub-topics that do not align linearly with existing consensus vectors.
Applied Content Transformation Example
Target Topic: Troubleshooting PostgreSQL Connection Pools
Existing SERP Consensus: Recommends installing PgBouncer, setting max_connections = 100, and monitoring CPU utilization.
Low Information Gain (Parroting Consensus)
“To fix PostgreSQL connection spikes, set your max_connections limit to 100 in postgresql.conf and install PgBouncer to manage pooling. High connection counts consume extra RAM and CPU.”
Analysis: H(D | St) ≈ 0. Adds no new entity triplets; orthogonal vector projection yields zero expansion.
High Information Gain (Subspace Expansion + ΔE Delta)
“While setting max_connections = 100 is common baseline advice, on PostgreSQL 15+ running on Linux kernel 5.15+, connection overhead stems primarily from backend memory allocation per process (2 MB base state overhead). Instead of standard session pooling, deploy PgBouncer in Transaction Pooling mode with pool_mode = transaction and calibrate default_pool_size using the formula:
In our load-testing benchmark across 10,000 concurrent clients, transaction pooling reduced backend thread context switching by 68% compared to default session pooling, sustaining under 4ms query latency.”
Analysis: Adds unindexed entity triplets (PgBouncer, pool_mode, transaction), introduces an explicit mathematical calibration formula, provides primary benchmark empirical metrics, and projects heavily onto the orthogonal subspace of the SERP consensus.
5. Editorial Quality-Assurance (QA) Checklist
To systematically scale this engineering approach across your writing team, use this operational checklist during content production and editing phases.
This framework ensures every document satisfies index-time filters and maximises orthogonal vector space expansion prior to publication.
Phase 1: Static Entity & Factual Novelty Audit (ΔE Layer)
Objective: Ensure the document injects unindexed entity-attribute-value triplets to bypass basic index-time similarity filters.
- [ ] Identify the SERP Consensus Triplets: List the top 5 ranking competitors. Identify the core facts, tools, and steps they all share (e.g., “All competitors recommend Tool X and Setting Y”).
- [ ] Inject Net-New Factual Nodes: Add at least two relevant entities, tools, libraries, or sub-topics completely absent from the top 5 competing URLs.
- [ ] Calibrate Property Precision (diff): Replace generalised claims with precise parameters.
- Change: “Using this tool makes queries faster.”
- To: “Deploying this tool in transaction mode reduced backend context switching by 68% in our tests.”
- [ ] Provide Proprietary Source Verification: Anchor new facts to first-party data, an in-house sandbox test, a unique case study, or proprietary survey data.
Phase 2: Perceived Algorithmic Labour (contentEffort & Encoders)
Objective: Maximise cross-modal asset density and structural complexity signals parsed by multimodal models like MUM.
- [ ] Eliminate Stock Media Redundancy: Remove all generic stock photography or unedited vendor screenshots that carry zero net-new visual information.
- [ ] Deploy Custom Visual Assets: Include at least one proprietary diagram, custom workflow schematic, data visualisation chart, or explanatory video mapping the core problem.
- [ ] Embed Functional Utility Assets: Where applicable, integrate interactive tools, dynamic calculators, code blocks with custom variables, or downloadable configuration templates to elevate the page layout complexity.
- [ ] Optimise Information Density: Format layout structures using distinct scannable hooks (nested Markdown tables, clean code blocks, visual quote blocks) to signal deep content structuring to index parsers.
Phase 3: Semantic Subspace Expansion (ΔGain Layer)
Objective: Project heavily onto the orthogonal subspace of the SERP consensus to catch users looking for alternative, deeper answers.
- [ ] Execute the “Anti-Parrot” Text Scan: Review all introductory and summary paragraphs. Ensure they do not simply use synonyms to rephrase the consensus advice found on page one of Google.
- [ ] Address Niche Edge-Cases: Dedicate a specific section to solving an obscure error, hardware limitation, or operational failure condition that top results gloss over.
- [ ] Introduce a Dissenting or Advanced Framework: Break away from the standard linear narrative of the keyword. Contrast theoretical best practices with practical implementation friction or alternative technical methodologies.
Phase 4: Dynamic Session Friction Reduction (I(D; S_t) Layer)
Objective: Secure the “Last Longest Click” (lastLongestClicks) and prevent user bounce-backs to the SERP.
- [ ] Verify Hook Alignment: Ensure the exact primary search intent is addressed comprehensively within the first viewport, eliminating fluff that forces users to scroll or bounce immediately.
- [ ] Implement Frictionless Internal Trajectories: If the page references advanced prerequisite concepts, provide clean internal link pathways to keep user session-state discovery safely constrained inside your domain.
Disclosure: I use generative AI when specifically writing about my own experiences, ideas, stories, concepts, tools, tool documentation or research. My tool of choice for this process is Google Gemini. All content was verified as correct. See Hobo Web AI policy.
Disclaimer: This is not official advice from Google. It is SEO theory based on patents and code leaks. The metrics equations are borrowed from known IR (information retrieval) models rather than being extracted from leaks. Any article (like this) dealing with the Google Content Data Warehouse leak requires a lot of logical inference when putting together the framework for SEOs, as I have done with this article. I urge you to double-check my work and use critical thinking when applying anything from the leaks or patents to your site. My aim with these articles is essentially to confirm that Google does, as it claims, try to identify trusted sites to rank in its index. The aim is to irrefutably confirm white hat SEO has purpose in 2026 – and that purpose is to build high-quality websites. Feedback and corrections welcome.