Publishing as a data pipeline problem — with contracts, validation and reproducible output
Everything upstream of a generated page is a dataset question: fill rates, drift, entity resolution and provenance. That is your domain, not marketing's.
- Fill-rate floor
- Enforced
- Entity resolution
- Canonical IDs
- Drift alerts
- Per field
- Provenance
- Per page
Generation refuses to run below contract thresholds.
One entity, one page, across all sources.
Distribution checks on every ingest.
Data version + template version recorded.
Modelled at current defaults: 18,600 indexed URLs → 744 conversions per month.
What actually hurts in this seat
The interesting work in programmatic publishing is upstream. Which fields have high enough coverage to render, how do you resolve the same entity across three sources, what happens when a supplier changes a unit, and how do you keep output reproducible when the input drifts. We build the publishing layer to be driven by a validated dataset with a versioned contract, so page generation is deterministic given the data.
Content pipelines with no schema validation, so bad rows become bad pages silently.
Entity duplication across sources creating near-identical pages nobody wants.
No provenance, so nobody can say which data version produced a given page.
The four jobs this engagement owns
Contract the data
A versioned schema with required fields, type constraints, unit normalisation and minimum fill rates that generation refuses to run below.
Resolve entities properly
Deduplication and canonical-entity assignment across sources, so 'Acme Corp' and 'ACME Corporation' produce one page, not two thin ones.
Monitor drift
Distribution checks on incoming data with alerting, so a supplier format change surfaces as an alert rather than 4,000 broken pages.
Keep it reproducible
Every generated page records its input data version and template version, so any output can be reproduced or explained.
Four stages, each with a named output
Contract definition
Required fields, types, units, allowed ranges and fill-rate floors, versioned in the repository alongside the transformation code.
Versioned data contract with tests.
Data Scientists workstreams
Training data pipelines
Using LLMs in a publishing pipeline where they help, and refusing them where they hallucinate.
OpenAnalytics integration
Closing the loop: performance data back into the warehouse so the next generation run is informed by the last.
OpenQuery shapes this surface answers
These are patterns, not a keyword list. Each one multiplies against the entities in your own dataset — which is where a 30,000-URL first batch comes from.
Are you ready to build this?
Not yet. Fix the unchecked items first; publishing now would create pages we would later consolidate.
Size it with your own numbers
Data-heavy catalogue and directory builds with validated pipelines and entity resolution. Indexation is held at a conservative 62%.
- Indexed URLs
- 18,600
- Monthly organic clicks
- 74,400
- conversions per month
- 744
- Modelled monthly value
- $156,240
A model, not a forecast. Move the sliders to your own conversion economics — we will run the same maths against your data on the call.
A supplier unit change caught before publishing
- Setup
- A catalogue pipeline where one supplier silently switched from millimetres to centimetres.
- Mechanism
- A distribution check on the dimension field flagged a 10× shift against the trailing baseline and halted the DAG before generation.
- Result
- Zero incorrect pages published; the previous quarter, an equivalent change had reached 6,000 live URLs before a customer reported it.
How data scientists usually solve this — and what changes
| Dimension | The usual approach | With WpBulkPublishing |
|---|---|---|
| Validation | None until a human notices | Suites that block the run |
| Entity handling | String matching | Canonical IDs with confidence scores |
| Reproducibility | Not possible | Data + template version per URL |
| Drift | Discovered by customers | Alerted at ingest |
What the first weeks look like
- Week 1
Scoping call and data review
We look at what data scientists already hold — systems, exports, APIs — and score each axis for demand and defensibility.
- Weeks 3–4
Contract and template design
The data contract is written and the first template is designed against real rows, not placeholders.
- Weeks 5–8
First tranche live
4,500–10,500 URLs published with schema, internal links, sitemap entries and IndexNow.
- Weeks 9–14
Read, widen, hand over
Indexation and impression data decides what widens and what gets cut. Templates, gates and runbook transfer to you.
Questions we get asked in this seat
Yes — dbt models or a warehouse view can be the contract surface, with generation pulling on a schedule or on a completed-run webhook.
Every seat we build for
Each role gets its own data reality, its own template families and its own definition of a good outcome. Pick the seat you sit in.
You do not have a content team. You have a spreadsheet, a product and limited runway. That is enough to publish a few thousand pages that answer real searches.
See the buildYou know the query shapes, the entity model and the internal-link plan. What you do not have is a publishing layer that will do it at 40,000 URLs without breaking canonicals.
See the build
The bottleneck in an agency is never the pitch. It is fulfilment cost per client and the senior hours that disappear into repeatable work.
See the build
Your team can produce 12 excellent pieces a month. The keyword plan needs 400. Programmatic covers the pattern work so your writers cover the judgement work.
See the build
You are the one who gets paged when a bulk job locks the database at 2am. So the generation layer needs to be queue-based, idempotent and inspectable.
See the build
The affiliate sites that got hit had one thing in common: pages that existed to hold a link, not to answer a comparison.
See the buildVolume is not your problem. Consistency of markup, byline integrity, archive structure and crawl efficiency across 100,000 URLs is.
See the buildA 12,000-product catalogue is already a programmatic dataset. The question is which slices deserve a URL and which quietly cannibalise the ones that convert.
See the buildThe honest answer for some startups is no. A validation tranche tells you in eight weeks rather than after four quarterly board decks.
See the buildYour product already generates the data these pages need: features, integrations, permissions, limits, changelogs and docs. Most SaaS companies never publish any of it as a page family.
See the buildThe marketing team wants 40,000 pages. You want to know what that does to your infrastructure, your security posture and your team's pager.
See the buildThe board question is never 'how many pages'. It is what this channel returns, when, and what happens to it if you cut headcount.
See the buildWhen pages are generated from data, page families behave like features. They deserve the same discovery, the same metrics and the same willingness to sunset.
See the build
Listings churn weekly. Neighbourhood, school-catchment, commute and price-trend pages do not — and they are what people search before they search a property.
See the build
In YMYL, an unreviewed page is not a small risk. The architecture has to make clinical sign-off a gate, not a nice-to-have.
See the buildLegal content dies in approval, not production. Build the review path into the template and the volume problem solves itself.
See the buildIn finance, a stale number is a compliance incident. Freshness is not an SEO tactic here — it is the product.
See the buildApplicants compare entry requirements, fees, duration, delivery mode and outcomes. Most education sites publish prose about campus life instead.
See the build
Travel search is the most seasonal, most comparison-driven category there is — and the most punishing to generic destination prose.
See the build
Fourteen locations does not mean fourteen copies of one page with the town name swapped. That is the exact pattern that gets filtered.
See the buildYou do not have a content team. You have a spreadsheet, a product and limited runway. That is enough to publish a few thousand pages that answer real searches.
See the buildYou know the query shapes, the entity model and the internal-link plan. What you do not have is a publishing layer that will do it at 40,000 URLs without breaking canonicals.
See the buildThe bottleneck in an agency is never the pitch. It is fulfilment cost per client and the senior hours that disappear into repeatable work.
See the buildYour team can produce 12 excellent pieces a month. The keyword plan needs 400. Programmatic covers the pattern work so your writers cover the judgement work.
See the buildYou are the one who gets paged when a bulk job locks the database at 2am. So the generation layer needs to be queue-based, idempotent and inspectable.
See the buildThe affiliate sites that got hit had one thing in common: pages that existed to hold a link, not to answer a comparison.
See the buildVolume is not your problem. Consistency of markup, byline integrity, archive structure and crawl efficiency across 100,000 URLs is.
See the buildA 12,000-product catalogue is already a programmatic dataset. The question is which slices deserve a URL and which quietly cannibalise the ones that convert.
See the buildThe honest answer for some startups is no. A validation tranche tells you in eight weeks rather than after four quarterly board decks.
See the buildYour product already generates the data these pages need: features, integrations, permissions, limits, changelogs and docs. Most SaaS companies never publish any of it as a page family.
See the buildThe marketing team wants 40,000 pages. You want to know what that does to your infrastructure, your security posture and your team's pager.
See the buildThe board question is never 'how many pages'. It is what this channel returns, when, and what happens to it if you cut headcount.
See the buildWhen pages are generated from data, page families behave like features. They deserve the same discovery, the same metrics and the same willingness to sunset.
See the buildListings churn weekly. Neighbourhood, school-catchment, commute and price-trend pages do not — and they are what people search before they search a property.
See the buildIn YMYL, an unreviewed page is not a small risk. The architecture has to make clinical sign-off a gate, not a nice-to-have.
See the buildLegal content dies in approval, not production. Build the review path into the template and the volume problem solves itself.
See the buildIn finance, a stale number is a compliance incident. Freshness is not an SEO tactic here — it is the product.
See the buildApplicants compare entry requirements, fees, duration, delivery mode and outcomes. Most education sites publish prose about campus life instead.
See the buildTravel search is the most seasonal, most comparison-driven category there is — and the most punishing to generic destination prose.
See the buildFourteen locations does not mean fourteen copies of one page with the town name swapped. That is the exact pattern that gets filtered.
See the buildWant this scoped for data scientists before you commit?
We audit your data, size the first batch, model the economics and tell you honestly when programmatic is the wrong tool for the job.
