DataForSEO vs a link-only API: what changes in your R pipeline
A teardown of one imaginary pipeline, step by step. We list what moves, what stays, and what you lose by narrowing the source to links.
CranSEO research team ·
The pipeline, stage by stage
Authenticate
Our key travels as a query parameter or an Authorization Bearer header. Free actions need none, so the first cells of a notebook can run for anyone.
Request
One GET per action, with q for the domain. Paging uses page and per (10 to 5000), sorting uses sort and order.
Parse
Every response is JSON with an ok field. Authenticated ones add _credits, showing the cost and the balance left.
Store
Write the tibble and the snapshot date together. The graph refreshes about monthly, so the date is part of the data.
Report
Render from the stored file, not from a live call, so the quarto document gives the same chart tomorrow.
What you gain and what you give up
| Concern | Link-only API | Broader provider |
|---|---|---|
| Scope | Backlinks, referring domains, anchors, top pages, DR | Usually adds SERP and keyword data |
| Billing | Per call, 1 credit = $0.0001, no subscription | Check its own price list |
| Freshness | Roughly monthly refresh | Check its own documentation |
| Rows per call | Up to 1,000, billed once | Varies by endpoint |
| Failure signals | HTTP 401 bad key, HTTP 402 no credit | Varies |
The mistake we see in teardowns like this
Most pipelines fail on the money, not the code. A loop that calls an endpoint once per row turns a cheap job into an expensive one.
With us, a call costs the same for 10 rows or 1,000, so the rule flips: ask for the biggest page you can use. Set per to 1000, sort by what you need, and stop paging when the question is answered.
Deep pagination is capped at 50,000 rows, which is a feature for analysts who are tempted to pull everything. If a domain has more links than that, you probably want toppages or anchors instead.
Compare the spend of your old loop with the single-call version before you celebrate. Sometimes the saving is zero because the old provider already billed per request.
Verdict on this teardown
Narrow the source only if the report is narrow
If most columns in your tibble are links, a link-only API removes parsing code and cost. If half the columns are SERP data, keep your current provider and add ours for the rest.
- Keep the demo cells on free actions
- Pin the snapshot date in the output
- Expect a monthly rhythm, not a daily one
Questions from the teardown
Do I need to rewrite my whole pipeline?
No. Replace only the link stage. Keep your SERP and keyword stages where they are, write the link tibble to its own file, and join the two downstream on the domain column.
Is CORS open for browser notebooks?
Yes. CORS is open, so a Shiny app or an Observable cell can call the free actions directly. Keep paid keys server-side, because anything in browser code can be read by visitors.
How do I test failure handling cheaply?
Send a deliberately wrong key to see HTTP 401 without spending anything. The HTTP 402 case only appears when the balance is empty, so mock it in your tests rather than draining credit.
Will totals match another provider exactly?
No. Every provider crawls differently and refreshes on its own schedule. Compare orders of magnitude and trends, and treat DR here as a separate, in-house log-scale metric.
A short test plan before you switch
Run both sources for one cycle. Same domains, same day.
Take 30 domains you know well, pull referring-domain counts from your current provider and from CranSEO, and plot them against each other on log axes. You should see a rough diagonal, not an exact one, because crawls differ and ours refreshes about monthly; DataForSEO, Ahrefs and Semrush each see a slightly different web.
Read the outliers, not the average. Where our count is far below, check whether the domain is new or recently moved. Where it is far above, check subdomains. The glossary defines what we count as a referring domain, and the runbook shows how to store both pulls side by side.