Most scraping projects don’t fail on day one. The first extraction works, the demo looks good, and the contract gets signed. Then a target site changes its layout, a bot-protection page starts returning empty fields, and nobody is quite sure whose job it is to fix it.
That is the real test when you choose a web scraping vendor: not whether it can pull data once, but whether it keeps delivering accurate data in month six. This guide covers how to evaluate a provider, which US legal questions to settle before you sign, how pricing works, and how to run a pilot that tells you the truth.
Quick answer: To choose a web scraping vendor, check seven things: how it defines and tests data quality, who fixes breakages, how it handles JavaScript-heavy and protected sources, its approach to US legal and privacy risk, delivery formats and integrations, who owns the code and data, and how the team communicates. Then test your shortlist with a pilot on your hardest source before committing.
What a scraping vendor actually does
A scraping vendor builds and runs the systems that collect public web data for you and hands it over in a usable shape: a CSV or JSON file, an API feed, or a database table. Writing the scraper is the smaller part. The harder work is everything around it: proxies, scheduling, retries, validation and monitoring that keep the pipeline alive after launch.
Providers fall into two broad groups. Tool and API vendors sell you the infrastructure, and your own developers do the engineering. Managed or custom providers do the engineering and take responsibility for delivery. Neither is better in general. The right one depends on how much of the work you want on your own team.
Do you need an outside vendor at all?
Sometimes the honest answer is no. Before you start vendor calls, compare the five usual ways to get web data.
| Option | Works best when | Watch out for |
| Official API or licensed dataset | The source offers one and it covers the fields you need | Missing fields, rate caps, per-call costs |
| No-code scraping tool | A few simple pages and an analyst who will run it | Logins, heavy JavaScript, anything at scale |
| Scraping API or proxy platform | You have developers and want to skip proxy and unblocking work | Parsing, QA and maintenance still land on your team |
| In-house team | Scraping is core to your product and you have engineers to maintain it | Upkeep quietly eats engineering time as sites change |
| Managed or custom provider | Many sources, protected sites, documents, or no spare engineering capacity | Lock-in if you don’t own the code; wide quality gaps between vendors |
Plenty of mature pipelines mix approaches: an official feed where one exists and scrapers for everything else. Our comparison of scraping and data APIs walks through how to split the work.
The hidden cost of keeping scrapers alive in-house
Building a first scraper is rarely the expensive part. Keeping twenty of them accurate for two years is. When teams price an in-house build, they usually count the developer time to write the code and forget the recurring work that follows it.
That recurring work has a familiar shape:
- Layout changes. Selectors break and fields shift. Someone has to notice and fix them, often urgently, because a downstream report is already wrong.
- Sites tighten bot protection, so proxy pools, browser behavior and retry rules need regular tuning.
- Without alerts for empty fields and sudden drops in record counts, bad data flows downstream without anyone noticing.
- Schedulers, queues, storage and logs need an owner, plus someone on call when a run fails overnight.
- If the engineer who wrote the scrapers leaves, most of the understanding of why each one works the way it does leaves too.
A rough test helps: multiply the number of sources by how often they change. A handful of stable sources suits an in-house script. Dozens of sources that change monthly start to behave like a small product with its own maintenance backlog, and that is usually the point where a specialist vendor costs less than the engineering hours you would otherwise spend. If scraping is the core of what you sell, owning it in-house can still be the right call.
Seven checks to run before you shortlist a provider

1. Data quality, defined in numbers
“We deliver clean data” is not a standard. Ask how accuracy is measured (per field or per record), what the target is, and what happens when a batch misses it. A capable team can describe its checks: limits on empty required fields, format and range validation, duplicate detection and freshness windows. The right bar also depends on the job. Exploratory research can tolerate gaps that a pricing feed cannot.
Ask: How do you measure accuracy, and can I see a QA report from a past project?
2. Who fixes it when a site changes
Sites change layouts, add bot protection and move content to new endpoints. A scraper nobody watches will quietly return blanks. Find out who monitors for failures, how fast you are told, and whether maintenance is part of the agreement or a separate invoice each time.
Ask: What alert fires when a field goes missing, and what is your usual turnaround on a fix?
3. Your hardest sources, not the easy ones
Any provider can scrape a static page. The differences show up on JavaScript-rendered sites, pages behind bot protection, infinite scroll, mobile-app endpoints and PDFs. Ask which tools they use for each: Scrapy for large crawls, Playwright or Selenium for dynamic pages, OCR for scanned documents. Be wary of anyone who promises to get past any block on any site. Good teams tell you where the limits are.
Ask: Which of my sources do you expect to be difficult, and why?
4. A written position on legal and privacy risk
Ask how the provider treats terms of service, login-protected content and personal data, and who carries which responsibility under the contract. A vendor with a clear written position is a safer bet than one that says scraping is “always legal.” The next section covers the US basics.
Ask: What will you refuse to collect, and where does your responsibility end and mine begin?
5. Delivery and integration
Data in the wrong shape just moves the work to your analysts. Confirm the formats (CSV, JSON, Excel, API, SQL or NoSQL tables), the schedule, schema versioning, and whether history stays consistent when a field changes. If you need a live view rather than a nightly file, our guide to real-time data extraction explains what changes when freshness matters.
Ask: How will the data reach my systems, and what happens to old records if the schema changes?
6. Ownership and security
Settle who owns the scraper code, the infrastructure and the data before work starts. Some providers run everything on their own platform, which is fine until you want to leave. Ask about NDAs, how target lists and credentials are stored, and what you receive if the engagement ends: source code, documentation, run books.
Ask: If we part ways in a year, what do I walk away with?
7. Communication and continuity
Scraping projects are full of small decisions. Which fields count? How are product variants handled? What happens when a site shows different prices by region? You want a team that asks these questions and writes the answers down. Check time-zone overlap with your working hours, who your point of contact is, and whether the engineers who scoped the work are the ones who build it.
Ask: Who will I talk to week to week, and when are they online?
US-based or offshore: does vendor location matter?
Buyers in the US often ask whether the vendor should be US-based. The honest answer is that location matters less than three things it sometimes affects.
Working-hours overlap. Scraping problems tend to arrive as emergencies: a feed went empty, or a competitor’s site changed overnight. Ask for the exact hours an engineer is online in your time zone and how out-of-hours failures are handled.
Accountability in the contract. Check which entity you are contracting with, which law governs the agreement, and how intellectual property transfers to you. Those terms matter more than the address on the invoice.
Data handling. Ask where scraped data and credentials are stored, who can access them, and how deletion works when the engagement ends.
Offshore teams can be a strong fit when overlap, communication and contract terms are handled properly, and a poor fit when they are not. Kanhasoft’s engineering team is based in India and works with US clients, so we would rather you ask these questions than take our word for it. A US address does not prove the engineering happens in the US either. Whoever you hire, ask who actually writes and maintains the code, and speak to that person before you sign.
US legal and compliance questions to settle before you sign
This is orientation, not legal advice. Use it to know what to ask your counsel and your vendor.
Public pages and the federal computer-crime law. The statute at the center of early scraping fights is the Computer Fraud and Abuse Act. After the Supreme Court’s 2021 decision in Van Buren and the Ninth Circuit’s rulings in hiQ v. LinkedIn, courts have generally treated collecting pages anyone can view without logging in as outside that law’s reach.
Contract claims are a separate risk. hiQ itself ultimately lost on contract grounds tied to LinkedIn’s user agreement. In January 2024, a federal court in Northern California held in Meta v. Bright Data that Meta’s terms did not bar logged-out scraping of public Facebook and Instagram pages. It is one district-court decision, and a related claim was still pending when law firms analyzed it, so treat it as a signal, not a license.
Logins change the picture. If you hold an account, you have accepted its terms, and those terms can restrict automated access. Any source that needs a login should be discussed openly with your vendor and your lawyer first.
robots.txt is a convention, not a permission slip. The IETF’s Robots Exclusion Protocol standard states that its rules are not a form of access authorization. A careful provider still reads the file, respects reasonable crawl limits, and tells you when a source’s rules conflict with what you are asking for.
Personal data needs its own review. If the data includes names, emails or anything tied to an individual, state privacy laws such as California’s CCPA and CPRA, and GDPR where EU residents are involved, may come into play depending on the data and how you use it. Ask what the vendor collects, what it strips out and how long it keeps it.
How web scraping pricing works
Quotes vary widely because projects vary widely. These are the models you will run into:
- Fixed project fee for the build, plus a monthly maintenance fee.
- Monthly fee per source or per scraper, often with a record or frequency cap.
- Usage-based pricing per request, record or credit, common with API platforms.
- Dedicated engineer or time-and-materials for ongoing work where scope keeps changing.
What actually moves the number is less about the model and more about the work:
- How many sources you need, and how well each one is protected
- JavaScript rendering, logins, CAPTCHAs and geo-targeting
- Refresh frequency and total volume
- PDFs, scans and other documents that need OCR or AI-assisted parsing
- Depth of validation and QA
- Integrations, dashboards and alerts
- Monitoring, maintenance and response-time commitments
A very low quote usually leaves something out, and it is typically monitoring, maintenance or QA. Ask for a line-by-line scope. Serious providers also scope against your actual URLs before quoting, because a public price list reflects the easy case.
Red flags that should slow you down
- A promise to access any site, or to deliver without ever breaking
- A demo built only on easy, static pages
- No written position on terms of service or personal data
- No sample, QA report or reference you can check
- A quote with no line for monitoring, or maintenance only “on request”
- Code and data that live only on the vendor’s systems, with no exit plan
Run a pilot before you commit
A short pilot reveals more than any sales call. A practical way to run one:
- Write a one-page data brief. List the fields, sources, refresh rate, delivery format and what “correct” means for each field.
- Choose the hardest two or three sources. Test the difficult ones, because the easy ones will work for everybody.
- Ask for a sample with a QA report. Look at empty-field rates, duplicates and a spot-check against the live pages.
- Ask what happens when something changes. Have the vendor walk through a recent layout change and how it was caught and fixed.
- Read the contract for the unglamorous parts. Ownership, maintenance, data handling and exit terms.
We run our own projects this way. Sample data comes first, and you continue only if it meets the brief.
What you should receive at handover
Many buyers only discover what they were missing when something breaks six months in. Agree the handover list before the build starts, so “delivered” means more than a first data file.
- Source code and a short run book explaining how to deploy and restart it
- A field dictionary: every column, its type, and how blanks and variants are handled
- The monitoring and alert setup, including who receives alerts and what triggers them
- A QA report from the final acceptance run
- Documentation of proxies, schedules and any third-party services the pipeline depends on
- Clear notes on known limits, such as sources that block often or fields that are only sometimes present
If a vendor hesitates to put this in writing, that tells you how they will behave when you need help later.
How these checks play out on real projects
Three of our builds show why the criteria matter.
- Healthcare event data across 1,000+ US websites. Sources mixed JavaScript-heavy pages with PDFs in inconsistent formats, so the pipeline pairs browser automation with AI-assisted PDF parsing. We report a 95%+ extraction success rate. Lesson: document-heavy sources need a plan beyond HTML.
- Dental product stock tracking across 8+ platforms. Stock and pricing are tracked at variant level (size, type), with proxy rotation to handle anti-bot protection and daily automated reporting. We report 98%+ data accuracy. Lesson: define “correct” at the level you actually make decisions.
- Hotel rate benchmarking from Booking.com. Room rates and availability for 30+ competitors per hotel across 365-day windows, delivered as JSON through an API into hotel dashboards. Lesson: delivery format decides whether the data gets used.
Conclusion: Judge a Provider by Month Six, Not Day One
The right web scraping vendor is not the one with the boldest claims or the lowest quote. It defines data quality in numbers, takes ownership of breakages, is honest about difficult sources and legal limits, and lets you test all of that before you commit. Use the seven checks above to build a shortlist, then let a pilot on your hardest source make the decision.
How Kanhasoft Can Help
If you are weighing build, buy or outsource for a scraping project, Kanhasoft can review your sources and data requirements and tell you which approach fits, including when an existing API or tool is enough. Our custom web scraping services start with free sample data, so you can judge the output before you commit.
Ready to test a provider on your hardest source?
Share your target sites, data fields and refresh frequency. We’ll scope the project and send sample data before you commit.
