The Problem With Manual Lead Scraping
Most sales teams still find leads the hard way: open a directory site, scan for businesses that fit, copy the name and email into a spreadsheet, and then spend another hour deciding if the lead is worth a call. It is slow, it is error-prone, and the best leads go cold before anyone reaches out.
I built a system that replaces the entire manual loop. It scrapes public directories on a schedule, enriches every lead with business data from secondary sources, runs each lead through an AI qualification model, and pushes the qualified ones into a CRM with a score and a reason. The sales team only sees leads that are ready to talk.
Here is the architecture, the technical decisions that mattered, and the code structure that makes it reliable.
The Pipeline: Crawl, Enrich, Qualify, Route
The system has four stages, and each one is a separate, independently testable step. If any stage fails, the pipeline pauses and alerts rather than silently dropping leads.
Stage one is the crawler. It targets public directory sites and extracts structured business data: name, address, phone, website, category, and any publicly listed reviews or ratings. I use Playwright for sites that require JavaScript rendering and plain HTTP requests for static pages. Rate limiting is non-negotiable — every request respects a configurable delay, and proxy rotation handles IP-level blocks.
Stage two is enrichment. The raw scraped data is thin, so I hit secondary APIs to fill in gaps: business registration data, social profiles, employee counts, and technology stacks. This turns a name-and-email into a business profile the AI can actually evaluate.
Stage three is AI qualification. An LLM receives the enriched profile and the client's ideal customer criteria, then returns a structured score: hot, warm, or cold, with a one-sentence reason. This is the step that replaces human judgment on first-pass filtering.
Stage four is routing. Hot leads go to the CRM immediately with the score and reason attached. Warm leads enter a nurture sequence. Cold leads are archived. Nothing gets lost.
The key insight: the AI does not replace the sales team. It replaces the three hours they spend deciding which leads are worth calling.
The Qualification Prompt: Why Structure Matters
The qualification step is where most AI scraping projects fail. They send the lead data to an LLM and ask 'is this a good lead?' — and get vague, inconsistent answers.
The fix is structured output. I define the exact schema the AI must return, including the score, the confidence level, and the reasoning. The prompt includes the client's ICP (ideal customer profile) as a rubric, not a vague description.
Qualification prompt structure
{
"system": "You are a lead qualification analyst. Score each lead against the ICP below.",
"icp": {
"industry": ["SaaS", "Agencies", "E-commerce"],
"team_size": "5-50",
"signals": ["active hiring", "recent funding", "manual processes visible"]
},
"output_schema": {
"score": "hot | warm | cold",
"confidence": "high | medium | low",
"reason": "one sentence",
"recommended_action": "call | nurture | archive"
}
}Reliability: Idempotency and Error Handling
A scraping pipeline that runs on a schedule will encounter the same leads repeatedly. Without idempotency, you get duplicates in the CRM and wasted qualification calls. I solve this with a deduplication layer: every lead gets a fingerprint based on name + domain, and the pipeline checks the fingerprint store before processing.
Error handling is equally important. Each stage wraps its work in a try-catch with structured logging. If the crawler fails on a page, it logs the URL and moves on. If the AI qualification returns malformed JSON, the lead is flagged for manual review rather than silently dropped. Every run produces a summary: leads found, leads qualified, leads routed, errors encountered.
The pipeline runs on n8n, which gives me visual monitoring, retry logic, and webhook triggers for real-time processing when a new lead source is added.
What This Looks Like in Practice
A typical run: the crawler visits three directory sites, finds 200 businesses, enriches 180 of them (20 had incomplete data), qualifies them in under two minutes, and pushes 12 hot leads to the CRM with scores and reasons. The sales team opens their CRM and sees 12 leads that are ready to call, each with a note explaining why they scored hot.
The entire system runs unattended. The only human involvement is reviewing the weekly summary and adjusting the ICP criteria as the business evolves.
The ROI is not just time saved — it is the leads that would have gone cold during the three hours your team spent manually qualifying.