Why Raw Diffs Are Useless
Most competitor monitoring tools do one thing: they take a snapshot of a page, compare it to the previous snapshot, and highlight what changed. The problem is that most changes are noise — a CSS update, a footer tweak, a blog post that does not matter to your strategy.
What you actually want is not 'what changed' but 'what changed that matters.' A pricing page update matters. A new product launch matters. A job posting for a role that signals a strategic pivot matters. A font change does not.
I built a crawler that solves this by combining change detection with AI summarization. It watches target pages on a schedule, detects changes at the DOM level, filters out noise, and then uses an LLM to classify and summarize the meaningful changes into an intelligence briefing.
The Detection Layer: DOM Hashing and Content Extraction
The first layer is pure technical: take a snapshot of the target page's DOM, extract the meaningful content (text, structured data, images), and hash it. On the next run, compare the new hash to the stored hash. If they differ, extract the new content and compute a diff.
I use Playwright for JavaScript-heavy sites and cheerio for static pages. The key decision is what to hash. Hashing the entire DOM is too noisy — every ad rotation triggers a change. Instead, I extract specific content zones: the main article body, the pricing table, the product listing, the team page. Each zone gets its own hash.
This means I can monitor a pricing page and only get alerted when the pricing table actually changes, not when the header banner rotates.
Zone-based change detection
// Each monitored page has a config that defines content zones
const zones = [
{ selector: ".pricing-table", name: "pricing" },
{ selector: ".product-list", name: "products" },
{ selector: "article.main", name: "blog-latest" },
];
for (const zone of zones) {
const content = await page.textContent(zone.selector);
const hash = createHash(content);
const previous = await getStoredHash(targetUrl, zone.name);
if (hash !== previous) {
changes.push({
zone: zone.name,
previousContent: await getStoredContent(targetUrl, zone.name),
currentContent: content,
});
await storeHash(targetUrl, zone.name, hash);
}
}The Intelligence Layer: AI Classification and Summary
Once the detection layer identifies a real change, the intelligence layer takes over. It receives the before-and-after content for each changed zone and asks the LLM to classify the change and summarize its significance.
The classification uses a fixed taxonomy: pricing_change, product_launch, hiring_signal, partnership, content_update, cosmetic. Only the first four trigger alerts. The summary is written in plain language and includes the strategic implication — not just 'they changed their pricing' but 'they lowered their entry tier by 30%, which may indicate a push for volume over margin.'
I feed the AI additional context: the competitor's recent changes (so it can spot trends), the client's own positioning (so it can flag competitive threats), and the industry context. This turns a raw diff into a briefing a strategist would write.
The AI does not just detect change — it explains why the change matters to your specific business.
Scheduling and Alerting
The crawler runs on a configurable schedule per target: pricing pages might be checked daily, blog pages weekly, careers pages bi-weekly. I use cron-style scheduling in n8n with exponential backoff on failures.
Alerts are delivered through the client's preferred channel: email digest, Slack message, or dashboard notification. Each alert includes the change classification, the AI summary, a link to the diff, and a confidence score. Low-confidence classifications are flagged for human review rather than sent as alerts.
The system maintains a change history for each target, which enables trend analysis: 'Competitor X has made three pricing changes in six months, each time lowering the entry tier.' That kind of pattern is invisible in raw diffs.