Learn / Lead lists
Scraping any website with Clay's AI agent
Clay's AI web scraper will visit any website, read the page, and hand you the one fact you asked for, on every row of a list at once. Hiring signals, case studies, investment criteria, office addresses, all pulled in minutes instead of the hours it would take to open a thousand tabs yourself.
The catch is that the Clay web scraper does exactly what you tell it. A lazy prompt returns lazy data, and a research column with no fallback behind it will happily ship an email with a hole in the middle of it. This is the setup, the prompts, the conditional logic that stops that happening, and where the money actually goes.
First, a list worth scraping
Clay does not source the list, and pointing an AI researcher at the wrong 6,000 companies is an expensive way to be wrong. The pull in this training started in Apollo: decision-maker job titles, US only, a custom employee band of 20 to 100, and a set of domains lifted from an agency directory so the pool was already narrow before a single filter ran. That search URL goes into a cheap scraping tool, in this case Ample Leads, which returned 6.2K contacts. Those tools rotate constantly, so use whichever one is working the week you read this. Download the CSV, upload it to Clay, delete every column you will not use, and you have a table. Sourcing is covered properly in the Apollo guide, and the full data pipeline around this step is the Clay cold email workflow.
The config screen is where the bill gets decided
Add column, add enrichment, search Clay Agent. Three choices on that screen decide everything downstream:
- Mode. Web research sends the agent out to visit pages. Create or modify content works only on text already sitting in your table, which costs a fraction of the same job done live. Most people pick web research for everything and then wonder why the run cost what it cost.
- Model. 4o Mini, every time. Cheapest thing that still does this job properly. The model Clay suggested by default quoted 3 credits per row, which is the gap between a test and a bill.
- Billing. If you have your own OpenAI key, put it in. The research bills to your OpenAI account instead of burning Clay credits, and on a list this size that is the single biggest cost lever on the page.
Then write the prompt (the generate button does a fine job), test it on 10 rows, map the response into a column, and set an only-run-if condition if the column should not fire on every row. Any prompt you run more than twice gets saved as a template, so you are never rewriting the hiring column from scratch again.
Write the prompt like a checklist
The generated prompts come out structured: objective, instructions, then steps. Keep that shape. The hiring column reads like directions rather than a request: go to the website, find the careers or jobs section, go to the LinkedIn page, open the jobs tab, scan every job title for the word marketing, return whether the role exists and what it is called.
Two habits save entire runs:
- Hand it the variables. Company name, website, and LinkedIn URL in the prompt means the agent never has to guess which company you meant.
- Iterate after the sample, not before.Write it, run 10 rows, read the output, fix what actually broke. And when a column comes back wrong twice, rewrite the prompt from scratch instead of patching it. "Find the headquarters" returned city-level noise in the video. "Find the exact address of the headquarters for {{company}}" returned an address.
Four things worth scraping
The signal has to be something your offer answers. Anything else is trivia with a merge tag around it.
| Signal | Where you point the agent | The line it earns |
|---|---|---|
| Hiring | Careers page, then the LinkedIn jobs tab, scanning titles for your keyword | Notice you're hiring for {{role}}. If you're open to alternatives, we built that exact function for a consumer products brand and beat their last in-house hire by 3x in 90 days. |
| Case studies and clients | Our work, case studies, testimonials, portfolio, clients, reviews pages | If we could build lookalike audiences of your best-fit prospects like {{best_client}} and open new opportunities with them, would that be worth a chat? |
| Offer-specific criteria | The site first, the open web second, fed by a company summary column | Based on the information on your website, you guys seem to primarily invest in {{criteria}}. |
| Office location | Exact HQ address, then a second column for the nearest coffee shop | Notice you're also in {{city}}. Had a look at your funnel and have a few ideas, maybe we grab a coffee and I run you through them. |
The last one only works if you are genuinely local to them, so build the list by state or proximity first. Pitching a coffee in Orlando from California is worse than sending nothing. The first two are the workhorses, and the third is the one that reads like you wrote the whole email for one company.
A prompt is cheap to fix. A broken column across 6,200 rows is not.
Conditional logic, so no row ships broken
Run 10 rows of a hiring column and most come back not hiring. That is not a failed column, that is the real state of the market, and it is why every research column needs a second answer behind it and usually a third.
The mechanism is the only-run-if field. On the case study column, set it to run only when the hiring response equals not hiring or not found, and let Clay generate that formula. Now the copy branches: the hiring signal writes the email where it exists, the case study writes it where it does not, and a third fallback catches the rest. Rows that would have gone out with a blank variable never go out at all.
Order matters as much as the condition. Free and cheap columns first, expensive research last, running only on the rows that survived. A blank variable in a live campaign is worse than no personalization at all, because it is the one thing every recipient notices.
The second pass, where facts become a sentence
Scraping the fact is half the job. The other half is a column that never touches the web.
In the private equity example the table carried three layers. A company summary written by 4o Mini off the site, 100 to 200 words on what the firm does, who it serves, and its typical customer base. Then a web research column pulling investment criteria, told which page types to check and instructed to search the open web only if the site stayed quiet. Then a create or modify column that reads both and returns one line:
"Based on the information on your website, you guys seem to primarily invest in founder-owned B2B software companies with 10 to 30 million in ARR."
That column browses nothing, so it costs a fraction of the ones above it, and it is the column that makes an email read like it was written for one firm instead of one variable dropped into a template. Same trick applies to formatting: joining two industries with the word and, or stripping the .com off a scraped company name, is a formula, and formulas are free. Never pay a model to delete a period.
Where this breaks
Three failure modes, all avoidable. The open-web fallback pulls a different company with the same name, which is why the prompt has to confirm the result belongs to that domain. The wrong column gets mapped into the prompt, which happened live in the video and only showed up because someone looked at the output. And the local plays need a local list, not a national one with a coffee shop bolted on.
Then the honest one. This is personalization, and personalization is one input, not the system. The hype around Clay hyper-personalization is BS: list precision plus a real offer do the heavy lifting, and 587 positive replies once came from one buyer profile in a single month with no Clay research behind it. Scrape when the data changes what you can credibly offer, which is exactly what intent signals do. Otherwise send it plain and put the effort in the offer.
tl;dr: point the agent at a URL you trust, prompt it like a checklist, test 10 rows, put a fallback behind every research column, and do the writing in a create or modify column that costs almost nothing.
FAQ
What is Clay's web scraper?
Clay Agent is an AI research column inside a Clay table. You point it at a website or a LinkedIn URL, tell it what to look for, and it visits the page, reads it, and returns the answer for every row. Instead of opening a thousand sites by hand, you write the instructions once and test them on ten rows.
Web research or create-or-modify content: which mode should I pick?
Web research sends the agent out to browse, so use it only when the data is not already in your table: hiring pages, case study pages, investment criteria. Create or modify content works on text you already pulled, which makes it much cheaper. Get the company summary with web research, then create or modify to turn it into an email line.
How much does a Clay agent column cost per row?
It depends on the model and whose credits you burn. The model Clay suggested by default in this run quoted 3 credits per row. Switching to a mini model and plugging in your own OpenAI key bills the research to your OpenAI account instead, which is cheaper. Formula columns cost nothing, so never pay a model to reformat text.
How do I stop Clay from scraping the wrong company?
Point it at a URL you already trust, the prospect's own website or LinkedIn, and it cannot wander. The risk shows up when you let it search the open web as a fallback, because two companies share a name more often than you think. Add an instruction to confirm the result belongs to that exact domain, then spot check ten rows before running the list.
What should I scrape for cold email?
Whatever your offer actually answers. Hiring pages tell you a team is growing in the area you serve. Case study pages tell you which clients they brag about. For a private equity list, investment criteria tells you the exact deal they want. Generic facts make generic emails, so scrape the thing your pitch responds to.
Does scraped personalization actually book more calls?
Sometimes, and not always. One ideal buyer profile with a plain email and a real offer produced 587 positive replies in a single month with no Clay research or personalization behind it. Scraping wins when the data changes what you can credibly offer, like an open role you could fill cheaper. It loses when it decorates the same generic pitch.
If you would rather have the whole thing run for you, lists, research, copy, infrastructure and a setter on the replies, that is what we do: a two-person firm whose previous vendor produced nothing got 42 qualified conversations in their first 30 days. Every training grouped by topic lives on the trainings hub, and the full video for this one is on YouTube.
PS - the most useful column in that whole table was the free one. The formula that formats the scraped text so it sits inside a sentence is what stops the email looking automated, and it costs nothing.