Firecrawl Inc., a provider of web scraping tools for artificial intelligence agents, today announced that it has raised $75 million in funding.

Los Angeles-based venture capital firm Smash Ventures led the Series B round. It was joined by Y Combinator, Altos Ventures, Nexus Venture Partners, Freestyle and Offline Ventures.

AI agents rely on information from the web for a wide range of tasks. In theory, they can access that data using a regular browser. In practice, however, collecting large volumes of website data involves multiple technical challenges. Firecrawl has built a platform that eases those challenges for developers. 

The first prerequisite to collecting the data an AI agent requires for a task is finding the web page that contain it. Firecrawl provides an agent-optimized search engine that can filter web pages based on their creation date, the type of files they host and other parameters. The company says that the tool is more efficient than search engines geared towards human users.

Some web pages load in multiple phases instead of all at once. When traditional web scrapes encounter such pages, they often attempt to download their contents before loading is complete, which can lead to data quality errors. Firecrawl says that its platform avoids such issues thanks to a so-called smart wait feature.

In many cases, certain parts of a website only load if users scroll, click an interface element or submit a form. Firecrawl says that its platform can perform those actions automatically. Developers can customize how the software interacts with web pages by entering prompts that describe what it should do.

The platform also promises to ease several related data collection tasks. Firecrawl can extract key details from website-hosted PDFs, Word documents and other files that pose a challenge for traditional web scrapers. It’s also capable of monitoring web pages of interest for changes. A software-as-a-service startup, for example, could configure Firecrawl to track updates to a rival’s product pricing page.

The company announced today’s funding milestone alongside a new cloud service called Alexandria. It combines web data with information from third-party providers and datasets curated by Firecrawl. One dataset contains tens of millions of scientific paper abstracts, while another includes a similar number of code and documentation files.

AI agents can use Alexandria to fill gaps in web data. A programming assistant, for instance, could enrich code examples from a developer forum with explanations from one of Firecrawl’s curated technical datasets. 

Agents also have other ways of combining information from multiple sources. However, those approaches often require developers to build a custom connector for each source. Firecrawl says that all of Alexandria’s data is available through a single application interface, which reduces the need for custom code.

The company will use its new funding to enhance the service. According to Firewall, much of the capital will go toward adding information from more third-party data providers. It plans to launch a self-service content licensing system “soon” to expedite the process. 

Image: Unsplash