SiteCrawler

SiteCrawler answers the very first question a new customer has: "how do I tell this platform about my business?". The answer is: you don't. You give us a URL.

The crawl starts on the homepage. Metadata is parsed, the logo located, the business name extracted, contact info captured. Then the crawler navigates to "services", "products", "about", or their equivalents and harvests up to thirty items with names, prices, and images.

Progress streams live. The user sees stages run in real time, which kills the anxiety of "is anything happening?" and builds trust before the first interaction. If the site sits behind bot protection, the crawler has specialised mechanics to politely negotiate around it.

Parsing is not a regex job. A modern AI model handles content separation — distinguishing real copy from navigation, ads, cookie banners, and boilerplate. The output is therefore clean and usable, not a dump of HTML noise.

The artefact is a polished business card: name, description, logo, full service and product lists. The owner can open it, share the link, hand it to a partner. In parallel, the crawler pushes the same data into SetupWizard, so the user never has to copy anything manually.

Every language AIM supports is covered. The output card is emitted in the language of the source site, and if that site is written in an RTL script the card is rendered right-to-left automatically. The crawler hub lists every scan the user has run, with timestamps and actions — including re-crawl, which refreshes the business profiles of every chat the user owns.

Admins can see every scan across the platform to help users debug issues. A JSON export is available for anyone who wants to take the data elsewhere. SiteCrawler is built to be usable on its own, but it shines when connected: every new user's journey on the platform begins with a single URL here.