Web Scrape

Crawl your website into a Raffle source yourself: choose a scrape plan, exclude pages, and fine-tune the scrape with CSS selectors and sitemaps.

A Web Scrape source crawls the pages of your website and puts them in Raffle, so your widgets can search and chat with them. You set it up yourself under Sources. How the crawler finds pages, and why a page may be missing, is explained in Sources.

Before you start

Check that the pages you want are:

  • published, and not marked noindex or nofollow (unless you turn on Disable Robot Tags, see Scrape options);
  • linked from another page, or listed in your sitemap;
  • not blocked for crawlers in your robots.txt.

If your server only lets known visitors in, see Whitelisting.

Create a Web Scrape source

  1. Go to Sources and click Add new source
  2. Select Web Scrape and click Next
  3. Choose your setup:
    • Recommended gets you started with the standard configuration.
    • Advanced adds two more steps, Scrape options and Advanced settings, for websites that need more detailed configuration.
  4. Enter your source details:
    • Source Title, a name to recognise the source by
    • Knowledge Base, if your account has more than one
    • Your website, the main URL to scrape, e.g. https://www.example.com/. Raffle crawls the pages under it. You can’t change this URL later.
    • Scrape Plan, how the source is kept up to date (see Scrape plans)
    • Privacy Settings, whether the content is Internal or Public
  5. Choose pages to exclude. Raffle reads your sitemap and shows its pages as a tree of areas. Tick the areas or pages you don’t want in the source, or search for one with Search for an area or specific URL. The list is based on the sitemap, so the pages found by the crawl can differ slightly.
  6. With Advanced, continue through Scrape options and Advanced settings.
  7. Click Create & Sync to create the source and start the first crawl, or Create to crawl later.

Scrape plans

Scrape Plan What it does
Weekly The whole website is crawled again every week.
Daily The whole website is crawled again every day. Shown only if your plan includes daily sync.
First crawl only The website is crawled once, when you create the source, and not again.

To refresh single pages between crawls, see Watchlist & Sync.

Change the settings later

On the Sources page, right-click the source or open its options and click Settings. All the steps are there, including Scrape options and Advanced settings, whichever setup you chose at the start. Click Update to save.

Scrape options

Option What it does
Rich Scrape Renders JavaScript before scraping a page. Use it for pages that build their content in the browser. It is more accurate but slower.
Scrape Files Includes files such as .pdf, .docx, .xlsx, .pptx and .txt found on your website.
Parse PDF Titles Uses the title inside a PDF instead of its file name.
Allow URL Parameters Allows scraping URLs with query parameters (?…).
Disable Robot Tags Ignores the noindex and nofollow directives in your pages.
Disable Sitemap Crawling Finds pages by following links only, without your sitemap.
Disable Content De-noising Turns off the automatic removal of repeated content, such as cookie banners, that appears on many pages. De-noising can sometimes remove valid content.

Advanced settings

Which pages are crawled

  • Entrypoints: the URLs the crawler starts from. Add more if parts of your site aren’t linked from the main URL.
  • Allowed Domains: the domains and subdomains the crawler may visit, e.g. www.example.com (no https:// and no /).
  • Sitemap Sources: specific sitemaps to crawl, e.g. https://example.com/sitemap.xml. You can scope each sitemap to one or more Languages; then turn on Also include URLs with no language if pages without a language tag should be kept too. Click Add sitemap source.
  • Included URL Filters: regular expressions for the URLs to include. By default, everything under the entrypoints is included, e.g. ^https://www.example.com/.*$.
  • Excluded URL Filters: regular expressions for the URLs to leave out, e.g. ^https://www.example.com/en.*$. The pages you ticked in Exclude pages are added here.
  • No Follow URL Filters: pages whose links the crawler doesn’t follow.
  • No Index URL Filters: pages the crawler visits to find links, but doesn’t put in the source.

Test your regular expressions with a tool such as regex101.

Which part of each page is used

  • CSS Whitelist: CSS selectors for the part of the page to keep, e.g. main or .article-body. When set, only what matches is used.
  • CSS Blacklist: CSS selectors for parts of the page to ignore, e.g. nav, footer or .breadcrumbs.
  • CSS Title Selectors: a CSS selector whose text becomes the page title, instead of the <title> tag.
  • CSS Metadata Selectors: a CSS selector and a label, to pick up extra information from the page as metadata.

Comment: any additional information about the source.

Next Page

File