You are using an unsupported browser. Please update your browser to the latest version on or before July 31, 2020.
close
You are viewing the article in preview mode. It is not live at the moment.
Home > RAW > Support > RAW Data FAQs > Scrape Runs: Frequency, Duration, and Timing
Scrape Runs: Frequency, Duration, and Timing
print icon

Frequency: How Often Do Scrapes Run?

By default, a scrape runs once every 48 hours after the previous scrape run has completed. There are, however, a few specific exceptions to this rule.

  • Scrapes for S&P 500 companies and their subsidiaries have a run frequency of every 24 hours
  • A very small number of scrapes have their frequency lowered to 168 hours (one week) to avoid volatility issues

 

If a scrape does not fall under one of these four aforementioned exceptions, assume it has the default 48 hour frequency.

 

Duration: How Long Does It Take A Scrape To Run?

How long a scrape can take to run from start to finish varies astronomically and most of the factors are not something we can control. The following is a list of some—but not all—variables that can affect scrape run duration.

 

  • How many jobs are there?
  • Are the jobs embedded in the page itself or being loaded in via API?
  • Are all the jobs on one page or multiple pages?
    • If multiple pages, how many pages are there?
    • If multiple pages, how many jobs are on each page?
  • Are the jobs split between multiple "portals" based on industry/category/location?
    • If so, how many different ways are the jobs split?
    • If so, are the different portals using the same ATS/solution or different ones?
  • Are we capturing the jobs with pattern matching or decoding another format such as XML or JSON?
  • Is the scrape running Full each time or Delta?
  • Does scraping the site require us to use a proxy?
  • Are the jobs on the employers website itself or are they using an ATS?
    • If they're using an ATS, which one are they using?
  • Does the site have issues with stability or receiving lots of page requests in a short period of time?
  • Is the site under maintenance?
  • What's the turnover rate on the company's listings?

 

Timing: When Do Scrapes Run?

There isn't a specific time when any given scrape will run. Also, as can be extrapolated from the two previous sections, the precise time a scrape runs is likely to change from day to day. Given that the next scrape run starts a number of hours after the previous run completes and how long the scrape takes to run can be anywhere from a matter of seconds to multiple hours, it may vary wildly.

Feedback
1 out of 1 found this helpful

scroll to top icon