5522
Raw Job Records
This is the core of our data. Generally, models are built off of this data, or aggregates of the data. In general, when using any other file, do a left join with this file - dropping any reference information that does not have a job record (ie companies with no job records).
| Daily File | CSV | Parquet |
|---|---|---|
| Location | Standard/Feeds/Raw Daily Job Records/ | Standard/Feeds/Parquet/Raw Daily Job Records/ |
| File Name | raw_daily_records_YYYY-MM-DD.csv.gz | raw_daily_records_YYYY-MM-DD-part-###.parquet |
| Monthly File | CSV | Parquet |
| Location | Standard/Feeds/Raw Full Job Records v2/ | Standard/Feeds/Parquet/Raw Full Job Records v2/ |
| File Name | raw_job_archive_v2_full_YYYY-MM-DD.tar.gz | raw_job_archive_v2_YYYY-MM-DD-part-####.parquet |
Job Records Table
| Field Name | Data Type | Description |
| hash | string/varchar | The unique identifier for job records. This is used to join job descriptions to job records. |
| title | string/varchar | Job title scraped from the unique url for the job post |
| company_id | string/varchar | Unique identifier for a company-scrape. This is is used to join to reference files with job records. |
| company_name | string/varchar | This is the name of the company pertaining to the company_ID. |
| city | string/varchar | City location for the job posting. |
| state | string/varchar | State or region for the job posting. |
| zip | string/varchar | Postal code for the job posting. |
| country | string/varchar | |
| created | timestamp | The first time this job was observed and scraped. |
| last_checked | timestamp | The most recent time this site was scraped and this job posting was observed. |
| last_updated | timestamp | The most recent time that this site was scraped and observed a change in the job posting. |
| delete_date | timestamp | The most recent time this site was scraped and this job posting was not found. |
| unmapped_location | boolean | If True, this indicates that the job posting location was not able to be accurately identified. |
| url | string | The unique URL for the job posting. |
Raw Job Record Structured Fields
This table provides several additional fields typically found on a job portal or job record for any one job listing. Join this data to the job records table by hash.
What is a structured field?
A structured field is a data point found on a job portal or job record and is defined by the employer consistently across their job listings. The consistency of structured fields allows our web scrapes to capture additional data points not found in the job title or description.
Will every employer have all structured fields?
The structured fields on any job listing are provided by the employer at their discretion. It is not expected that all employers will provide all structured field data.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | Standard/Feeds/Raw Daily Structured Fields/ | Standard/Feeds/Parquet/Raw Daily Structured Fields/ |
| File Name | raw_daily_structured_fields_YYYY-MM-DD.csv.gz | raw_daily_structured_fields_YYYY-MM-DD-part-###.parquet |
| Monthly File | CSV | Parquet |
| Location | Standard/Feeds/Raw Full Structured Fields/ | Standard/Feeds/Parquet/Raw Full Structured Fields/ |
| File Name | raw_full_structured_fields_YYYY-MM-DD.tar.gz | raw_full_extended_fields_YYYY-MM-DD-part-####.parquet |
Structured Fields Table
| Field Name | Data Type | Description |
| hash | string/varchar | The unique identifier for job records. This is used to join job descriptions to job records. |
| reqid | string/varchar | The job ID or requisition ID for the job opening defined by the employer. This field can be used to track a job opening across multiple employer-owned career portals. |
| category | string/varchar |
A classification or group which the job is categorized into by the employer on the career portal. Category could be a team, department, or brand defined by the employer on the job listing or career portal. |
| subcategory | string/varchar |
Sub-category is a secondary or additional category to the primary category. The sub-category field is defined by the employer as a secondary classification for their open roles. An example of sub-category would further define the role under a category. Where a category may be Retail the sub-category could be Shift-Supervisor. |
| address | string/varchar | The address for the role provided on the job listing |
| certifications | string/varchar | Any required certifications or licenses for the role provided on the job listing. |
| posted_date | timestamp | The date shown on the job listing as the posted date or the date the job became available on the career page in YYYY-MM-DD format. |
| close_date | timestamp | The date shown on the job listing as the close date or the date the employer stops accepting applications, in YYYY-MM-DD format. |
| commission_eligible | string/varchar | Information provided on the job listing about commission eligibility. |
| compensation | string/varchar | The compensation amount or range provided on the job listing. |
| contract_length | string/varchar | Information detailing the length of a contracted position. |
| division | string/varchar | The brand or division of a company the job listing falls under. |
| education_requirements | string/varchar | Educational requirements provided on the job listing. |
| employment_type | string/varchar | Describes the role as permanent or temporary. |
| experience_required | string/varchar | Minimum experience required for the role. |
| signing_bonus | string/varchar | Provides any signing bonus offered by the employer on the job listing. |
| shift | string/varchar | What shift or shifts the job listing is hiring for. |
| site_id | string/varchar | The store number or facility associated with the job listing. |
| time_type | string/varchar | Describes the work schedule. Typically full-time, part-time, Internship, Contract, Seasonal or Project based work. |
| travel_requirements | string/varchar | Any travel requirements for the role provided on the job listing. |
| vacancy_count | string/varchar | The count of hires being made for the job listing. |
| work_location | string/varchar | Describes where the work for this role takes place, such as on-site, hybrid, or work from home. |
Raw Job Descriptions
This contains only the job descriptions. To use job descriptions, you will join with Job records on hash. For NLP applications you would typically join this with job descriptions and use a combination of job title from job records file and job description.
| Daily File | XML | Parquet |
|---|---|---|
| Location | /Feeds/Raw Daily Job Descriptions/ | /Feeds/Parquet/Raw Daily Job Descriptions/ |
| File Name | raw_daily_descriptions_YYYY-MM-DD.xml | raw_daily_descriptions_YYYY-MM-DD.parquet |
| Monthly File | XML | Parquet |
| Location | /Feeds/Raw Full Job Descriptions/ | |
| File Name | linkup_job_descriptions_YYYY-M-DD.tar.gz |
Raw Job Descriptions Table
| Field Name | Data Type | Description |
| hash | string/varchar | The unique identifier for job records. This is used to join job descriptions to job records. |
| description | string/varchar | The full job description scraped from the website for the job posting. |
Company Scrape Log
This is a record of when scrapes run and changes to scrapes. The primary use is identifying outliers. This can be used to classify outliers as legitimate or due to a scrape break. This can eliminate noise and false signals in job data.
Scrape changes can provide meaningful information as well as just reducing noise. For example after an influx of financing, a company may change Applicant Tracking Systems (e.g., Charming Charlie coming out of bankruptcy). A change was recognized because a change was made to the scrape to fix it and the documentation of when that change occurred is in the scrape log.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Raw Daily Company Scrape Log/ | /Feeds/Parquet/Raw Daily Company Scrape Log/ |
| File Name | raw_company_scrape_log_daily_YYYY-MM-DD.csv | raw_company_scrape_log_daily_YYYY-MM-DD.parquet |
| Monthly File | CSV | Parquet |
| Location | /Feeds/Raw Full Company Scrape Log/ | /Feeds/Parquet/Raw Full Company Scrape Log/ |
| File Name | raw_company_scrape_log_full_YYYY-MM-DD.csv | raw_company_scrape_log_daily_YYYY-MM-DD.parquet |
Company Scrape Log Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| date | date | The date the change occurred for the scrape. |
| scrape_run_complete | boolean | If True, this indicates the company-id was scraped on this date |
| scrape_changed | boolean | If True, this indicates that the code was modified for the scrape on this date |
Company Ticker Reference
This file shows point-in-time ticker information for company ids. This can be joined to job records to understand mergers and acquisitions.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Company Ticker Reference/ | /Feeds/Parquet/Company Ticker Reference/ |
| File Name | company_ticker_YYYY-MM-DD.csv | company_ticker_YYYY-MM-DD.parquet |
Company Ticker Reference Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| start_date | date | The start date for this row. This is used for joining point-in-time information. Please see the join tutorial for assistance with join. |
| end_date | date | The end date for this row. This is used for joining point-in-time information. Please see join tutorial for assistance with join. |
| ticker_symbol | string/varchar | The ticker symbol |
| stock_exchange_country | string/varchar | The country of the stock exchange that the ticker symbol is traded on. |
| stock_exchange_name | string/varchar | The stock exchange symbol that the ticker is traded on. |
| primary_flag | boolean | If True, this is the primary ticker for the company_id. |
Company ISIN* Reference
This file shows point-in-time ISIN (International Securities Identification Number), mapped via FactSet concordance. Please note, a license is required to receive this file.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Company ISIN Reference/ | /Feeds/Parquet/Company ISIN Reference/ |
| File Name | company_isin_YYYY-MM-DD.csv | company_isin_YYYY-MM-DD.parquet |
Company ISIN Reference Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| start_date | date | The start date for this row. This is used for joining point-in-time information. Please see the join tutorial for assistance with join. |
| end_date | date | The end date for this row. This is used for joining point-in-time information. Please see join tutorial for assistance with join. |
| isin | string/varchar | International Securities Identification Number, mapped using FactSet |
| primary_flag | boolean | If True, this is the primary ISIN for the company_id. |
Company CUSIP* Reference
This file shows point-in-time CUSIP (Committee on Uniform Securities Identification Procedures), mapped via FactSet concordance. This identifier is primarily used for publicly traded organizations in the United States. Please note, a license is required to receive this file.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Company CUSIP Reference/ | /Feeds/Parquet/Company CUSIP Reference/ |
| File Name | company_cusip_YYYY-MM-DD.csv | company_cusip_YYYY-MM-DD.parquet |
Company CUSIP Reference Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| start_date | date | The start date for this row. This is used for joining point-in-time information. Please see the join tutorial for assistance with join. |
| end_date | date | The end date for this row. This is used for joining point-in-time information. Please see join tutorial for assistance with join. |
| cusip | string/varchar | Committee on Uniform Securities Identification procedures number, mapped via FactSet. |
| primary_flag | boolean | If True, this is the primary CUSIP for the company_id. |
Company Sedol Reference
This file shows point-in-time SEDOL (Stock Exchange Daily Official List), mapped via FactSet concordance. This identifier is managed by the London Stock Exchange. Please note a license is required to receive this file.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Company Sedol Reference/ | /Feeds/Parquet/Company Sedol Reference/ |
| File Name | company_sedol_YYYY-MM-DD.csv | company_sedol_YYYY-MM-DD.parquet |
Company SEDOL Reference Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| start_date | date | The start date for this row. This is used for joining point-in-time information. Please see the join tutorial for assistance with join. |
| end_date | date | The end date for this row. This is used for joining point-in-time information. Please see join tutorial for assistance with join. |
| sedol | string/varchar | Stock exchange daily official list number, mapped via FactSet. |
| primary_flag | boolean | If True, this is the primary SEDOL for the company_id. |
PIT Company Reference
This file shows point-in-time company information for a company_id. This can be joined to job records to understand some corporate change (e.g, corporate name change, url changes, etc.). This would be joined to job records or aggregates to be used, dropping any company ids with no job records.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/Raw Daily PIT Company Reference/ | /Feeds/Parquet/Raw Daily PIT Company Reference/ |
| File Name | raw_pit_company_reference_daily_YYYY-MM-DD.csv | raw_pit_company_reference_daily_YYYY-MM-DD.parquet |
| Monthly File | CSV | Parquet |
| Location | /Feeds/Raw Full PIT Company Reference/ | /Feeds/Parquet/Raw Full PIT Company Reference/ |
| File Name | raw_pit_company_reference_full_YYYY-MM-DD.csv | raw_pit_company_reference_full_YYYY-MM-DD.parquet |
PIT Company Reference Table
| Field Name | Data Type | Description |
| company_id | string/varchar | The unique identifier for job records. This is used to join to reference files or aggregates. |
| start_date | date | The start date for this row. This is used for joining point-in-time information. Please see the join tutorial for assistance with join. |
| end_date | date | The end date for this row. This is used for joining point-in-time information. Please see join tutorial for assistance with join. |
| company_name | string/varchar | The name for the company_id. |
| company_url | string/varchar | The URL for the company_id. |
| lei | string/varchar | Legal Entity Identifier |
| open_perm_id | string/varchar | Open source company identifier used for joining to other data. |
| naics_code | string/varchar | Industry classification. This can be used to join to the Bureau of Labor Statistics salary data. |
O*Net-SOC Taxonomy 2019 Reference
This file provides ONET-SOC code by job_hash, currently available in 2010 and 2019 taxonomy which supports databases 25.1 through the latest release, 26.1. ONET is the primary source for standardized occupation information in the US for over 1,000 occupations covering the entire US Economy.
NOTE: 2010 ONet data is available up to October 1st, 2022.
| Daily File | CSV | Parquet |
|---|---|---|
| Location | /Feeds/ONet Taxonomy 2019 Daily v2/ | /Feeds/Parquet/ONet Taxonomy 2019 Daily v2/ |
| File Name | onet_taxonomy_2019_daily_v2_YYYY-MM-DD.csv | onet_taxonomy_2019_daily_YYYY-MM-DD.csv |
| Monthly File | CSV | Parquet |
| Location | /Feeds/ONet Taxonomy 2019 Full v2/ | /Feeds/Parquet/ONet Taxonomy 2019 Full v2/ |
| File Name | onet_taxonomy_2019_full_v2_YYYY-MM-DD.csv | onet_taxonomy_2019_full_v2_YYYY-MM-DD-part-###.parquet |
ONET Taxonomy Table
| Field Name | Data Type | Description |
| hash | string/varchar | The unique identifier for job records. This is used to join job descriptions to job records. |
| onet_occupation_code | string | The ONET classification of the job. It is one way of normalizing job titles. |
*Copyright © 2020, American Bankers Association CUSIP Database provided by S&P Global Market Intelligence LLC. All rights reserved.
