ESSnet WP1: Webscraping Job Vacancy advancement review - France

Transcription

1 ESSnet WP1: Webscraping Job Vacancy advancement review - France September , Thessaloniki Paul ANDREY, Dares

2 Job Vacancy data

3 Institutional actors - Official job enrollment declarations - Job Vacancy Survey - Survey on enrollment procedures - Data on registered job seekers - Collected job offers - Job offers from online partners

4 Hundreds of online actors

5 Webscraping Job Offers

6 Webscraping solution Python 3 (mainly standard library) General Framework (simple, stable & scalable) Configurable (control over all the specifics)

7 Webscraping Framework Step one : crawl job offers search results page {1,, n} Job offer title some details Job offer title some details {url, 1, url } m Job offer title some details

8 Webscraping Framework Step two : scrape job offers details pages {url, 1, url } m Job offer title Structured information Structured information Structured information Structured information Long and interesting job offer description. { } title :, info ₁ :, info 2 :, desc. :...

9 Webscraping Framework Two possible models urls Search results crawling iterate over search pages Search results crawling iterate over search pages urls Job details scraping urls Job details scraping iterate over stored urls details details offers Sequential model offers Embedded model

10 Webscraping Program Prototype GUI

11 Webscraping Program Search results crawling

12 Webscraping Program Embedded model running

13 Webscraping issues Maintainance costs (scraped sites evolution) Time costly (depending on adopted policy) Remotely closed connections

14 Cleaning textual data

15 Cleaning textual data The task Our great company, «Super café» is recruiting! You will have a lot of tasks: -You will prepare and serve coffee -You (and your colleagues ) will be friends.and more! great company super café recruit have lot task prepare serve coffee colleagues friends more

16 Cleaning textual data Text cleaning framework - Normalization - special characters translation - punctuation spacing (or removal) é é coffee <br />-You coffee. You - Lemmatization - [optional part-of-speech tagging] - replace identified forms by their lemma recruiting recruit [verb] tasks task - Filtering - stopwords removal - unwanted characters removal prepare and serve prepare serve coffee. Colleagues coffee colleagues

17 Cleaning textual data Lemmatization tools Morphalou - word forms dictionary - French only - xml corpus TreeTagger - POS-tagging tool - multiple languages - binary using Perl => used on titles => used on descriptions

18 Cleaning textual data Typical cleaning steps Our great company, «Super café» is recruiting! You will have a lot of tasks: -You will prepare and serve coffee Our great company, Super café, is recruiting! You will have a lot of tasks: You will prepare and serve coffee. You and your colleagues will be friends. And more! -You (and your colleagues ) will be friends.and more! great company super café recruit have lot task prepare serve coffee colleague friend more Your great company, super café, be recruit! You have a lot of task: you prepare and serve coffee. You and your colleague be friend. And more!

19 Classifying job offers

20 Classifying job offers Job nomenclatures FAP Professional family 22 professional domains 87 aggregate professional families 225 detailed professional families ROME Operational repertory of jobs and professions 14 professional families 110 professional domains 532 «fiches métier» (job card)

21 Classifying job offers ROME matching technique Job title Waiter / waitress in a tea room ROME code G Barman / barmaid G job titles / 535 codes ~ 4000 words long vocabulary (once cleaned) ROME job titles referential

22 Classifying job offers ROME matching technique waiter tea room downtown filter words belonging to the reference vocabulary waiter / tea / room select job titles containing one of those words Similarity.875 Cleaned job title waiter tea room ROME code G compute titles similarity... waiter restaurant G1803 select best ROME Score Matched G waiter tea room

23 Classifying job offers ROME matching similarity function similarity(t ₁, t ₂ ) :=.5 * [ {w ₁ in t ₁ } f ₂ (w ₁ ) / Card(t ₁ )] +.5 * [ {w ₂ in t ₂ } f ₁ (w ₂ ) / Card(t ₂ )] t ₁ : words in the offer s title t ₂ : word in the reference job title f ₁ (w) : w ₁ in t ₁ ; jaro_winkler(w, w ₁ ).9 f ₂ (w) : w ₂ in t ₂ ; jaro_winkler(w, w ₂ ).9 jaro_winkler : Jaro-Winkler edit distance similarity(«waiter tea room downtown», «waiter restaurant») =.5 * (1 / 4) +.5 * (1 / 2) =.375

24 Classifying job offers ROME matching planned improvements - Improve offers titles cleaning (company names, etc.) [brand] employee employee - Mine description for additional details [brand] is recruiting a waiter employee waiter employee - Use a training set to improve the similarity function

25 Classifying job offers Bag of words approach - Document-term matrix based on textual information - Potential TF-IDF weighting - Trained classifier to assign aggregate codes based on such vectors work in progress!

26 Towards statistical production

27 Some concerns about aggregation - How to measure data representativity? - How to handle selection biases when targetting entreprise websites? - How to monitor structural evolutions of (online) job offers? thrilling challenges ahead!

28 Thank your for your attention.