Web Scraping Specialist

About the Company

We build infrastructure that delivers massive amounts of web data to the companies training the world’s most powerful AI models. We power and support Grass, a bandwidth-sharing network that operates a massive distributed crawler, providing unique access to high-quality public web data at global scale.

Responsibilities

• Write, test, and refine code that extracts data from various online sources
• Perform data retrieval tasks, handling pagination and dynamic content loaded with AJAX
• Clean and format extracted data to meet quality standards
• Store and manage scraped data in appropriate databases, optimizing for speed and integrity
• Monitor scraping processes and resolve issues to maintain continuous data flow

Requirements

• Demonstrated ability to extract data from complex websites with minimal supervision
• Proficiency in Python or JavaScript
• Experience with BeautifulSoup, Scrapy, or Selenium
• Knowledge of asynchronous programming, multithreading, and distributed scraping
• In-depth knowledge of HTML, CSS, JavaScript, and the DOM
• Experience with NoSQL databases such as MongoDB or Cassandra
• Experience with cloud services like AWS, Google Cloud, or Azure

Preferred Qualifications

• Ability to apply machine learning algorithms for data cleaning or categorization
• Active participation in open-source projects related to web scraping or data processing