A few days ago I wrote about a scraper for gathering economic calendar data. Well, I’m back again to write about another aspect of the same project: acquiring earnings calendar data.
I split the task into two parts:
- a Python scraper for the calendar data, which dumps its output to CSV files; and
- a BASH script that consolidates the individual CSV files into a single master file.
In retrospect I’m not sure why I didn’t do the whole thing in Python. However, it works and I’m not going to mess with it.
Scraper
The data were scraped from the Company Earnings Calendar on Yahoo! Finance.
The data are divided by date and each page is accessible via an URL parameter. If there are many announcements on a specific day then the listing is paginated in batches of 100 announcements. Fortunately the pages are statically rendered, which means that I could do the scrape with just requests and BeautifulSoup. Data for each date was aggregated and written to a CSV file.
The scraper accepts two command line arguments:
--week— gather announcements for next week or--month— gather announcements for next month.
In the absence of either argument it will hyst gather announcements for the current day.
Consolidator
This is a simple BASH script that gather all of the individual CSV files and concatenates them together. The only interesting component of the script is stripping off the header for all but the first file so that the header is not replicated.
The resulting CSV file is available here and will be updated daily.