Downloading content from SharePoint can be tricky. It might appear that a Microsoft login is required. You might attempt to automate the login process but run into other challenges.
If you are lucky though it might not be all that hard.
For the purpose of illustration I’ll document the process I went through for downloading a CSV document. The URL for the document is stored as URL in a module called const.py.
Simple Static
You can try simply retrieving the page via a static request. It’s important to follow redirects because the SharePoint URL will likely result in a 302 redirect.
import httpx
from const import URL
response = httpx.get(URL, follow_redirects=True)
response.raise_for_status()But this is not useful because the page is a heap of JavaScript, which of course is not executed. You don’t get the actual content of the CSV document. The JavaScript needs to be executed. Perhaps browser automation with Playwright or Selenium would help?
Full Dynamic
The CSV file is presented as a Spreadsheet in Excel on SharePoint. Since this is clearly a dynamic page (via the JavaScript mentioned before) one could use Playwright.
from playwright.sync_api import sync_playwright
from const import CHROME_PATH, URL
def main():
with sync_playwright() as p:
browser = p.chromium.launch(headless=False, executable_path=CHROME_PATH)
context = browser.new_context()
# Extend timeout (currently on a slow connection).
context.set_default_timeout(60000)
page = context.new_page()
page.goto(URL)
# Do the actual download here...
browser.close()
if __name__ == "__main__":
main()That code simply (and successfully) opens the page. You’d then need to automate the clicking of some buttons to export the data as CSV (or ODS or PDF). This might work. But it would probably be fragile. And you might get kicked out and confronted with a login page.
Downloading Done Right!
Appending the download parameter to the URL does the job.
from io import StringIO
import httpx
import polars as pl
from const import URL
url = f"{URL}?download=1"
response = httpx.get(url, follow_redirects=True)
response.raise_for_status()
df = pl.read_csv(StringIO(response.text), separator=";")
print(df.shape)(29, 22)Great success.
This is the original motivation for the post:
https://puertoaviles.sharepoint.com/:x:/s/WEB_DOCS/EQiczIHvWF1Dk0Oc7vD4tJcBxmfQDQsmSCLn8RRmgGY-LA https://puertoaviles.sharepoint.com/:x:/s/WEB_DOCS/EZ3B4_gDnDFMrMVroa-s1PQBjn0iX9Gif2IaVmKBzNeLpQ
Here’s a shortcode for one of those links (not currently used in post): https://bit.ly/sharepoint-spreadsheet.
Links are from this page: https://www.puertoaviles.es/en-US/Prestaci%C3%B3n-de-servicios/Buques-en-el-Puerto/
This is some useful background:
https://sharepoint.stackexchange.com/questions/240021/sharepoint-online-enable-direct-download-instead-of-opening-document-when-clic