You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 

1.6 KiB

Single file link data formatting (something)

It's split into several parts from out of which only 1 is fully controllable by you. Those parts are:

  1. Matching URL based on it's domain name and setting up the needed functions for upcoming tasks
    • this function is your filehost link parser
    • if it doesn't match with any of the hosts it will discard the link
    • before it processes the URL it gets verified by urllib
  2. Fetching the contents of URL by calling a function url2bs(link) that will return a BeautifulSoup object
  3. Passes the object to your filehost parser function and assigns it to a private variable called result
    • Task of your function is to return a 3-element array [state, url, headers]
    • state - an integer that communicates the state of the link, it should contain values ranging from 0 to 2
      • 0 = no data returned
      • 1 = file link found
      • 2 = folder link found
        • if folder link offers a zip download function, please point it there appropriately and use state = 1 instead
    • url - return a direct link to the file
    • headers - headers necessary to install via _declare_opener(additional_header), this variable is optional and it's there for future-proofing
  4. Passes result to a file downloader function, it will pass a referrer (the referrer is the link itself) by default - If the result has state = 2, it will pass the result once more to a specialized function for parsing folders (your job as well) - The function will bypass the fifth step as well, you must call for the download functions yourself
  5. Downloads the file and job's done