flambe.compile.downloader

Module Contents

flambe.compile.downloader.logger[source]
flambe.compile.downloader.s3_exists(url: ParseResult) → bool[source]

Return is an S3 resource exists.

Parameters:url (ParseResult) – The parsed URL.
Returns:True if it exists. False otherwise.
Return type:bool
flambe.compile.downloader.s3_remote_file(url: ParseResult) → bool[source]

Check if an existing S3 hosted artifact is a file or a folder.

Parameters:url (ParseResult) – The parsed URL.
Returns:True if it’s a file, False if it’s a folder.
Return type:bool
flambe.compile.downloader.download_s3_file(url: str, destination: str) → None[source]

Download an S3 file.

Parameters:
  • url (str) – The S3 URL. Should follow the format: ‘s3://<bucket-name>[/path/to/file]’
  • destination (str) – The output file where to copy the content
flambe.compile.downloader.http_exists(url: str) → bool[source]

Check if an HTTP/HTTPS file exists.

Parameters:url (str) – The HTTP/HTTPS URL.
Returns:True if the HTTP file exists
Return type:bool
flambe.compile.downloader.download_http_file(url: str, destination: str) → None[source]

Download an HTTP/HTTPS file.

Parameters:
  • url (str) – The HTTP/HTTPS URL.
  • destination (str) – The output file where to copy the content. Needs to support binary writing.
flambe.compile.downloader.download_s3_folder(url: str, destination: str) → None[source]

Download an S3 folder.

Parameters:
  • url (str) – The S3 URL. Should follow the format: ‘s3://<bucket-name>[/path/to/folder]’
  • destination (str) – The output folder where to copy the content
flambe.compile.downloader.download_manager(path: str, destination: Optional[str] = None)[source]

Manager for downloading remote URLs

Parameters:
  • path (str) – The remote URL to download. Currently, only S3 and http/https URLs are supported. In case it’s already a local path, it yields the same path.
  • destination (Optional[str]) – The path where the artifact will be downloaded (this includes the file/folder name also). In case of not given, a temporary directory will be used and the name of the artifact will be inferred from the path.

Examples

>>> with download_manager("https://host.com/my/file.zip") as path:
>>>     os.path.exists(path)
>>> True
Yields:str – The new local path