Reference

async_s3

S3 Bucket helper utils. Async list objects by folders

The file is mandatory for build system to find the package.

Classes

async_s3.S3BucketObjects

Attributes

async_s3.S3BucketObjects.semaphore instance-attribute

semaphore = Semaphore(parallelism)

Functions

async_s3.S3BucketObjects.iter async

iter(prefix: str = '/', *, max_level: Optional[int] = None, max_folders: Optional[int] = None, delimiter: str = '/') -> AsyncIterator[List[Dict[str, Any]]]

Generator that yields objects in the bucket with the given prefix.

Yield objects by partial chunks (list of AWS S3 object dicts) as they are collected from AWS asynchronously.

max_level: The maximum folders depth to traverse in separate requests. If None, traverse all levels. max_folders: The maximum number of folders to load in separate requests. If None, requests all folders. Otherwise, the folders are grouped by prefixes before loading in separate requests. Try to group in the given number of folders if possible. delimiter: The delimiter for "folders".

async_s3.S3BucketObjects.list async

list(prefix: str = '/', *, max_level: Optional[int] = None, max_folders: Optional[int] = None, delimiter: str = '/') -> List[Dict[str, Any]]

List all objects in the bucket with the given prefix.

max_level: The maximum folders depth to traverse in separate requests. If None, traverse all levels. max_folders: The maximum number of folders to load in separate requests. If None, requests all folders. Otherwise, the folders are grouped by prefixes before loading in separate requests. Try to group to the given max_folders if possible. delimiter: The delimiter for "folders".

Modules

async_s3.group_by_prefix

Functions

async_s3.group_by_prefix.find_longest_common_prefix

find_longest_common_prefix(words: List[str]) -> str

Finds the longest common prefix among a list of words.

async_s3.group_by_prefix.group_by_prefix

group_by_prefix(words: List[str], desired_group_count: int) -> List[str]

Groups words by prefixes to create a desired number of word groups.

Try to create the desired number of groups if possible.

async_s3.main

async-s3.

Attributes

async_s3.main.PROGRESS_REFRESH_INTERVAL module-attribute

PROGRESS_REFRESH_INTERVAL = 0.5

async_s3.main.S3PROTO module-attribute

S3PROTO = 's3://'

Classes

Functions

async_s3.main.as3

as3() -> None

Async S3.

async_s3.main.du

du(s3_url: str, max_level: Optional[int], max_folders: Optional[int], repeat: int, parallelism: int, delimiter: str) -> None

Show count and size for objects in an S3 bucket.

Example: as3 du s3://bucket/key

async_s3.main.error

error(message: str) -> None

Print an error message and exit.

async_s3.main.human_readable_size

human_readable_size(size: float, decimal_places: int = 2) -> str

Convert bytes to a human-readable format.

async_s3.main.list_objects

list_objects(s3_url: str, max_level: Optional[int] = None, max_folders: Optional[int] = None, repeat: int = 1, parallelism: int = 100, delimiter: str = '/') -> Iterable[Dict[str, Any]]

List objects in an S3 bucket.

async_s3.main.list_objects_async async

list_objects_async(s3_url: str, max_level: Optional[int], max_folders: Optional[int], repeat: int, parallelism: int, delimiter: str) -> Iterable[Dict[str, Any]]

List objects in an S3 bucket.

async_s3.main.list_objects_options

list_objects_options(func: Callable[[Any], None]) -> Callable[[Any], None]

Add common options to commands using list_objects.

async_s3.main.list_objects_with_progress async

list_objects_with_progress(s3_list: S3BucketObjects, key: str, max_level: Optional[int], max_folders: Optional[int], delimiter: str) -> Tuple[Iterable[Dict[str, Any]], float]

List objects in an S3 bucket with a progress bar.

Returns: (The objects, the elapsed time)

async_s3.main.ls

ls(s3_url: str, max_level: Optional[int], max_folders: Optional[int], repeat: int, parallelism: int, delimiter: str) -> None

List objects in an S3 bucket.

Example: as3 ls s3://bucket/key

async_s3.main.print_attempt_info

print_attempt_info(attempt: int, duration: float) -> None

Print the elapsed time for an attempt.

async_s3.main.print_average_time

print_average_time(total_time: float, repeat: int) -> None

Print the average elapsed time.

async_s3.main.print_start_info

print_start_info(s3_url: str, max_level: Optional[int], max_folders: Optional[int], delimiter: str, parallelism: int, repeat: int) -> None

Print the command parameters.

async_s3.main.print_summary

print_summary(objects: Iterable[Dict[str, Any]]) -> None

Print a summary of the objects.

async_s3.main.split_s3_url

split_s3_url(s3_url: str) -> Iterable[str]

Split an S3 URL into bucket and key.

async_s3.main.validate_delimiter

validate_delimiter(ctx: Context, param: Parameter, value: str) -> str

Validate the Delimiter option.

async_s3.s3_bucket_objects

Attributes

async_s3.s3_bucket_objects.DEFAULT_PARALLELISM module-attribute

DEFAULT_PARALLELISM = 100

Classes

async_s3.s3_bucket_objects.S3BucketObjects

Attributes

async_s3.s3_bucket_objects.S3BucketObjects.semaphore instance-attribute

semaphore = Semaphore(parallelism)

Functions

async_s3.s3_bucket_objects.S3BucketObjects.iter async

iter(prefix: str = '/', *, max_level: Optional[int] = None, max_folders: Optional[int] = None, delimiter: str = '/') -> AsyncIterator[List[Dict[str, Any]]]

Generator that yields objects in the bucket with the given prefix.

Yield objects by partial chunks (list of AWS S3 object dicts) as they are collected from AWS asynchronously.

max_level: The maximum folders depth to traverse in separate requests. If None, traverse all levels. max_folders: The maximum number of folders to load in separate requests. If None, requests all folders. Otherwise, the folders are grouped by prefixes before loading in separate requests. Try to group in the given number of folders if possible. delimiter: The delimiter for "folders".

async_s3.s3_bucket_objects.S3BucketObjects.list async

list(prefix: str = '/', *, max_level: Optional[int] = None, max_folders: Optional[int] = None, delimiter: str = '/') -> List[Dict[str, Any]]

List all objects in the bucket with the given prefix.

max_level: The maximum folders depth to traverse in separate requests. If None, traverse all levels. max_folders: The maximum number of folders to load in separate requests. If None, requests all folders. Otherwise, the folders are grouped by prefixes before loading in separate requests. Try to group to the given max_folders if possible. delimiter: The delimiter for "folders".

Functions

async_s3.s3_bucket_objects.create_session cached

create_session() -> AioSession

Create a session object.

async_s3.s3_bucket_objects.get_s3_client

get_s3_client() -> AioBaseClient

Get S3 client.