semvec/pypi-packages huggingface dataset:
lk-utils 3.4.3
LK Utils is a set of utility wrappers made for data processing.
Requires Python: >=3.8

# LK Utils

[中文版](./README.zh.md)

lk-utils is a set of utilities to make data processing more simple and fluent.

# Install

```shell
pip install lk-utils
```

lk-utils requires python 3.8 or higher version.

# Usage

## subproc

### new thread decorator

```python
from lk_utils import new_thread


def main(files: list[str]) -> None:
    for f in files:
        handle_file(f)


@new_thread()
def handle_file(file: str) -> None:
    # do something
    ...
```

### fetch results from threads

```python
from lk_utils import new_thread


def main(files: list[str]) -> None:
    pool = []
    for f in files:
        thread = handle_file(f)
        pool.append(thread)
    
    ...
    
    for thread in pool:
        result = thread.join()
        print(result)


@new_thread()
def handle_file(file: str) -> str:
    # do something
    ...
```

### run cmd args

```python
from lk_utils import run_cmd_args
from lk_utils import run_cmd_shell
run_cmd_args('python', '-m', 'pip', 'list')
run_cmd_shell('python -m pip list')
```

advanced filter:

```python
from lk_utils import run_cmd_args


def pip_install(
        dest: str, 
        url_index: str = None
) -> None:
    run_cmd_args(
        ('python', '-m', 'pip'),
        ('install', '-r', 'requirements.txt'),
        ('-t', dest),
        ('-i', url_index),
    )
```

### mklink, mklinks

```python
from lk_utils import mklink, mklinks
mklink('/from_dir', '/to_dir_1')
mklinks('/from_dir', '/to_dir_2')
```

## filesniff

### get current dir, get relative path

```python
import os
from lk_utils import filesys as fs
print(fs.currdir() == os.path.dirname(__file__).replace('\\', '/'))  # -> True
print(fs.relpath('..') == os.path.dirname(fs.currdir()))  # -> True
```

### list files/dirs

```python
from lk_utils import filesys as fs

for path, name in fs.find_files('.'):  # this is an generator.
    print(path, name)
    #   the first element is the **abspath**, the second is path's
    #   basename (<- os.path.basename(path))

for path in fs.find_file_paths('.'):  # this is a list[str]
    print(path)

for name in fs.find_file_names('.'):  # this is a list[str]
    print(name)

# more:
#   fs.findall_files
#   fs.findall_file_paths
#   fs.findall_file_names
#
#   fs.find_dirs
#   fs.find_dir_paths
#   fs.find_dir_names
#
#   fs.findall_dirs
#   fs.findall_dir_paths
#   fs.findall_dir_names
```

## read_and_write

### loads and dumps

```python
from lk_utils import read_and_write as rw

data_r = rw.loads(file_i)
#   it recognizes json, yaml, pkl as sturctured data. others are treated as
#   plain text.

data_w = ...
rw.dumps(data_w, file_o)
#   it recognizes json, yaml, pkl as sturctured data. others are treated as
#   plain text.
```

---

<font color="red">below are marked as deprecated.</font>

## excel

### excel reader and writer

```python
from lk_utils import excel as exl

reader = exl.ExcelReader(file_i)
#   accepts '.xls' and '.xlsx' files.
...  # TODO:CompleteExample

writer = exl.ExcelWriter(file_o)
#   accepts only '.xlsx' files.
...  # TODO:CompleteExample
writer.save()

```

## nlp

*TODO*
smartpool-core 0.1.0
Auto-selecting thread/process pool utilities.
Requires Python: >=3.9
Keywords: thread, process, pool, concurrency

# smartpool

Auto-selecting thread/process pool utilities for parameterized tasks.

[English](README.md) | [简体中文](README_zh.md)

[![Python](https://img.shields.io/badge/Python-3.9%2B-blue.svg)](https://www.python.org/downloads/)
[![Typing](https://img.shields.io/badge/Typing-PEP%20561-informational.svg)](https://peps.python.org/pep-0561/)
[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

## ✨ Why smartpool

smartpool provides a single entry point to run parameterized tasks with either a thread pool or a process pool. It keeps the API small, chooses a sensible executor in `auto` mode, and stays close to the stdlib so you can drop it into existing scripts.

## 🎯 Target users & scope

Smartpool is designed for lightweight batch execution in scripts and services where you want a small, predictable API.

- Best for: IO-heavy tasks, CPU-heavy tasks, short batch jobs
- Not for: distributed scheduling, persistent queues, complex retry pipelines

## ✅ Features

- Unified interface for thread/process pools
- `auto` mode picks a reasonable executor for the runtime
- Ordered or completion-order results
- Timeout control per task and per run
- Lightweight stats collection via `RunStats`
- Resource guard with `max_tasks`
- PEP 561 typing with `py.typed`

## 📦 Installation

```bash
pip install smartpool
```

## 🚀 Quickstart

```python
from smartpool import ThreadUtils


def work(x: int) -> int:
    return x * 2

result = ThreadUtils.run_parameterized_task(
    work,
    [1, 2, 3, 4],
    mode="auto",
    ordered=True,
)
print(result)
```

## 📘 API

```python
ThreadUtils.run_parameterized_task(
    task,
    params,
    *,
    mode="auto",          # auto | cpu | io | thread | process
    max_workers=None,
    thread_name_prefix="default",
    timeout=None,
    timeout_total=None,
    ordered=True,
    result_order=None,    # "input" | "completed"
    chunksize=1,          # meaningful for process + ordered=True
    max_tasks=None,
    return_exceptions=False,
    stats=None,           # RunStats
)
```

### 🧭 Mode selection

- `auto` / `cpu`: use thread pool on free-threaded runtimes, otherwise process pool
- `io` / `thread`: always use a thread pool
- `process`: always use a process pool

### 📝 Behavior notes

- Process pools require `task` to be a top-level function and parameters to be pickle-able.
- With `ordered=True`, process pool uses `map`; per-item timeout is not available in this path.
- With `ordered=False`, results are returned by completion order.
- For ordered process pools, enabling timeout/return_exceptions/timeout_total switches to `submit` and ignores `chunksize`.
- `timeout_total` enforces a total runtime limit; it may raise `TimeoutError`.
- When `return_exceptions=True`, exceptions are returned in the result list.

## 🧱 Project structure

```
smartpool/
├── __init__.py
├── thread.py
└── py.typed
```

## 📄 License

MIT License. See `LICENSE` for details.
hiagent-observe 2.0.2
HiAgent Observe SDK
Requires Python: >=3.10

# HiAgent Observe SDK

## OpenTelemetry Trace

OpenTelemetry（OTel）是一个开源的、具备厂商中立性的可观测性框架。它提供了一组规范、API 和库，基于特定结构化数据建模，以输出应用程序的分布式追踪（Trace）、指标（Metrics）和日志（Log）数据。​
HiAgent 支持接收来自 OpenTelemetry 客户端上报的 Trace 数据并可视化展示，从而实现对应用程序的深度监控和分析。​
基于 OpenTelemetry 客户端上报的流程图如下所示：​
​
<img alt="observe" src="./assets/observe.png" width="40%"/>
​

## OpenTelemetry Trace 集成方式​

通过 OpenTelemetry SDK 上报 Trace​，你可以使用各种语言的 OpenTelemetry SDK 将 Trace 数据上报到 HiAgent。通过 OpenTelemetry SDK 上报时，开发者需要直接在代码中引入 OpenTelemetry SDK 来创建和管理 Trace 数据（例如创建 Span、设置 Attribute、记录 Event 等）。这种方式需要开发者对 OpenTelemetry 的 SDK 有一定的了解，并在代码中明确实现埋点逻辑。

### 常见问题​

#### HiAgent 支持什么协议？​

HiAgent 不支持 GRPC 协议的 OpenTelemetry Endpoint，需使用 http/protobuf 协议。​以 Python 为例，配置示例如下：​
​

``` python
import os
import time

from dotenv import load_dotenv
from hiagent_observe import client
from opentelemetry import trace

load_dotenv()


if __name__ == "__main__":
  try:
    # 初始化 OpenTelemetry Client，并且注入鉴权的 header
    # 鉴权的 header 每个小时会自动刷新
    client.init(
      # 配置为 HiAgent 的 OpenTelemetry Endpoint
      trace_endpoint=os.getenv("HIAGENT_TRACE_ENDPOINT"),
      # 配置为 HiAgent 的 Top Endpoint
      top_endpoint=os.getenv("HIAGENT_TOP_ENDPOINT"),
      # 配置为 HiAgent 的个人中心申请的 access key
      ak=os.getenv("VOLC_ACCESSKEY"),
      # 配置为 HiAgent 的个人中心申请的 secret key
      sk=os.getenv("VOLC_SECRETKEY"),
      # 配置为 HiAgent 的自定义应用的 workspace id
      workspace_id=os.getenv("WORKSPACE_ID"),
      # 配置为 HiAgent 的自定义应用的 app id
      app_id=os.getenv("CUSTOM_APP_ID"),
    )
  except Exception as e:
    raise RuntimeError(e)

  tracer = trace.get_tracer("example-tracer") # 创建一个 Tracer

  for i in range(100000):
    with tracer.start_as_current_span("main-operation") as root_span:
      with tracer.start_as_current_span("do-work") as span:
        span.set_attributes(
          {
            "http.method": "GET",
            "http.route": "/api/work",
            "http.status_code": 200,
          }
        )
    print("push trace")
    time.sleep(2)
```

更完整的例子参考[这里](https://github.com/volcengine/hiagent-python-sdk/blob/feat/span_type/libs/observe/samples/sample_chainlit.py)。
aspen-dynamo 0.1.3
Async DynamoDB table helper based on aioboto3.
Requires Python: >=3.10

# aspen-dynamo

Async DynamoDB table helper built on `aioboto3`. It wraps a DynamoDB table
resource with small conveniences for key handling, pagination, and optional
Pydantic model validation.

## Installation

```bash
pip install aspen-dynamo
```

Optional extras:

```bash
pip install "aspen-dynamo[benchmark]"
```

## Usage

Create a `DynamoDBTable` with a boto3 DynamoDB resource:

```python
import aioboto3
from aspen_dynamo import DynamoDBTable

session = aioboto3.Session()

async with session.resource("dynamodb") as resource:
    table = DynamoDBTable("MyTable", "pk", resource=resource)
    item = await table.get_item(123)
    print(item)
```

### Partition + sort key (composite key)

Pass a tuple of attribute names for a table with a partition key and sort key:

```python
import aioboto3
from aspen_dynamo import DynamoDBTable

session = aioboto3.Session()

async with session.resource("dynamodb") as resource:
    table = DynamoDBTable("MyTable", ("pk", "sk"), resource=resource)
    item = await table.get_item(123, "v1")
    print(item)
```

### Usage with a Pydantic model

Provide a model to coerce DynamoDB items into typed objects:

```python
import aioboto3
from pydantic import BaseModel
from aspen_dynamo import DynamoDBTable

class Widget(BaseModel):
    pk: int
    name: str

session = aioboto3.Session()

async with session.resource("dynamodb") as resource:
    table = DynamoDBTable("MyTable", "pk", resource=resource, model=Widget)
    widget = await table.get_item(123)
    print(widget.name)
```

## Benchmark

The `benchmark.py` script compares `aspen-dynamo` to other async DynamoDB
clients. It uses the same `DynamoDBTable` API as above:

```bash
python -m aspen_dynamo.benchmark MyTable pk 123
```

Results from two runs:

* `uvloop` 0.19.0
* `aioboto3` 13.0.0
* `aiobotocore` 2.13.0
* `aiodynamo` 24.7
* `aiohttp` 3.9.5
* `boto3` 1.34.106
* `httpx` 0.28.1

| Test | Avg latency 1 | Avg latency 2 | CPU time 1 | CPU time 2 |
| --- | --- | --- | --- | --- |
| aioboto3 (client)     | 3.12ms | 2.67ms | 0.75ms | 0.71ms |
| aioboto3 (resource)   | 3.39ms | 2.64ms | 0.80ms | 0.78ms |
| aspen-dynamo          | 2.61ms | 3.24ms | 0.78ms | 0.79ms |
| raw request (aiohttp) | 2.43ms | 2.17ms | 0.26ms | 0.28ms |
| aiodynamo (aiohttp)   | 2.34ms | 2.15ms | 0.32ms | 0.32ms |
| raw request (httpx)   | 2.53ms | 2.54ms | 2.05ms | 1.99ms |
| aiodynamo (httpx)     | 3.02ms | 3.01ms | 2.07ms | 2.23ms |