semvec/pypi-packages huggingface dataset:
fustor-benchmark 0.8.17
Performance benchmarking tool for Fustor platform
Requires Python: >=3.11

# Fustor 性能基准测试工具 (Benchmark)

该模块是 Fustor 平台的自动化压力测试和性能量化工具。它通过模拟大规模文件系统元数据，量化 Fusion API 相比于操作系统原生文件系统调用的性能优势。

## 核心设计目标

1.  **量化优势**: 对比 Fusion 内存索引与 Linux 原生 `find` 命令在递归元数据检索下的延迟与吞吐量。
2.  **百万级规模**: 支持生成并同步超过 1,000,000 个文件的元数据。
3.  **全自动流程**: 自动编排 Fusion 和 Agent，实现一键式从环境部署到报告生成。
4.  **生产环境巡检**: 支持对接已有的 Fustor 集群进行性能实时量化。

## 目录结构规范与安全

为了保护生产数据免受误删，Benchmark 实施了严格的路径校验：

*   **运行沙箱**: 所有的日志、中间数据库和压测报告都固定在当前目录下的 **`fustor-benchmark-run/`**。
*   **安全逻辑**: `generate` 命令如果发现目标目录非空，会强制停止并报错，防止覆盖已有数据。

## 快速使用

### 1. 数据生成
数据生成器通过以下三个维度的乘积来确定最终的文件总量：
*   `--num-dirs` (默认 1000): UUID 目录的数量。
*   `--num-subdirs` (默认 4): 每个 UUID 目录下的子目录数量。
*   `--files-per-subdir` (默认 250): 每个子目录下的文件数量。

**默认规模**: $1000 \times 4 \times 250 = 1,000,000$ 个文件。

构建一个包含 1000 个 UUID 目录（总计 100 万文件）的测试集：
```bash
uv run fustor-benchmark generate fustor-benchmark-run/data --num-dirs 1000
```

### 2. 执行压测 (全自动模式)
此模式会自动在本地启动 Fusion 和 Agent，并在压测结束后自动停止：
```bash
uv run fustor-benchmark run fustor-benchmark-run/data -d 5 -c 20 -n 100
```

### 3. 执行压测 (外部服务模式)
对接已在运行的 Fustor 集群（如分布式部署环境）：
```bash
uv run fustor-benchmark run /mnt/nfs_data \
    --fusion-api http://10.0.0.1:18102 \
    --api-key YOUR_SECRET_KEY \
    -d 5 -c 50 -n 1000
```

## 参数说明

| 参数 | 缩写 | 默认值 | 说明 |
| :--- | :--- | :--- | :--- |
| `TARGET-DIR` | - | (必填) | 目标数据路径（本地或 NFS 挂载点） |
| `--concurrency` | `-c` | 20 | 并发工作进程/线程数 |
| `--num-requests` | `-n` | 200 | 总压测迭代次数 |
| `--target-depth` | `-d` | 5 | 探测并测试的目录相对深度 |
| `--fusion-api` | - | - | 外部 Fusion API 地址（跳过本地服务启动） |
| `--api-key` | - | - | 外部 Fusion API 的身份验证 Key |

## 报告与指标

压测完成后，将在 `fustor-benchmark-run/results/` 下生成以下产出：

1.  **`stress-find.html`**: 交互式可视化报表。
    *   **Latency Distribution**: 展示 Avg, P50, P95, P99 的柱状对比。
    *   **Latency Percentiles**: 展现延迟分布曲线。
    *   **Speedup Factor**: 自动计算 Fusion 相比 OS 的加速倍数。
2.  **`stress-find.json`**: 结构化的指标数据，包含所有原始延迟序列。

## 安全保护说明

Benchmark 及其底层驱动 `source_fs` 遵循 **100% 纯读取** 准则。压测过程仅通过 API 获取元数据或执行 `find` 命令，绝对不会修改用户的生产数据或在监控目录下创建任何临时文件。
fustor-agent 0.8.17
realtime data sync agent for various data sources
Requires Python: >=3.11

# Fustor Agent 服务

Fustor Agent 是一款轻量、可扩展的数据采集与推送工具。它负责监听数据源变更，并将其实时推送到 Fustor Fusion 服务。

## 安装

```bash
pip install fustor-agent
# 安装文件系统源驱动
pip install fustor-source-fs
```

### 1. 配置

Fustor Agent 使用一个主目录来存放配置和状态。
*   **默认路径**: `~/.fustor`
*   **自定义路径**: 设置 `FUSTOR_HOME` 环境变量。

Agent 的核心配置文件位于 Fustor 主目录下的 `agent-config.yaml`。你需要定义 `sources` (数据源)、`senders` (推送目标) 和 `pipes` (同步任务)。

### 1. 配置 Source (数据源)

以文件系统 (FS) 为例：

```yaml
sources:
  - id: "my-local-files"       # 唯一 ID
    type: "fs"                 # 驱动类型
    config:
      uri: "/data/research"    # 监控的绝对路径
      driver_params:
        # 可选：文件过滤模式
        file_pattern: "*"      
```

### 2. 配置 Sender (推送目标)

通常推送到 Fusion 服务：

```yaml
senders:
  - id: "to-fusion"            # 唯一 ID
    type: "fusion"             # 驱动类型
    config:
      # Fusion 服务的 Ingest API 基准地址
      endpoint: "http://localhost:8102"
      # 从 Fusion 管理员处获取的 API Key，用于鉴权
      credential: "YOUR_API_KEY_HERE"
```

### 3. 配置 Pipe (数据管道任务)

将 Source 和 Sender 绑定：

```yaml
pipes:
  - id: "phase-files-to-fusion"
    source_id: "my-local-files"
    sender_id: "to-fusion"
    enabled: true              # 设置为 true 以自动启动
```

## 命令指南

*   **启动服务**: `fustor-agent start -D` (后台运行) 或 `fustor-agent start` (前台运行)
*   **停止服务**: `fustor-agent stop`
*   **查看状态**: 访问 `http://localhost:8100` 查看 Web 控制台。

## 数据可靠性保证 (Data Reliability)

Agent 遵循 **"瘦 Agent 感知 + 胖 Fusion 裁决"** 架构。

### Leader/Follower 模式

| 角色 | Realtime Sync Phase | Snapshot Sync Phase | Audit Sync Phase | Sentinel Sweep |
|------|---------------|---------------|------------|----------------|
| **Leader** | ✅ | ✅ | ✅ | ✅ |
| **Follower** | ✅ | ❌ | ❌ | ❌ |

- **先到先得**：第一个建立 Session 的 Agent 成为 Leader
- **故障转移**：仅当 Leader 心跳超时后，Fusion 才释放 Leader 锁

### 消息类型 (`message_source`)

| 类型 | 说明 |
|------|------|
| `realtime` | inotify 事件，优先级最高 |
| `snapshot` | Agent 启动时全量扫描 |
| `audit` | 定时审计，发现盲区变更 |

详见 `docs/CONSISTENCY_DESIGN.md`。

## 更多文档

*   **驱动开发**: 详见 `docs/driver_design.md`
*   **一致性设计**: 详见 `docs/CONSISTENCY_DESIGN.md`
fustor-sender-http 0.8.17
Fustor HTTP Sender - Transport layer for Agent to Fusion communication
Requires Python: >=3.11

# fustor-sender-http

HTTP Sender for Fustor Agent - implements the transport layer for Agent to Fusion communication.

## Overview

This package provides an HTTP-based implementation of the `Sender` transport abstraction. It uses the Fusion SDK client to communicate with Fusion's REST API.

## Installation

```bash
pip install fustor-sender-http
```

## Usage

```python
from fustor_sender_http import HTTPSender

sender = HTTPSender(
    sender_id="my-sender",
    endpoint="http://fusion.example.com:8000",
    credential={"api_key": "your-api-key"}
)

# Create session
await sender.connect()
session = await sender.create_session("my-task-id")

# Send events
await sender.send_events(events, source_type="message")

# Heartbeat
await sender.heartbeat()

# Cleanup
await sender.close()
```

## Entry Points

This package registers itself as:
- `fustor.senders:http` - New sender registry
- `fustor_agent.drivers.senders:fusion` - Legacy sender registry (backward compat)

## Migration from fustor-sender-fusion

The `fustor-sender-fusion` package is deprecated. To migrate:

1. Replace `from fustor_sender_fusion import FusionDriver` with `from fustor_sender_http import HTTPSender`
2. Update configuration to use `sender` instead of `sender` terminology
3. The `HTTPSender` class implements the new `Sender` interface but maintains API compatibility
fustor-receiver-http 0.8.17
Fustor HTTP Receiver - Transport layer for Fusion to receive events from Agents
Requires Python: >=3.11

# fustor-receiver-http

HTTP Receiver for Fustor Fusion - implements the transport layer for receiving events from Agents.

## Overview

This package provides an HTTP-based implementation of the `Receiver` transport abstraction. It creates FastAPI routers that handle session management and event ingestion.

## Installation

```bash
pip install fustor-receiver-http
```

## Usage

```python
from fustor_receiver_http import HTTPReceiver, SessionInfo

# Create receiver
receiver = HTTPReceiver(
    receiver_id="main-receiver",
    config={"session_timeout_seconds": 30}
)

# Register API keys for pipes
receiver.register_api_key("fk_abc123", "pipe-1")

# Register callbacks
async def on_session_created(session_id, task_id, pipe_id, client_info):
    # Handle session creation
    return SessionInfo(
        session_id=session_id,
        task_id=task_id,
        pipe_id=pipe_id,
        role="leader",
        created_at=time.time(),
        last_heartbeat=time.time()
    )

async def on_event_received(session_id, events, source_type, is_end):
    # Process events
    return True

receiver.register_callbacks(
    on_session_created=on_session_created,
    on_event_received=on_event_received,
)

# Mount routers in FastAPI app
app.include_router(receiver.get_session_router(), prefix="/api/v1/pipe/session")
app.include_router(receiver.get_ingestion_router(), prefix="/api/v1/pipe/ingest")
```

## API Endpoints

### Session Router

- `POST /` - Create a new session
- `POST /{session_id}/heartbeat` - Send heartbeat
- `DELETE /{session_id}` - Terminate session

### Ingestion Router

- `POST /{session_id}/events` - Ingest event batch

## Entry Points

This package registers itself as:
- `fustor.receivers:http` - Receiver registry
fustor-sender-echo 0.8.17
Echo sender driver for Fustor
Requires Python: >=3.10

# fustor-sender-echo

This package provides an "echo" sender driver for the Fustor Agent service. It serves as a basic example and debugging tool for `SenderDriver` implementations. Instead of sending data to an external system, it simply logs all received events and control flags to the Fustor Agent's log output.

## Features

*   **Echo Functionality**: Logs all incoming events, including `realtime` and `snapshot` data, to the console/log.
*   **Control Flag Visibility**: Displays control flags like `is_snapshot_end` and `snapshot_sync_suggested` for debugging data flow.
*   **Session Management**: Implements `create_session` and `heartbeat` to demonstrate session lifecycle.
*   **No Configuration Needed**: The `get_needed_fields` method returns an empty schema, indicating it accepts all fields without specific requirements.
*   **Wizard Definition**: Provides a simple wizard step for UI integration.

## Installation

This package is part of the Fustor monorepo and is typically installed in editable mode within the monorepo's development environment using `uv sync`. It is registered as a `fustor_agent.drivers.senders` entry point.

## Usage

To use the `fustor-sender-echo` driver, configure a Sender in your Fustor Agent setup with the driver type `echo`. When a pipe involves this sender, all data processed by the Agent will be logged by this driver.

This driver is particularly useful for:
*   **Debugging**: Understanding the exact data and control signals being sent by the Fustor Agent.
*   **Development**: As a template for creating new `SenderDriver` implementations.
*   **Testing**: Verifying that the Fustor Agent's data pipe is correctly delivering events.

Example (conceptual configuration in Fustor Agent):

```yaml
# Fustor 主目录下的 agent-config.yaml
senders:
  my-echo-sender:
    driver_type: echo
    # No specific configuration parameters needed for the echo driver
```

## Dependencies

*   `fustor-core`: Provides the `SenderDriver` abstract base class and other core components.
*   `fustor-event-model`: Provides `EventBase` for event data structures.
fustor-schema-fs 0.8.17
Fustor File System Schema - Contract between source-fs and view-fs
Requires Python: >=3.11

# fustor-schema-fs

File System Schema for Fustor - defines the data contract between `fustor-source-fs` (Agent) and `fustor-view-fs` (Fusion).

## Overview

This package contains only type definitions and validation logic, with **no runtime dependencies**. It serves as the single source of truth for the file system event format.

## Schema

### Event Row Format

```python
{
    "path": str,              # Absolute file path
    "file_name": str,         # Base file name
    "size": int,              # File size in bytes
    "modified_time": float,   # Last modification time (Unix timestamp)
    "is_directory": bool,     # True if this is a directory
    # Optional fields
    "created_time": float,    # Creation time
    "parent_path": str,       # Parent directory path
    "parent_mtime": float,    # Parent directory mtime (for audit)
}
```

## Usage

```python
from fustor_schema_fs import (
    SCHEMA_NAME,
    SCHEMA_VERSION,
    validate_row,
    get_path,
    get_mtime,
)

# Check schema version
print(f"Schema: {SCHEMA_NAME} v{SCHEMA_VERSION}")

# Validate event row
row = {"path": "/data/file.txt", "file_name": "file.txt", ...}
if validate_row(row):
    path = get_path(row)
    mtime = get_mtime(row)
```

## Versioning

This schema follows semantic versioning:
- **Major** version changes indicate breaking changes to required fields
- **Minor** version changes add optional fields
- **Patch** version changes are documentation-only

Current version: `1.0`
snoopy-bv 2.5.0
DR C++/Python library
Requires Python: >=3.10

.. image:: https://gitlab.com/bv-dr/Snoopy/badges/master/pipeline.svg?branch=master
   :target: https://gitlab.com/bv-dr/Snoopy/badges/master/pipeline.svg
   :alt: Build Status


Snoopy library
==============

Documentation and tutorials : `https://bv-dr.gitlab.io/Snoopy <https://bv-dr.gitlab.io/Snoopy>`_

Snoopy is an open source library for marine applications. Among other things, it contains the following sub-packages

* 
  Spectral : Routines for spectral analysis of sea-keeping results

  * Wave spectra (Jonswap, Ochihubble, spreading function...)
  * Manipulation of transfer function (RAOs)
  * Response spectrum
  * 2nd order response spectrum


* 
  TimeDomain

  * Reconstruction of seakeeping results in time domain (1st and 2nd order)
  * Convolution for wave radiation in time domain (from added-mass).
  * Upcrossing analysis

* 
  Fatigue

  * Rainflow counting
  * SN-Curves
  * Spectral fatigue

* 
  Meshing

* 
   WaveKinematics : (Linear, wheeler stretching, 2nd order...)
  
* 
   Statistics

   * Peak Over Threshold
   * Long term analysis
   * IFORM contour
  


Example of use
--------------

.. code-block:: python

   >>> from Snoopy import Spectral as sp
   >>> spec = sp.Jonswap( hs = 1.0 , tp = 10.0 , gamma = 1.0)
   >>> spec.plot()

.. image:: https://gitlab.com/bv-dr/Snoopy/-/raw/master/Doc/pictures/Jonswap.png
   :width: 400
   :alt: Jonswap picture



How to install
--------------

Snoopy is available on pypi.org and can be using pip: 

.. code-block:: sh

   pip install snoopy-bv


In case conda is used, it is recommended to install dependencies using conda (hence the --no-deps in above command line):

.. code-block:: sh

   conda install xarray numpy scipy pandas matplotlib h5py tqdm Deprecated
   pip install snoopy-bv --no-deps



How to build from sources
-------------------------

To build the package from sources, instructions are given in `readme_dev.rst <https://gitlab.com/bv-dr/Snoopy/-/blob/master/readme_dev.rst>`_