goyimnose/Medios-Macina
1
0
Fork 0
Code Issues Pull Requests Packages Releases 1 Wiki Activity

update

Browse Source
This commit is contained in:
Nose
2026-05-16 15:26:08 -07:00
parent 5048729b0c
commit 02d84f423e
10 changed files with 488 additions and 470 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CHANGELOG.md
+3 -3
View File
@@ -2,8 +2,8 @@
## Unreleased (2026-01-05)
- **docs:** Add `docs/provider_authoring.md` with a Quick Start, examples, and testing guidance for providers that integrate with the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn).
- **docs:** Add link to `docs/result_table.md` pointing to the provider authoring guide.
- **tests:** Add `tests/test_provider_author_examples.py` validating example provider registration and adapter behavior.
- **docs:** Add `docs/provider_authoring.md` as a plugin authoring Quick Start for the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn). The file keeps its legacy name for compatibility.
- **docs:** Add a link from `docs/result_table.md` to the plugin authoring guide.
- **tests:** Add `tests/test_provider_author_examples.py` validating example plugin registration and adapter behavior. The test file keeps its legacy name for compatibility.
- **notes:** Existing example plugins (`plugins/example_provider.py`, `plugins/vimm/__init__.py`) are referenced as canonical patterns.
docs/PR_PROVIDER_AUTHORING.md
+6 -6
View File
@@ -1,15 +1,15 @@
PR Title: docs: Add Provider authoring doc, examples, and tests
PR Title: docs: Add plugin authoring doc, examples, and tests
Summary:
- Add `docs/provider_authoring.md` describing the strict `ResultModel`-based provider adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping.
- Link new doc from `docs/result_table.md`.
- Add `tests/test_provider_author_examples.py` to validate `Provider/example_provider.py` and `Provider/vimm.py` integration with the registry.
- Add `docs/provider_authoring.md` describing the strict `ResultModel`-based plugin adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping.
- Link the new doc from `docs/result_table.md`.
- Add `tests/test_provider_author_examples.py` to validate the example plugin integration with the registry.
Why:
- Provide a short, focused Quick Start to help contributors author providers that integrate with the new strict ResultTable API.
- Provide a short, focused Quick Start to help contributors author plugins that integrate with the strict ResultTable API.
Testing:
- New tests pass locally (provider-related subset).
- New tests pass locally (plugin-related subset).
Notes:
- The change is documentation-first and non-functional, with tests ensuring examples remain valid.
docs/ftp_plugin_tutorial.md
+31 -53
View File
@@ -1,6 +1,6 @@
# FTP Plugin Walkthrough
This walkthrough adds a real bundled `ftp` plugin so users can:
This walkthrough covers the bundled `ftp` plugin. It lets users:
- run `search-file -plugin ftp -instance <name> ...`
- browse remote folders as result tables
@@ -10,9 +10,10 @@ This walkthrough adds a real bundled `ftp` plugin so users can:
The implementation lives in [plugins/ftp/__init__.py](plugins/ftp/__init__.py).
## What The Plugin Does
## What the plugin does
The FTP plugin demonstrates the main provider hooks that matter for a storage-style integration:
The FTP plugin demonstrates the main plugin hooks that matter for a
storage-style integration:
- `config_schema()` exposes host, credentials, base path, TLS, and search depth.
- `extract_query_arguments()` supports inline query fields like `path:` and `depth:`.
@@ -22,9 +23,10 @@ The FTP plugin demonstrates the main provider hooks that matter for a storage-st
- `resolve_pipe_result_download()` lets `@N | add-file -instance ...` materialize a remote FTP file first.
- `upload()` lets `add-file -plugin ftp -instance <name> -path ...` push a local file to the configured FTP server.
## Example Config
## Example config
Add one or more named FTP provider instances to your config:
Add one or more named FTP plugin instances to your config. The current stored
key path remains `provider.ftp.<instance>` for legacy compatibility:
```toml
[provider.ftp.work]
@@ -48,121 +50,97 @@ tls = true
```
Notes:
- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`.
- `work` and `archive` are instance names.
- `host` is the only required field for each instance to validate.
- `username` defaults to `anonymous` and `password` defaults to `anonymous@`.
- `base_path` is both the default search root and the upload target directory.
- `search_depth` controls how many folder levels `search-file -plugin ftp` scans by default.
- You can browse configured instances from `.config plugins` in the CLI.
## Search Flow
Basic listing from the configured base path:
## Search flow
```powershell
search-file -plugin ftp -instance work "*"
```
Search by filename fragment:
```powershell
search-file -plugin ftp -instance work "invoice"
```
Search a different subtree and recurse deeper:
```powershell
search-file -plugin ftp -instance work "path:/pub depth:2 invoice"
```
Filter to folders only:
```powershell
search-file -plugin ftp -instance work "path:/pub type:folder *"
```
The plugin returns rows with explicit columns for name, type, directory, size, and modification time.
The plugin returns rows with explicit columns for name, type, directory, size,
and modification time.
## Selection Flow
## Selection flow
Folder rows are navigation rows. If the selected row is a directory, plain `@N` opens a new FTP table for that directory:
Folder rows are navigation rows:
```powershell
search-file -plugin ftp -instance work "*"
@2
```
File rows carry an explicit row action:
File rows carry an explicit row action equivalent to:
```powershell
download-file -plugin ftp -instance work -url ftp://ftp.example.com/incoming/report.pdf
```
That means plain `@N` on a file row downloads it immediately:
So plain `@N` on a file row downloads it immediately:
```powershell
search-file -plugin ftp -instance work "report"
@1
```
## Download And Add-File Flow
If you want the downloaded file in a specific local directory:
## Download and add-file flow
```powershell
search-file -plugin ftp -instance work "report"
@1 | download-file -path C:\Downloads
```
If you want to ingest the selected FTP file into a configured instance backend:
```powershell
search-file -plugin ftp -instance work "report"
@1 | add-file -instance tutorial
```
Why this works:
- the file row advertises a `download-file` row action
- the pipeline auto-inserts that download before `add-file`
- the FTP plugin also implements `resolve_pipe_result_download()` so provider-owned FTP rows can be materialized for ingestion
- file rows also carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target
- the FTP plugin also implements `resolve_pipe_result_download()` so plugin-owned FTP rows can be materialized for ingestion
- file rows carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target
## Upload Flow
## Upload flow
Uploading uses the same provider name, but through `add-file -plugin ftp -instance <name>`:
Uploading uses the same plugin, through `add-file -plugin ftp -instance <name>`:
```powershell
add-file -plugin ftp -instance archive -path C:\Media\report.pdf
```
That sends the file to the selected instance's FTP `base_path` and returns the FTP URL as the uploaded result.
That sends the file to the selected instance's FTP `base_path` and returns the
FTP URL as the uploaded result.
## Why The Row Metadata Matters
## Why the row metadata matters
The critical part of this plugin is the file-row metadata:
- file rows emit `_selection_args` as `['-instance', '<name>', '-url', '<ftp-url>']`
- file rows emit `_selection_action` as `['download-file', '-plugin', 'ftp', '-instance', '<name>', '-url', '<ftp-url>']`
- folder rows do not emit a download action, so `selector()` can own drill-in behavior instead
That split is what keeps these two user experiences compatible:
- folder rows do not emit a download action, so `selector()` owns drill-in behavior instead
That keeps these flows compatible:
- `@N` on a folder opens a new table
- `@N` on a file downloads the file
- `@N | add-file -instance ...` first downloads, then ingests
## Implementation Notes
## Implementation notes
The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus directory probes when the server does not support machine-readable listings.
The code is intentionally small and uses only Python stdlib pieces:
The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus
directory probes when the server does not support machine-readable listings.
The code intentionally stays small and uses only Python stdlib pieces:
- `ftplib` for FTP and FTPS
- `fnmatch` for wildcard-style search tokens
- `tempfile` for `add-file` handoff downloads
## Recommended Commands To Demo The Walkthrough
## Recommended demo commands
```powershell
search-file -plugin ftp -instance work "*"
docs/provider_authoring.md
+66 -58
View File
@@ -1,37 +1,48 @@
# Plugin authoring: ResultTable & plugin adapters
# Plugin authoring: ResultTable and plugin adapters
This short guide explains how to write plugins that integrate with the *strict* ResultTable API: adapters must yield `ResultModel` instances and plugins register via `SYS.result_table_adapters.register_plugin` with a column specification and a `selection_fn`.
This short guide explains how to write plugins that integrate with the strict
ResultTable API: adapters yield `ResultModel` instances, and plugins register
via `SYS.result_table_adapters.register_plugin` with columns and a
`selection_fn`.
Note: this file keeps its historical `provider_authoring` name, but the public
terminology is plugin-first. Some internal classes and metadata fields still use
`Provider` naming.
---
## Quick summary
- Plugins register a *plugin adapter* (callable that yields `ResultModel`).
- Plugins must also provide `columns` (static list or factory) and a `selection_fn` that returns CLI args for a selected row.
- For simple HTML table/list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers` to fetch and extract rows using `SYS.html_table.extract_records`.
- Plugins register a plugin adapter, a `columns` definition, and a `selection_fn`.
- `selection_fn` returns CLI args for a selected row.
- For HTML table or list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers`.
## Runtime dependency policy
- Treat required runtime dependencies (e.g., **Playwright**) as mandatory: import them unconditionally and let missing dependencies fail fast at import time. Avoid adding per-call try/except import guards for required modules—these silently hide configuration errors and add bloat.
- Use guarded imports only for truly optional dependencies (e.g., `pandas` for enhanced table parsing) and provide meaningful fallbacks or helpful error messages in those cases.
- Keep provider code minimal and explicit: fail early and document required runtime dependencies in README/installation notes.
- Treat required runtime dependencies such as Playwright as mandatory: import them unconditionally and let missing dependencies fail fast.
- Use guarded imports only for truly optional dependencies such as `pandas`.
- Keep plugin code minimal and explicit: fail early and document required runtime dependencies in README and installation notes.
---
## Minimal provider template (copy/paste)
## Minimal plugin template
```py
# plugins/my_plugin.py
from typing import Any, Dict, Iterable, List
from SYS.result_table_api import ResultModel, ColumnSpec, title_column, metadata_column
from SYS.result_table_api import ResultModel, ColumnSpec, title_column
from SYS.result_table_adapters import register_plugin
# Example adapter: convert provider-specific items into ResultModel instances
SAMPLE_ITEMS = [
{"name": "Example File.pdf", "path": "https://example.com/x.pdf", "ext": "pdf", "size": 1024, "source": "myprovider"},
{
"name": "Example File.pdf",
"path": "https://example.com/x.pdf",
"ext": "pdf",
"size": 1024,
"source": "myplugin",
},
]
def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]:
for it in items:
title = it.get("name") or it.get("title") or str(it.get("path") or "")
@@ -41,39 +52,38 @@ def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]:
ext=str(it.get("ext")) if it.get("ext") else None,
size_bytes=int(it.get("size")) if it.get("size") is not None else None,
metadata=dict(it),
source=str(it.get("source")) if it.get("source") else "myprovider",
source=str(it.get("source")) if it.get("source") else "myplugin",
)
# Optional: build columns dynamically from sample rows
def columns_factory(rows: List[ResultModel]) -> List[ColumnSpec]:
cols = [title_column()]
# add extra columns if metadata keys exist
if any((r.metadata or {}).get("size") for r in rows):
cols.append(ColumnSpec("size", "Size", lambda r: r.size_bytes or ""))
if any((row.metadata or {}).get("size") for row in rows):
cols.append(ColumnSpec("size", "Size", lambda row: row.size_bytes or ""))
return cols
# Selection args for `@N` expansion or `select` cmdlet
def selection_fn(row: ResultModel) -> List[str]:
# prefer -path when available
if row.path:
return ["-path", row.path]
return ["-title", row.title or ""]
# Register plugin (done at import time)
register_plugin("myprovider", adapter, columns=columns_factory, selection_fn=selection_fn)
register_plugin("myplugin", adapter, columns=columns_factory, selection_fn=selection_fn)
```
---
## Table scraping: using TableProviderMixin (HTML tables / list-results)
## Table scraping with `TableProviderMixin`
If your provider scrapes HTML tables or list-like results (common on web search pages), use `TableProviderMixin`:
If your plugin scrapes HTML tables or list-like results, use `TableProviderMixin`:
```py
from ProviderCore.base import Provider
from SYS.provider_helpers import TableProviderMixin
class MyTableProvider(TableProviderMixin, Provider):
class MyTablePlugin(TableProviderMixin, Provider):
URL = ("https://example.org/search",)
def validate(self) -> bool:
@@ -84,58 +94,56 @@ class MyTableProvider(TableProviderMixin, Provider):
return self.search_table_from_url(url, limit=limit)
```
`TableProviderMixin.search_table_from_url` returns `ProviderCore.base.SearchResult` entries. If you want to integrate this plugin with the strict `ResultTable` registry, add a small adapter that converts `SearchResult` -> `ResultModel` and register it using `register_plugin` (see `plugins/vimm/__init__.py` for a real example).
`TableProviderMixin.search_table_from_url` returns
`ProviderCore.base.SearchResult` entries. If you want to integrate the plugin
with the strict `ResultTable` registry, add a small adapter that converts
`SearchResult` to `ResultModel` and register it using `register_plugin`.
---
## Columns & selection
- `columns` may be a static `List[ColumnSpec]` or a factory `def cols(rows: List[ResultModel]) -> List[ColumnSpec]` that inspects sample rows.
- `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args (e.g., `['-path', row.path]`). These args are used by `select` and `@N` expansion.
**Tip:** for plugins that produce downloadable file rows prefer returning explicit URL args (e.g., `['-url', row.path]`) so the selected URL is clearly identified by downstream downloaders and to avoid ambiguous parsing when plugin hints (like `-plugin`) are present.
- Ensure your `ResultModel.source` is set (either in the model or rely on the provider name set by `serialize_row`).
## Columns and selection
- `columns` may be a static `List[ColumnSpec]` or a factory that inspects sample rows.
- `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args.
- For downloadable file rows, prefer explicit URL args such as `['-url', row.path]` so downstream downloaders interpret the row unambiguously.
- Ensure `ResultModel.source` is set directly or falls back to the registered plugin name during serialization.
---
## Optional: pandas path for `<table>` extraction
`SYS.html_table.extract_records` prefers a pure-lxml path but will use `pandas.read_html` if pandas is installed and the helper detects it works for the input table. This is optional and **not required** to author a provider — document in your provider whether it requires `pandas` and add an informative error/log message when it is missing.
## Optional pandas support
`SYS.html_table.extract_records` prefers a pure-lxml path but can fall back to
`pandas.read_html` when pandas is installed and the helper detects it works for
the input table. This is optional. Document whether your plugin requires
`pandas` and emit a clear error or log message when it is missing.
---
## Testing & examples
## Testing and examples
- Write `tests/test_plugin_<name>.py` or follow the repo's older naming conventions when extending existing tests.
- Verify `plugin.build_table(...)` produces a `ResultTable` with rows and columns.
- Verify `serialize_rows()` yields `_selection_args`, `_selection_action` when applicable, and `source`.
- When you need an exact CLI stage sequence, call `table.set_row_selection_action(row_index, tokens)` so replay uses the row action verbatim.
- For table-oriented plugins, test `search_table_from_url` with a local HTML fixture or a mocked `HTTPClient`.
- Write `tests/test_provider_<name>.py` that imports your provider and verifies `provider.build_table(...)` produces a `ResultTable` (has `.rows` and `.columns`) and that `serialize_rows()` yields dicts with `_selection_args`, `_selection_action` when applicable, and `source`.
- When you need to guarantee a specific CLI stage sequence (e.g., `download-file -url <path> -plugin <name>`), call `table.set_row_selection_action(row_index, tokens)` so the serialized payload emits `_selection_action` and the CLI can run the row exactly as intended.
- For table providers you can test `search_table_from_url` using a local HTML fixture or by mocking `HTTPClient` to return a small sample page.
- If you rely on pandas, add a test that monkeypatches `sys.modules['pandas']` to a simple shim to validate the pandas path.
**Example test skeleton**
Example test skeleton:
```py
from SYS.result_table_adapters import get_provider
from SYS.result_table_adapters import get_plugin
from plugins import example_provider
def test_example_provider_registration():
def test_example_plugin_registration():
plugin = get_plugin("example")
rows = list(provider.adapter(example_provider.SAMPLE_ITEMS))
rows = list(plugin.adapter(example_provider.SAMPLE_ITEMS))
assert rows and rows[0].title
cols = provider.get_columns(rows)
assert any(c.name == "title" for c in cols)
table = provider.build_table(example_provider.SAMPLE_ITEMS)
cols = plugin.get_columns(rows)
assert any(col.name == "title" for col in cols)
table = plugin.build_table(example_provider.SAMPLE_ITEMS)
assert table.provider == "example" and table.rows
```
---
## References & examples
## References and examples
- Read `plugins/example_provider.py` for a compact example of a strict adapter and dynamic columns.
- Read `plugins/vimm/__init__.py` for a table-provider that uses `TableProviderMixin` and converts `SearchResult` → `ResultModel` for registration.
- See `docs/provider_guide.md` for a broader provider development checklist.
---
If you want, I can also add a small `plugins/myprovider_template.py` file and unit tests for it.
- Read `plugins/vimm/__init__.py` for a table-oriented plugin that uses `TableProviderMixin` and converts `SearchResult` to `ResultModel` for registration.
- See `docs/provider_guide.md` for a broader plugin development checklist.
docs/provider_guide.md
+72 -73
View File
@@ -1,62 +1,69 @@
# Plugin Development Guide
## 🎯 Purpose
This guide describes how to write, test, and register a plugin so the application can discover and use it as a pluggable component.
## Purpose
This guide describes how to write, test, and register a plugin so the
application can discover and use it as a pluggable component.
> Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in plugins share the same `plugins/` layout.
Note: this file keeps its historical `provider_guide` name, but the public
model is plugin-first. Some runtime classes still use `Provider` naming
internally.
Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in
plugins share the same `plugins/` layout.
---
## 🔧 Anatomy of a Plugin
A plugin is a Python class that extends `ProviderCore.base.Provider` and implements a few key methods and attributes.
## Anatomy of a plugin
A plugin is a Python class that currently extends the internal base class
`ProviderCore.base.Provider` and implements a few key methods and attributes.
Minimum expectations:
- `class MyPlugin(Provider):` — subclass the base plugin class
- `URL` / `URL_DOMAINS` or `url_patterns()` — to let the registry route URLs
- `validate(self) -> bool` — return True when the plugin is configured and usable
- `search(self, query, limit=50, filters=None, **kwargs)` — return a list of `SearchResult`
- `class MyPlugin(Provider):` subclasses the current internal base plugin class.
- `URL`, `URL_DOMAINS`, or `url_patterns()` let the registry route URLs.
- `validate(self) -> bool` returns `True` when the plugin is configured and usable.
- `search(self, query, limit=50, filters=None, **kwargs)` returns a list of `SearchResult` items.
Optional but common:
- `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]` — download a plugin result
- `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool` — handle `@N` selections
- `download_url(self, url, output_dir, progress_cb=None)` — direct URL-handling helper
- `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]`
- `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool`
- `download_url(self, url, output_dir, progress_cb=None)`
---
## 🧩 SearchResult
## SearchResult
Use `ProviderCore.base.SearchResult` to describe results returned by `search()`.
Important fields:
- `table` (str) — provider table name
- `title` (str) — short human title
- `path` (str) — canonical URL / link the provider/dl may use
- `media_kind` (str) — `file`, `folder`, `book`, etc.
- `columns` (list[tuple[str,str]]) — extra key/value pairs to display
- `full_metadata` (dict) — provider-specific metadata for downstream stages
- `annotations` / `tag` — simple metadata for filtering
- `table` (str): plugin table name
- `title` (str): short human title
- `path` (str): canonical URL or link the plugin or downloader may use
- `media_kind` (str): `file`, `folder`, `book`, and similar values
- `columns` (list[tuple[str, str]]): extra key/value pairs to display
- `full_metadata` (dict): plugin-specific metadata for downstream stages
- `annotations` or `tag`: simple metadata for filtering
Return a list of `SearchResult(...)` objects or simple dicts convertible with `.to_dict()`.
---
## ✅ Implementing search()
## Implementing `search()`
- Parse and sanitize `query` and `filters`.
- Return no more than `limit` results.
- Use `columns` to provide table columns (TITLE, Seeds, Size, etc.).
- Keep `search()` fast and predictable (apply reasonable timeouts).
- Use `columns` to provide table columns such as `TITLE`, `Seeds`, or `Size`.
- Keep `search()` fast and predictable by using reasonable timeouts.
Example:
```python
from ProviderCore.base import Provider, SearchResult
class HelloProvider(Provider):
class HelloPlugin(Provider):
def search(self, query, limit=50, filters=None, **kwargs):
q = (query or "").strip()
if not q:
return []
results = []
# Build up results
results.append(
results = [
SearchResult(
table="hello",
title=f"Hit for {q}",
@@ -64,102 +71,94 @@ class HelloProvider(Provider):
columns=[("Info", "example")],
full_metadata={"source": "hello"},
)
)
]
return results[:max(0, int(limit))]
```
---
## ⬇️ Implementing download() and download_url()
## Implementing `download()` and `download_url()`
- Prefer plugin `download(self, result, output_dir)` for piped plugin items.
- For plugin-provided URLs, implement `download_url` to allow `download-file` to route downloads through plugins.
- For plugin-provided URLs, implement `download_url` so `download-file` can route downloads through the plugin.
- Use the repo `_download_direct_file` helper for HTTP downloads when possible.
Example download():
Example download method:
```python
def download(self, result: SearchResult, output_dir: Path) -> Optional[Path]:
# Validate config
url = getattr(result, "path", None)
if not url or not url.startswith("http"):
return None
# use existing helpers to fetch the file
return _download_direct_file(url, output_dir)
```
---
## 🧭 URL routing
## URL routing
Plugins can declare:
- `URL = ("magnet:",)` or similar prefix list
- `URL = ("magnet:",)` or similar prefix lists
- `URL_DOMAINS = ("example.com",)` to match hosts
- Or override `@classmethod def url_patterns(cls):` to combine static and dynamic patterns
- `@classmethod def url_patterns(cls):` to combine static and dynamic patterns
The registry uses these to match `download-file <url>` or to pick which plugin should handle the URL.
The registry uses these declarations to match `download-file <url>` and to pick
which plugin should handle a URL.
---
## 🛠 Selector (handling `@N` picks)
- Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or to enqueue downloads.
- Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-ups.
- Return `True` when you handled the selection and the pipeline should pause or proceed accordingly.
## Selector behavior and `@N`
- Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or enqueue downloads.
- Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-up tables.
- Return `True` when the selector handled the selection and the pipeline should stop expanding that row.
---
## 🧪 Testing plugins
- Keep tests small and local. Create `tests/test_provider_<name>.py` or another tracked test target.
- Test `search()` with mock HTTP responses (use `requests-mock` or similar).
## Testing plugins
- Keep tests small and local.
- Create `tests/test_plugin_<name>.py` or follow the existing repo naming when extending older tests.
- Test `search()` with mock HTTP responses.
- Test `download()` using a temp directory and a small file server or by mocking `_download_direct_file`.
- Test `selector()` by constructing a fake result and `ctx` object.
Example PowerShell commands to run tests (repo root):
Example PowerShell commands from the repo root:
```powershell
# Run a single test file
pytest tests/test_provider_hello.py -q
# Run all tests
pytest tests/test_plugin_hello.py -q
pytest -q
```
---
## 📦 Registration & packaging
## Registration and packaging
- Bundled plugins live under `plugins/` and are auto-discovered from that package.
- External user plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` / `MEDEIA_PLUGIN_PATH`.
- Package directories are preferred so plugin-specific files can travel with the plugin.
- External plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` or `MEDEIA_PLUGIN_PATH`.
- Package directories are preferred so plugin-specific files travel with the plugin.
- Plugin authors should import from `ProviderCore.*`.
If a plugin supports multiple configured endpoints or accounts, the user-facing
concept is a plugin instance. Some stored config still lives under legacy key
paths such as `provider.<plugin>.<instance>`.
---
## 💡 Best practices & tips
- Use `debug()` / `log()` appropriately; avoid noisy stderr output in normal runs.
## Best practices
- Use `debug()` and `log()` appropriately; avoid noisy stderr output in normal runs.
- Prefer returning `SearchResult` objects to provide consistent UX.
- Keep `search()` tolerant (timeouts, malformed responses) and avoid raising for expected network problems.
- Keep `search()` tolerant of timeouts and malformed responses.
- Use `full_metadata` to pass non-display data to `download()` and `selector()`.
- Respect the `limit` parameter in `search()`.
---
## 🧾 Example provider checklist
- [ ] Implement `search()` and return `SearchResult` items
- [ ] Implement `validate()` to check essential config (API keys, credentials)
- [ ] Provide `URL` / `URL_DOMAINS` or `url_patterns()` for routing
- [ ] Add `download()` or `download_url()` for piped/passed URL downloads
- [ ] Add tests under `tests/`
- [ ] Add the plugin under `plugins/<name>/` for bundled or plug-and-play installs
## Example plugin checklist
- [ ] Implement `search()` and return `SearchResult` items.
- [ ] Implement `validate()` to check essential config such as API keys or credentials.
- [ ] Provide `URL`, `URL_DOMAINS`, or `url_patterns()` for routing.
- [ ] Add `download()` or `download_url()` for piped or passed URL downloads.
- [ ] Add tests under `tests/`.
- [ ] Add the plugin under `plugins/<name>/` for bundled or plug-and-play installs.
---
## 🔗 Further reading
## Further reading
- See existing bundled plugins in `plugins/` for patterns and edge cases.
- Check `API/` helpers for HTTP and debrid clients.
---
If you'd like, I can:
- Add an example plugin file under `plugins/` as a template (see `plugins/hello/__init__.py`), and
- Create unit tests for it (see `tests/test_provider_hello.py`).
I have added a minimal example plugin and tests in this repository; use them as a starting point for new plugins.
- Check `API/` helpers for HTTP and debrid clients used by plugins.
docs/result_table.md
+108 -153
View File
@@ -1,229 +1,184 @@
# ResultTable system — Overview & usage
# ResultTable system: overview and usage
This document explains the `ResultTable` system used across the CLI and TUI: how tables are built, how providers integrate with them, and how `@N` selection/expansion and provider selectors work.
This document explains the `ResultTable` system used across the CLI: how tables
are built, how plugins integrate with them, and how `@N` selection, row replay,
and plugin selectors work.
## TL;DR ✅
- `ResultTable` is the unified object used to render tabular results and drive selection (`@N`) behavior.
- Providers should return `SearchResult` objects (or dicts) and can either supply `selection_args` per row or implement a `selector()` method to handle `@N` selections.
- Table metadata (`set_table_metadata`) helps providers attach context (e.g., `provider_view`, `magnet_id`) that selectors can use.
## TL;DR
- `ResultTable` is the unified object used to render tabular results and drive `@N` behavior.
- Plugins return `SearchResult` objects or dicts and can either supply row selection args or implement a `selector()` method.
- Rows can also carry an explicit `selection_action`, which the CLI replays verbatim.
- Table metadata helps plugins attach context. Some metadata keys still use older `provider` names internally.
---
## Key concepts
- **ResultTable** (`SYS/result_table.py`)
- Renders rows as a rich table and stores metadata used for selection expansion.
- Important APIs: `add_result()`, `set_table()`, `set_source_command()`, `set_row_selection_args()`, `set_row_selection_action()`, `set_table_metadata()`, and `select_interactive()`.
### ResultTable
`SYS/result_table.py` renders rows as a rich table and stores metadata used for
selection expansion.
- **ResultRow**
- Holds columns plus `selection_args` (used for `@N` expansion) and `payload` (original object).
- Optionally stores `selection_action`, a full list of CLI tokens to run when `@N` selects this row. When present the CLI honors the explicit action instead of reconstructing it from `source_command` and `selection_args`.
Important APIs:
- `add_result()`
- `set_table()`
- `set_source_command()`
- `set_row_selection_args()`
- `set_row_selection_action()`
- `set_table_metadata()`
- **Provider selector**
- If a provider implements `selector(selected_items, ctx=..., stage_is_last=True)`, it is run first when `@N` is used; if the selector returns `True` it has handled the selection (e.g., drilling into a folder and publishing a new ResultTable).
### ResultRow
A row holds columns, payload data, selection args, and optionally a full
`selection_action` token list. When `selection_action` is present, `@N` uses it
instead of reconstructing a command from the source command and row args.
- **Pipeline / CLI expansion**
- When the user types `@N`, CLI tries provider selectors first. If none handle it, CLI re-runs `source_command + source_args + row_selection_args` (for single-selection) or pipes items downstream for multi-selection.
### Plugin selector
If a plugin implements `selector(selected_items, ctx=..., stage_is_last=True)`,
that selector runs first when `@N` is used. If the selector returns `True`, it
has handled the selection, for example by publishing a nested table.
- **Table metadata**
- `ResultTable.set_table_metadata(dict)` allows attaching provider-specific context (for example: `{"provider":"alldebrid","view":"files","magnet_id":123}`) for selectors and other code to use.
### Table metadata
`ResultTable.set_table_metadata(dict)` attaches plugin-specific context for
selectors and follow-up stages. You will still see legacy keys such as
`provider` in some metadata because parts of the runtime still consume them.
---
## How to build a table (provider pattern)
## Building a table
Typical provider flow (pseudocode):
Typical plugin flow:
```py
from SYS.result_table import ResultTable
table = ResultTable("Provider: X result").set_preserve_order(True)
table.set_table("provider_name")
table.set_table_metadata({"provider":"provider_name","view":"folders"})
table.set_source_command("search-file", ["-plugin","provider_name","query"])
table = ResultTable("Plugin: X results").set_preserve_order(True)
table.set_table("plugin_name")
table.set_table_metadata({"provider": "plugin_name", "view": "folders"})
table.set_source_command("search-file", ["-plugin", "plugin_name", "query"])
for r in results:
table.add_result(r) # r can be a SearchResult, dict, or PipeObject
for result in results:
table.add_result(result)
ctx.set_last_result_table(table, payloads)
ctx.set_current_stage_table(table)
```
Notes:
- To drive a direct `@N` re-run, call `table.set_row_selection_args(row_index, ["-open", "<id>"])`.
- For more advanced or interactive behavior (e.g., drill-into, fetch more rows), implement `provider.selector()` and return `True` when handled.
- To drive a direct replay, call `table.set_row_selection_args(row_index, ["-open", "<id>"])`.
- For drill-in or interactive behavior, implement `plugin.selector()` and return `True` when handled.
- For exact row replay, call `table.set_row_selection_action(row_index, tokens)`.
---
## Selection (@N) flow (brief)
## Selection flow
1. User enters `@N` in the CLI.
2. CLI chooses the appropriate table (overlay > last table > history) and gathers the selected payload(s).
3. `PipelineExecutor._maybe_run_class_selector()` runs provider `selector()` hooks for the provider inferred from table or payloads. If any selector returns `True`, expansion stops.
4. Otherwise, for single selections, CLI grabs `row.selection_args` and expands: `source_command + source_args + row_selection_args` and inserts it as the expanded stage. For multi-selections, items are piped downstream.
When a user enters `@N`:
1. The CLI chooses the active table.
2. It gathers the selected payloads.
3. `PipelineExecutor._maybe_run_class_selector()` runs plugin `selector()` hooks for the plugin inferred from the table or payloads.
4. If no selector handles the row, the CLI checks `row.selection_action` first.
5. If no explicit row action exists, the CLI expands `source_command + source_args + row_selection_args`.
6. Multi-selection results are piped downstream instead of being collapsed to one row replay.
---
## Columns & display
- Providers can pass a `columns` list ([(name, value), ...]) in the result dict/SearchResult to control which columns are shown and their order.
- Otherwise, `ResultTable` uses a priority list (title/store/size/ext) and sensible defaults.
- The table rendering functions (`to_rich`, `format_json`, `format_compact`) are available for different UIs.
## Columns and display
- Plugins can pass a `columns` list in the result dict or `SearchResult` to control column order and display.
- Otherwise, `ResultTable` uses sensible defaults.
- Rendering helpers such as `to_rich`, `format_json`, and `format_compact` support different CLI display paths.
---
## Provider-specific examples
## Plugin-specific examples
### AllDebrid (debrid file hosting)
### AllDebrid
AllDebrid exposes a list of magnets (folder rows) and the files inside each magnet. The provider returns `folder` SearchResults for magnets and `file` SearchResults for individual files. The provider includes a `selector()` that drills into a magnet by calling `search(..., filters={"view":"files","magnet_id":...})` and builds a new `ResultTable` of files.
AllDebrid exposes a list of magnets as folder rows and the files inside each
magnet as file rows. The plugin uses `selector()` to drill into a magnet and
publish a new `ResultTable` of files.
Example commands:
```
# List magnets in your account
```powershell
search-file -plugin alldebrid "*"
# Open magnet id 123 and list its files
search-file -plugin alldebrid -open 123 "*"
# Or expand via @ selection (selector handles drilling):
search-file -plugin alldebrid "*"
@3 # selector will open the magnet referenced by row #3 and show the file table
@3
```
Illustrative folder (magnet) SearchResult:
Illustrative magnet row:
```py
SearchResult(
table="alldebrid",
title="My Magnet Title",
path="alldebrid:magnet:123",
detail="OK",
annotations=["folder", "ready"],
media_kind="folder",
columns=[("Folder", "My Magnet Title"), ("ID", "123"), ("Status", "ready"), ("Ready", "yes")],
full_metadata={
"magnet": {...},
"magnet_id": 123,
"provider": "alldebrid",
"provider_view": "folders",
"magnet_name": "My Magnet Title",
},
table="alldebrid",
title="My Magnet Title",
path="alldebrid:magnet:123",
media_kind="folder",
full_metadata={
"magnet_id": 123,
"provider": "alldebrid",
"provider_view": "folders",
},
)
```
4. Otherwise, for single selections, CLI checks for `row.selection_action` and runs that verbatim if present; otherwise it expands `source_command + source_args + row_selection_args`. For multi-selections, items are piped downstream.
Illustrative file row:
```py
SearchResult(
table="alldebrid",
title="Episode 01.mkv",
path="https://.../unlocked_direct_url",
detail="My Magnet Title",
annotations=["file"],
media_kind="file",
size_bytes=123456789,
columns=[("File", "Episode 01.mkv"), ("Folder", "My Magnet Title"), ("ID", "123")],
full_metadata={
"magnet": {...},
"magnet_id": 123,
"magnet_name": "My Magnet Title",
"relpath": "Season 1/E01.mkv",
"provider": "alldebrid",
"provider_view": "files",
"file": {...},
},
table="alldebrid",
title="Episode 01.mkv",
path="https://.../unlocked_direct_url",
media_kind="file",
full_metadata={
"magnet_id": 123,
"provider": "alldebrid",
"provider_view": "files",
},
)
```
Selection & download flows
Selection and download behavior:
- `@3` on a magnet row runs the plugin selector and shows a file table.
- `@2 | download-file` downloads the selected file row.
- `@1 | add-file -plugin local -instance <dest>` triggers add-file's plugin-aware handoff flow.
- Drill-in (selector): `@3` on a magnet row runs the provider's `selector()` to build a new file table and show it. The selector uses `search(..., filters={"view":"files","magnet_id":...})` to fetch file rows.
Configure the AllDebrid plugin in `.config plugins` and add its API key before
testing these flows.
- `download-file` integration: With a file row (http(s) path), `@2 | download-file` will download the file. The `download-file` cmdlet expands AllDebrid magnet folders and will call the provider layer to fetch file bytes as appropriate.
### Bandcamp
- `add-file` convenience: Piping a file row into `add-file -plugin local -instance <dest>` will trigger add-file's provider-aware logic. If the piped item has `table == 'alldebrid'` and a http(s) `path`, `add-file` will call `provider.download()` into a temporary directory and then ingest the downloaded file, cleaning up the temp when done. Example:
Bandcamp search supports `artist:` queries. The Bandcamp plugin implements a
`selector()` that detects artist rows and expands them into a discography table.
```
# Expand magnet and add first file to local directory
search-file -plugin alldebrid "*"
@3 # view files
@1 | add-file -plugin local -instance C:\mydir
```
Example:
Notes & troubleshooting
- Configure an AllDebrid API key (see `Provider/alldebrid._get_debrid_api_key()`).
- If a magnet isn't ready the selector or `download-file` will log the magnet status and avoid attempting file downloads.
---
### Bandcamp (artist → discography drill-in)
Bandcamp search supports `artist:` queries. Bandcamp's provider implements a `selector()` that detects `artist` results and scrapes the artist's page using Playwright to build a discography `ResultTable`.
Example usage:
```
# Search for an artist
```powershell
search-file -plugin bandcamp "artist:radiohead"
# Select an artist row to expand into releases
@1
```
Bandcamp SearchResult (artist / album rows):
```py
SearchResult(
table="bandcamp",
title="Album Title",
path="https://bandcamp.com/album_url",
detail="By: Artist",
annotations=["album"],
media_kind="audio",
columns=[("Title","Album Title"), ("Location","Artist"), ("Type","album"), ("Url","https://...")],
full_metadata={"artist":"Artist","type":"album","url":"https://..."}
)
```
Notes:
- Playwright is required for Bandcamp scraping. The selector will log an informative message if Playwright is missing.
- Provider selectors are ideal when you need to replace one table with another (artist → discography).
Plugin selectors are the right tool when you need to replace one table with
another, such as artist to discography drill-in.
---
## Provider author checklist (short)
- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts; include useful `full_metadata` (IDs, view names) for selection/drilling.
- If you support fetching downloadable file bytes, implement `download(result, output_dir) -> Optional[Path]`.
- For drill-in or interactive transforms, implement `selector(selected_items, ctx=..., stage_is_last=True)` and call `ctx.set_last_result_table(...)` / `ctx.set_current_stage_table(...)`; return `True` when handled.
- Add tests (unit/integration) that exercise search → select → download flows.
## Plugin author checklist
- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts.
- Include useful `full_metadata` values for drill-in and selection replay.
- Implement `download(result, output_dir)` when the plugin can materialize file bytes.
- Implement `selector(...)` for drill-in or interactive transforms.
- Add tests that cover search, select, and download flows.
---
## Debugging tips
- Use `ctx.set_last_result_table(table, payloads)` to immediately show a table while developing a selector.
- Add `log(...)` messages in provider code to capture fail points.
- Check `full_metadata` attached to SearchResults to pass extra context (IDs, view names, provider names).
- Use `ctx.set_last_result_table(table, payloads)` to publish a table while developing a selector.
- Add `log(...)` messages in plugin code to capture fail points.
- Inspect `full_metadata` when a selector or replay path is missing required context.
---
## Quick reference
- ResultTable location: `SYS/result_table.py`
- Pipeline helpers: `SYS/pipeline.py` (`set_last_result_table`, `set_current_stage_table`, `get_current_stage_table_row_selection_args`)
- CLI expansion: `CLI.py` (handles `@N`, provider selectors, and insertion of expanded stages)
- Provider selector pattern: Implement `.selector(selected_items, ctx=..., stage_is_last=True)` in provider class.
---
For more detail on ResultTable provider authoring, see `docs/provider_authoring.md`.
If you'd like, I can also:
- Add provider-specific examples (AllDebrid, Bandcamp) into this doc ✅
- Add a short checklist for PR reviewers when adding new providers
---
Created by GitHub Copilot (Raptor mini - Preview) — brief guide to the ResultTable system. Feedback welcome!
- ResultTable implementation: `SYS/result_table.py`
- Pipeline helpers: `SYS/pipeline.py`
- Row replay and selection flow: `SYS/pipeline.py`
- Plugin authoring guide: `docs/provider_authoring.md`
docs/result_table_selector.md
+15 -12
View File
@@ -1,22 +1,25 @@
Selector & plugin-table usage
Selector and `plugin-table` usage
This project provides a small plugin/table/selector flow that allows plugins
and cmdlets to interact via a simple, pipable API.
This project provides a small plugin/table/selector flow that lets plugins and
cmdlets interact through a simple pipable API.
Key ideas
- `plugin-table` renders a plugin result set and *emits* pipeline-friendly dicts for each row. Each emitted item includes `_selection_args`, a list of args the plugin suggests for selecting that row (e.g., `['-path', '/tmp/file']`).
- Use the `@N` syntax to select an item from a table and chain it to the next cmdlet.
Key ideas:
- `plugin-table` renders a plugin result set and emits pipeline-friendly dicts for each row.
- Emitted rows can include `_selection_args` and, when needed, `_selection_action` for exact row replay.
- Use `@N` to select an item from a table and chain it to the next cmdlet.
Example:
plugin-table -plugin example -sample | @1 | add-file -instance default
What plugins must implement
- An adapter that yields `ResultModel` objects (breaking API).
- Optionally supply a `columns` factory and `selection_fn` (see `plugins/example_provider.py`).
What plugins must implement:
- An adapter that yields `ResultModel` objects.
- Optionally a `columns` factory and `selection_fn`.
Implementation notes
Implementation notes:
- `plugin-table` emits dicts like `{ 'title': ..., 'path': ..., 'metadata': ..., '_selection_args': [...] }`.
- Selection syntax (`@1`) will prefer `_selection_args` if present; otherwise it will fall back to plugin selection logic or sensible defaults (`-path` or `-title`).
- When a row includes `_selection_action`, `@1` prefers that exact action.
- Otherwise `@1` falls back to `_selection_args`, then plugin selection logic, then sensible defaults such as `-path` or `-title`.
This design keeps the selector-focused UX small and predictable while enabling full cmdlet interoperability via piping and `-run-cmd`.
This design keeps selector-focused workflows small and predictable while still
allowing full cmdlet interoperability through piping and `-run-cmd`.
docs/scp_plugin_tutorial.md
+26 -48
View File
@@ -1,23 +1,26 @@
# SCP Plugin Walkthrough
This walkthrough adds a bundled `scp` plugin backed by existing SSH libraries:
This walkthrough covers the bundled `scp` plugin, backed by existing SSH
libraries:
- `paramiko` for SSH and SFTP directory listing
- `scp` for file transfers
The implementation lives in [plugins/scp/__init__.py](plugins/scp/__init__.py).
## What The Plugin Does
## What the plugin does
The SCP plugin mirrors the FTP walkthrough, but on top of SSH:
- `search-file -plugin scp -instance <name> ...` lists remote files and folders over SFTP
- plain `@N` on a folder drills into that directory
- plain `@N` on a file runs `download-file -plugin scp -instance <name> -url ...`
- `@N | add-file -instance ...` downloads first, then ingests the local temp file
- `add-file -plugin scp -instance <name> -path ...` uploads a local file to the configured remote path
- `search-file -plugin scp -instance <name> ...` lists remote files and folders over SFTP.
- plain `@N` on a folder drills into that directory.
- plain `@N` on a file runs `download-file -plugin scp -instance <name> -url ...`.
- `@N | add-file -instance ...` downloads first, then ingests the local temp file.
- `add-file -plugin scp -instance <name> -path ...` uploads a local file to the configured remote path.
## Example config
## Example Config
Add one or more named SCP plugin instances to your config. The current stored
key path remains `provider.scp.<instance>` for legacy compatibility:
```toml
[provider.scp.work]
@@ -42,39 +45,22 @@ timeout = 20
```
Notes:
- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`.
- `work` and `archive` are instance names.
- `host` and `username` are required for each instance to validate.
- You can use password auth, key auth, or both.
- `base_path` is both the default search root and the default upload directory.
- You can browse configured instances from `.config plugins` in the CLI.
## Search Flow
List the configured base path:
## Search flow
```powershell
search-file -plugin scp -instance work "*"
```
Search by filename:
```powershell
search-file -plugin scp -instance work "invoice"
```
Search another subtree with deeper recursion:
```powershell
search-file -plugin scp -instance work "path:/srv/files/releases depth:2 *.zip"
```
Show only folders:
```powershell
search-file -plugin scp -instance work "path:/srv/files type:folder *"
```
## Selection Flow
## Selection flow
Folder rows are navigation rows:
@@ -83,7 +69,8 @@ search-file -plugin scp -instance work "*"
@2
```
File rows carry an explicit row action, so terminal selection downloads directly:
File rows carry an explicit row action, so terminal selection downloads
directly:
```powershell
search-file -plugin scp -instance work "report"
@@ -96,45 +83,36 @@ That expands to the equivalent of:
download-file -plugin scp -instance work -url scp://ssh.example.com/srv/files/report.pdf
```
## Download And Add-File Flow
Download into a local folder:
## Download and add-file flow
```powershell
search-file -plugin scp -instance work "report"
@1 | download-file -path C:\Downloads
```
Ingest a selected remote file into a configured instance backend:
```powershell
search-file -plugin scp -instance work "report"
@1 | add-file -instance tutorial
```
Why this works:
- file rows advertise `_selection_action` for `download-file`
- `add-file` selection replay inserts that provider download stage before ingest
- the plugin also implements `resolve_pipe_result_download()` for provider-owned SCP rows
- file rows also carry the chosen `instance`, so replay stays bound to the same SSH target
- `add-file` selection replay inserts that plugin download stage before ingest
- the plugin also implements `resolve_pipe_result_download()` for plugin-owned SCP rows
- file rows carry the chosen `instance`, so replay stays bound to the same SSH target
## Upload Flow
Upload a local file to the configured remote `base_path`:
## Upload flow
```powershell
add-file -plugin scp -instance archive -path C:\Media\report.pdf
```
## Implementation Notes
The plugin uses SFTP for directory listing because SCP itself is a transfer protocol, not a browse/search protocol. That split keeps the provider simple:
## Implementation notes
The plugin uses SFTP for directory listing because SCP itself is a transfer
protocol, not a browse/search protocol. That split keeps the plugin simple:
- browse and metadata via Paramiko SFTP
- file transfer via the `scp` package
## Recommended Demo Commands
## Recommended demo commands
```powershell
search-file -plugin scp -instance work "*"