This commit is contained in:
2026-05-16 15:26:08 -07:00
parent 5048729b0c
commit 02d84f423e
10 changed files with 488 additions and 470 deletions
+3 -3
View File
@@ -2,8 +2,8 @@
## Unreleased (2026-01-05) ## Unreleased (2026-01-05)
- **docs:** Add `docs/provider_authoring.md` with a Quick Start, examples, and testing guidance for providers that integrate with the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn). - **docs:** Add `docs/provider_authoring.md` as a plugin authoring Quick Start for the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn). The file keeps its legacy name for compatibility.
- **docs:** Add link to `docs/result_table.md` pointing to the provider authoring guide. - **docs:** Add a link from `docs/result_table.md` to the plugin authoring guide.
- **tests:** Add `tests/test_provider_author_examples.py` validating example provider registration and adapter behavior. - **tests:** Add `tests/test_provider_author_examples.py` validating example plugin registration and adapter behavior. The test file keeps its legacy name for compatibility.
- **notes:** Existing example plugins (`plugins/example_provider.py`, `plugins/vimm/__init__.py`) are referenced as canonical patterns. - **notes:** Existing example plugins (`plugins/example_provider.py`, `plugins/vimm/__init__.py`) are referenced as canonical patterns.
+6 -6
View File
@@ -1,15 +1,15 @@
PR Title: docs: Add Provider authoring doc, examples, and tests PR Title: docs: Add plugin authoring doc, examples, and tests
Summary: Summary:
- Add `docs/provider_authoring.md` describing the strict `ResultModel`-based provider adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping. - Add `docs/provider_authoring.md` describing the strict `ResultModel`-based plugin adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping.
- Link new doc from `docs/result_table.md`. - Link the new doc from `docs/result_table.md`.
- Add `tests/test_provider_author_examples.py` to validate `Provider/example_provider.py` and `Provider/vimm.py` integration with the registry. - Add `tests/test_provider_author_examples.py` to validate the example plugin integration with the registry.
Why: Why:
- Provide a short, focused Quick Start to help contributors author providers that integrate with the new strict ResultTable API. - Provide a short, focused Quick Start to help contributors author plugins that integrate with the strict ResultTable API.
Testing: Testing:
- New tests pass locally (provider-related subset). - New tests pass locally (plugin-related subset).
Notes: Notes:
- The change is documentation-first and non-functional, with tests ensuring examples remain valid. - The change is documentation-first and non-functional, with tests ensuring examples remain valid.
+31 -53
View File
@@ -1,6 +1,6 @@
# FTP Plugin Walkthrough # FTP Plugin Walkthrough
This walkthrough adds a real bundled `ftp` plugin so users can: This walkthrough covers the bundled `ftp` plugin. It lets users:
- run `search-file -plugin ftp -instance <name> ...` - run `search-file -plugin ftp -instance <name> ...`
- browse remote folders as result tables - browse remote folders as result tables
@@ -10,9 +10,10 @@ This walkthrough adds a real bundled `ftp` plugin so users can:
The implementation lives in [plugins/ftp/__init__.py](plugins/ftp/__init__.py). The implementation lives in [plugins/ftp/__init__.py](plugins/ftp/__init__.py).
## What The Plugin Does ## What the plugin does
The FTP plugin demonstrates the main provider hooks that matter for a storage-style integration: The FTP plugin demonstrates the main plugin hooks that matter for a
storage-style integration:
- `config_schema()` exposes host, credentials, base path, TLS, and search depth. - `config_schema()` exposes host, credentials, base path, TLS, and search depth.
- `extract_query_arguments()` supports inline query fields like `path:` and `depth:`. - `extract_query_arguments()` supports inline query fields like `path:` and `depth:`.
@@ -22,9 +23,10 @@ The FTP plugin demonstrates the main provider hooks that matter for a storage-st
- `resolve_pipe_result_download()` lets `@N | add-file -instance ...` materialize a remote FTP file first. - `resolve_pipe_result_download()` lets `@N | add-file -instance ...` materialize a remote FTP file first.
- `upload()` lets `add-file -plugin ftp -instance <name> -path ...` push a local file to the configured FTP server. - `upload()` lets `add-file -plugin ftp -instance <name> -path ...` push a local file to the configured FTP server.
## Example Config ## Example config
Add one or more named FTP provider instances to your config: Add one or more named FTP plugin instances to your config. The current stored
key path remains `provider.ftp.<instance>` for legacy compatibility:
```toml ```toml
[provider.ftp.work] [provider.ftp.work]
@@ -48,121 +50,97 @@ tls = true
``` ```
Notes: Notes:
- `work` and `archive` are instance names.
- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`.
- `host` is the only required field for each instance to validate. - `host` is the only required field for each instance to validate.
- `username` defaults to `anonymous` and `password` defaults to `anonymous@`. - `username` defaults to `anonymous` and `password` defaults to `anonymous@`.
- `base_path` is both the default search root and the upload target directory. - `base_path` is both the default search root and the upload target directory.
- `search_depth` controls how many folder levels `search-file -plugin ftp` scans by default. - `search_depth` controls how many folder levels `search-file -plugin ftp` scans by default.
- You can browse configured instances from `.config plugins` in the CLI.
## Search Flow ## Search flow
Basic listing from the configured base path:
```powershell ```powershell
search-file -plugin ftp -instance work "*" search-file -plugin ftp -instance work "*"
```
Search by filename fragment:
```powershell
search-file -plugin ftp -instance work "invoice" search-file -plugin ftp -instance work "invoice"
```
Search a different subtree and recurse deeper:
```powershell
search-file -plugin ftp -instance work "path:/pub depth:2 invoice" search-file -plugin ftp -instance work "path:/pub depth:2 invoice"
```
Filter to folders only:
```powershell
search-file -plugin ftp -instance work "path:/pub type:folder *" search-file -plugin ftp -instance work "path:/pub type:folder *"
``` ```
The plugin returns rows with explicit columns for name, type, directory, size, and modification time. The plugin returns rows with explicit columns for name, type, directory, size,
and modification time.
## Selection Flow ## Selection flow
Folder rows are navigation rows. If the selected row is a directory, plain `@N` opens a new FTP table for that directory: Folder rows are navigation rows:
```powershell ```powershell
search-file -plugin ftp -instance work "*" search-file -plugin ftp -instance work "*"
@2 @2
``` ```
File rows carry an explicit row action: File rows carry an explicit row action equivalent to:
```powershell ```powershell
download-file -plugin ftp -instance work -url ftp://ftp.example.com/incoming/report.pdf download-file -plugin ftp -instance work -url ftp://ftp.example.com/incoming/report.pdf
``` ```
That means plain `@N` on a file row downloads it immediately: So plain `@N` on a file row downloads it immediately:
```powershell ```powershell
search-file -plugin ftp -instance work "report" search-file -plugin ftp -instance work "report"
@1 @1
``` ```
## Download And Add-File Flow ## Download and add-file flow
If you want the downloaded file in a specific local directory:
```powershell ```powershell
search-file -plugin ftp -instance work "report" search-file -plugin ftp -instance work "report"
@1 | download-file -path C:\Downloads @1 | download-file -path C:\Downloads
```
If you want to ingest the selected FTP file into a configured instance backend:
```powershell
search-file -plugin ftp -instance work "report" search-file -plugin ftp -instance work "report"
@1 | add-file -instance tutorial @1 | add-file -instance tutorial
``` ```
Why this works: Why this works:
- the file row advertises a `download-file` row action - the file row advertises a `download-file` row action
- the pipeline auto-inserts that download before `add-file` - the pipeline auto-inserts that download before `add-file`
- the FTP plugin also implements `resolve_pipe_result_download()` so provider-owned FTP rows can be materialized for ingestion - the FTP plugin also implements `resolve_pipe_result_download()` so plugin-owned FTP rows can be materialized for ingestion
- file rows also carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target - file rows carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target
## Upload Flow ## Upload flow
Uploading uses the same provider name, but through `add-file -plugin ftp -instance <name>`: Uploading uses the same plugin, through `add-file -plugin ftp -instance <name>`:
```powershell ```powershell
add-file -plugin ftp -instance archive -path C:\Media\report.pdf add-file -plugin ftp -instance archive -path C:\Media\report.pdf
``` ```
That sends the file to the selected instance's FTP `base_path` and returns the FTP URL as the uploaded result. That sends the file to the selected instance's FTP `base_path` and returns the
FTP URL as the uploaded result.
## Why The Row Metadata Matters ## Why the row metadata matters
The critical part of this plugin is the file-row metadata: The critical part of this plugin is the file-row metadata:
- file rows emit `_selection_args` as `['-instance', '<name>', '-url', '<ftp-url>']` - file rows emit `_selection_args` as `['-instance', '<name>', '-url', '<ftp-url>']`
- file rows emit `_selection_action` as `['download-file', '-plugin', 'ftp', '-instance', '<name>', '-url', '<ftp-url>']` - file rows emit `_selection_action` as `['download-file', '-plugin', 'ftp', '-instance', '<name>', '-url', '<ftp-url>']`
- folder rows do not emit a download action, so `selector()` can own drill-in behavior instead - folder rows do not emit a download action, so `selector()` owns drill-in behavior instead
That split is what keeps these two user experiences compatible:
That keeps these flows compatible:
- `@N` on a folder opens a new table - `@N` on a folder opens a new table
- `@N` on a file downloads the file - `@N` on a file downloads the file
- `@N | add-file -instance ...` first downloads, then ingests - `@N | add-file -instance ...` first downloads, then ingests
## Implementation Notes ## Implementation notes
The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus directory probes when the server does not support machine-readable listings. The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus
directory probes when the server does not support machine-readable listings.
The code is intentionally small and uses only Python stdlib pieces:
The code intentionally stays small and uses only Python stdlib pieces:
- `ftplib` for FTP and FTPS - `ftplib` for FTP and FTPS
- `fnmatch` for wildcard-style search tokens - `fnmatch` for wildcard-style search tokens
- `tempfile` for `add-file` handoff downloads - `tempfile` for `add-file` handoff downloads
## Recommended Commands To Demo The Walkthrough ## Recommended demo commands
```powershell ```powershell
search-file -plugin ftp -instance work "*" search-file -plugin ftp -instance work "*"
+66 -58
View File
@@ -1,37 +1,48 @@
# Plugin authoring: ResultTable & plugin adapters # Plugin authoring: ResultTable and plugin adapters
This short guide explains how to write plugins that integrate with the *strict* ResultTable API: adapters must yield `ResultModel` instances and plugins register via `SYS.result_table_adapters.register_plugin` with a column specification and a `selection_fn`. This short guide explains how to write plugins that integrate with the strict
ResultTable API: adapters yield `ResultModel` instances, and plugins register
via `SYS.result_table_adapters.register_plugin` with columns and a
`selection_fn`.
Note: this file keeps its historical `provider_authoring` name, but the public
terminology is plugin-first. Some internal classes and metadata fields still use
`Provider` naming.
--- ---
## Quick summary ## Quick summary
- Plugins register a plugin adapter, a `columns` definition, and a `selection_fn`.
- Plugins register a *plugin adapter* (callable that yields `ResultModel`). - `selection_fn` returns CLI args for a selected row.
- Plugins must also provide `columns` (static list or factory) and a `selection_fn` that returns CLI args for a selected row. - For HTML table or list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers`.
- For simple HTML table/list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers` to fetch and extract rows using `SYS.html_table.extract_records`.
## Runtime dependency policy ## Runtime dependency policy
- Treat required runtime dependencies such as Playwright as mandatory: import them unconditionally and let missing dependencies fail fast.
- Treat required runtime dependencies (e.g., **Playwright**) as mandatory: import them unconditionally and let missing dependencies fail fast at import time. Avoid adding per-call try/except import guards for required modules—these silently hide configuration errors and add bloat. - Use guarded imports only for truly optional dependencies such as `pandas`.
- Use guarded imports only for truly optional dependencies (e.g., `pandas` for enhanced table parsing) and provide meaningful fallbacks or helpful error messages in those cases. - Keep plugin code minimal and explicit: fail early and document required runtime dependencies in README and installation notes.
- Keep provider code minimal and explicit: fail early and document required runtime dependencies in README/installation notes.
--- ---
## Minimal provider template (copy/paste) ## Minimal plugin template
```py ```py
# plugins/my_plugin.py # plugins/my_plugin.py
from typing import Any, Dict, Iterable, List from typing import Any, Dict, Iterable, List
from SYS.result_table_api import ResultModel, ColumnSpec, title_column, metadata_column from SYS.result_table_api import ResultModel, ColumnSpec, title_column
from SYS.result_table_adapters import register_plugin from SYS.result_table_adapters import register_plugin
# Example adapter: convert provider-specific items into ResultModel instances
SAMPLE_ITEMS = [ SAMPLE_ITEMS = [
{"name": "Example File.pdf", "path": "https://example.com/x.pdf", "ext": "pdf", "size": 1024, "source": "myprovider"}, {
"name": "Example File.pdf",
"path": "https://example.com/x.pdf",
"ext": "pdf",
"size": 1024,
"source": "myplugin",
},
] ]
def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]: def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]:
for it in items: for it in items:
title = it.get("name") or it.get("title") or str(it.get("path") or "") title = it.get("name") or it.get("title") or str(it.get("path") or "")
@@ -41,39 +52,38 @@ def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]:
ext=str(it.get("ext")) if it.get("ext") else None, ext=str(it.get("ext")) if it.get("ext") else None,
size_bytes=int(it.get("size")) if it.get("size") is not None else None, size_bytes=int(it.get("size")) if it.get("size") is not None else None,
metadata=dict(it), metadata=dict(it),
source=str(it.get("source")) if it.get("source") else "myprovider", source=str(it.get("source")) if it.get("source") else "myplugin",
) )
# Optional: build columns dynamically from sample rows
def columns_factory(rows: List[ResultModel]) -> List[ColumnSpec]: def columns_factory(rows: List[ResultModel]) -> List[ColumnSpec]:
cols = [title_column()] cols = [title_column()]
# add extra columns if metadata keys exist if any((row.metadata or {}).get("size") for row in rows):
if any((r.metadata or {}).get("size") for r in rows): cols.append(ColumnSpec("size", "Size", lambda row: row.size_bytes or ""))
cols.append(ColumnSpec("size", "Size", lambda r: r.size_bytes or ""))
return cols return cols
# Selection args for `@N` expansion or `select` cmdlet
def selection_fn(row: ResultModel) -> List[str]: def selection_fn(row: ResultModel) -> List[str]:
# prefer -path when available
if row.path: if row.path:
return ["-path", row.path] return ["-path", row.path]
return ["-title", row.title or ""] return ["-title", row.title or ""]
# Register plugin (done at import time)
register_plugin("myprovider", adapter, columns=columns_factory, selection_fn=selection_fn) register_plugin("myplugin", adapter, columns=columns_factory, selection_fn=selection_fn)
``` ```
--- ---
## Table scraping: using TableProviderMixin (HTML tables / list-results) ## Table scraping with `TableProviderMixin`
If your provider scrapes HTML tables or list-like results (common on web search pages), use `TableProviderMixin`: If your plugin scrapes HTML tables or list-like results, use `TableProviderMixin`:
```py ```py
from ProviderCore.base import Provider from ProviderCore.base import Provider
from SYS.provider_helpers import TableProviderMixin from SYS.provider_helpers import TableProviderMixin
class MyTableProvider(TableProviderMixin, Provider):
class MyTablePlugin(TableProviderMixin, Provider):
URL = ("https://example.org/search",) URL = ("https://example.org/search",)
def validate(self) -> bool: def validate(self) -> bool:
@@ -84,58 +94,56 @@ class MyTableProvider(TableProviderMixin, Provider):
return self.search_table_from_url(url, limit=limit) return self.search_table_from_url(url, limit=limit)
``` ```
`TableProviderMixin.search_table_from_url` returns `ProviderCore.base.SearchResult` entries. If you want to integrate this plugin with the strict `ResultTable` registry, add a small adapter that converts `SearchResult` -> `ResultModel` and register it using `register_plugin` (see `plugins/vimm/__init__.py` for a real example). `TableProviderMixin.search_table_from_url` returns
`ProviderCore.base.SearchResult` entries. If you want to integrate the plugin
with the strict `ResultTable` registry, add a small adapter that converts
`SearchResult` to `ResultModel` and register it using `register_plugin`.
--- ---
## Columns & selection ## Columns and selection
- `columns` may be a static `List[ColumnSpec]` or a factory that inspects sample rows.
- `columns` may be a static `List[ColumnSpec]` or a factory `def cols(rows: List[ResultModel]) -> List[ColumnSpec]` that inspects sample rows. - `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args.
- `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args (e.g., `['-path', row.path]`). These args are used by `select` and `@N` expansion. - For downloadable file rows, prefer explicit URL args such as `['-url', row.path]` so downstream downloaders interpret the row unambiguously.
- Ensure `ResultModel.source` is set directly or falls back to the registered plugin name during serialization.
**Tip:** for plugins that produce downloadable file rows prefer returning explicit URL args (e.g., `['-url', row.path]`) so the selected URL is clearly identified by downstream downloaders and to avoid ambiguous parsing when plugin hints (like `-plugin`) are present.
- Ensure your `ResultModel.source` is set (either in the model or rely on the provider name set by `serialize_row`).
--- ---
## Optional: pandas path for `<table>` extraction ## Optional pandas support
`SYS.html_table.extract_records` prefers a pure-lxml path but can fall back to
`SYS.html_table.extract_records` prefers a pure-lxml path but will use `pandas.read_html` if pandas is installed and the helper detects it works for the input table. This is optional and **not required** to author a provider — document in your provider whether it requires `pandas` and add an informative error/log message when it is missing. `pandas.read_html` when pandas is installed and the helper detects it works for
the input table. This is optional. Document whether your plugin requires
`pandas` and emit a clear error or log message when it is missing.
--- ---
## Testing & examples ## Testing and examples
- Write `tests/test_plugin_<name>.py` or follow the repo's older naming conventions when extending existing tests.
- Verify `plugin.build_table(...)` produces a `ResultTable` with rows and columns.
- Verify `serialize_rows()` yields `_selection_args`, `_selection_action` when applicable, and `source`.
- When you need an exact CLI stage sequence, call `table.set_row_selection_action(row_index, tokens)` so replay uses the row action verbatim.
- For table-oriented plugins, test `search_table_from_url` with a local HTML fixture or a mocked `HTTPClient`.
- Write `tests/test_provider_<name>.py` that imports your provider and verifies `provider.build_table(...)` produces a `ResultTable` (has `.rows` and `.columns`) and that `serialize_rows()` yields dicts with `_selection_args`, `_selection_action` when applicable, and `source`. Example test skeleton:
- When you need to guarantee a specific CLI stage sequence (e.g., `download-file -url <path> -plugin <name>`), call `table.set_row_selection_action(row_index, tokens)` so the serialized payload emits `_selection_action` and the CLI can run the row exactly as intended.
- For table providers you can test `search_table_from_url` using a local HTML fixture or by mocking `HTTPClient` to return a small sample page.
- If you rely on pandas, add a test that monkeypatches `sys.modules['pandas']` to a simple shim to validate the pandas path.
**Example test skeleton**
```py ```py
from SYS.result_table_adapters import get_provider from SYS.result_table_adapters import get_plugin
from plugins import example_provider from plugins import example_provider
def test_example_provider_registration(): def test_example_plugin_registration():
plugin = get_plugin("example") plugin = get_plugin("example")
rows = list(provider.adapter(example_provider.SAMPLE_ITEMS)) rows = list(plugin.adapter(example_provider.SAMPLE_ITEMS))
assert rows and rows[0].title assert rows and rows[0].title
cols = provider.get_columns(rows) cols = plugin.get_columns(rows)
assert any(c.name == "title" for c in cols) assert any(col.name == "title" for col in cols)
table = provider.build_table(example_provider.SAMPLE_ITEMS) table = plugin.build_table(example_provider.SAMPLE_ITEMS)
assert table.provider == "example" and table.rows assert table.provider == "example" and table.rows
``` ```
--- ---
## References & examples ## References and examples
- Read `plugins/example_provider.py` for a compact example of a strict adapter and dynamic columns. - Read `plugins/example_provider.py` for a compact example of a strict adapter and dynamic columns.
- Read `plugins/vimm/__init__.py` for a table-provider that uses `TableProviderMixin` and converts `SearchResult` `ResultModel` for registration. - Read `plugins/vimm/__init__.py` for a table-oriented plugin that uses `TableProviderMixin` and converts `SearchResult` to `ResultModel` for registration.
- See `docs/provider_guide.md` for a broader provider development checklist. - See `docs/provider_guide.md` for a broader plugin development checklist.
---
If you want, I can also add a small `plugins/myprovider_template.py` file and unit tests for it.
+72 -73
View File
@@ -1,62 +1,69 @@
# Plugin Development Guide # Plugin Development Guide
## 🎯 Purpose ## Purpose
This guide describes how to write, test, and register a plugin so the application can discover and use it as a pluggable component. This guide describes how to write, test, and register a plugin so the
application can discover and use it as a pluggable component.
> Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in plugins share the same `plugins/` layout. Note: this file keeps its historical `provider_guide` name, but the public
model is plugin-first. Some runtime classes still use `Provider` naming
internally.
Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in
plugins share the same `plugins/` layout.
--- ---
## 🔧 Anatomy of a Plugin ## Anatomy of a plugin
A plugin is a Python class that extends `ProviderCore.base.Provider` and implements a few key methods and attributes. A plugin is a Python class that currently extends the internal base class
`ProviderCore.base.Provider` and implements a few key methods and attributes.
Minimum expectations: Minimum expectations:
- `class MyPlugin(Provider):` subclass the base plugin class - `class MyPlugin(Provider):` subclasses the current internal base plugin class.
- `URL` / `URL_DOMAINS` or `url_patterns()` — to let the registry route URLs - `URL`, `URL_DOMAINS`, or `url_patterns()` let the registry route URLs.
- `validate(self) -> bool` return True when the plugin is configured and usable - `validate(self) -> bool` returns `True` when the plugin is configured and usable.
- `search(self, query, limit=50, filters=None, **kwargs)` return a list of `SearchResult` - `search(self, query, limit=50, filters=None, **kwargs)` returns a list of `SearchResult` items.
Optional but common: Optional but common:
- `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]` — download a plugin result - `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]`
- `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool` — handle `@N` selections - `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool`
- `download_url(self, url, output_dir, progress_cb=None)` — direct URL-handling helper - `download_url(self, url, output_dir, progress_cb=None)`
--- ---
## 🧩 SearchResult ## SearchResult
Use `ProviderCore.base.SearchResult` to describe results returned by `search()`. Use `ProviderCore.base.SearchResult` to describe results returned by `search()`.
Important fields: Important fields:
- `table` (str) — provider table name - `table` (str): plugin table name
- `title` (str) short human title - `title` (str): short human title
- `path` (str) canonical URL / link the provider/dl may use - `path` (str): canonical URL or link the plugin or downloader may use
- `media_kind` (str) `file`, `folder`, `book`, etc. - `media_kind` (str): `file`, `folder`, `book`, and similar values
- `columns` (list[tuple[str,str]]) extra key/value pairs to display - `columns` (list[tuple[str, str]]): extra key/value pairs to display
- `full_metadata` (dict) — provider-specific metadata for downstream stages - `full_metadata` (dict): plugin-specific metadata for downstream stages
- `annotations` / `tag` simple metadata for filtering - `annotations` or `tag`: simple metadata for filtering
Return a list of `SearchResult(...)` objects or simple dicts convertible with `.to_dict()`. Return a list of `SearchResult(...)` objects or simple dicts convertible with `.to_dict()`.
--- ---
## Implementing search() ## Implementing `search()`
- Parse and sanitize `query` and `filters`. - Parse and sanitize `query` and `filters`.
- Return no more than `limit` results. - Return no more than `limit` results.
- Use `columns` to provide table columns (TITLE, Seeds, Size, etc.). - Use `columns` to provide table columns such as `TITLE`, `Seeds`, or `Size`.
- Keep `search()` fast and predictable (apply reasonable timeouts). - Keep `search()` fast and predictable by using reasonable timeouts.
Example: Example:
```python ```python
from ProviderCore.base import Provider, SearchResult from ProviderCore.base import Provider, SearchResult
class HelloProvider(Provider):
class HelloPlugin(Provider):
def search(self, query, limit=50, filters=None, **kwargs): def search(self, query, limit=50, filters=None, **kwargs):
q = (query or "").strip() q = (query or "").strip()
if not q: if not q:
return [] return []
results = [] results = [
# Build up results
results.append(
SearchResult( SearchResult(
table="hello", table="hello",
title=f"Hit for {q}", title=f"Hit for {q}",
@@ -64,102 +71,94 @@ class HelloProvider(Provider):
columns=[("Info", "example")], columns=[("Info", "example")],
full_metadata={"source": "hello"}, full_metadata={"source": "hello"},
) )
) ]
return results[:max(0, int(limit))] return results[:max(0, int(limit))]
``` ```
--- ---
## ⬇️ Implementing download() and download_url() ## Implementing `download()` and `download_url()`
- Prefer plugin `download(self, result, output_dir)` for piped plugin items. - Prefer plugin `download(self, result, output_dir)` for piped plugin items.
- For plugin-provided URLs, implement `download_url` to allow `download-file` to route downloads through plugins. - For plugin-provided URLs, implement `download_url` so `download-file` can route downloads through the plugin.
- Use the repo `_download_direct_file` helper for HTTP downloads when possible. - Use the repo `_download_direct_file` helper for HTTP downloads when possible.
Example download(): Example download method:
```python ```python
def download(self, result: SearchResult, output_dir: Path) -> Optional[Path]: def download(self, result: SearchResult, output_dir: Path) -> Optional[Path]:
# Validate config
url = getattr(result, "path", None) url = getattr(result, "path", None)
if not url or not url.startswith("http"): if not url or not url.startswith("http"):
return None return None
# use existing helpers to fetch the file
return _download_direct_file(url, output_dir) return _download_direct_file(url, output_dir)
``` ```
--- ---
## 🧭 URL routing ## URL routing
Plugins can declare: Plugins can declare:
- `URL = ("magnet:",)` or similar prefix list - `URL = ("magnet:",)` or similar prefix lists
- `URL_DOMAINS = ("example.com",)` to match hosts - `URL_DOMAINS = ("example.com",)` to match hosts
- Or override `@classmethod def url_patterns(cls):` to combine static and dynamic patterns - `@classmethod def url_patterns(cls):` to combine static and dynamic patterns
The registry uses these to match `download-file <url>` or to pick which plugin should handle the URL. The registry uses these declarations to match `download-file <url>` and to pick
which plugin should handle a URL.
--- ---
## 🛠 Selector (handling `@N` picks) ## Selector behavior and `@N`
- Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or to enqueue downloads. - Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or enqueue downloads.
- Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-ups. - Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-up tables.
- Return `True` when you handled the selection and the pipeline should pause or proceed accordingly. - Return `True` when the selector handled the selection and the pipeline should stop expanding that row.
--- ---
## 🧪 Testing plugins ## Testing plugins
- Keep tests small and local. Create `tests/test_provider_<name>.py` or another tracked test target. - Keep tests small and local.
- Test `search()` with mock HTTP responses (use `requests-mock` or similar). - Create `tests/test_plugin_<name>.py` or follow the existing repo naming when extending older tests.
- Test `search()` with mock HTTP responses.
- Test `download()` using a temp directory and a small file server or by mocking `_download_direct_file`. - Test `download()` using a temp directory and a small file server or by mocking `_download_direct_file`.
- Test `selector()` by constructing a fake result and `ctx` object. - Test `selector()` by constructing a fake result and `ctx` object.
Example PowerShell commands to run tests (repo root): Example PowerShell commands from the repo root:
```powershell ```powershell
# Run a single test file pytest tests/test_plugin_hello.py -q
pytest tests/test_provider_hello.py -q
# Run all tests
pytest -q pytest -q
``` ```
--- ---
## 📦 Registration & packaging ## Registration and packaging
- Bundled plugins live under `plugins/` and are auto-discovered from that package. - Bundled plugins live under `plugins/` and are auto-discovered from that package.
- External user plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` / `MEDEIA_PLUGIN_PATH`. - External plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` or `MEDEIA_PLUGIN_PATH`.
- Package directories are preferred so plugin-specific files can travel with the plugin. - Package directories are preferred so plugin-specific files travel with the plugin.
- Plugin authors should import from `ProviderCore.*`. - Plugin authors should import from `ProviderCore.*`.
If a plugin supports multiple configured endpoints or accounts, the user-facing
concept is a plugin instance. Some stored config still lives under legacy key
paths such as `provider.<plugin>.<instance>`.
--- ---
## 💡 Best practices & tips ## Best practices
- Use `debug()` / `log()` appropriately; avoid noisy stderr output in normal runs. - Use `debug()` and `log()` appropriately; avoid noisy stderr output in normal runs.
- Prefer returning `SearchResult` objects to provide consistent UX. - Prefer returning `SearchResult` objects to provide consistent UX.
- Keep `search()` tolerant (timeouts, malformed responses) and avoid raising for expected network problems. - Keep `search()` tolerant of timeouts and malformed responses.
- Use `full_metadata` to pass non-display data to `download()` and `selector()`. - Use `full_metadata` to pass non-display data to `download()` and `selector()`.
- Respect the `limit` parameter in `search()`. - Respect the `limit` parameter in `search()`.
--- ---
## 🧾 Example provider checklist ## Example plugin checklist
- [ ] Implement `search()` and return `SearchResult` items - [ ] Implement `search()` and return `SearchResult` items.
- [ ] Implement `validate()` to check essential config (API keys, credentials) - [ ] Implement `validate()` to check essential config such as API keys or credentials.
- [ ] Provide `URL` / `URL_DOMAINS` or `url_patterns()` for routing - [ ] Provide `URL`, `URL_DOMAINS`, or `url_patterns()` for routing.
- [ ] Add `download()` or `download_url()` for piped/passed URL downloads - [ ] Add `download()` or `download_url()` for piped or passed URL downloads.
- [ ] Add tests under `tests/` - [ ] Add tests under `tests/`.
- [ ] Add the plugin under `plugins/<name>/` for bundled or plug-and-play installs - [ ] Add the plugin under `plugins/<name>/` for bundled or plug-and-play installs.
--- ---
## 🔗 Further reading ## Further reading
- See existing bundled plugins in `plugins/` for patterns and edge cases. - See existing bundled plugins in `plugins/` for patterns and edge cases.
- Check `API/` helpers for HTTP and debrid clients. - Check `API/` helpers for HTTP and debrid clients used by plugins.
---
If you'd like, I can:
- Add an example plugin file under `plugins/` as a template (see `plugins/hello/__init__.py`), and
- Create unit tests for it (see `tests/test_provider_hello.py`).
I have added a minimal example plugin and tests in this repository; use them as a starting point for new plugins.
+90 -135
View File
@@ -1,229 +1,184 @@
# ResultTable system — Overview & usage # ResultTable system: overview and usage
This document explains the `ResultTable` system used across the CLI and TUI: how tables are built, how providers integrate with them, and how `@N` selection/expansion and provider selectors work. This document explains the `ResultTable` system used across the CLI: how tables
are built, how plugins integrate with them, and how `@N` selection, row replay,
and plugin selectors work.
## TL;DR ## TL;DR
- `ResultTable` is the unified object used to render tabular results and drive selection (`@N`) behavior. - `ResultTable` is the unified object used to render tabular results and drive `@N` behavior.
- Providers should return `SearchResult` objects (or dicts) and can either supply `selection_args` per row or implement a `selector()` method to handle `@N` selections. - Plugins return `SearchResult` objects or dicts and can either supply row selection args or implement a `selector()` method.
- Table metadata (`set_table_metadata`) helps providers attach context (e.g., `provider_view`, `magnet_id`) that selectors can use. - Rows can also carry an explicit `selection_action`, which the CLI replays verbatim.
- Table metadata helps plugins attach context. Some metadata keys still use older `provider` names internally.
--- ---
## Key concepts ## Key concepts
- **ResultTable** (`SYS/result_table.py`) ### ResultTable
- Renders rows as a rich table and stores metadata used for selection expansion. `SYS/result_table.py` renders rows as a rich table and stores metadata used for
- Important APIs: `add_result()`, `set_table()`, `set_source_command()`, `set_row_selection_args()`, `set_row_selection_action()`, `set_table_metadata()`, and `select_interactive()`. selection expansion.
- **ResultRow** Important APIs:
- Holds columns plus `selection_args` (used for `@N` expansion) and `payload` (original object). - `add_result()`
- Optionally stores `selection_action`, a full list of CLI tokens to run when `@N` selects this row. When present the CLI honors the explicit action instead of reconstructing it from `source_command` and `selection_args`. - `set_table()`
- `set_source_command()`
- `set_row_selection_args()`
- `set_row_selection_action()`
- `set_table_metadata()`
- **Provider selector** ### ResultRow
- If a provider implements `selector(selected_items, ctx=..., stage_is_last=True)`, it is run first when `@N` is used; if the selector returns `True` it has handled the selection (e.g., drilling into a folder and publishing a new ResultTable). A row holds columns, payload data, selection args, and optionally a full
`selection_action` token list. When `selection_action` is present, `@N` uses it
instead of reconstructing a command from the source command and row args.
- **Pipeline / CLI expansion** ### Plugin selector
- When the user types `@N`, CLI tries provider selectors first. If none handle it, CLI re-runs `source_command + source_args + row_selection_args` (for single-selection) or pipes items downstream for multi-selection. If a plugin implements `selector(selected_items, ctx=..., stage_is_last=True)`,
that selector runs first when `@N` is used. If the selector returns `True`, it
has handled the selection, for example by publishing a nested table.
- **Table metadata** ### Table metadata
- `ResultTable.set_table_metadata(dict)` allows attaching provider-specific context (for example: `{"provider":"alldebrid","view":"files","magnet_id":123}`) for selectors and other code to use. `ResultTable.set_table_metadata(dict)` attaches plugin-specific context for
selectors and follow-up stages. You will still see legacy keys such as
`provider` in some metadata because parts of the runtime still consume them.
--- ---
## How to build a table (provider pattern) ## Building a table
Typical provider flow (pseudocode): Typical plugin flow:
```py ```py
from SYS.result_table import ResultTable from SYS.result_table import ResultTable
table = ResultTable("Provider: X result").set_preserve_order(True) table = ResultTable("Plugin: X results").set_preserve_order(True)
table.set_table("provider_name") table.set_table("plugin_name")
table.set_table_metadata({"provider":"provider_name","view":"folders"}) table.set_table_metadata({"provider": "plugin_name", "view": "folders"})
table.set_source_command("search-file", ["-plugin","provider_name","query"]) table.set_source_command("search-file", ["-plugin", "plugin_name", "query"])
for r in results: for result in results:
table.add_result(r) # r can be a SearchResult, dict, or PipeObject table.add_result(result)
ctx.set_last_result_table(table, payloads) ctx.set_last_result_table(table, payloads)
ctx.set_current_stage_table(table) ctx.set_current_stage_table(table)
``` ```
Notes: Notes:
- To drive a direct `@N` re-run, call `table.set_row_selection_args(row_index, ["-open", "<id>"])`. - To drive a direct replay, call `table.set_row_selection_args(row_index, ["-open", "<id>"])`.
- For more advanced or interactive behavior (e.g., drill-into, fetch more rows), implement `provider.selector()` and return `True` when handled. - For drill-in or interactive behavior, implement `plugin.selector()` and return `True` when handled.
- For exact row replay, call `table.set_row_selection_action(row_index, tokens)`.
--- ---
## Selection (@N) flow (brief) ## Selection flow
1. User enters `@N` in the CLI. When a user enters `@N`:
2. CLI chooses the appropriate table (overlay > last table > history) and gathers the selected payload(s).
3. `PipelineExecutor._maybe_run_class_selector()` runs provider `selector()` hooks for the provider inferred from table or payloads. If any selector returns `True`, expansion stops. 1. The CLI chooses the active table.
4. Otherwise, for single selections, CLI grabs `row.selection_args` and expands: `source_command + source_args + row_selection_args` and inserts it as the expanded stage. For multi-selections, items are piped downstream. 2. It gathers the selected payloads.
3. `PipelineExecutor._maybe_run_class_selector()` runs plugin `selector()` hooks for the plugin inferred from the table or payloads.
4. If no selector handles the row, the CLI checks `row.selection_action` first.
5. If no explicit row action exists, the CLI expands `source_command + source_args + row_selection_args`.
6. Multi-selection results are piped downstream instead of being collapsed to one row replay.
--- ---
## Columns & display ## Columns and display
- Plugins can pass a `columns` list in the result dict or `SearchResult` to control column order and display.
- Providers can pass a `columns` list ([(name, value), ...]) in the result dict/SearchResult to control which columns are shown and their order. - Otherwise, `ResultTable` uses sensible defaults.
- Otherwise, `ResultTable` uses a priority list (title/store/size/ext) and sensible defaults. - Rendering helpers such as `to_rich`, `format_json`, and `format_compact` support different CLI display paths.
- The table rendering functions (`to_rich`, `format_json`, `format_compact`) are available for different UIs.
--- ---
## Provider-specific examples ## Plugin-specific examples
### AllDebrid (debrid file hosting) ### AllDebrid
AllDebrid exposes a list of magnets (folder rows) and the files inside each magnet. The provider returns `folder` SearchResults for magnets and `file` SearchResults for individual files. The provider includes a `selector()` that drills into a magnet by calling `search(..., filters={"view":"files","magnet_id":...})` and builds a new `ResultTable` of files. AllDebrid exposes a list of magnets as folder rows and the files inside each
magnet as file rows. The plugin uses `selector()` to drill into a magnet and
publish a new `ResultTable` of files.
Example commands: Example commands:
``` ```powershell
# List magnets in your account
search-file -plugin alldebrid "*" search-file -plugin alldebrid "*"
@3
# Open magnet id 123 and list its files
search-file -plugin alldebrid -open 123 "*"
# Or expand via @ selection (selector handles drilling):
search-file -plugin alldebrid "*"
@3 # selector will open the magnet referenced by row #3 and show the file table
``` ```
Illustrative folder (magnet) SearchResult: Illustrative magnet row:
```py ```py
SearchResult( SearchResult(
table="alldebrid", table="alldebrid",
title="My Magnet Title", title="My Magnet Title",
path="alldebrid:magnet:123", path="alldebrid:magnet:123",
detail="OK",
annotations=["folder", "ready"],
media_kind="folder", media_kind="folder",
columns=[("Folder", "My Magnet Title"), ("ID", "123"), ("Status", "ready"), ("Ready", "yes")],
full_metadata={ full_metadata={
"magnet": {...},
"magnet_id": 123, "magnet_id": 123,
"provider": "alldebrid", "provider": "alldebrid",
"provider_view": "folders", "provider_view": "folders",
"magnet_name": "My Magnet Title",
}, },
) )
``` ```
4. Otherwise, for single selections, CLI checks for `row.selection_action` and runs that verbatim if present; otherwise it expands `source_command + source_args + row_selection_args`. For multi-selections, items are piped downstream. Illustrative file row:
```py ```py
SearchResult( SearchResult(
table="alldebrid", table="alldebrid",
title="Episode 01.mkv", title="Episode 01.mkv",
path="https://.../unlocked_direct_url", path="https://.../unlocked_direct_url",
detail="My Magnet Title",
annotations=["file"],
media_kind="file", media_kind="file",
size_bytes=123456789,
columns=[("File", "Episode 01.mkv"), ("Folder", "My Magnet Title"), ("ID", "123")],
full_metadata={ full_metadata={
"magnet": {...},
"magnet_id": 123, "magnet_id": 123,
"magnet_name": "My Magnet Title",
"relpath": "Season 1/E01.mkv",
"provider": "alldebrid", "provider": "alldebrid",
"provider_view": "files", "provider_view": "files",
"file": {...},
}, },
) )
``` ```
Selection & download flows Selection and download behavior:
- `@3` on a magnet row runs the plugin selector and shows a file table.
- `@2 | download-file` downloads the selected file row.
- `@1 | add-file -plugin local -instance <dest>` triggers add-file's plugin-aware handoff flow.
- Drill-in (selector): `@3` on a magnet row runs the provider's `selector()` to build a new file table and show it. The selector uses `search(..., filters={"view":"files","magnet_id":...})` to fetch file rows. Configure the AllDebrid plugin in `.config plugins` and add its API key before
testing these flows.
- `download-file` integration: With a file row (http(s) path), `@2 | download-file` will download the file. The `download-file` cmdlet expands AllDebrid magnet folders and will call the provider layer to fetch file bytes as appropriate. ### Bandcamp
- `add-file` convenience: Piping a file row into `add-file -plugin local -instance <dest>` will trigger add-file's provider-aware logic. If the piped item has `table == 'alldebrid'` and a http(s) `path`, `add-file` will call `provider.download()` into a temporary directory and then ingest the downloaded file, cleaning up the temp when done. Example: Bandcamp search supports `artist:` queries. The Bandcamp plugin implements a
`selector()` that detects artist rows and expands them into a discography table.
``` Example:
# Expand magnet and add first file to local directory
search-file -plugin alldebrid "*"
@3 # view files
@1 | add-file -plugin local -instance C:\mydir
```
Notes & troubleshooting ```powershell
- Configure an AllDebrid API key (see `Provider/alldebrid._get_debrid_api_key()`).
- If a magnet isn't ready the selector or `download-file` will log the magnet status and avoid attempting file downloads.
---
### Bandcamp (artist → discography drill-in)
Bandcamp search supports `artist:` queries. Bandcamp's provider implements a `selector()` that detects `artist` results and scrapes the artist's page using Playwright to build a discography `ResultTable`.
Example usage:
```
# Search for an artist
search-file -plugin bandcamp "artist:radiohead" search-file -plugin bandcamp "artist:radiohead"
# Select an artist row to expand into releases
@1 @1
``` ```
Bandcamp SearchResult (artist / album rows): Plugin selectors are the right tool when you need to replace one table with
another, such as artist to discography drill-in.
```py
SearchResult(
table="bandcamp",
title="Album Title",
path="https://bandcamp.com/album_url",
detail="By: Artist",
annotations=["album"],
media_kind="audio",
columns=[("Title","Album Title"), ("Location","Artist"), ("Type","album"), ("Url","https://...")],
full_metadata={"artist":"Artist","type":"album","url":"https://..."}
)
```
Notes:
- Playwright is required for Bandcamp scraping. The selector will log an informative message if Playwright is missing.
- Provider selectors are ideal when you need to replace one table with another (artist → discography).
--- ---
## Provider author checklist (short) ## Plugin author checklist
- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts.
- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts; include useful `full_metadata` (IDs, view names) for selection/drilling. - Include useful `full_metadata` values for drill-in and selection replay.
- If you support fetching downloadable file bytes, implement `download(result, output_dir) -> Optional[Path]`. - Implement `download(result, output_dir)` when the plugin can materialize file bytes.
- For drill-in or interactive transforms, implement `selector(selected_items, ctx=..., stage_is_last=True)` and call `ctx.set_last_result_table(...)` / `ctx.set_current_stage_table(...)`; return `True` when handled. - Implement `selector(...)` for drill-in or interactive transforms.
- Add tests (unit/integration) that exercise search select download flows. - Add tests that cover search, select, and download flows.
--- ---
## Debugging tips ## Debugging tips
- Use `ctx.set_last_result_table(table, payloads)` to immediately show a table while developing a selector. - Use `ctx.set_last_result_table(table, payloads)` to publish a table while developing a selector.
- Add `log(...)` messages in provider code to capture fail points. - Add `log(...)` messages in plugin code to capture fail points.
- Check `full_metadata` attached to SearchResults to pass extra context (IDs, view names, provider names). - Inspect `full_metadata` when a selector or replay path is missing required context.
--- ---
## Quick reference ## Quick reference
- ResultTable location: `SYS/result_table.py` - ResultTable implementation: `SYS/result_table.py`
- Pipeline helpers: `SYS/pipeline.py` (`set_last_result_table`, `set_current_stage_table`, `get_current_stage_table_row_selection_args`) - Pipeline helpers: `SYS/pipeline.py`
- CLI expansion: `CLI.py` (handles `@N`, provider selectors, and insertion of expanded stages) - Row replay and selection flow: `SYS/pipeline.py`
- Provider selector pattern: Implement `.selector(selected_items, ctx=..., stage_is_last=True)` in provider class. - Plugin authoring guide: `docs/provider_authoring.md`
---
For more detail on ResultTable provider authoring, see `docs/provider_authoring.md`.
If you'd like, I can also:
- Add provider-specific examples (AllDebrid, Bandcamp) into this doc ✅
- Add a short checklist for PR reviewers when adding new providers
---
Created by GitHub Copilot (Raptor mini - Preview) — brief guide to the ResultTable system. Feedback welcome!
+15 -12
View File
@@ -1,22 +1,25 @@
Selector & plugin-table usage Selector and `plugin-table` usage
This project provides a small plugin/table/selector flow that allows plugins This project provides a small plugin/table/selector flow that lets plugins and
and cmdlets to interact via a simple, pipable API. cmdlets interact through a simple pipable API.
Key ideas Key ideas:
- `plugin-table` renders a plugin result set and *emits* pipeline-friendly dicts for each row. Each emitted item includes `_selection_args`, a list of args the plugin suggests for selecting that row (e.g., `['-path', '/tmp/file']`). - `plugin-table` renders a plugin result set and emits pipeline-friendly dicts for each row.
- Use the `@N` syntax to select an item from a table and chain it to the next cmdlet. - Emitted rows can include `_selection_args` and, when needed, `_selection_action` for exact row replay.
- Use `@N` to select an item from a table and chain it to the next cmdlet.
Example: Example:
plugin-table -plugin example -sample | @1 | add-file -instance default plugin-table -plugin example -sample | @1 | add-file -instance default
What plugins must implement What plugins must implement:
- An adapter that yields `ResultModel` objects (breaking API). - An adapter that yields `ResultModel` objects.
- Optionally supply a `columns` factory and `selection_fn` (see `plugins/example_provider.py`). - Optionally a `columns` factory and `selection_fn`.
Implementation notes Implementation notes:
- `plugin-table` emits dicts like `{ 'title': ..., 'path': ..., 'metadata': ..., '_selection_args': [...] }`. - `plugin-table` emits dicts like `{ 'title': ..., 'path': ..., 'metadata': ..., '_selection_args': [...] }`.
- Selection syntax (`@1`) will prefer `_selection_args` if present; otherwise it will fall back to plugin selection logic or sensible defaults (`-path` or `-title`). - When a row includes `_selection_action`, `@1` prefers that exact action.
- Otherwise `@1` falls back to `_selection_args`, then plugin selection logic, then sensible defaults such as `-path` or `-title`.
This design keeps the selector-focused UX small and predictable while enabling full cmdlet interoperability via piping and `-run-cmd`. This design keeps selector-focused workflows small and predictable while still
allowing full cmdlet interoperability through piping and `-run-cmd`.
+26 -48
View File
@@ -1,23 +1,26 @@
# SCP Plugin Walkthrough # SCP Plugin Walkthrough
This walkthrough adds a bundled `scp` plugin backed by existing SSH libraries: This walkthrough covers the bundled `scp` plugin, backed by existing SSH
libraries:
- `paramiko` for SSH and SFTP directory listing - `paramiko` for SSH and SFTP directory listing
- `scp` for file transfers - `scp` for file transfers
The implementation lives in [plugins/scp/__init__.py](plugins/scp/__init__.py). The implementation lives in [plugins/scp/__init__.py](plugins/scp/__init__.py).
## What The Plugin Does ## What the plugin does
The SCP plugin mirrors the FTP walkthrough, but on top of SSH: The SCP plugin mirrors the FTP walkthrough, but on top of SSH:
- `search-file -plugin scp -instance <name> ...` lists remote files and folders over SFTP
- plain `@N` on a folder drills into that directory
- plain `@N` on a file runs `download-file -plugin scp -instance <name> -url ...`
- `@N | add-file -instance ...` downloads first, then ingests the local temp file
- `add-file -plugin scp -instance <name> -path ...` uploads a local file to the configured remote path
- `search-file -plugin scp -instance <name> ...` lists remote files and folders over SFTP. ## Example config
- plain `@N` on a folder drills into that directory.
- plain `@N` on a file runs `download-file -plugin scp -instance <name> -url ...`.
- `@N | add-file -instance ...` downloads first, then ingests the local temp file.
- `add-file -plugin scp -instance <name> -path ...` uploads a local file to the configured remote path.
## Example Config Add one or more named SCP plugin instances to your config. The current stored
key path remains `provider.scp.<instance>` for legacy compatibility:
```toml ```toml
[provider.scp.work] [provider.scp.work]
@@ -42,39 +45,22 @@ timeout = 20
``` ```
Notes: Notes:
- `work` and `archive` are instance names.
- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`.
- `host` and `username` are required for each instance to validate. - `host` and `username` are required for each instance to validate.
- You can use password auth, key auth, or both. - You can use password auth, key auth, or both.
- `base_path` is both the default search root and the default upload directory. - `base_path` is both the default search root and the default upload directory.
- You can browse configured instances from `.config plugins` in the CLI.
## Search Flow ## Search flow
List the configured base path:
```powershell ```powershell
search-file -plugin scp -instance work "*" search-file -plugin scp -instance work "*"
```
Search by filename:
```powershell
search-file -plugin scp -instance work "invoice" search-file -plugin scp -instance work "invoice"
```
Search another subtree with deeper recursion:
```powershell
search-file -plugin scp -instance work "path:/srv/files/releases depth:2 *.zip" search-file -plugin scp -instance work "path:/srv/files/releases depth:2 *.zip"
```
Show only folders:
```powershell
search-file -plugin scp -instance work "path:/srv/files type:folder *" search-file -plugin scp -instance work "path:/srv/files type:folder *"
``` ```
## Selection Flow ## Selection flow
Folder rows are navigation rows: Folder rows are navigation rows:
@@ -83,7 +69,8 @@ search-file -plugin scp -instance work "*"
@2 @2
``` ```
File rows carry an explicit row action, so terminal selection downloads directly: File rows carry an explicit row action, so terminal selection downloads
directly:
```powershell ```powershell
search-file -plugin scp -instance work "report" search-file -plugin scp -instance work "report"
@@ -96,45 +83,36 @@ That expands to the equivalent of:
download-file -plugin scp -instance work -url scp://ssh.example.com/srv/files/report.pdf download-file -plugin scp -instance work -url scp://ssh.example.com/srv/files/report.pdf
``` ```
## Download And Add-File Flow ## Download and add-file flow
Download into a local folder:
```powershell ```powershell
search-file -plugin scp -instance work "report" search-file -plugin scp -instance work "report"
@1 | download-file -path C:\Downloads @1 | download-file -path C:\Downloads
```
Ingest a selected remote file into a configured instance backend:
```powershell
search-file -plugin scp -instance work "report" search-file -plugin scp -instance work "report"
@1 | add-file -instance tutorial @1 | add-file -instance tutorial
``` ```
Why this works: Why this works:
- file rows advertise `_selection_action` for `download-file` - file rows advertise `_selection_action` for `download-file`
- `add-file` selection replay inserts that provider download stage before ingest - `add-file` selection replay inserts that plugin download stage before ingest
- the plugin also implements `resolve_pipe_result_download()` for provider-owned SCP rows - the plugin also implements `resolve_pipe_result_download()` for plugin-owned SCP rows
- file rows also carry the chosen `instance`, so replay stays bound to the same SSH target - file rows carry the chosen `instance`, so replay stays bound to the same SSH target
## Upload Flow ## Upload flow
Upload a local file to the configured remote `base_path`:
```powershell ```powershell
add-file -plugin scp -instance archive -path C:\Media\report.pdf add-file -plugin scp -instance archive -path C:\Media\report.pdf
``` ```
## Implementation Notes ## Implementation notes
The plugin uses SFTP for directory listing because SCP itself is a transfer protocol, not a browse/search protocol. That split keeps the provider simple:
The plugin uses SFTP for directory listing because SCP itself is a transfer
protocol, not a browse/search protocol. That split keeps the plugin simple:
- browse and metadata via Paramiko SFTP - browse and metadata via Paramiko SFTP
- file transfer via the `scp` package - file transfer via the `scp` package
## Recommended Demo Commands ## Recommended demo commands
```powershell ```powershell
search-file -plugin scp -instance work "*" search-file -plugin scp -instance work "*"
+12 -4
View File
@@ -3,6 +3,11 @@
This folder is the primary home for bundled plugins and also the default search This folder is the primary home for bundled plugins and also the default search
path for drop-in plugins. path for drop-in plugins.
User-facing docs and CLI flows treat these integrations as plugins. Some Python
types and some stored config keys still use older `Provider` naming internally,
but that is a legacy implementation detail rather than the preferred public
term.
Preferred layout: Preferred layout:
- Put each plugin in its own folder under `plugins/<name>/` with an `__init__.py`. - Put each plugin in its own folder under `plugins/<name>/` with an `__init__.py`.
- Keep plugin-specific assets beside the code in that same folder. - Keep plugin-specific assets beside the code in that same folder.
@@ -24,8 +29,8 @@ drop-in plugin search paths are:
Plugin rules: Plugin rules:
- A plugin can be a single `.py` file or a package directory with `__init__.py`. - A plugin can be a single `.py` file or a package directory with `__init__.py`.
- Define a class that inherits from `ProviderCore.base.Provider`. - Current plugin classes inherit from `ProviderCore.base.Provider`.
- Give it a stable name using `PLUGIN_NAME` or the class name. - Give the plugin a stable name using `PLUGIN_NAME` or the class name.
Example skeleton: Example skeleton:
@@ -52,9 +57,12 @@ class MyPlugin(Provider):
Bundled walkthrough: Bundled walkthrough:
- Providers can now expose named config instances under `provider.<plugin>.<instance>` and cmdlets can target them with `-instance <name>`. - Plugins can expose named config instances. The current stored config may still
use legacy key paths such as `provider.<plugin>.<instance>`, and cmdlets target
instances with `-instance <name>`.
- Use `.config plugins` in the CLI to browse configured plugin instances.
- The repo now includes a real FTP example plugin in [plugins/ftp/__init__.py](plugins/ftp/__init__.py). - The repo now includes a real FTP example plugin in [plugins/ftp/__init__.py](plugins/ftp/__init__.py).
- The walkthrough is in [docs/ftp_plugin_tutorial.md](docs/ftp_plugin_tutorial.md) and shows `search-file -plugin ftp -instance <name>`, folder drill-in via `@N`, file download routing, `@N | add-file -instance ...`, and `add-file -plugin ftp -instance <name>` uploads. - The walkthrough is in [docs/ftp_plugin_tutorial.md](docs/ftp_plugin_tutorial.md) and shows `search-file -plugin ftp -instance <name>`, folder drill-in via `@N`, file download routing, `@N | add-file -instance ...`, and `add-file -plugin ftp -instance <name>` uploads.
- The repo also includes an SCP example plugin in [plugins/scp/__init__.py](plugins/scp/__init__.py). - The repo also includes an SCP example plugin in [plugins/scp/__init__.py](plugins/scp/__init__.py).
- The walkthrough is in [docs/scp_plugin_tutorial.md](docs/scp_plugin_tutorial.md) and shows `search-file -plugin scp -instance <name>`, SSH-backed directory drill-in, file download routing, `@N | add-file -instance ...`, and `add-file -plugin scp -instance <name>` uploads. - The walkthrough is in [docs/scp_plugin_tutorial.md](docs/scp_plugin_tutorial.md) and shows `search-file -plugin scp -instance <name>`, SSH-backed directory drill-in, file download routing, `@N | add-file -instance ...`, and `add-file -plugin scp -instance <name>` uploads.
- The repo now also includes a built-in HydrusNetwork provider in [plugins/hydrusnetwork/__init__.py](plugins/hydrusnetwork/__init__.py). Its Hydrus client API now lives in the plugin-owned package [plugins/hydrusnetwork/api/__init__.py](plugins/hydrusnetwork/api/__init__.py), its registry-facing store adapter lives in [plugins/hydrusnetwork/store_proxy.py](plugins/hydrusnetwork/store_proxy.py), and its heavy internal operations live in [plugins/hydrusnetwork/store_backend.py](plugins/hydrusnetwork/store_backend.py). This `plugins/<name>/api/` package shape is the intended pattern for plugin-owned API helpers going forward. The provider now resolves configured Hydrus instances directly from plugin config instead of routing back through `Store.registry`; the proxy exists only so generic store callers can still target configured Hydrus stores. [API/HydrusNetwork.py](API/HydrusNetwork.py) and [Store/HydrusNetwork.py](Store/HydrusNetwork.py) are legacy compatibility shims only, and store discovery prefers the plugin-owned Hydrus hook over those shims. - The repo also includes a built-in HydrusNetwork plugin in [plugins/hydrusnetwork/__init__.py](plugins/hydrusnetwork/__init__.py). Its Hydrus client API now lives in the plugin-owned package [plugins/hydrusnetwork/api/__init__.py](plugins/hydrusnetwork/api/__init__.py), its registry-facing store adapter lives in [plugins/hydrusnetwork/store_proxy.py](plugins/hydrusnetwork/store_proxy.py), and its heavy internal operations live in [plugins/hydrusnetwork/store_backend.py](plugins/hydrusnetwork/store_backend.py). This `plugins/<name>/api/` package shape is the intended pattern for plugin-owned API helpers going forward. The plugin resolves configured Hydrus instances directly from plugin config instead of routing back through `Store.registry`; the proxy exists only so generic store callers can still target configured Hydrus stores. [API/HydrusNetwork.py](API/HydrusNetwork.py) and [Store/HydrusNetwork.py](Store/HydrusNetwork.py) are legacy compatibility shims only, and store discovery prefers the plugin-owned Hydrus hook over those shims.
+148 -59
View File
@@ -1,69 +1,158 @@
<div align="center"> # MEDEIA-MACINA
<h1>MEDEIA-MACINA</h1>
<img src="https://code.glowers.club/goyimnose/Medios-Macina/raw/branch/main/docs/img/MM.png"></img>
<h3>4 TEXT BASED FILE ONTOLOGY </h3>
</div>
Medios-Macina is a API driven file media manager and virtual toolbox capable of downloading, tagging, archiving, sharing, and connecting you to HydrusNetwork backends. It is designed around a compact, pipeable command language ("cmdlets") so complex workflows can be composed simply and repeatably. ![Medeia-Macina logo](https://code.glowers.club/goyimnose/Medios-Macina/raw/branch/main/docs/img/MM.png)
<h3>ELEVATED PITCH</h3> Medeia-Macina is a text-first media manager and plugin runtime for searching, downloading, tagging, archiving, replaying, and moving media through one CLI. It is built around pipeable commands, rich result tables, and row replay so you can move from search to action without leaving the terminal.
<ul>
<li><i>Have you ever wanted one app that can manage all your media files and is completely in your control?</i></li>
<li><i>Do you want a no-brainer one-stop shop for finding & downloading applications?</i></li>
<li><i>Are you one that has an unorganized & unapologetic mess of files that are loosely organized in random folders?</i></li>
<li><i>Does it take you several brainfarts until you get a scent of where that file is at that your looking for?</i></li>
<li><i>Do you have trouble struggling with filenames so that you can find the file you want later?</i></li>
</ul>
<li><i>Would you like to have your media library available with you even when you are away from home?</i></li>
</ul>
<hr>
<h2>CONTENTS</H2>
<a href="#features">FEATURES</a><br>
<a href="#installation">INSTALLATION</a><br>
<a href="docs/tag_template_syntax.md">TAG TEMPLATE SYNTAX</a><br>
<a href="https://code.glowers.club/goyimnose/Medios-Macina/wiki/Config.conf">CONFIG</a><br>
<a href="https://code.glowers.club/goyimnose/Medios-Macina/wiki/Hydrus-Network">HYDRUS NETWORK</a><br>
<a href="https://code.glowers.club/goyimnose/Medios-Macina/wiki/cookies.txt">COOKIES</a><br>
<a href="https://code.glowers.club/goyimnose/Medios-Macina/wiki/Tutorial">TUTORIAL</a><br>
<BR>
<div id="features"><h2> Features</h2> The current UX is plugin-first. Older internal names like `provider` and `tool` still exist in some code paths and authoring APIs, but the user-facing model is now centered on plugins.
<ul>
<li><b>Connects to HydrusNetwork, which is an open-source privacy orientated database-driven file manager</li>
<li><i>Medios-Macina uses multiple python module downloaders for your specific needs, it is highly configurable and customizable</i></li>
<li><i>no opening of folders neccessary! You can add multiple tags to a file and use the search engine to immediately find and retrieve that file your looking for</i></li>
<li><b>Flexible syntax structure:</b> chain commands with `|` and select options from tables with `@N`.</li>
<li><b>Multiple file stores:</b> *HYDRUSNETWORK*
- **Plugin integration:** *YOUTUBE, OPENLIBRARY, INTERNETARCHIVE, SOULSEEK, LIBGEN, ALLDEBRID, TELEGRAM, BANDCAMP*</li>
<li><b>Module Mixing:</b> *[Playwright](https://github.com/microsoft/playwright), [yt-dlp](https://github.com/yt-dlp/yt-dlp), [typer](https://github.com/fastapi/typer)*</li>
<li><b>Optional stacks:</b> Telethon (Telegram), aioslsk (Soulseek), and the FlorenceVision tooling install automatically when you configure the corresponding plugin/tool blocks.
<li><b>MPV Manager:</b> Play audio, video, and even images in a custom designed MPV with trimming, screenshotting, and more built right in!</li>
<li><i>Supports remote access and networked setups for offsite servers and sharing workflows.</i></li>
<li><b>Reusable tag templates:</b> derive new tags from existing ones with placeholder and padding syntax documented in <a href="docs/tag_template_syntax.md">docs/tag_template_syntax.md</a>.</li>
</ul>
</ul
</div> ## What Medeia-Macina Does
<div id="installation"><h2> INSTALLATION</h2>
<h4><b>Requirements:</b> <a href="https://www.python.org/downloads/">Python</a>
<h4><a href="https://git-scm.com/">GIT</a>
<br>
<details> - Search local and remote sources through plugins.
<summary>COMMAND LINE</summary> - Browse results as tables and replay rows with `@N`.
- Chain follow-up actions with `|`.
- Add or move files into configured backends such as HydrusNetwork.
- Inspect metadata, tags, URLs, and file details from the same CLI.
- Hand media off to MPV for playback, screenshots, and related workflows.
- Load bundled plugins or drop-in plugins from external paths.
<pre><code> ## How The App Works
curl -sSL https://code.glowers.club/goyimnose/Medios-Macina/raw/branch/main/scripts/bootstrap.py | python -
</code></pre>
</details> The basic interaction loop is:
you may need to change python3 to python depending on your python installation 1. Run a command that returns a table.
<br> 2. Use `@N` to select a row.
<b>After install, start the CLI by simply inputting "mm" into terminal/console, once the application is up and running you will need to connect to a HydrusNetwork sever to get the full experience. To access the config simply input ".config" while the application is running</b> 3. Let the row replay its plugin-defined action, or pipe it into another command.
</div> 4. Use `.config` any time to inspect or update configuration.
<img src="https://avatars.githubusercontent.com/u/79589310?s=48&v=4">ytdlp</img>
<img src="https://avatars.githubusercontent.com/u/3878678?s=48&v=4 That means rows are not just display data. A row can carry selection arguments or a full replay action. Depending on the plugin and row type, plain `@N` might open a nested table, download a file, show details, or trigger another plugin-specific workflow.
">hydrusnetwork</img>
## Plugin System
Plugins are the main integration surface for the app.
- Built-in integrations such as HydrusNetwork, yt-dlp/YouTube, FTP, SCP, Soulseek, Telegram, Internet Archive, OpenLibrary, Bandcamp, and others are treated as plugins.
- Plugins can expose named instances, so one plugin can target multiple endpoints, accounts, or servers via `-instance <name>`.
- Bundled and external plugins use the same `plugins/<name>/` layout.
- External plugin search paths include the repo `plugins/` folder, the current working directory `plugins/` folder, `MM_PLUGIN_PATH`, and `MEDEIA_PLUGIN_PATH`.
- Plugin authoring still uses the current Python base class name `ProviderCore.base.Provider`. That is an implementation detail rather than the preferred user-facing term.
See [plugins/README.md](plugins/README.md) for plugin packaging and discovery details.
## Configuration
The old interactive TUI config editor has been discontinued.
Use `.config` from inside the CLI instead:
- `.config` opens the root configuration table.
- `@N` drills into a selected section.
- `@..` goes back.
- `.config plugins` jumps straight to the user-facing plugin section.
- Selecting a value row and running `.config <value>` updates that setting.
This keeps configuration inside the same table-and-selection model as the rest of the app.
## Installation From A Git Checkout
Requirements:
- Python 3.9 through 3.13
- Git
- PowerShell on Windows
- `mpv` recommended for playback workflows
From the repository root, run:
```powershell
Set-Location C:\path\to\Medios-Macina
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\scripts\bootstrap.ps1 -Editable
.\.venv\Scripts\Activate.ps1
mm
```
Notes:
- The bootstrap script creates `.venv`, installs the project, and exposes the `mm` and `medeia` console commands.
- On Windows, the bootstrap script will try to ensure `mpv` is available.
- Playwright browser support is installed by default unless you pass `-NoPlaywright`.
- The repository also ships `scripts/bootstrap.py` and `scripts/bootstrap.sh`, but the examples here use PowerShell.
## First Run
After launching `mm`, start with configuration:
```powershell
.config
.config plugins
```
If you want the full archive/tag/search workflow, HydrusNetwork is usually the first plugin to configure.
## Example Workflows
Browse plugin configuration:
```powershell
.config
.config plugins
```
Search a configured plugin instance:
```powershell
search-file -plugin ftp -instance work "invoice"
```
Replay a selected row:
```powershell
@1
```
Ingest a selected remote result into a configured backend:
```powershell
@1 | add-file -instance tutorial
```
Upload a local file through a plugin:
```powershell
add-file -plugin ftp -instance archive -path C:\Media\report.pdf
```
The exact meaning of `@1` depends on the current table and plugin. For example, one row may open a nested directory table while another row may download or replay a file-specific action.
## Core Concepts
- `search-file`: search a plugin, source, or configured backend and produce a table.
- `@N`: replay or select row `N` from the most recent table.
- `|`: pipe the selected result into the next command.
- `add-file`: ingest a file into a configured backend or upload through a plugin.
- `.config`: browse and edit configuration from inside the CLI.
- `.mpv`: hand media off to the integrated MPV workflow.
## Documentation
- [docs/tag_template_syntax.md](docs/tag_template_syntax.md)
- [plugins/README.md](plugins/README.md)
- [docs/provider_guide.md](docs/provider_guide.md)
- [docs/provider_authoring.md](docs/provider_authoring.md)
- [docs/ftp_plugin_tutorial.md](docs/ftp_plugin_tutorial.md)
- [docs/scp_plugin_tutorial.md](docs/scp_plugin_tutorial.md)
- [docs/BOOTSTRAP_TROUBLESHOOTING.md](docs/BOOTSTRAP_TROUBLESHOOTING.md)
## Current Direction
Medeia-Macina is moving toward one canonical plugin model:
- user-facing integrations are described as plugins
- plugin rows carry their own replay actions when needed
- nested tables use the same `@N` and `@..` interaction model across config and plugin workflows
- Hydrus-backed archiving, metadata, and playback flows are treated as part of the same CLI pipeline rather than separate apps