diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index f8df2e2..ec4824f 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -2,8 +2,8 @@ ## Unreleased (2026-01-05) -- **docs:** Add `docs/provider_authoring.md` with a Quick Start, examples, and testing guidance for providers that integrate with the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn). -- **docs:** Add link to `docs/result_table.md` pointing to the provider authoring guide. -- **tests:** Add `tests/test_provider_author_examples.py` validating example provider registration and adapter behavior. +- **docs:** Add `docs/provider_authoring.md` as a plugin authoring Quick Start for the strict `ResultTable` API (ResultModel/ColumnSpec/selection_fn). The file keeps its legacy name for compatibility. +- **docs:** Add a link from `docs/result_table.md` to the plugin authoring guide. +- **tests:** Add `tests/test_provider_author_examples.py` validating example plugin registration and adapter behavior. The test file keeps its legacy name for compatibility. - **notes:** Existing example plugins (`plugins/example_provider.py`, `plugins/vimm/__init__.py`) are referenced as canonical patterns. diff --git a/docs/PR_PROVIDER_AUTHORING.md b/docs/PR_PROVIDER_AUTHORING.md index a15eea3..ec6ec7a 100644 --- a/docs/PR_PROVIDER_AUTHORING.md +++ b/docs/PR_PROVIDER_AUTHORING.md @@ -1,15 +1,15 @@ -PR Title: docs: Add Provider authoring doc, examples, and tests +PR Title: docs: Add plugin authoring doc, examples, and tests Summary: -- Add `docs/provider_authoring.md` describing the strict `ResultModel`-based provider adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping. -- Link new doc from `docs/result_table.md`. -- Add `tests/test_provider_author_examples.py` to validate `Provider/example_provider.py` and `Provider/vimm.py` integration with the registry. +- Add `docs/provider_authoring.md` describing the strict `ResultModel`-based plugin adapter pattern, `ColumnSpec` usage, `selection_fn`, and `TableProviderMixin` for HTML table scraping. +- Link the new doc from `docs/result_table.md`. +- Add `tests/test_provider_author_examples.py` to validate the example plugin integration with the registry. Why: -- Provide a short, focused Quick Start to help contributors author providers that integrate with the new strict ResultTable API. +- Provide a short, focused Quick Start to help contributors author plugins that integrate with the strict ResultTable API. Testing: -- New tests pass locally (provider-related subset). +- New tests pass locally (plugin-related subset). Notes: - The change is documentation-first and non-functional, with tests ensuring examples remain valid. diff --git a/docs/ftp_plugin_tutorial.md b/docs/ftp_plugin_tutorial.md index be0327b..09d93b9 100644 --- a/docs/ftp_plugin_tutorial.md +++ b/docs/ftp_plugin_tutorial.md @@ -1,6 +1,6 @@ # FTP Plugin Walkthrough -This walkthrough adds a real bundled `ftp` plugin so users can: +This walkthrough covers the bundled `ftp` plugin. It lets users: - run `search-file -plugin ftp -instance ...` - browse remote folders as result tables @@ -10,9 +10,10 @@ This walkthrough adds a real bundled `ftp` plugin so users can: The implementation lives in [plugins/ftp/__init__.py](plugins/ftp/__init__.py). -## What The Plugin Does +## What the plugin does -The FTP plugin demonstrates the main provider hooks that matter for a storage-style integration: +The FTP plugin demonstrates the main plugin hooks that matter for a +storage-style integration: - `config_schema()` exposes host, credentials, base path, TLS, and search depth. - `extract_query_arguments()` supports inline query fields like `path:` and `depth:`. @@ -22,9 +23,10 @@ The FTP plugin demonstrates the main provider hooks that matter for a storage-st - `resolve_pipe_result_download()` lets `@N | add-file -instance ...` materialize a remote FTP file first. - `upload()` lets `add-file -plugin ftp -instance -path ...` push a local file to the configured FTP server. -## Example Config +## Example config -Add one or more named FTP provider instances to your config: +Add one or more named FTP plugin instances to your config. The current stored +key path remains `provider.ftp.` for legacy compatibility: ```toml [provider.ftp.work] @@ -48,121 +50,97 @@ tls = true ``` Notes: - -- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`. +- `work` and `archive` are instance names. - `host` is the only required field for each instance to validate. - `username` defaults to `anonymous` and `password` defaults to `anonymous@`. - `base_path` is both the default search root and the upload target directory. - `search_depth` controls how many folder levels `search-file -plugin ftp` scans by default. +- You can browse configured instances from `.config plugins` in the CLI. -## Search Flow - -Basic listing from the configured base path: +## Search flow ```powershell search-file -plugin ftp -instance work "*" -``` - -Search by filename fragment: - -```powershell search-file -plugin ftp -instance work "invoice" -``` - -Search a different subtree and recurse deeper: - -```powershell search-file -plugin ftp -instance work "path:/pub depth:2 invoice" -``` - -Filter to folders only: - -```powershell search-file -plugin ftp -instance work "path:/pub type:folder *" ``` -The plugin returns rows with explicit columns for name, type, directory, size, and modification time. +The plugin returns rows with explicit columns for name, type, directory, size, +and modification time. -## Selection Flow +## Selection flow -Folder rows are navigation rows. If the selected row is a directory, plain `@N` opens a new FTP table for that directory: +Folder rows are navigation rows: ```powershell search-file -plugin ftp -instance work "*" @2 ``` -File rows carry an explicit row action: +File rows carry an explicit row action equivalent to: ```powershell download-file -plugin ftp -instance work -url ftp://ftp.example.com/incoming/report.pdf ``` -That means plain `@N` on a file row downloads it immediately: +So plain `@N` on a file row downloads it immediately: ```powershell search-file -plugin ftp -instance work "report" @1 ``` -## Download And Add-File Flow - -If you want the downloaded file in a specific local directory: +## Download and add-file flow ```powershell search-file -plugin ftp -instance work "report" @1 | download-file -path C:\Downloads -``` -If you want to ingest the selected FTP file into a configured instance backend: - -```powershell search-file -plugin ftp -instance work "report" @1 | add-file -instance tutorial ``` Why this works: - - the file row advertises a `download-file` row action - the pipeline auto-inserts that download before `add-file` -- the FTP plugin also implements `resolve_pipe_result_download()` so provider-owned FTP rows can be materialized for ingestion -- file rows also carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target +- the FTP plugin also implements `resolve_pipe_result_download()` so plugin-owned FTP rows can be materialized for ingestion +- file rows carry the chosen `instance`, so selection replay and `@N | add-file ...` keep the same FTP target -## Upload Flow +## Upload flow -Uploading uses the same provider name, but through `add-file -plugin ftp -instance `: +Uploading uses the same plugin, through `add-file -plugin ftp -instance `: ```powershell add-file -plugin ftp -instance archive -path C:\Media\report.pdf ``` -That sends the file to the selected instance's FTP `base_path` and returns the FTP URL as the uploaded result. +That sends the file to the selected instance's FTP `base_path` and returns the +FTP URL as the uploaded result. -## Why The Row Metadata Matters +## Why the row metadata matters The critical part of this plugin is the file-row metadata: - - file rows emit `_selection_args` as `['-instance', '', '-url', '']` - file rows emit `_selection_action` as `['download-file', '-plugin', 'ftp', '-instance', '', '-url', '']` -- folder rows do not emit a download action, so `selector()` can own drill-in behavior instead - -That split is what keeps these two user experiences compatible: +- folder rows do not emit a download action, so `selector()` owns drill-in behavior instead +That keeps these flows compatible: - `@N` on a folder opens a new table - `@N` on a file downloads the file - `@N | add-file -instance ...` first downloads, then ingests -## Implementation Notes +## Implementation notes -The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus directory probes when the server does not support machine-readable listings. - -The code is intentionally small and uses only Python stdlib pieces: +The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus +directory probes when the server does not support machine-readable listings. +The code intentionally stays small and uses only Python stdlib pieces: - `ftplib` for FTP and FTPS - `fnmatch` for wildcard-style search tokens - `tempfile` for `add-file` handoff downloads -## Recommended Commands To Demo The Walkthrough +## Recommended demo commands ```powershell search-file -plugin ftp -instance work "*" diff --git a/docs/provider_authoring.md b/docs/provider_authoring.md index 93a30ee..37be18b 100644 --- a/docs/provider_authoring.md +++ b/docs/provider_authoring.md @@ -1,37 +1,48 @@ -# Plugin authoring: ResultTable & plugin adapters +# Plugin authoring: ResultTable and plugin adapters -This short guide explains how to write plugins that integrate with the *strict* ResultTable API: adapters must yield `ResultModel` instances and plugins register via `SYS.result_table_adapters.register_plugin` with a column specification and a `selection_fn`. +This short guide explains how to write plugins that integrate with the strict +ResultTable API: adapters yield `ResultModel` instances, and plugins register +via `SYS.result_table_adapters.register_plugin` with columns and a +`selection_fn`. + +Note: this file keeps its historical `provider_authoring` name, but the public +terminology is plugin-first. Some internal classes and metadata fields still use +`Provider` naming. --- ## Quick summary - -- Plugins register a *plugin adapter* (callable that yields `ResultModel`). -- Plugins must also provide `columns` (static list or factory) and a `selection_fn` that returns CLI args for a selected row. -- For simple HTML table/list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers` to fetch and extract rows using `SYS.html_table.extract_records`. +- Plugins register a plugin adapter, a `columns` definition, and a `selection_fn`. +- `selection_fn` returns CLI args for a selected row. +- For HTML table or list scraping, prefer `TableProviderMixin` from `SYS.provider_helpers`. ## Runtime dependency policy - -- Treat required runtime dependencies (e.g., **Playwright**) as mandatory: import them unconditionally and let missing dependencies fail fast at import time. Avoid adding per-call try/except import guards for required modulesβ€”these silently hide configuration errors and add bloat. -- Use guarded imports only for truly optional dependencies (e.g., `pandas` for enhanced table parsing) and provide meaningful fallbacks or helpful error messages in those cases. -- Keep provider code minimal and explicit: fail early and document required runtime dependencies in README/installation notes. +- Treat required runtime dependencies such as Playwright as mandatory: import them unconditionally and let missing dependencies fail fast. +- Use guarded imports only for truly optional dependencies such as `pandas`. +- Keep plugin code minimal and explicit: fail early and document required runtime dependencies in README and installation notes. --- -## Minimal provider template (copy/paste) +## Minimal plugin template ```py # plugins/my_plugin.py from typing import Any, Dict, Iterable, List -from SYS.result_table_api import ResultModel, ColumnSpec, title_column, metadata_column +from SYS.result_table_api import ResultModel, ColumnSpec, title_column from SYS.result_table_adapters import register_plugin -# Example adapter: convert provider-specific items into ResultModel instances SAMPLE_ITEMS = [ - {"name": "Example File.pdf", "path": "https://example.com/x.pdf", "ext": "pdf", "size": 1024, "source": "myprovider"}, + { + "name": "Example File.pdf", + "path": "https://example.com/x.pdf", + "ext": "pdf", + "size": 1024, + "source": "myplugin", + }, ] + def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]: for it in items: title = it.get("name") or it.get("title") or str(it.get("path") or "") @@ -41,39 +52,38 @@ def adapter(items: Iterable[Dict[str, Any]]) -> Iterable[ResultModel]: ext=str(it.get("ext")) if it.get("ext") else None, size_bytes=int(it.get("size")) if it.get("size") is not None else None, metadata=dict(it), - source=str(it.get("source")) if it.get("source") else "myprovider", + source=str(it.get("source")) if it.get("source") else "myplugin", ) -# Optional: build columns dynamically from sample rows + def columns_factory(rows: List[ResultModel]) -> List[ColumnSpec]: cols = [title_column()] - # add extra columns if metadata keys exist - if any((r.metadata or {}).get("size") for r in rows): - cols.append(ColumnSpec("size", "Size", lambda r: r.size_bytes or "")) + if any((row.metadata or {}).get("size") for row in rows): + cols.append(ColumnSpec("size", "Size", lambda row: row.size_bytes or "")) return cols -# Selection args for `@N` expansion or `select` cmdlet + def selection_fn(row: ResultModel) -> List[str]: - # prefer -path when available if row.path: return ["-path", row.path] return ["-title", row.title or ""] -# Register plugin (done at import time) -register_plugin("myprovider", adapter, columns=columns_factory, selection_fn=selection_fn) + +register_plugin("myplugin", adapter, columns=columns_factory, selection_fn=selection_fn) ``` --- -## Table scraping: using TableProviderMixin (HTML tables / list-results) +## Table scraping with `TableProviderMixin` -If your provider scrapes HTML tables or list-like results (common on web search pages), use `TableProviderMixin`: +If your plugin scrapes HTML tables or list-like results, use `TableProviderMixin`: ```py from ProviderCore.base import Provider from SYS.provider_helpers import TableProviderMixin -class MyTableProvider(TableProviderMixin, Provider): + +class MyTablePlugin(TableProviderMixin, Provider): URL = ("https://example.org/search",) def validate(self) -> bool: @@ -84,58 +94,56 @@ class MyTableProvider(TableProviderMixin, Provider): return self.search_table_from_url(url, limit=limit) ``` -`TableProviderMixin.search_table_from_url` returns `ProviderCore.base.SearchResult` entries. If you want to integrate this plugin with the strict `ResultTable` registry, add a small adapter that converts `SearchResult` -> `ResultModel` and register it using `register_plugin` (see `plugins/vimm/__init__.py` for a real example). +`TableProviderMixin.search_table_from_url` returns +`ProviderCore.base.SearchResult` entries. If you want to integrate the plugin +with the strict `ResultTable` registry, add a small adapter that converts +`SearchResult` to `ResultModel` and register it using `register_plugin`. --- -## Columns & selection - -- `columns` may be a static `List[ColumnSpec]` or a factory `def cols(rows: List[ResultModel]) -> List[ColumnSpec]` that inspects sample rows. -- `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args (e.g., `['-path', row.path]`). These args are used by `select` and `@N` expansion. - - **Tip:** for plugins that produce downloadable file rows prefer returning explicit URL args (e.g., `['-url', row.path]`) so the selected URL is clearly identified by downstream downloaders and to avoid ambiguous parsing when plugin hints (like `-plugin`) are present. -- Ensure your `ResultModel.source` is set (either in the model or rely on the provider name set by `serialize_row`). +## Columns and selection +- `columns` may be a static `List[ColumnSpec]` or a factory that inspects sample rows. +- `selection_fn` must accept a `ResultModel` and return a `List[str]` representing CLI args. +- For downloadable file rows, prefer explicit URL args such as `['-url', row.path]` so downstream downloaders interpret the row unambiguously. +- Ensure `ResultModel.source` is set directly or falls back to the registered plugin name during serialization. --- -## Optional: pandas path for `` extraction - -`SYS.html_table.extract_records` prefers a pure-lxml path but will use `pandas.read_html` if pandas is installed and the helper detects it works for the input table. This is optional and **not required** to author a provider β€” document in your provider whether it requires `pandas` and add an informative error/log message when it is missing. +## Optional pandas support +`SYS.html_table.extract_records` prefers a pure-lxml path but can fall back to +`pandas.read_html` when pandas is installed and the helper detects it works for +the input table. This is optional. Document whether your plugin requires +`pandas` and emit a clear error or log message when it is missing. --- -## Testing & examples +## Testing and examples +- Write `tests/test_plugin_.py` or follow the repo's older naming conventions when extending existing tests. +- Verify `plugin.build_table(...)` produces a `ResultTable` with rows and columns. +- Verify `serialize_rows()` yields `_selection_args`, `_selection_action` when applicable, and `source`. +- When you need an exact CLI stage sequence, call `table.set_row_selection_action(row_index, tokens)` so replay uses the row action verbatim. +- For table-oriented plugins, test `search_table_from_url` with a local HTML fixture or a mocked `HTTPClient`. -- Write `tests/test_provider_.py` that imports your provider and verifies `provider.build_table(...)` produces a `ResultTable` (has `.rows` and `.columns`) and that `serialize_rows()` yields dicts with `_selection_args`, `_selection_action` when applicable, and `source`. -- When you need to guarantee a specific CLI stage sequence (e.g., `download-file -url -plugin `), call `table.set_row_selection_action(row_index, tokens)` so the serialized payload emits `_selection_action` and the CLI can run the row exactly as intended. -- For table providers you can test `search_table_from_url` using a local HTML fixture or by mocking `HTTPClient` to return a small sample page. -- If you rely on pandas, add a test that monkeypatches `sys.modules['pandas']` to a simple shim to validate the pandas path. - -**Example test skeleton** +Example test skeleton: ```py -from SYS.result_table_adapters import get_provider +from SYS.result_table_adapters import get_plugin from plugins import example_provider -def test_example_provider_registration(): +def test_example_plugin_registration(): plugin = get_plugin("example") - rows = list(provider.adapter(example_provider.SAMPLE_ITEMS)) + rows = list(plugin.adapter(example_provider.SAMPLE_ITEMS)) assert rows and rows[0].title - cols = provider.get_columns(rows) - assert any(c.name == "title" for c in cols) - table = provider.build_table(example_provider.SAMPLE_ITEMS) + cols = plugin.get_columns(rows) + assert any(col.name == "title" for col in cols) + table = plugin.build_table(example_provider.SAMPLE_ITEMS) assert table.provider == "example" and table.rows ``` --- -## References & examples - +## References and examples - Read `plugins/example_provider.py` for a compact example of a strict adapter and dynamic columns. -- Read `plugins/vimm/__init__.py` for a table-provider that uses `TableProviderMixin` and converts `SearchResult` β†’ `ResultModel` for registration. -- See `docs/provider_guide.md` for a broader provider development checklist. - ---- - -If you want, I can also add a small `plugins/myprovider_template.py` file and unit tests for it. +- Read `plugins/vimm/__init__.py` for a table-oriented plugin that uses `TableProviderMixin` and converts `SearchResult` to `ResultModel` for registration. +- See `docs/provider_guide.md` for a broader plugin development checklist. diff --git a/docs/provider_guide.md b/docs/provider_guide.md index 696f479..4532b3c 100644 --- a/docs/provider_guide.md +++ b/docs/provider_guide.md @@ -1,62 +1,69 @@ # Plugin Development Guide -## 🎯 Purpose -This guide describes how to write, test, and register a plugin so the application can discover and use it as a pluggable component. +## Purpose +This guide describes how to write, test, and register a plugin so the +application can discover and use it as a pluggable component. -> Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in plugins share the same `plugins/` layout. +Note: this file keeps its historical `provider_guide` name, but the public +model is plugin-first. Some runtime classes still use `Provider` naming +internally. + +Keep plugin code small, focused, and well-tested. Bundled plugins and drop-in +plugins share the same `plugins/` layout. --- -## πŸ”§ Anatomy of a Plugin -A plugin is a Python class that extends `ProviderCore.base.Provider` and implements a few key methods and attributes. +## Anatomy of a plugin +A plugin is a Python class that currently extends the internal base class +`ProviderCore.base.Provider` and implements a few key methods and attributes. Minimum expectations: -- `class MyPlugin(Provider):` β€” subclass the base plugin class -- `URL` / `URL_DOMAINS` or `url_patterns()` β€” to let the registry route URLs -- `validate(self) -> bool` β€” return True when the plugin is configured and usable -- `search(self, query, limit=50, filters=None, **kwargs)` β€” return a list of `SearchResult` +- `class MyPlugin(Provider):` subclasses the current internal base plugin class. +- `URL`, `URL_DOMAINS`, or `url_patterns()` let the registry route URLs. +- `validate(self) -> bool` returns `True` when the plugin is configured and usable. +- `search(self, query, limit=50, filters=None, **kwargs)` returns a list of `SearchResult` items. Optional but common: -- `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]` β€” download a plugin result -- `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool` β€” handle `@N` selections -- `download_url(self, url, output_dir, progress_cb=None)` β€” direct URL-handling helper +- `download(self, result: SearchResult, output_dir: Path) -> Optional[Path]` +- `selector(self, selected_items, *, ctx, stage_is_last=True, **kwargs) -> bool` +- `download_url(self, url, output_dir, progress_cb=None)` --- -## 🧩 SearchResult +## SearchResult Use `ProviderCore.base.SearchResult` to describe results returned by `search()`. + Important fields: -- `table` (str) β€” provider table name -- `title` (str) β€” short human title -- `path` (str) β€” canonical URL / link the provider/dl may use -- `media_kind` (str) β€” `file`, `folder`, `book`, etc. -- `columns` (list[tuple[str,str]]) β€” extra key/value pairs to display -- `full_metadata` (dict) β€” provider-specific metadata for downstream stages -- `annotations` / `tag` β€” simple metadata for filtering +- `table` (str): plugin table name +- `title` (str): short human title +- `path` (str): canonical URL or link the plugin or downloader may use +- `media_kind` (str): `file`, `folder`, `book`, and similar values +- `columns` (list[tuple[str, str]]): extra key/value pairs to display +- `full_metadata` (dict): plugin-specific metadata for downstream stages +- `annotations` or `tag`: simple metadata for filtering Return a list of `SearchResult(...)` objects or simple dicts convertible with `.to_dict()`. --- -## βœ… Implementing search() +## Implementing `search()` - Parse and sanitize `query` and `filters`. - Return no more than `limit` results. -- Use `columns` to provide table columns (TITLE, Seeds, Size, etc.). -- Keep `search()` fast and predictable (apply reasonable timeouts). +- Use `columns` to provide table columns such as `TITLE`, `Seeds`, or `Size`. +- Keep `search()` fast and predictable by using reasonable timeouts. Example: ```python from ProviderCore.base import Provider, SearchResult -class HelloProvider(Provider): + +class HelloPlugin(Provider): def search(self, query, limit=50, filters=None, **kwargs): q = (query or "").strip() if not q: return [] - results = [] - # Build up results - results.append( + results = [ SearchResult( table="hello", title=f"Hit for {q}", @@ -64,102 +71,94 @@ class HelloProvider(Provider): columns=[("Info", "example")], full_metadata={"source": "hello"}, ) - ) + ] return results[:max(0, int(limit))] ``` --- -## ⬇️ Implementing download() and download_url() +## Implementing `download()` and `download_url()` - Prefer plugin `download(self, result, output_dir)` for piped plugin items. -- For plugin-provided URLs, implement `download_url` to allow `download-file` to route downloads through plugins. +- For plugin-provided URLs, implement `download_url` so `download-file` can route downloads through the plugin. - Use the repo `_download_direct_file` helper for HTTP downloads when possible. -Example download(): +Example download method: ```python def download(self, result: SearchResult, output_dir: Path) -> Optional[Path]: - # Validate config url = getattr(result, "path", None) if not url or not url.startswith("http"): return None - # use existing helpers to fetch the file return _download_direct_file(url, output_dir) ``` --- -## 🧭 URL routing +## URL routing Plugins can declare: -- `URL = ("magnet:",)` or similar prefix list +- `URL = ("magnet:",)` or similar prefix lists - `URL_DOMAINS = ("example.com",)` to match hosts -- Or override `@classmethod def url_patterns(cls):` to combine static and dynamic patterns +- `@classmethod def url_patterns(cls):` to combine static and dynamic patterns -The registry uses these to match `download-file ` or to pick which plugin should handle the URL. +The registry uses these declarations to match `download-file ` and to pick +which plugin should handle a URL. --- -## πŸ›  Selector (handling `@N` picks) -- Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or to enqueue downloads. -- Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-ups. -- Return `True` when you handled the selection and the pipeline should pause or proceed accordingly. +## Selector behavior and `@N` +- Implement `selector(self, selected_items, *, ctx, stage_is_last=True)` to present a sub-table or enqueue downloads. +- Use `ctx.set_last_result_table()` and `ctx.set_current_stage_table()` to display follow-up tables. +- Return `True` when the selector handled the selection and the pipeline should stop expanding that row. --- -## πŸ§ͺ Testing plugins -- Keep tests small and local. Create `tests/test_provider_.py` or another tracked test target. -- Test `search()` with mock HTTP responses (use `requests-mock` or similar). +## Testing plugins +- Keep tests small and local. +- Create `tests/test_plugin_.py` or follow the existing repo naming when extending older tests. +- Test `search()` with mock HTTP responses. - Test `download()` using a temp directory and a small file server or by mocking `_download_direct_file`. - Test `selector()` by constructing a fake result and `ctx` object. -Example PowerShell commands to run tests (repo root): +Example PowerShell commands from the repo root: ```powershell -# Run a single test file -pytest tests/test_provider_hello.py -q - -# Run all tests +pytest tests/test_plugin_hello.py -q pytest -q ``` --- -## πŸ“¦ Registration & packaging +## Registration and packaging - Bundled plugins live under `plugins/` and are auto-discovered from that package. -- External user plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` / `MEDEIA_PLUGIN_PATH`. -- Package directories are preferred so plugin-specific files can travel with the plugin. +- External plugins can be dropped into `plugins/` or any directory listed in `MM_PLUGIN_PATH` or `MEDEIA_PLUGIN_PATH`. +- Package directories are preferred so plugin-specific files travel with the plugin. - Plugin authors should import from `ProviderCore.*`. +If a plugin supports multiple configured endpoints or accounts, the user-facing +concept is a plugin instance. Some stored config still lives under legacy key +paths such as `provider..`. + --- -## πŸ’‘ Best practices & tips -- Use `debug()` / `log()` appropriately; avoid noisy stderr output in normal runs. +## Best practices +- Use `debug()` and `log()` appropriately; avoid noisy stderr output in normal runs. - Prefer returning `SearchResult` objects to provide consistent UX. -- Keep `search()` tolerant (timeouts, malformed responses) and avoid raising for expected network problems. +- Keep `search()` tolerant of timeouts and malformed responses. - Use `full_metadata` to pass non-display data to `download()` and `selector()`. - Respect the `limit` parameter in `search()`. --- -## 🧾 Example provider checklist -- [ ] Implement `search()` and return `SearchResult` items -- [ ] Implement `validate()` to check essential config (API keys, credentials) -- [ ] Provide `URL` / `URL_DOMAINS` or `url_patterns()` for routing -- [ ] Add `download()` or `download_url()` for piped/passed URL downloads -- [ ] Add tests under `tests/` -- [ ] Add the plugin under `plugins//` for bundled or plug-and-play installs +## Example plugin checklist +- [ ] Implement `search()` and return `SearchResult` items. +- [ ] Implement `validate()` to check essential config such as API keys or credentials. +- [ ] Provide `URL`, `URL_DOMAINS`, or `url_patterns()` for routing. +- [ ] Add `download()` or `download_url()` for piped or passed URL downloads. +- [ ] Add tests under `tests/`. +- [ ] Add the plugin under `plugins//` for bundled or plug-and-play installs. --- -## πŸ”— Further reading +## Further reading - See existing bundled plugins in `plugins/` for patterns and edge cases. -- Check `API/` helpers for HTTP and debrid clients. - - ---- - -If you'd like, I can: -- Add an example plugin file under `plugins/` as a template (see `plugins/hello/__init__.py`), and -- Create unit tests for it (see `tests/test_provider_hello.py`). - -I have added a minimal example plugin and tests in this repository; use them as a starting point for new plugins. +- Check `API/` helpers for HTTP and debrid clients used by plugins. diff --git a/docs/result_table.md b/docs/result_table.md index 45b93e1..ad64ea9 100644 --- a/docs/result_table.md +++ b/docs/result_table.md @@ -1,229 +1,184 @@ -# ResultTable system β€” Overview & usage +# ResultTable system: overview and usage -This document explains the `ResultTable` system used across the CLI and TUI: how tables are built, how providers integrate with them, and how `@N` selection/expansion and provider selectors work. +This document explains the `ResultTable` system used across the CLI: how tables +are built, how plugins integrate with them, and how `@N` selection, row replay, +and plugin selectors work. -## TL;DR βœ… -- `ResultTable` is the unified object used to render tabular results and drive selection (`@N`) behavior. -- Providers should return `SearchResult` objects (or dicts) and can either supply `selection_args` per row or implement a `selector()` method to handle `@N` selections. -- Table metadata (`set_table_metadata`) helps providers attach context (e.g., `provider_view`, `magnet_id`) that selectors can use. +## TL;DR +- `ResultTable` is the unified object used to render tabular results and drive `@N` behavior. +- Plugins return `SearchResult` objects or dicts and can either supply row selection args or implement a `selector()` method. +- Rows can also carry an explicit `selection_action`, which the CLI replays verbatim. +- Table metadata helps plugins attach context. Some metadata keys still use older `provider` names internally. --- ## Key concepts -- **ResultTable** (`SYS/result_table.py`) - - Renders rows as a rich table and stores metadata used for selection expansion. - - Important APIs: `add_result()`, `set_table()`, `set_source_command()`, `set_row_selection_args()`, `set_row_selection_action()`, `set_table_metadata()`, and `select_interactive()`. +### ResultTable +`SYS/result_table.py` renders rows as a rich table and stores metadata used for +selection expansion. -- **ResultRow** - - Holds columns plus `selection_args` (used for `@N` expansion) and `payload` (original object). - - Optionally stores `selection_action`, a full list of CLI tokens to run when `@N` selects this row. When present the CLI honors the explicit action instead of reconstructing it from `source_command` and `selection_args`. +Important APIs: +- `add_result()` +- `set_table()` +- `set_source_command()` +- `set_row_selection_args()` +- `set_row_selection_action()` +- `set_table_metadata()` -- **Provider selector** - - If a provider implements `selector(selected_items, ctx=..., stage_is_last=True)`, it is run first when `@N` is used; if the selector returns `True` it has handled the selection (e.g., drilling into a folder and publishing a new ResultTable). +### ResultRow +A row holds columns, payload data, selection args, and optionally a full +`selection_action` token list. When `selection_action` is present, `@N` uses it +instead of reconstructing a command from the source command and row args. -- **Pipeline / CLI expansion** - - When the user types `@N`, CLI tries provider selectors first. If none handle it, CLI re-runs `source_command + source_args + row_selection_args` (for single-selection) or pipes items downstream for multi-selection. +### Plugin selector +If a plugin implements `selector(selected_items, ctx=..., stage_is_last=True)`, +that selector runs first when `@N` is used. If the selector returns `True`, it +has handled the selection, for example by publishing a nested table. -- **Table metadata** - - `ResultTable.set_table_metadata(dict)` allows attaching provider-specific context (for example: `{"provider":"alldebrid","view":"files","magnet_id":123}`) for selectors and other code to use. +### Table metadata +`ResultTable.set_table_metadata(dict)` attaches plugin-specific context for +selectors and follow-up stages. You will still see legacy keys such as +`provider` in some metadata because parts of the runtime still consume them. --- -## How to build a table (provider pattern) +## Building a table -Typical provider flow (pseudocode): +Typical plugin flow: ```py from SYS.result_table import ResultTable -table = ResultTable("Provider: X result").set_preserve_order(True) -table.set_table("provider_name") -table.set_table_metadata({"provider":"provider_name","view":"folders"}) -table.set_source_command("search-file", ["-plugin","provider_name","query"]) +table = ResultTable("Plugin: X results").set_preserve_order(True) +table.set_table("plugin_name") +table.set_table_metadata({"provider": "plugin_name", "view": "folders"}) +table.set_source_command("search-file", ["-plugin", "plugin_name", "query"]) -for r in results: - table.add_result(r) # r can be a SearchResult, dict, or PipeObject +for result in results: + table.add_result(result) ctx.set_last_result_table(table, payloads) ctx.set_current_stage_table(table) ``` Notes: -- To drive a direct `@N` re-run, call `table.set_row_selection_args(row_index, ["-open", ""])`. -- For more advanced or interactive behavior (e.g., drill-into, fetch more rows), implement `provider.selector()` and return `True` when handled. +- To drive a direct replay, call `table.set_row_selection_args(row_index, ["-open", ""])`. +- For drill-in or interactive behavior, implement `plugin.selector()` and return `True` when handled. +- For exact row replay, call `table.set_row_selection_action(row_index, tokens)`. --- -## Selection (@N) flow (brief) +## Selection flow -1. User enters `@N` in the CLI. -2. CLI chooses the appropriate table (overlay > last table > history) and gathers the selected payload(s). -3. `PipelineExecutor._maybe_run_class_selector()` runs provider `selector()` hooks for the provider inferred from table or payloads. If any selector returns `True`, expansion stops. -4. Otherwise, for single selections, CLI grabs `row.selection_args` and expands: `source_command + source_args + row_selection_args` and inserts it as the expanded stage. For multi-selections, items are piped downstream. +When a user enters `@N`: + +1. The CLI chooses the active table. +2. It gathers the selected payloads. +3. `PipelineExecutor._maybe_run_class_selector()` runs plugin `selector()` hooks for the plugin inferred from the table or payloads. +4. If no selector handles the row, the CLI checks `row.selection_action` first. +5. If no explicit row action exists, the CLI expands `source_command + source_args + row_selection_args`. +6. Multi-selection results are piped downstream instead of being collapsed to one row replay. --- -## Columns & display - -- Providers can pass a `columns` list ([(name, value), ...]) in the result dict/SearchResult to control which columns are shown and their order. -- Otherwise, `ResultTable` uses a priority list (title/store/size/ext) and sensible defaults. -- The table rendering functions (`to_rich`, `format_json`, `format_compact`) are available for different UIs. +## Columns and display +- Plugins can pass a `columns` list in the result dict or `SearchResult` to control column order and display. +- Otherwise, `ResultTable` uses sensible defaults. +- Rendering helpers such as `to_rich`, `format_json`, and `format_compact` support different CLI display paths. --- -## Provider-specific examples +## Plugin-specific examples -### AllDebrid (debrid file hosting) +### AllDebrid -AllDebrid exposes a list of magnets (folder rows) and the files inside each magnet. The provider returns `folder` SearchResults for magnets and `file` SearchResults for individual files. The provider includes a `selector()` that drills into a magnet by calling `search(..., filters={"view":"files","magnet_id":...})` and builds a new `ResultTable` of files. +AllDebrid exposes a list of magnets as folder rows and the files inside each +magnet as file rows. The plugin uses `selector()` to drill into a magnet and +publish a new `ResultTable` of files. Example commands: -``` -# List magnets in your account +```powershell search-file -plugin alldebrid "*" - -# Open magnet id 123 and list its files -search-file -plugin alldebrid -open 123 "*" - -# Or expand via @ selection (selector handles drilling): -search-file -plugin alldebrid "*" -@3 # selector will open the magnet referenced by row #3 and show the file table +@3 ``` -Illustrative folder (magnet) SearchResult: +Illustrative magnet row: ```py SearchResult( - table="alldebrid", - title="My Magnet Title", - path="alldebrid:magnet:123", - detail="OK", - annotations=["folder", "ready"], - media_kind="folder", - columns=[("Folder", "My Magnet Title"), ("ID", "123"), ("Status", "ready"), ("Ready", "yes")], - full_metadata={ - "magnet": {...}, - "magnet_id": 123, - "provider": "alldebrid", - "provider_view": "folders", - "magnet_name": "My Magnet Title", - }, + table="alldebrid", + title="My Magnet Title", + path="alldebrid:magnet:123", + media_kind="folder", + full_metadata={ + "magnet_id": 123, + "provider": "alldebrid", + "provider_view": "folders", + }, ) ``` -4. Otherwise, for single selections, CLI checks for `row.selection_action` and runs that verbatim if present; otherwise it expands `source_command + source_args + row_selection_args`. For multi-selections, items are piped downstream. +Illustrative file row: ```py SearchResult( - table="alldebrid", - title="Episode 01.mkv", - path="https://.../unlocked_direct_url", - detail="My Magnet Title", - annotations=["file"], - media_kind="file", - size_bytes=123456789, - columns=[("File", "Episode 01.mkv"), ("Folder", "My Magnet Title"), ("ID", "123")], - full_metadata={ - "magnet": {...}, - "magnet_id": 123, - "magnet_name": "My Magnet Title", - "relpath": "Season 1/E01.mkv", - "provider": "alldebrid", - "provider_view": "files", - "file": {...}, - }, + table="alldebrid", + title="Episode 01.mkv", + path="https://.../unlocked_direct_url", + media_kind="file", + full_metadata={ + "magnet_id": 123, + "provider": "alldebrid", + "provider_view": "files", + }, ) ``` -Selection & download flows +Selection and download behavior: +- `@3` on a magnet row runs the plugin selector and shows a file table. +- `@2 | download-file` downloads the selected file row. +- `@1 | add-file -plugin local -instance ` triggers add-file's plugin-aware handoff flow. -- Drill-in (selector): `@3` on a magnet row runs the provider's `selector()` to build a new file table and show it. The selector uses `search(..., filters={"view":"files","magnet_id":...})` to fetch file rows. +Configure the AllDebrid plugin in `.config plugins` and add its API key before +testing these flows. -- `download-file` integration: With a file row (http(s) path), `@2 | download-file` will download the file. The `download-file` cmdlet expands AllDebrid magnet folders and will call the provider layer to fetch file bytes as appropriate. +### Bandcamp -- `add-file` convenience: Piping a file row into `add-file -plugin local -instance ` will trigger add-file's provider-aware logic. If the piped item has `table == 'alldebrid'` and a http(s) `path`, `add-file` will call `provider.download()` into a temporary directory and then ingest the downloaded file, cleaning up the temp when done. Example: +Bandcamp search supports `artist:` queries. The Bandcamp plugin implements a +`selector()` that detects artist rows and expands them into a discography table. -``` -# Expand magnet and add first file to local directory -search-file -plugin alldebrid "*" -@3 # view files -@1 | add-file -plugin local -instance C:\mydir -``` +Example: -Notes & troubleshooting - -- Configure an AllDebrid API key (see `Provider/alldebrid._get_debrid_api_key()`). -- If a magnet isn't ready the selector or `download-file` will log the magnet status and avoid attempting file downloads. - ---- - -### Bandcamp (artist β†’ discography drill-in) - -Bandcamp search supports `artist:` queries. Bandcamp's provider implements a `selector()` that detects `artist` results and scrapes the artist's page using Playwright to build a discography `ResultTable`. - -Example usage: - -``` -# Search for an artist +```powershell search-file -plugin bandcamp "artist:radiohead" - -# Select an artist row to expand into releases @1 ``` -Bandcamp SearchResult (artist / album rows): - -```py -SearchResult( - table="bandcamp", - title="Album Title", - path="https://bandcamp.com/album_url", - detail="By: Artist", - annotations=["album"], - media_kind="audio", - columns=[("Title","Album Title"), ("Location","Artist"), ("Type","album"), ("Url","https://...")], - full_metadata={"artist":"Artist","type":"album","url":"https://..."} -) -``` - -Notes: -- Playwright is required for Bandcamp scraping. The selector will log an informative message if Playwright is missing. -- Provider selectors are ideal when you need to replace one table with another (artist β†’ discography). +Plugin selectors are the right tool when you need to replace one table with +another, such as artist to discography drill-in. --- -## Provider author checklist (short) - -- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts; include useful `full_metadata` (IDs, view names) for selection/drilling. -- If you support fetching downloadable file bytes, implement `download(result, output_dir) -> Optional[Path]`. -- For drill-in or interactive transforms, implement `selector(selected_items, ctx=..., stage_is_last=True)` and call `ctx.set_last_result_table(...)` / `ctx.set_current_stage_table(...)`; return `True` when handled. -- Add tests (unit/integration) that exercise search β†’ select β†’ download flows. +## Plugin author checklist +- Implement `search(query, limit, filters)` and return `SearchResult` objects or dicts. +- Include useful `full_metadata` values for drill-in and selection replay. +- Implement `download(result, output_dir)` when the plugin can materialize file bytes. +- Implement `selector(...)` for drill-in or interactive transforms. +- Add tests that cover search, select, and download flows. --- ## Debugging tips -- Use `ctx.set_last_result_table(table, payloads)` to immediately show a table while developing a selector. -- Add `log(...)` messages in provider code to capture fail points. -- Check `full_metadata` attached to SearchResults to pass extra context (IDs, view names, provider names). +- Use `ctx.set_last_result_table(table, payloads)` to publish a table while developing a selector. +- Add `log(...)` messages in plugin code to capture fail points. +- Inspect `full_metadata` when a selector or replay path is missing required context. --- ## Quick reference -- ResultTable location: `SYS/result_table.py` -- Pipeline helpers: `SYS/pipeline.py` (`set_last_result_table`, `set_current_stage_table`, `get_current_stage_table_row_selection_args`) -- CLI expansion: `CLI.py` (handles `@N`, provider selectors, and insertion of expanded stages) -- Provider selector pattern: Implement `.selector(selected_items, ctx=..., stage_is_last=True)` in provider class. - ---- - -For more detail on ResultTable provider authoring, see `docs/provider_authoring.md`. - -If you'd like, I can also: -- Add provider-specific examples (AllDebrid, Bandcamp) into this doc βœ… -- Add a short checklist for PR reviewers when adding new providers - ---- - -Created by GitHub Copilot (Raptor mini - Preview) β€” brief guide to the ResultTable system. Feedback welcome! +- ResultTable implementation: `SYS/result_table.py` +- Pipeline helpers: `SYS/pipeline.py` +- Row replay and selection flow: `SYS/pipeline.py` +- Plugin authoring guide: `docs/provider_authoring.md` diff --git a/docs/result_table_selector.md b/docs/result_table_selector.md index 80372e7..cd0f179 100644 --- a/docs/result_table_selector.md +++ b/docs/result_table_selector.md @@ -1,22 +1,25 @@ -Selector & plugin-table usage +Selector and `plugin-table` usage -This project provides a small plugin/table/selector flow that allows plugins -and cmdlets to interact via a simple, pipable API. +This project provides a small plugin/table/selector flow that lets plugins and +cmdlets interact through a simple pipable API. -Key ideas -- `plugin-table` renders a plugin result set and *emits* pipeline-friendly dicts for each row. Each emitted item includes `_selection_args`, a list of args the plugin suggests for selecting that row (e.g., `['-path', '/tmp/file']`). -- Use the `@N` syntax to select an item from a table and chain it to the next cmdlet. +Key ideas: +- `plugin-table` renders a plugin result set and emits pipeline-friendly dicts for each row. +- Emitted rows can include `_selection_args` and, when needed, `_selection_action` for exact row replay. +- Use `@N` to select an item from a table and chain it to the next cmdlet. Example: plugin-table -plugin example -sample | @1 | add-file -instance default -What plugins must implement -- An adapter that yields `ResultModel` objects (breaking API). -- Optionally supply a `columns` factory and `selection_fn` (see `plugins/example_provider.py`). +What plugins must implement: +- An adapter that yields `ResultModel` objects. +- Optionally a `columns` factory and `selection_fn`. -Implementation notes +Implementation notes: - `plugin-table` emits dicts like `{ 'title': ..., 'path': ..., 'metadata': ..., '_selection_args': [...] }`. -- Selection syntax (`@1`) will prefer `_selection_args` if present; otherwise it will fall back to plugin selection logic or sensible defaults (`-path` or `-title`). +- When a row includes `_selection_action`, `@1` prefers that exact action. +- Otherwise `@1` falls back to `_selection_args`, then plugin selection logic, then sensible defaults such as `-path` or `-title`. -This design keeps the selector-focused UX small and predictable while enabling full cmdlet interoperability via piping and `-run-cmd`. +This design keeps selector-focused workflows small and predictable while still +allowing full cmdlet interoperability through piping and `-run-cmd`. diff --git a/docs/scp_plugin_tutorial.md b/docs/scp_plugin_tutorial.md index dfd9196..e56ca54 100644 --- a/docs/scp_plugin_tutorial.md +++ b/docs/scp_plugin_tutorial.md @@ -1,23 +1,26 @@ # SCP Plugin Walkthrough -This walkthrough adds a bundled `scp` plugin backed by existing SSH libraries: +This walkthrough covers the bundled `scp` plugin, backed by existing SSH +libraries: - `paramiko` for SSH and SFTP directory listing - `scp` for file transfers The implementation lives in [plugins/scp/__init__.py](plugins/scp/__init__.py). -## What The Plugin Does +## What the plugin does The SCP plugin mirrors the FTP walkthrough, but on top of SSH: +- `search-file -plugin scp -instance ...` lists remote files and folders over SFTP +- plain `@N` on a folder drills into that directory +- plain `@N` on a file runs `download-file -plugin scp -instance -url ...` +- `@N | add-file -instance ...` downloads first, then ingests the local temp file +- `add-file -plugin scp -instance -path ...` uploads a local file to the configured remote path -- `search-file -plugin scp -instance ...` lists remote files and folders over SFTP. -- plain `@N` on a folder drills into that directory. -- plain `@N` on a file runs `download-file -plugin scp -instance -url ...`. -- `@N | add-file -instance ...` downloads first, then ingests the local temp file. -- `add-file -plugin scp -instance -path ...` uploads a local file to the configured remote path. +## Example config -## Example Config +Add one or more named SCP plugin instances to your config. The current stored +key path remains `provider.scp.` for legacy compatibility: ```toml [provider.scp.work] @@ -42,39 +45,22 @@ timeout = 20 ``` Notes: - -- `work` and `archive` are instance names; use them with `-instance work` or `-instance archive`. +- `work` and `archive` are instance names. - `host` and `username` are required for each instance to validate. - You can use password auth, key auth, or both. - `base_path` is both the default search root and the default upload directory. +- You can browse configured instances from `.config plugins` in the CLI. -## Search Flow - -List the configured base path: +## Search flow ```powershell search-file -plugin scp -instance work "*" -``` - -Search by filename: - -```powershell search-file -plugin scp -instance work "invoice" -``` - -Search another subtree with deeper recursion: - -```powershell search-file -plugin scp -instance work "path:/srv/files/releases depth:2 *.zip" -``` - -Show only folders: - -```powershell search-file -plugin scp -instance work "path:/srv/files type:folder *" ``` -## Selection Flow +## Selection flow Folder rows are navigation rows: @@ -83,7 +69,8 @@ search-file -plugin scp -instance work "*" @2 ``` -File rows carry an explicit row action, so terminal selection downloads directly: +File rows carry an explicit row action, so terminal selection downloads +directly: ```powershell search-file -plugin scp -instance work "report" @@ -96,45 +83,36 @@ That expands to the equivalent of: download-file -plugin scp -instance work -url scp://ssh.example.com/srv/files/report.pdf ``` -## Download And Add-File Flow - -Download into a local folder: +## Download and add-file flow ```powershell search-file -plugin scp -instance work "report" @1 | download-file -path C:\Downloads -``` -Ingest a selected remote file into a configured instance backend: - -```powershell search-file -plugin scp -instance work "report" @1 | add-file -instance tutorial ``` Why this works: - - file rows advertise `_selection_action` for `download-file` -- `add-file` selection replay inserts that provider download stage before ingest -- the plugin also implements `resolve_pipe_result_download()` for provider-owned SCP rows -- file rows also carry the chosen `instance`, so replay stays bound to the same SSH target +- `add-file` selection replay inserts that plugin download stage before ingest +- the plugin also implements `resolve_pipe_result_download()` for plugin-owned SCP rows +- file rows carry the chosen `instance`, so replay stays bound to the same SSH target -## Upload Flow - -Upload a local file to the configured remote `base_path`: +## Upload flow ```powershell add-file -plugin scp -instance archive -path C:\Media\report.pdf ``` -## Implementation Notes - -The plugin uses SFTP for directory listing because SCP itself is a transfer protocol, not a browse/search protocol. That split keeps the provider simple: +## Implementation notes +The plugin uses SFTP for directory listing because SCP itself is a transfer +protocol, not a browse/search protocol. That split keeps the plugin simple: - browse and metadata via Paramiko SFTP - file transfer via the `scp` package -## Recommended Demo Commands +## Recommended demo commands ```powershell search-file -plugin scp -instance work "*" diff --git a/plugins/README.md b/plugins/README.md index 5ca22f8..b4c93cf 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -3,11 +3,16 @@ This folder is the primary home for bundled plugins and also the default search path for drop-in plugins. +User-facing docs and CLI flows treat these integrations as plugins. Some Python +types and some stored config keys still use older `Provider` naming internally, +but that is a legacy implementation detail rather than the preferred public +term. + Preferred layout: - Put each plugin in its own folder under `plugins//` with an `__init__.py`. - Keep plugin-specific assets beside the code in that same folder. - Single-file `.py` plugins are still supported, but package folders are the - recommended plug-and-play format. + recommended plug-and-play format. That means a plugin can ship as a drag-and-drop folder with extras such as: - `cookies.txt` @@ -24,8 +29,8 @@ drop-in plugin search paths are: Plugin rules: - A plugin can be a single `.py` file or a package directory with `__init__.py`. -- Define a class that inherits from `ProviderCore.base.Provider`. -- Give it a stable name using `PLUGIN_NAME` or the class name. +- Current plugin classes inherit from `ProviderCore.base.Provider`. +- Give the plugin a stable name using `PLUGIN_NAME` or the class name. Example skeleton: @@ -52,9 +57,12 @@ class MyPlugin(Provider): Bundled walkthrough: -- Providers can now expose named config instances under `provider..` and cmdlets can target them with `-instance `. +- Plugins can expose named config instances. The current stored config may still + use legacy key paths such as `provider..`, and cmdlets target + instances with `-instance `. +- Use `.config plugins` in the CLI to browse configured plugin instances. - The repo now includes a real FTP example plugin in [plugins/ftp/__init__.py](plugins/ftp/__init__.py). - The walkthrough is in [docs/ftp_plugin_tutorial.md](docs/ftp_plugin_tutorial.md) and shows `search-file -plugin ftp -instance `, folder drill-in via `@N`, file download routing, `@N | add-file -instance ...`, and `add-file -plugin ftp -instance ` uploads. - The repo also includes an SCP example plugin in [plugins/scp/__init__.py](plugins/scp/__init__.py). - The walkthrough is in [docs/scp_plugin_tutorial.md](docs/scp_plugin_tutorial.md) and shows `search-file -plugin scp -instance `, SSH-backed directory drill-in, file download routing, `@N | add-file -instance ...`, and `add-file -plugin scp -instance ` uploads. -- The repo now also includes a built-in HydrusNetwork provider in [plugins/hydrusnetwork/__init__.py](plugins/hydrusnetwork/__init__.py). Its Hydrus client API now lives in the plugin-owned package [plugins/hydrusnetwork/api/__init__.py](plugins/hydrusnetwork/api/__init__.py), its registry-facing store adapter lives in [plugins/hydrusnetwork/store_proxy.py](plugins/hydrusnetwork/store_proxy.py), and its heavy internal operations live in [plugins/hydrusnetwork/store_backend.py](plugins/hydrusnetwork/store_backend.py). This `plugins//api/` package shape is the intended pattern for plugin-owned API helpers going forward. The provider now resolves configured Hydrus instances directly from plugin config instead of routing back through `Store.registry`; the proxy exists only so generic store callers can still target configured Hydrus stores. [API/HydrusNetwork.py](API/HydrusNetwork.py) and [Store/HydrusNetwork.py](Store/HydrusNetwork.py) are legacy compatibility shims only, and store discovery prefers the plugin-owned Hydrus hook over those shims. \ No newline at end of file +- The repo also includes a built-in HydrusNetwork plugin in [plugins/hydrusnetwork/__init__.py](plugins/hydrusnetwork/__init__.py). Its Hydrus client API now lives in the plugin-owned package [plugins/hydrusnetwork/api/__init__.py](plugins/hydrusnetwork/api/__init__.py), its registry-facing store adapter lives in [plugins/hydrusnetwork/store_proxy.py](plugins/hydrusnetwork/store_proxy.py), and its heavy internal operations live in [plugins/hydrusnetwork/store_backend.py](plugins/hydrusnetwork/store_backend.py). This `plugins//api/` package shape is the intended pattern for plugin-owned API helpers going forward. The plugin resolves configured Hydrus instances directly from plugin config instead of routing back through `Store.registry`; the proxy exists only so generic store callers can still target configured Hydrus stores. [API/HydrusNetwork.py](API/HydrusNetwork.py) and [Store/HydrusNetwork.py](Store/HydrusNetwork.py) are legacy compatibility shims only, and store discovery prefers the plugin-owned Hydrus hook over those shims. \ No newline at end of file diff --git a/readme.md b/readme.md index 7e114cd..75b4a3c 100644 --- a/readme.md +++ b/readme.md @@ -1,69 +1,158 @@ -
-

MEDEIA-MACINA

- -

4 TEXT BASED FILE ONTOLOGY

-
+# MEDEIA-MACINA -Medios-Macina is a API driven file media manager and virtual toolbox capable of downloading, tagging, archiving, sharing, and connecting you to HydrusNetwork backends. It is designed around a compact, pipeable command language ("cmdlets") so complex workflows can be composed simply and repeatably. +![Medeia-Macina logo](https://code.glowers.club/goyimnose/Medios-Macina/raw/branch/main/docs/img/MM.png) -

ELEVATED PITCH

-
    -
  • Have you ever wanted one app that can manage all your media files and is completely in your control?
    • -
  • Do you want a no-brainer one-stop shop for finding & downloading applications?
    • -
  • Are you one that has an unorganized & unapologetic mess of files that are loosely organized in random folders?
    • -
  • Does it take you several brainfarts until you get a scent of where that file is at that your looking for?
    • -
  • Do you have trouble struggling with filenames so that you can find the file you want later?
    • -
-
  • Would you like to have your media library available with you even when you are away from home?
    • - -
    -

    CONTENTS

    -FEATURES
    -INSTALLATION
    -TAG TEMPLATE SYNTAX
    -CONFIG
    -HYDRUS NETWORK
    -COOKIES
    -TUTORIAL
    -
    +Medeia-Macina is a text-first media manager and plugin runtime for searching, downloading, tagging, archiving, replaying, and moving media through one CLI. It is built around pipeable commands, rich result tables, and row replay so you can move from search to action without leaving the terminal. -

    Features

    -
      -
    • Connects to HydrusNetwork, which is an open-source privacy orientated database-driven file manager
      • -
    • Medios-Macina uses multiple python module downloaders for your specific needs, it is highly configurable and customizable
      • -
    • no opening of folders neccessary! You can add multiple tags to a file and use the search engine to immediately find and retrieve that file your looking for
      • -
    • Flexible syntax structure: chain commands with `|` and select options from tables with `@N`.
      • -
    • Multiple file stores: *HYDRUSNETWORK* -- **Plugin integration:** *YOUTUBE, OPENLIBRARY, INTERNETARCHIVE, SOULSEEK, LIBGEN, ALLDEBRID, TELEGRAM, BANDCAMP*
      • -
    • Module Mixing: *[Playwright](https://github.com/microsoft/playwright), [yt-dlp](https://github.com/yt-dlp/yt-dlp), [typer](https://github.com/fastapi/typer)*
      • -
    • Optional stacks: Telethon (Telegram), aioslsk (Soulseek), and the FlorenceVision tooling install automatically when you configure the corresponding plugin/tool blocks. -
    • MPV Manager: Play audio, video, and even images in a custom designed MPV with trimming, screenshotting, and more built right in!
      • -
    • Supports remote access and networked setups for offsite servers and sharing workflows.
      • -
    • Reusable tag templates: derive new tags from existing ones with placeholder and padding syntax documented in docs/tag_template_syntax.md.
      • -
    - -

    INSTALLATION

    -

    Requirements: Python -

    GIT -
    +## What Medeia-Macina Does -
    -COMMAND LINE +- Search local and remote sources through plugins. +- Browse results as tables and replay rows with `@N`. +- Chain follow-up actions with `|`. +- Add or move files into configured backends such as HydrusNetwork. +- Inspect metadata, tags, URLs, and file details from the same CLI. +- Hand media off to MPV for playback, screenshots, and related workflows. +- Load bundled plugins or drop-in plugins from external paths. -
     
-curl -sSL https://code.glowers.club/goyimnose/Medios-Macina/raw/branch/main/scripts/bootstrap.py | python -
-
    +## How The App Works -
    +The basic interaction loop is: -you may need to change python3 to python depending on your python installation -
    -After install, start the CLI by simply inputting "mm" into terminal/console, once the application is up and running you will need to connect to a HydrusNetwork sever to get the full experience. To access the config simply input ".config" while the application is running -

    -ytdlp -hydrusnetwork +1. Run a command that returns a table. +2. Use `@N` to select a row. +3. Let the row replay its plugin-defined action, or pipe it into another command. +4. Use `.config` any time to inspect or update configuration. + +That means rows are not just display data. A row can carry selection arguments or a full replay action. Depending on the plugin and row type, plain `@N` might open a nested table, download a file, show details, or trigger another plugin-specific workflow. + +## Plugin System + +Plugins are the main integration surface for the app. + +- Built-in integrations such as HydrusNetwork, yt-dlp/YouTube, FTP, SCP, Soulseek, Telegram, Internet Archive, OpenLibrary, Bandcamp, and others are treated as plugins. +- Plugins can expose named instances, so one plugin can target multiple endpoints, accounts, or servers via `-instance `. +- Bundled and external plugins use the same `plugins//` layout. +- External plugin search paths include the repo `plugins/` folder, the current working directory `plugins/` folder, `MM_PLUGIN_PATH`, and `MEDEIA_PLUGIN_PATH`. +- Plugin authoring still uses the current Python base class name `ProviderCore.base.Provider`. That is an implementation detail rather than the preferred user-facing term. + +See [plugins/README.md](plugins/README.md) for plugin packaging and discovery details. + +## Configuration + +The old interactive TUI config editor has been discontinued. + +Use `.config` from inside the CLI instead: + +- `.config` opens the root configuration table. +- `@N` drills into a selected section. +- `@..` goes back. +- `.config plugins` jumps straight to the user-facing plugin section. +- Selecting a value row and running `.config ` updates that setting. + +This keeps configuration inside the same table-and-selection model as the rest of the app. + +## Installation From A Git Checkout + +Requirements: + +- Python 3.9 through 3.13 +- Git +- PowerShell on Windows +- `mpv` recommended for playback workflows + +From the repository root, run: + +```powershell +Set-Location C:\path\to\Medios-Macina + +Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass +.\scripts\bootstrap.ps1 -Editable + +.\.venv\Scripts\Activate.ps1 +mm +``` + +Notes: + +- The bootstrap script creates `.venv`, installs the project, and exposes the `mm` and `medeia` console commands. +- On Windows, the bootstrap script will try to ensure `mpv` is available. +- Playwright browser support is installed by default unless you pass `-NoPlaywright`. +- The repository also ships `scripts/bootstrap.py` and `scripts/bootstrap.sh`, but the examples here use PowerShell. + +## First Run + +After launching `mm`, start with configuration: + +```powershell +.config +.config plugins +``` + +If you want the full archive/tag/search workflow, HydrusNetwork is usually the first plugin to configure. + +## Example Workflows + +Browse plugin configuration: + +```powershell +.config +.config plugins +``` + +Search a configured plugin instance: + +```powershell +search-file -plugin ftp -instance work "invoice" +``` + +Replay a selected row: + +```powershell +@1 +``` + +Ingest a selected remote result into a configured backend: + +```powershell +@1 | add-file -instance tutorial +``` + +Upload a local file through a plugin: + +```powershell +add-file -plugin ftp -instance archive -path C:\Media\report.pdf +``` + +The exact meaning of `@1` depends on the current table and plugin. For example, one row may open a nested directory table while another row may download or replay a file-specific action. + +## Core Concepts + +- `search-file`: search a plugin, source, or configured backend and produce a table. +- `@N`: replay or select row `N` from the most recent table. +- `|`: pipe the selected result into the next command. +- `add-file`: ingest a file into a configured backend or upload through a plugin. +- `.config`: browse and edit configuration from inside the CLI. +- `.mpv`: hand media off to the integrated MPV workflow. + +## Documentation + +- [docs/tag_template_syntax.md](docs/tag_template_syntax.md) +- [plugins/README.md](plugins/README.md) +- [docs/provider_guide.md](docs/provider_guide.md) +- [docs/provider_authoring.md](docs/provider_authoring.md) +- [docs/ftp_plugin_tutorial.md](docs/ftp_plugin_tutorial.md) +- [docs/scp_plugin_tutorial.md](docs/scp_plugin_tutorial.md) +- [docs/BOOTSTRAP_TROUBLESHOOTING.md](docs/BOOTSTRAP_TROUBLESHOOTING.md) + +## Current Direction + +Medeia-Macina is moving toward one canonical plugin model: + +- user-facing integrations are described as plugins +- plugin rows carry their own replay actions when needed +- nested tables use the same `@N` and `@..` interaction model across config and plugin workflows +- Hydrus-backed archiving, metadata, and playback flows are treated as part of the same CLI pipeline rather than separate apps