updated plugin refactor and added FTP and SCP plugins , also hydrusnetwork plugin migration
This commit is contained in:
@@ -0,0 +1,164 @@
|
||||
# FTP Plugin Walkthrough
|
||||
|
||||
This walkthrough adds a real bundled `ftp` plugin so users can:
|
||||
|
||||
- run `search-file -plugin ftp ...`
|
||||
- browse remote folders as result tables
|
||||
- select file rows to `download-file`
|
||||
- pipe selected file rows into `add-file`
|
||||
- upload local files with `add-file -plugin ftp`
|
||||
|
||||
The implementation lives in [plugins/ftp/__init__.py](plugins/ftp/__init__.py).
|
||||
|
||||
## What The Plugin Does
|
||||
|
||||
The FTP plugin demonstrates the main provider hooks that matter for a storage-style integration:
|
||||
|
||||
- `config_schema()` exposes host, credentials, base path, TLS, and search depth.
|
||||
- `extract_query_arguments()` supports inline query fields like `path:` and `depth:`.
|
||||
- `search()` walks an FTP directory tree and returns `SearchResult` rows.
|
||||
- `selector()` turns folder rows into a follow-up table when the user runs `@N`.
|
||||
- `download()` and `download_url()` fetch FTP files into `download-file` output paths.
|
||||
- `resolve_pipe_result_download()` lets `@N | add-file -store ...` materialize a remote FTP file first.
|
||||
- `upload()` lets `add-file -plugin ftp -path ...` push a local file to the configured FTP server.
|
||||
|
||||
## Example Config
|
||||
|
||||
Add an FTP provider block to your config:
|
||||
|
||||
```toml
|
||||
[provider.ftp]
|
||||
host = "ftp.example.com"
|
||||
port = 21
|
||||
username = "demo"
|
||||
password = "secret"
|
||||
base_path = "/incoming"
|
||||
tls = false
|
||||
passive = true
|
||||
timeout = 20
|
||||
search_depth = 1
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `host` is the only required field for the plugin to validate.
|
||||
- `username` defaults to `anonymous` and `password` defaults to `anonymous@`.
|
||||
- `base_path` is both the default search root and the upload target directory.
|
||||
- `search_depth` controls how many folder levels `search-file -plugin ftp` scans by default.
|
||||
|
||||
## Search Flow
|
||||
|
||||
Basic listing from the configured base path:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "*"
|
||||
```
|
||||
|
||||
Search by filename fragment:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "invoice"
|
||||
```
|
||||
|
||||
Search a different subtree and recurse deeper:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "path:/pub depth:2 invoice"
|
||||
```
|
||||
|
||||
Filter to folders only:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "path:/pub type:folder *"
|
||||
```
|
||||
|
||||
The plugin returns rows with explicit columns for name, type, directory, size, and modification time.
|
||||
|
||||
## Selection Flow
|
||||
|
||||
Folder rows are navigation rows. If the selected row is a directory, plain `@N` opens a new FTP table for that directory:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "*"
|
||||
@2
|
||||
```
|
||||
|
||||
File rows carry an explicit row action:
|
||||
|
||||
```powershell
|
||||
download-file -plugin ftp -url ftp://ftp.example.com/incoming/report.pdf
|
||||
```
|
||||
|
||||
That means plain `@N` on a file row downloads it immediately:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "report"
|
||||
@1
|
||||
```
|
||||
|
||||
## Download And Add-File Flow
|
||||
|
||||
If you want the downloaded file in a specific local directory:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "report"
|
||||
@1 | download-file -path C:\Downloads
|
||||
```
|
||||
|
||||
If you want to ingest the selected FTP file into a configured store backend:
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "report"
|
||||
@1 | add-file -store tutorial
|
||||
```
|
||||
|
||||
Why this works:
|
||||
|
||||
- the file row advertises a `download-file` row action
|
||||
- the pipeline auto-inserts that download before `add-file`
|
||||
- the FTP plugin also implements `resolve_pipe_result_download()` so provider-owned FTP rows can be materialized for ingestion
|
||||
|
||||
## Upload Flow
|
||||
|
||||
Uploading uses the same provider name, but through `add-file -plugin ftp`:
|
||||
|
||||
```powershell
|
||||
add-file -plugin ftp -path C:\Media\report.pdf
|
||||
```
|
||||
|
||||
That sends the file to the configured FTP `base_path` and returns the FTP URL as the uploaded result.
|
||||
|
||||
## Why The Row Metadata Matters
|
||||
|
||||
The critical part of this plugin is the file-row metadata:
|
||||
|
||||
- file rows emit `_selection_args` as `['-url', '<ftp-url>']`
|
||||
- file rows emit `_selection_action` as `['download-file', '-plugin', 'ftp', '-url', '<ftp-url>']`
|
||||
- folder rows do not emit a download action, so `selector()` can own drill-in behavior instead
|
||||
|
||||
That split is what keeps these two user experiences compatible:
|
||||
|
||||
- `@N` on a folder opens a new table
|
||||
- `@N` on a file downloads the file
|
||||
- `@N | add-file -store ...` first downloads, then ingests
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The plugin prefers `MLSD` for directory listings and falls back to `NLST` plus directory probes when the server does not support machine-readable listings.
|
||||
|
||||
The code is intentionally small and uses only Python stdlib pieces:
|
||||
|
||||
- `ftplib` for FTP and FTPS
|
||||
- `fnmatch` for wildcard-style search tokens
|
||||
- `tempfile` for `add-file` handoff downloads
|
||||
|
||||
## Recommended Commands To Demo The Walkthrough
|
||||
|
||||
```powershell
|
||||
search-file -plugin ftp "*"
|
||||
search-file -plugin ftp "path:/incoming depth:2 *.pdf"
|
||||
@1
|
||||
@1 | download-file -path C:\Downloads
|
||||
@1 | add-file -store tutorial
|
||||
add-file -plugin ftp -path C:\Media\report.pdf
|
||||
```
|
||||
@@ -0,0 +1,136 @@
|
||||
# SCP Plugin Walkthrough
|
||||
|
||||
This walkthrough adds a bundled `scp` plugin backed by existing SSH libraries:
|
||||
|
||||
- `paramiko` for SSH and SFTP directory listing
|
||||
- `scp` for file transfers
|
||||
|
||||
The implementation lives in [plugins/scp/__init__.py](plugins/scp/__init__.py).
|
||||
|
||||
## What The Plugin Does
|
||||
|
||||
The SCP plugin mirrors the FTP walkthrough, but on top of SSH:
|
||||
|
||||
- `search-file -plugin scp ...` lists remote files and folders over SFTP.
|
||||
- plain `@N` on a folder drills into that directory.
|
||||
- plain `@N` on a file runs `download-file -plugin scp -url ...`.
|
||||
- `@N | add-file -store ...` downloads first, then ingests the local temp file.
|
||||
- `add-file -plugin scp -path ...` uploads a local file to the configured remote path.
|
||||
|
||||
## Example Config
|
||||
|
||||
```toml
|
||||
[provider.scp]
|
||||
host = "ssh.example.com"
|
||||
port = 22
|
||||
username = "deploy"
|
||||
password = "secret"
|
||||
key_path = "C:/Users/Admin/.ssh/id_ed25519"
|
||||
base_path = "/srv/files"
|
||||
timeout = 20
|
||||
search_depth = 1
|
||||
allow_agent = true
|
||||
look_for_keys = true
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `host` and `username` are required for the plugin to validate.
|
||||
- You can use password auth, key auth, or both.
|
||||
- `base_path` is both the default search root and the default upload directory.
|
||||
|
||||
## Search Flow
|
||||
|
||||
List the configured base path:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "*"
|
||||
```
|
||||
|
||||
Search by filename:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "invoice"
|
||||
```
|
||||
|
||||
Search another subtree with deeper recursion:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "path:/srv/files/releases depth:2 *.zip"
|
||||
```
|
||||
|
||||
Show only folders:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "path:/srv/files type:folder *"
|
||||
```
|
||||
|
||||
## Selection Flow
|
||||
|
||||
Folder rows are navigation rows:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "*"
|
||||
@2
|
||||
```
|
||||
|
||||
File rows carry an explicit row action, so terminal selection downloads directly:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "report"
|
||||
@1
|
||||
```
|
||||
|
||||
That expands to the equivalent of:
|
||||
|
||||
```powershell
|
||||
download-file -plugin scp -url scp://ssh.example.com/srv/files/report.pdf
|
||||
```
|
||||
|
||||
## Download And Add-File Flow
|
||||
|
||||
Download into a local folder:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "report"
|
||||
@1 | download-file -path C:\Downloads
|
||||
```
|
||||
|
||||
Ingest a selected remote file into a configured store backend:
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "report"
|
||||
@1 | add-file -store tutorial
|
||||
```
|
||||
|
||||
Why this works:
|
||||
|
||||
- file rows advertise `_selection_action` for `download-file`
|
||||
- `add-file` selection replay inserts that provider download stage before ingest
|
||||
- the plugin also implements `resolve_pipe_result_download()` for provider-owned SCP rows
|
||||
|
||||
## Upload Flow
|
||||
|
||||
Upload a local file to the configured remote `base_path`:
|
||||
|
||||
```powershell
|
||||
add-file -plugin scp -path C:\Media\report.pdf
|
||||
```
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
The plugin uses SFTP for directory listing because SCP itself is a transfer protocol, not a browse/search protocol. That split keeps the provider simple:
|
||||
|
||||
- browse and metadata via Paramiko SFTP
|
||||
- file transfer via the `scp` package
|
||||
|
||||
## Recommended Demo Commands
|
||||
|
||||
```powershell
|
||||
search-file -plugin scp "*"
|
||||
search-file -plugin scp "path:/srv/files depth:2 *.zip"
|
||||
@1
|
||||
@1 | download-file -path C:\Downloads
|
||||
@1 | add-file -store tutorial
|
||||
add-file -plugin scp -path C:\Media\report.pdf
|
||||
```
|
||||
Reference in New Issue
Block a user