link preview v1

This commit is contained in:
Joren de Graaf
2026-10-21 15:25:33 +02:00
parent 4948d7645e
commit 9a5210a48f
30 changed files with 1173 additions and 4 deletions
+43
View File
@@ -0,0 +1,43 @@
# Knowledge Library
Knowledge Library is a separate Django app under `/knowledge/`, with its own models, routes and menu entry. It does not create or modify Project Manager cards. See [Pocket source review and schema rationale](research/knowledge-library-pocket.md).
## Capture
Run migrations, sign in, and open **Knowledge Library → Capture tokens**. The token is displayed only when created and can be revoked there. An Apple Shortcut can send a JSON POST to `/api/bookmarks/` with `Authorization: Bearer <token>` and `Content-Type: application/json`:
```json
{"url":"https://example.org/article","title":"Optional title","saved_reason":"Why this matters"}
```
The endpoint returns `201` for a new item or `200` with `duplicate: true` for an existing URL. URLs are normalized conservatively: scheme/host case and fragment are ignored, while path and query are preserved. Capture is owner-scoped and limited to 30 posts per token per minute. Token values are stored only as SHA-256 hashes. Browser forms use Django session authentication and CSRF; bearer tokens are accepted only at the capture endpoint. A saved link remains available when metadata has not been fetched.
New items show `pending` until `.venv/bin/python manage.py fetch_knowledge_metadata --limit 20` runs. A one-minute user-systemd timer is supplied in `deploy/systemd/common-ground-knowledge-metadata.{service,timer}`; link and enable it after the application and migration are deployed. The timer processes at most five links per run. The page polls pending cards for two minutes and replaces them when the worker completes. On this host, link both files from `deploy/systemd/` into `~/.config/systemd/user/`, run `systemctl --user daemon-reload`, then `systemctl --user enable --now common-ground-knowledge-metadata.timer` after migrations. Use `flatpak-spawn --host` before `systemctl` when running inside the Codex Flatpak shell. It checks every DNS answer and redirect target, pins the outbound connection to a validated public IP, restricts ports, limits redirects, total time and response bytes, and extracts Open Graph metadata first, then Twitter Card metadata, then HTML title/description and favicon. It stores no page HTML. YouTube URLs use the fixed YouTube oEmbed JSON endpoint; returned embed HTML is ignored. Failed fetches show `error` and use exponential retry, capped at five attempts. No arbitrary URL is fetched during the save request. Fetched HTML is never rendered or stored as article content.
## MCP discovery
Authorize the MCP client with `knowledge:read` to expose `find_relevant_saved_interests(context)`. It returns at most three links belonging to the OAuth user. The current matching is intentionally simple and deterministic; it searches saved titles, descriptions, reasons and tags, and can return no matches. It does not inspect conversations on its own. Clients must choose to call it, so automatic invocation cannot be guaranteed. Existing `projects:read` scope does not grant library access.
## ntfy prototype
TasksNest's `planner/services/notifications.py` at commit `743785c2e0e41514c8dd18c035b7ea21716d87de` uses a UTF-8 HTTP POST with a `Title` header; this app follows that transport convention. Configure the server environment, outside Git:
```sh
KNOWLEDGE_NTFY_ENABLED=1
KNOWLEDGE_NTFY_URL=https://your-ntfy-host/private-topic
KNOWLEDGE_NTFY_USER_ID=<owner database ID>
KNOWLEDGE_NTFY_MODE=forgotten
KNOWLEDGE_NTFY_WEEKLY_LIMIT=2
KNOWLEDGE_NTFY_QUIET_START=22
KNOWLEDGE_NTFY_QUIET_END=8
```
The feature is off by default. Run `.venv/bin/python manage.py rediscover_knowledge` from a scheduled service (for example, once daily); no timer is installed by the app. Modes are `forgotten`, `random`, and `timely` (recent saved item, not a claim that its price or availability changed). Items must be at least two days old, are cooled down for 14 days after delivery, and the weekly limit defaults to two. Quiet hours use Django's configured local timezone. The notification includes the original source URL and saved reason. Delivery events are recorded after success; failed sends do not consume a slot. Snooze and dismissal controls are future work.
## Current limits
This checkout uses SQLite. Search is scoped by owner but does not yet use PostgreSQL full-text search or semantic matching. The browser page supports save, search, edit of user title/description, preview/plain/embed modes, manual refresh and archive. Publisher metadata remains separate from user edits, so refresh preserves overrides. YouTube playback is opt-in and uses only a constructed `youtube-nocookie.com` URL after a click; no provider HTML or iframe loads automatically. Preview images load directly from their publisher with no referrer; broken images fall back to text. Editing tags, reasons, and snooze needs a later UI pass. No article body is stored. No project conversion happens automatically.
## Preview implementation choices
The current project uses SQLite, so preview metadata is cached on the owner-scoped `SavedItem` row rather than in PostgreSQL. A later PostgreSQL deployment can use the same migration fields and add a full-text index. `micawber` offers general oEmbed provider discovery, Beautiful Soup offers broader HTML parsing, and `httpx` offers a higher-level HTTP client. This feature needs only one allowlisted oEmbed provider and a narrow set of head tags. The standard-library parser and DNS-pinned connection avoid extra dependencies and make SSRF checks explicit. [Open Graph image alt and site name](https://ogp.me/) and the [YouTube oEmbed endpoint](https://oembed.org/providers/youtube) informed field mapping; [YouTube Help](https://support.google.com/youtube/answer/171780) documents the privacy-enhanced embed domain.
+22
View File
@@ -0,0 +1,22 @@
# Knowledge library: Pocket source review and design
Reviewed 2026-10-20 before implementing the library data layer. Source inspected at [Pocket/pocket-monorepo](https://github.com/Pocket/pocket-monorepo/tree/5e5268fd92e997bb3350eaeafb54285238178a8e), commit `5e5268fd92e997bb3350eaeafb54285238178a8e`.
## Verified upstream patterns
- [`servers/list-api/schema.graphql`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/list-api/schema.graphql) defines `SavedItem` as a user's saved URL with title, tags, archive/favorite state, timestamps and a linked underlying item. That separation is useful: the saved reason and state belong to the user, while fetched page metadata is fallible.
- [`servers/list-api/src/dataService/savedItemsService.ts`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/list-api/src/dataService/savedItemsService.ts) scopes saved-item queries by `user_id` and looks up `given_url` separately from the resolved item. [`servers/list-api/src/models/SavedItem.ts`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/list-api/src/models/SavedItem.ts) handles archive, tags and soft deletion. The published monorepo does not provide a complete migration history for the legacy list database, so its exact indexes and constraints cannot be inferred from the API alone.
- [`servers/list-api/src/externalCaller/parserCaller.ts`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/list-api/src/externalCaller/parserCaller.ts) calls a separate parser. [`servers/parser-graphql-wrapper/src/databases/pocketMetadataStore.ts`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/parser-graphql-wrapper/src/databases/pocketMetadataStore.ts) illustrates metadata storage. The many service and cache layers are excessive for this personal Django app.
- [`lambdas/user-list-search-corpus-parser-hydration`](https://github.com/Pocket/pocket-monorepo/tree/5e5268fd92e997bb3350eaeafb54285238178a8e/lambdas/user-list-search-corpus-parser-hydration) shows separate search hydration. Local title, URL, description, notes and tag search should come first.
## License and reuse boundary
The monorepo root [`LICENSE`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/LICENSE) is Apache-2.0, but [`servers/client-api/LICENSE`](https://github.com/Pocket/pocket-monorepo/blob/5e5268fd92e997bb3350eaeafb54285238178a8e/servers/client-api/LICENSE) is an MIT notice credited to Apollo Graph, Inc. Dependencies retain their own licenses. Apache-2.0 copying would require preserving license and applicable notices, marking changes, and observing its patent terms. Third-party article text and images are not granted by a source-code license. Pocket names and marks are not licensed as product branding. No upstream code or assets are copied here; the implementation uses independently written Django models and services based on general patterns. The [Pocket iOS repository](https://github.com/Pocket/pocket-ios) contains a Share extension, but that native code is unnecessary for an Apple Shortcut sending a POST.
## Proposed local schema and pipeline
Use a separate `knowledge` app. `SavedItem` has owner, original URL, conservative URL key, optional canonical URL, user title, extracted title/description/thumbnail, notes, tags, saved and deferral reasons, status, read date, metadata state, and exposure dates. Enforce `(owner, url_key)` uniqueness and index `(owner, status, created_at)` and `(owner, last_surfaced_at)`. `CaptureToken` stores only a hash, owner, scope, creation/revocation dates. `RediscoveryEvent` records attempted/delivered resurfacing and feedback. Do not model project cards as saved items. Optional project association can come later through an explicit link.
Save must commit without network access. A separate bounded worker may fetch metadata after validating each DNS resolution and redirect target against private, loopback, link-local and reserved addresses; cap redirects, time, response bytes and content type. Never render fetched HTML. Keep metadata pending until the separately invoked bounded worker runs. A failed fetch becomes an explicit error state. No arbitrary URL is fetched in the capture request. The current checkout uses SQLite, so local search uses indexed owner filtering plus simple text matching; PostgreSQL `SearchVector` and a GIN index are a deployment upgrade, not an assumed current capability. Do not add pgvector until relevance tests show a need.
MCP contextual retrieval uses only the authenticated owner's records, a relevance threshold, a maximum of three results, and a recent-exposure cooldown. A tool description can invite proactive calls, but the MCP server cannot compel ChatGPT to invoke it. ntfy scheduling must be opt-in with quiet hours, a low weekly budget, delivery history and duplicate suppression. The TasksNest ntfy implementation is outside this checkout and must be inspected before reusing its exact transport conventions.