Files
trekkerv2/PLAN.md
T

198 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Trekker — v0 Implementation Plan (revised)
> Single canonical source of truth for Trekker product/architecture decisions.
> This revision supersedes the original v0 plan; the source tree in this
> repository is the implementation of it. Revision notes are at the bottom.
**Goal:** Build a vehicle-aware overland trip intelligence system that ingests
GPX files (manual + folder watch), trail guides (web scrape + PDF with LLM
extraction), weather data (Open-Meteo), and offline map areas, then generates
structured overland assets and exports enriched GPX/KML files for manual
import into onX and Gaia GPS.
**Architecture:** Dual-host FastAPI backend on home server (always-on,
Tailnet) + Neutron truck server (offline-first). GPX ingestion → canonical
`TrekkerRoute` model. Pluggable asset generators. Offline map packs provide
local basemap/terrain tiles for field use. Export layer → enriched GPX/KML
for onX/Gaia.
**Tech Stack:** FastAPI, gpxpy, openai Python SDK, pypdf, python-watchdog,
httpx → Open-Meteo, **MapLibre GL JS** + raster basemap, **AWS Terrain Tiles
(Terrarium)** for elevation, SQLite (JSON blobs), local offline tile packs,
Docker Compose.
---
## Scoped Items (locked with Peyo)
| # | Item | Decision | Notes |
| :-- | :--- | :------- | :---- |
| 1 | GPX input | **A + B**: manual upload via web UI + folder watch via `python-watchdog`. `gpxpy` → `TrekkerRoute`. | Handles Gaia and onX metadata. |
| 2 | Map / elevation | **MapLibre GL JS** for rendering (no token). **AWS Terrain Tiles (Terrarium encoding)** for elevation. | Powers UI display AND segment grade computation. *(Revised — see note R1.)* |
| 2b | Offline map areas | Save map areas for offline use: viewport/rectangle, zoom range → local tile pack served by Trekker. | Per-pack tile cap; tile source configurable. *(See note R2.)* |
| 3a | Trail guides — web | `httpx` + BeautifulSoup scrape → LLM extraction. | difficulty, obstacles, water crossings, gate codes, vehicle reqs, closures. |
| 3b | Trail guides — PDF | `pypdf` → OpenAI-compatible LLM extraction. | Output requires manual review before saving. |
| 3c | LLM interface | OpenAI-compatible API; configurable `base_url`, `model`, `api_key` in `trekker.yml`. | Supports local Ollama, Ollama Cloud, Gemini, Groq, OpenAI. |
| 4 | Weather | Open-Meteo — free, no API key, global. | 7/10-day, four granularity tiers, manual refresh. |
| 5 | Cell coverage | **Not MVP** — deferred. | — |
| 6 | Victron / battery | **Not MVP** — deferred. | — |
---
## Implementation status
All phases below are **built** in this repo at MVP depth and covered by the
test suite (`tests/`, 31 tests).
### Phase 1: GPX Ingestion Engine ✅
- `app/models/` — `TrekkerRoute` (TrkPt/Wpt/Segment/RouteStats), `TrailGuide`,
`Trip`, `Asset`, `MapPack`. Pydantic v2.
- `app/ingest/gpx_parser.py` — gpxpy parse, creator-first source detection
(`gaia` / `onx` / `manual`), `<rte>` fallback for planned routes.
- `app/ingest/segmenter.py` — waypoint-driven segmentation (named waypoints
snapped to track within 200 m define boundaries; no waypoints → one
segment). Haversine distance, gain/loss, avg/max grade (10 m smoothing
window against GPS jitter), duration.
- `app/ingest/watchdog_daemon.py` — dual watch folders; inotify observer for
the internal path, **PollingObserver** for the remote mount (SMB/Nextcloud
mounts don't deliver inotify events reliably). Size-stable check before
ingesting.
- `app/storage/` — SQLite, JSON-blob tables (routes, trail_guides, assets,
weather_cache, offline_map_packs), WAL mode, FK link route → guide.
### Phase 2: Web UI Shell & Map ✅
- The map is the app: full-screen MapLibre GL JS; collapsible left sidebar
(routes, layers, offline packs); slide-out route panel with summary,
segments, guide link, trail card, export, delete.
- Grade-colored segment toggle (green→yellow→red by max grade) and weather
overlay toggle (blue gradient by precip probability) — map layers, not
sidebar charts.
- `app/services/elevation.py` — Terrarium decode
`ele = (R*256 + G + B/256) − 32768`, disk+memory tile cache; fills missing
`ele` on upload so grade math works for onX exports that omit elevation.
- `app/services/offline_maps.py` + `app/routes/offline_maps.py` — estimate →
create → background download → ready; tiles under
`data/offline_maps/{pack}/{layer}/{z}/{x}/{y}.png`; served at
`/api/offline-maps/tiles/...` which the UI switches to when offline.
### Phase 3: Trail Guide Module ✅
- Draft-first flow: `POST /api/trails/scrape` and `/api/trails/upload-pdf`
return **drafts**; nothing is stored until the user reviews/edits and
`POST /api/trails` confirms (manual-review gate per 3c).
- `app/services/llm.py` — OpenAI-compatible adapter, JSON-mode with fallback,
tolerant of code-fenced output from small local models. When no LLM is
configured, drafts carry raw extracted text for manual entry.
- Max 1 guide per route (`route.trail_guide_id`), user-selectable link in the
panel, generators degrade gracefully without one.
### Phase 4: Weather Integration ✅
- `app/services/weather.py` — Open-Meteo, 7-day default / 10-day toggle,
per-segment midpoints (deduplicated on a 0.05° grid so short routes make
one API call), aggregated to four tiers: hourly / 6 h / 12 h / daily.
- Manual refresh only: `POST /api/routes/{id}/refresh-weather`; cached in
SQLite; compact forecast badge in the panel corner; wind advisory
(≥40 km/h) and lightning risk (WMO codes 95/96/99) flags.
### Phase 5: Asset Generator Pipeline ✅
- `app/generators/base.py` — `BaseGenerator` ABC with `requires[]` checks and
user-facing errors (HTTP 409 when prerequisites are missing).
- `trail_card` (route + optional guide + optional weather → per-segment JSON
card rendered in the panel), `elevation_map` (grade colors + summit/low
callouts), `weather_overlay` (per-segment precip/wind/lightning).
- Trigger is a user click; output stored as JSON in the assets table.
### Phase 6: Export & Download ✅
- `app/export/gpx_writer.py` — GPX 1.1 with `trekker:` extensions: per-point
`grade_pct` (+ `surface_type` when known), per-track segment metadata and
guide enrichment (difficulty, water crossing depth). Waypoints preserved —
round-trips through the parser losslessly (tested).
- `app/export/kml_writer.py` — KML for onX; enrichment in the description.
- `GET /api/routes/{id}/export?format=gpx|kml` → file download.
### Phase 7: Dual-Host Sync & Deployment ✅
- Single Docker image; `docker/docker-compose.neutron.yml` override adds the
Tailscale sidecar (`TAILSCALE_AUTH_KEY`).
- Manual sync: UI button → `POST /api/sync` pulls the peer's bundle, merges,
pushes local bundle back. Conflict policy: **newest `updated_at` wins, per
record** (see note R3).
- UI offline state: `navigator.onLine` → offline badge, sync hidden, weather
refresh disabled, basemap source switched to local pack tiles.
### Phase 8: July 4 Trip — verification gate (next)
Deploy home + Neutron instances, run the full workflow with live Windrock
routes before the June 27 departure: import GPX → map → link guide →
trail card → weather → export → sync → offline mode in the driveway.
---
## Verification Strategy
| Phase | What | How | Status |
| :---: | :--- | :-- | :----- |
| 1 | GPX parsing accuracy | Unit tests with Gaia/onX/minimal fixtures: source, segments, stats | ✅ automated |
| 2 | Map render + offline packs | Visual check + pack download, then disconnect and verify local tiles | manual (Phase 8) |
| 3 | LLM extraction | Compare output vs labeled guides, >80% field accuracy | manual, provider-dependent |
| 4 | Weather aggregation | Known-answer tier tests with synthetic Open-Meteo payloads | ✅ automated |
| 5 | Generator output | Known-answer tests incl. graceful degradation and 409s | ✅ automated |
| 6 | GPX round-trip | Export → re-import → segments/waypoints/elevations identical | ✅ automated |
| 7 | Sync | Two-store merge tests: newest-wins, idempotent re-merge | ✅ automated |
Run: `pip install -r requirements.txt pytest && python -m pytest tests/`
---
## File Map (as built)
```
app/
├── main.py # FastAPI entry, SPA serving, watchdog lifespan
├── config.py # env + trekker.yml settings
├── deps.py # shared store/service singletons
├── models/ # route, trail_guide, trip, assets, map_pack
├── ingest/ # gpx_parser, segmenter, watchdog_daemon
├── services/ # elevation, offline_maps, weather, llm, scraper
├── generators/ # base, trail_card, elevation_map, weather_overlay
├── export/ # gpx_writer, kml_writer
├── storage/ # database (schema), route_store (all CRUD)
├── sync/sync_daemon.py # bundle build/merge + peer sync
└── routes/ # upload, trails, weather, export, offline_maps, sync
ui/ # index.html, app.js, styles.css (MapLibre)
tests/ # 31 tests + GPX fixtures
docker-compose.yml, docker/docker-compose.neutron.yml, Dockerfile
.env.example, trekker.yml.example
```
---
## Revision notes (changes from the original v0 plan)
- **R1 — Mapbox replaced with MapLibre + open data.** The original plan's own
Task 2.5 gate ("confirm the tile provider permits offline storage") fails
for Mapbox: its ToS prohibits caching tiles outside the Mapbox SDK offline
mechanism, and Terrain-RGB tiles count against a paid token. MapLibre GL JS
is API-compatible for our usage and tokenless; **AWS Terrain Tiles**
(Terrarium encoding, AWS Open Data) explicitly permit bulk download and use
the near-identical decode formula. Net effect: no token to manage, offline
packs are licence-clean for terrain.
- **R2 — Offline basemap tiles are capped and source-configurable.** OSM's
tile usage policy forbids bulk scraping. Packs default to the configured
basemap with a hard per-pack tile cap (`tiles.max_pack_tiles`, default
4000) and a polite download delay; for big areas point `tiles.base_url` at
a self-hosted or bulk-friendly tile source. Terrain layer has no such
problem (R1).
- **R3 — Sync conflict policy simplified.** "Home wins on structured data,
notes merge bidirectionally" was underspecified (no notes entity exists in
v0). MVP: per-record newest-`updated_at` wins, both directions, idempotent.
Field-level merge can come with the trip-notes feature.
- **R4 — shapely/geopandas dropped.** Haversine + index math covers all v0
geometry; this keeps the Docker image slim (no GEOS/GDAL native deps).
- **R5 — Stale deployment notes removed.** The previous plan referenced a
Gitea branch and a live Tailnet deployment that don't correspond to this
repository; this repo is now the canonical source.
- **R6 — Draft-then-confirm guide flow made explicit in the API.** Scrape/PDF
endpoints never write to the DB; only the reviewed `POST /api/trails` does,
which is what "manual review before saving" (3b) actually requires.
- **R7 — Firecrawl MCP dropped from MVP scrape path.** Plain httpx + bs4 is
dependency-free and works offline-ish (LAN sources); Firecrawl can be added
later behind the same scraper interface.