198 lines
11 KiB
Markdown
198 lines
11 KiB
Markdown
# Trekker — v0 Implementation Plan (revised)
|
||
|
||
> Single canonical source of truth for Trekker product/architecture decisions.
|
||
> This revision supersedes the original v0 plan; the source tree in this
|
||
> repository is the implementation of it. Revision notes are at the bottom.
|
||
|
||
**Goal:** Build a vehicle-aware overland trip intelligence system that ingests
|
||
GPX files (manual + folder watch), trail guides (web scrape + PDF with LLM
|
||
extraction), weather data (Open-Meteo), and offline map areas, then generates
|
||
structured overland assets and exports enriched GPX/KML files for manual
|
||
import into onX and Gaia GPS.
|
||
|
||
**Architecture:** Dual-host FastAPI backend on home server (always-on,
|
||
Tailnet) + Neutron truck server (offline-first). GPX ingestion → canonical
|
||
`TrekkerRoute` model. Pluggable asset generators. Offline map packs provide
|
||
local basemap/terrain tiles for field use. Export layer → enriched GPX/KML
|
||
for onX/Gaia.
|
||
|
||
**Tech Stack:** FastAPI, gpxpy, openai Python SDK, pypdf, python-watchdog,
|
||
httpx → Open-Meteo, **MapLibre GL JS** + raster basemap, **AWS Terrain Tiles
|
||
(Terrarium)** for elevation, SQLite (JSON blobs), local offline tile packs,
|
||
Docker Compose.
|
||
|
||
---
|
||
|
||
## Scoped Items (locked with Peyo)
|
||
|
||
| # | Item | Decision | Notes |
|
||
| :-- | :--- | :------- | :---- |
|
||
| 1 | GPX input | **A + B**: manual upload via web UI + folder watch via `python-watchdog`. `gpxpy` → `TrekkerRoute`. | Handles Gaia and onX metadata. |
|
||
| 2 | Map / elevation | **MapLibre GL JS** for rendering (no token). **AWS Terrain Tiles (Terrarium encoding)** for elevation. | Powers UI display AND segment grade computation. *(Revised — see note R1.)* |
|
||
| 2b | Offline map areas | Save map areas for offline use: viewport/rectangle, zoom range → local tile pack served by Trekker. | Per-pack tile cap; tile source configurable. *(See note R2.)* |
|
||
| 3a | Trail guides — web | `httpx` + BeautifulSoup scrape → LLM extraction. | difficulty, obstacles, water crossings, gate codes, vehicle reqs, closures. |
|
||
| 3b | Trail guides — PDF | `pypdf` → OpenAI-compatible LLM extraction. | Output requires manual review before saving. |
|
||
| 3c | LLM interface | OpenAI-compatible API; configurable `base_url`, `model`, `api_key` in `trekker.yml`. | Supports local Ollama, Ollama Cloud, Gemini, Groq, OpenAI. |
|
||
| 4 | Weather | Open-Meteo — free, no API key, global. | 7/10-day, four granularity tiers, manual refresh. |
|
||
| 5 | Cell coverage | **Not MVP** — deferred. | — |
|
||
| 6 | Victron / battery | **Not MVP** — deferred. | — |
|
||
|
||
---
|
||
|
||
## Implementation status
|
||
|
||
All phases below are **built** in this repo at MVP depth and covered by the
|
||
test suite (`tests/`, 31 tests).
|
||
|
||
### Phase 1: GPX Ingestion Engine ✅
|
||
- `app/models/` — `TrekkerRoute` (TrkPt/Wpt/Segment/RouteStats), `TrailGuide`,
|
||
`Trip`, `Asset`, `MapPack`. Pydantic v2.
|
||
- `app/ingest/gpx_parser.py` — gpxpy parse, creator-first source detection
|
||
(`gaia` / `onx` / `manual`), `<rte>` fallback for planned routes.
|
||
- `app/ingest/segmenter.py` — waypoint-driven segmentation (named waypoints
|
||
snapped to track within 200 m define boundaries; no waypoints → one
|
||
segment). Haversine distance, gain/loss, avg/max grade (10 m smoothing
|
||
window against GPS jitter), duration.
|
||
- `app/ingest/watchdog_daemon.py` — dual watch folders; inotify observer for
|
||
the internal path, **PollingObserver** for the remote mount (SMB/Nextcloud
|
||
mounts don't deliver inotify events reliably). Size-stable check before
|
||
ingesting.
|
||
- `app/storage/` — SQLite, JSON-blob tables (routes, trail_guides, assets,
|
||
weather_cache, offline_map_packs), WAL mode, FK link route → guide.
|
||
|
||
### Phase 2: Web UI Shell & Map ✅
|
||
- The map is the app: full-screen MapLibre GL JS; collapsible left sidebar
|
||
(routes, layers, offline packs); slide-out route panel with summary,
|
||
segments, guide link, trail card, export, delete.
|
||
- Grade-colored segment toggle (green→yellow→red by max grade) and weather
|
||
overlay toggle (blue gradient by precip probability) — map layers, not
|
||
sidebar charts.
|
||
- `app/services/elevation.py` — Terrarium decode
|
||
`ele = (R*256 + G + B/256) − 32768`, disk+memory tile cache; fills missing
|
||
`ele` on upload so grade math works for onX exports that omit elevation.
|
||
- `app/services/offline_maps.py` + `app/routes/offline_maps.py` — estimate →
|
||
create → background download → ready; tiles under
|
||
`data/offline_maps/{pack}/{layer}/{z}/{x}/{y}.png`; served at
|
||
`/api/offline-maps/tiles/...` which the UI switches to when offline.
|
||
|
||
### Phase 3: Trail Guide Module ✅
|
||
- Draft-first flow: `POST /api/trails/scrape` and `/api/trails/upload-pdf`
|
||
return **drafts**; nothing is stored until the user reviews/edits and
|
||
`POST /api/trails` confirms (manual-review gate per 3c).
|
||
- `app/services/llm.py` — OpenAI-compatible adapter, JSON-mode with fallback,
|
||
tolerant of code-fenced output from small local models. When no LLM is
|
||
configured, drafts carry raw extracted text for manual entry.
|
||
- Max 1 guide per route (`route.trail_guide_id`), user-selectable link in the
|
||
panel, generators degrade gracefully without one.
|
||
|
||
### Phase 4: Weather Integration ✅
|
||
- `app/services/weather.py` — Open-Meteo, 7-day default / 10-day toggle,
|
||
per-segment midpoints (deduplicated on a 0.05° grid so short routes make
|
||
one API call), aggregated to four tiers: hourly / 6 h / 12 h / daily.
|
||
- Manual refresh only: `POST /api/routes/{id}/refresh-weather`; cached in
|
||
SQLite; compact forecast badge in the panel corner; wind advisory
|
||
(≥40 km/h) and lightning risk (WMO codes 95/96/99) flags.
|
||
|
||
### Phase 5: Asset Generator Pipeline ✅
|
||
- `app/generators/base.py` — `BaseGenerator` ABC with `requires[]` checks and
|
||
user-facing errors (HTTP 409 when prerequisites are missing).
|
||
- `trail_card` (route + optional guide + optional weather → per-segment JSON
|
||
card rendered in the panel), `elevation_map` (grade colors + summit/low
|
||
callouts), `weather_overlay` (per-segment precip/wind/lightning).
|
||
- Trigger is a user click; output stored as JSON in the assets table.
|
||
|
||
### Phase 6: Export & Download ✅
|
||
- `app/export/gpx_writer.py` — GPX 1.1 with `trekker:` extensions: per-point
|
||
`grade_pct` (+ `surface_type` when known), per-track segment metadata and
|
||
guide enrichment (difficulty, water crossing depth). Waypoints preserved —
|
||
round-trips through the parser losslessly (tested).
|
||
- `app/export/kml_writer.py` — KML for onX; enrichment in the description.
|
||
- `GET /api/routes/{id}/export?format=gpx|kml` → file download.
|
||
|
||
### Phase 7: Dual-Host Sync & Deployment ✅
|
||
- Single Docker image; `docker/docker-compose.neutron.yml` override adds the
|
||
Tailscale sidecar (`TAILSCALE_AUTH_KEY`).
|
||
- Manual sync: UI button → `POST /api/sync` pulls the peer's bundle, merges,
|
||
pushes local bundle back. Conflict policy: **newest `updated_at` wins, per
|
||
record** (see note R3).
|
||
- UI offline state: `navigator.onLine` → offline badge, sync hidden, weather
|
||
refresh disabled, basemap source switched to local pack tiles.
|
||
|
||
### Phase 8: July 4 Trip — verification gate (next)
|
||
Deploy home + Neutron instances, run the full workflow with live Windrock
|
||
routes before the June 27 departure: import GPX → map → link guide →
|
||
trail card → weather → export → sync → offline mode in the driveway.
|
||
|
||
---
|
||
|
||
## Verification Strategy
|
||
|
||
| Phase | What | How | Status |
|
||
| :---: | :--- | :-- | :----- |
|
||
| 1 | GPX parsing accuracy | Unit tests with Gaia/onX/minimal fixtures: source, segments, stats | ✅ automated |
|
||
| 2 | Map render + offline packs | Visual check + pack download, then disconnect and verify local tiles | manual (Phase 8) |
|
||
| 3 | LLM extraction | Compare output vs labeled guides, >80% field accuracy | manual, provider-dependent |
|
||
| 4 | Weather aggregation | Known-answer tier tests with synthetic Open-Meteo payloads | ✅ automated |
|
||
| 5 | Generator output | Known-answer tests incl. graceful degradation and 409s | ✅ automated |
|
||
| 6 | GPX round-trip | Export → re-import → segments/waypoints/elevations identical | ✅ automated |
|
||
| 7 | Sync | Two-store merge tests: newest-wins, idempotent re-merge | ✅ automated |
|
||
|
||
Run: `pip install -r requirements.txt pytest && python -m pytest tests/`
|
||
|
||
---
|
||
|
||
## File Map (as built)
|
||
|
||
```
|
||
app/
|
||
├── main.py # FastAPI entry, SPA serving, watchdog lifespan
|
||
├── config.py # env + trekker.yml settings
|
||
├── deps.py # shared store/service singletons
|
||
├── models/ # route, trail_guide, trip, assets, map_pack
|
||
├── ingest/ # gpx_parser, segmenter, watchdog_daemon
|
||
├── services/ # elevation, offline_maps, weather, llm, scraper
|
||
├── generators/ # base, trail_card, elevation_map, weather_overlay
|
||
├── export/ # gpx_writer, kml_writer
|
||
├── storage/ # database (schema), route_store (all CRUD)
|
||
├── sync/sync_daemon.py # bundle build/merge + peer sync
|
||
└── routes/ # upload, trails, weather, export, offline_maps, sync
|
||
ui/ # index.html, app.js, styles.css (MapLibre)
|
||
tests/ # 31 tests + GPX fixtures
|
||
docker-compose.yml, docker/docker-compose.neutron.yml, Dockerfile
|
||
.env.example, trekker.yml.example
|
||
```
|
||
|
||
---
|
||
|
||
## Revision notes (changes from the original v0 plan)
|
||
|
||
- **R1 — Mapbox replaced with MapLibre + open data.** The original plan's own
|
||
Task 2.5 gate ("confirm the tile provider permits offline storage") fails
|
||
for Mapbox: its ToS prohibits caching tiles outside the Mapbox SDK offline
|
||
mechanism, and Terrain-RGB tiles count against a paid token. MapLibre GL JS
|
||
is API-compatible for our usage and tokenless; **AWS Terrain Tiles**
|
||
(Terrarium encoding, AWS Open Data) explicitly permit bulk download and use
|
||
the near-identical decode formula. Net effect: no token to manage, offline
|
||
packs are licence-clean for terrain.
|
||
- **R2 — Offline basemap tiles are capped and source-configurable.** OSM's
|
||
tile usage policy forbids bulk scraping. Packs default to the configured
|
||
basemap with a hard per-pack tile cap (`tiles.max_pack_tiles`, default
|
||
4000) and a polite download delay; for big areas point `tiles.base_url` at
|
||
a self-hosted or bulk-friendly tile source. Terrain layer has no such
|
||
problem (R1).
|
||
- **R3 — Sync conflict policy simplified.** "Home wins on structured data,
|
||
notes merge bidirectionally" was underspecified (no notes entity exists in
|
||
v0). MVP: per-record newest-`updated_at` wins, both directions, idempotent.
|
||
Field-level merge can come with the trip-notes feature.
|
||
- **R4 — shapely/geopandas dropped.** Haversine + index math covers all v0
|
||
geometry; this keeps the Docker image slim (no GEOS/GDAL native deps).
|
||
- **R5 — Stale deployment notes removed.** The previous plan referenced a
|
||
Gitea branch and a live Tailnet deployment that don't correspond to this
|
||
repository; this repo is now the canonical source.
|
||
- **R6 — Draft-then-confirm guide flow made explicit in the API.** Scrape/PDF
|
||
endpoints never write to the DB; only the reviewed `POST /api/trails` does,
|
||
which is what "manual review before saving" (3b) actually requires.
|
||
- **R7 — Firecrawl MCP dropped from MVP scrape path.** Plain httpx + bs4 is
|
||
dependency-free and works offline-ish (LAN sources); Firecrawl can be added
|
||
later behind the same scraper interface.
|