Files

11 KiB
Raw Permalink Blame History

Trekker — v0 Implementation Plan (revised)

Single canonical source of truth for Trekker product/architecture decisions. This revision supersedes the original v0 plan; the source tree in this repository is the implementation of it. Revision notes are at the bottom.

Goal: Build a vehicle-aware overland trip intelligence system that ingests GPX files (manual + folder watch), trail guides (web scrape + PDF with LLM extraction), weather data (Open-Meteo), and offline map areas, then generates structured overland assets and exports enriched GPX/KML files for manual import into onX and Gaia GPS.

Architecture: Dual-host FastAPI backend on home server (always-on, Tailnet) + Neutron truck server (offline-first). GPX ingestion → canonical TrekkerRoute model. Pluggable asset generators. Offline map packs provide local basemap/terrain tiles for field use. Export layer → enriched GPX/KML for onX/Gaia.

Tech Stack: FastAPI, gpxpy, openai Python SDK, pypdf, python-watchdog, httpx → Open-Meteo, MapLibre GL JS + raster basemap, AWS Terrain Tiles (Terrarium) for elevation, SQLite (JSON blobs), local offline tile packs, Docker Compose.


Scoped Items (locked with Peyo)

# Item Decision Notes
1 GPX input A + B: manual upload via web UI + folder watch via python-watchdog. gpxpy → TrekkerRoute. Handles Gaia and onX metadata.
2 Map / elevation MapLibre GL JS for rendering (no token). AWS Terrain Tiles (Terrarium encoding) for elevation. Powers UI display AND segment grade computation. (Revised — see note R1.)
2b Offline map areas Save map areas for offline use: viewport/rectangle, zoom range → local tile pack served by Trekker. Per-pack tile cap; tile source configurable. (See note R2.)
3a Trail guides — web httpx + BeautifulSoup scrape → LLM extraction. difficulty, obstacles, water crossings, gate codes, vehicle reqs, closures.
3b Trail guides — PDF pypdf → OpenAI-compatible LLM extraction. Output requires manual review before saving.
3c LLM interface OpenAI-compatible API; configurable base_url, model, api_key in trekker.yml. Supports local Ollama, Ollama Cloud, Gemini, Groq, OpenAI.
4 Weather Open-Meteo — free, no API key, global. 7/10-day, four granularity tiers, manual refresh.
5 Cell coverage Not MVP — deferred. —
6 Victron / battery Not MVP — deferred. —

Implementation status

All phases below are built in this repo at MVP depth and covered by the test suite (tests/, 31 tests).

Phase 1: GPX Ingestion Engine ✅

  • app/models/ — TrekkerRoute (TrkPt/Wpt/Segment/RouteStats), TrailGuide, Trip, Asset, MapPack. Pydantic v2.
  • app/ingest/gpx_parser.py — gpxpy parse, creator-first source detection (gaia / onx / manual), <rte> fallback for planned routes.
  • app/ingest/segmenter.py — waypoint-driven segmentation (named waypoints snapped to track within 200 m define boundaries; no waypoints → one segment). Haversine distance, gain/loss, avg/max grade (10 m smoothing window against GPS jitter), duration.
  • app/ingest/watchdog_daemon.py — dual watch folders; inotify observer for the internal path, PollingObserver for the remote mount (SMB/Nextcloud mounts don't deliver inotify events reliably). Size-stable check before ingesting.
  • app/storage/ — SQLite, JSON-blob tables (routes, trail_guides, assets, weather_cache, offline_map_packs), WAL mode, FK link route → guide.

Phase 2: Web UI Shell & Map ✅

  • The map is the app: full-screen MapLibre GL JS; collapsible left sidebar (routes, layers, offline packs); slide-out route panel with summary, segments, guide link, trail card, export, delete.
  • Grade-colored segment toggle (green→yellow→red by max grade) and weather overlay toggle (blue gradient by precip probability) — map layers, not sidebar charts.
  • app/services/elevation.py — Terrarium decode ele = (R*256 + G + B/256) − 32768, disk+memory tile cache; fills missing ele on upload so grade math works for onX exports that omit elevation.
  • app/services/offline_maps.py + app/routes/offline_maps.py — estimate → create → background download → ready; tiles under data/offline_maps/{pack}/{layer}/{z}/{x}/{y}.png; served at /api/offline-maps/tiles/... which the UI switches to when offline.

Phase 3: Trail Guide Module ✅

  • Draft-first flow: POST /api/trails/scrape and /api/trails/upload-pdf return drafts; nothing is stored until the user reviews/edits and POST /api/trails confirms (manual-review gate per 3c).
  • app/services/llm.py — OpenAI-compatible adapter, JSON-mode with fallback, tolerant of code-fenced output from small local models. When no LLM is configured, drafts carry raw extracted text for manual entry.
  • Max 1 guide per route (route.trail_guide_id), user-selectable link in the panel, generators degrade gracefully without one.

Phase 4: Weather Integration ✅

  • app/services/weather.py — Open-Meteo, 7-day default / 10-day toggle, per-segment midpoints (deduplicated on a 0.05° grid so short routes make one API call), aggregated to four tiers: hourly / 6 h / 12 h / daily.
  • Manual refresh only: POST /api/routes/{id}/refresh-weather; cached in SQLite; compact forecast badge in the panel corner; wind advisory (≥40 km/h) and lightning risk (WMO codes 95/96/99) flags.

Phase 5: Asset Generator Pipeline ✅

  • app/generators/base.py — BaseGenerator ABC with requires[] checks and user-facing errors (HTTP 409 when prerequisites are missing).
  • trail_card (route + optional guide + optional weather → per-segment JSON card rendered in the panel), elevation_map (grade colors + summit/low callouts), weather_overlay (per-segment precip/wind/lightning).
  • Trigger is a user click; output stored as JSON in the assets table.

Phase 6: Export & Download ✅

  • app/export/gpx_writer.py — GPX 1.1 with trekker: extensions: per-point grade_pct (+ surface_type when known), per-track segment metadata and guide enrichment (difficulty, water crossing depth). Waypoints preserved — round-trips through the parser losslessly (tested).
  • app/export/kml_writer.py — KML for onX; enrichment in the description.
  • GET /api/routes/{id}/export?format=gpx|kml → file download.

Phase 7: Dual-Host Sync & Deployment ✅

  • Single Docker image; docker/docker-compose.neutron.yml override adds the Tailscale sidecar (TAILSCALE_AUTH_KEY).
  • Manual sync: UI button → POST /api/sync pulls the peer's bundle, merges, pushes local bundle back. Conflict policy: newest updated_at wins, per record (see note R3).
  • UI offline state: navigator.onLine → offline badge, sync hidden, weather refresh disabled, basemap source switched to local pack tiles.

Phase 8: July 4 Trip — verification gate (next)

Deploy home + Neutron instances, run the full workflow with live Windrock routes before the June 27 departure: import GPX → map → link guide → trail card → weather → export → sync → offline mode in the driveway.


Verification Strategy

Phase What How Status
1 GPX parsing accuracy Unit tests with Gaia/onX/minimal fixtures: source, segments, stats ✅ automated
2 Map render + offline packs Visual check + pack download, then disconnect and verify local tiles manual (Phase 8)
3 LLM extraction Compare output vs labeled guides, >80% field accuracy manual, provider-dependent
4 Weather aggregation Known-answer tier tests with synthetic Open-Meteo payloads ✅ automated
5 Generator output Known-answer tests incl. graceful degradation and 409s ✅ automated
6 GPX round-trip Export → re-import → segments/waypoints/elevations identical ✅ automated
7 Sync Two-store merge tests: newest-wins, idempotent re-merge ✅ automated

Run: pip install -r requirements.txt pytest && python -m pytest tests/


File Map (as built)

app/
├── main.py                 # FastAPI entry, SPA serving, watchdog lifespan
├── config.py               # env + trekker.yml settings
├── deps.py                 # shared store/service singletons
├── models/                 # route, trail_guide, trip, assets, map_pack
├── ingest/                 # gpx_parser, segmenter, watchdog_daemon
├── services/               # elevation, offline_maps, weather, llm, scraper
├── generators/             # base, trail_card, elevation_map, weather_overlay
├── export/                 # gpx_writer, kml_writer
├── storage/                # database (schema), route_store (all CRUD)
├── sync/sync_daemon.py     # bundle build/merge + peer sync
└── routes/                 # upload, trails, weather, export, offline_maps, sync
ui/                          # index.html, app.js, styles.css (MapLibre)
tests/                       # 31 tests + GPX fixtures
docker-compose.yml, docker/docker-compose.neutron.yml, Dockerfile
.env.example, trekker.yml.example

Revision notes (changes from the original v0 plan)

  • R1 — Mapbox replaced with MapLibre + open data. The original plan's own Task 2.5 gate ("confirm the tile provider permits offline storage") fails for Mapbox: its ToS prohibits caching tiles outside the Mapbox SDK offline mechanism, and Terrain-RGB tiles count against a paid token. MapLibre GL JS is API-compatible for our usage and tokenless; AWS Terrain Tiles (Terrarium encoding, AWS Open Data) explicitly permit bulk download and use the near-identical decode formula. Net effect: no token to manage, offline packs are licence-clean for terrain.
  • R2 — Offline basemap tiles are capped and source-configurable. OSM's tile usage policy forbids bulk scraping. Packs default to the configured basemap with a hard per-pack tile cap (tiles.max_pack_tiles, default 4000) and a polite download delay; for big areas point tiles.base_url at a self-hosted or bulk-friendly tile source. Terrain layer has no such problem (R1).
  • R3 — Sync conflict policy simplified. "Home wins on structured data, notes merge bidirectionally" was underspecified (no notes entity exists in v0). MVP: per-record newest-updated_at wins, both directions, idempotent. Field-level merge can come with the trip-notes feature.
  • R4 — shapely/geopandas dropped. Haversine + index math covers all v0 geometry; this keeps the Docker image slim (no GEOS/GDAL native deps).
  • R5 — Stale deployment notes removed. The previous plan referenced a Gitea branch and a live Tailnet deployment that don't correspond to this repository; this repo is now the canonical source.
  • R6 — Draft-then-confirm guide flow made explicit in the API. Scrape/PDF endpoints never write to the DB; only the reviewed POST /api/trails does, which is what "manual review before saving" (3b) actually requires.
  • R7 — Firecrawl MCP dropped from MVP scrape path. Plain httpx + bs4 is dependency-free and works offline-ish (LAN sources); Firecrawl can be added later behind the same scraper interface.