11 KiB
Trekker — v0 Implementation Plan (revised)
Single canonical source of truth for Trekker product/architecture decisions. This revision supersedes the original v0 plan; the source tree in this repository is the implementation of it. Revision notes are at the bottom.
Goal: Build a vehicle-aware overland trip intelligence system that ingests GPX files (manual + folder watch), trail guides (web scrape + PDF with LLM extraction), weather data (Open-Meteo), and offline map areas, then generates structured overland assets and exports enriched GPX/KML files for manual import into onX and Gaia GPS.
Architecture: Dual-host FastAPI backend on home server (always-on,
Tailnet) + Neutron truck server (offline-first). GPX ingestion → canonical
TrekkerRoute model. Pluggable asset generators. Offline map packs provide
local basemap/terrain tiles for field use. Export layer → enriched GPX/KML
for onX/Gaia.
Tech Stack: FastAPI, gpxpy, openai Python SDK, pypdf, python-watchdog, httpx → Open-Meteo, MapLibre GL JS + raster basemap, AWS Terrain Tiles (Terrarium) for elevation, SQLite (JSON blobs), local offline tile packs, Docker Compose.
Scoped Items (locked with Peyo)
| # | Item | Decision | Notes |
|---|---|---|---|
| 1 | GPX input | A + B: manual upload via web UI + folder watch via python-watchdog. gpxpy → TrekkerRoute. |
Handles Gaia and onX metadata. |
| 2 | Map / elevation | MapLibre GL JS for rendering (no token). AWS Terrain Tiles (Terrarium encoding) for elevation. | Powers UI display AND segment grade computation. (Revised — see note R1.) |
| 2b | Offline map areas | Save map areas for offline use: viewport/rectangle, zoom range → local tile pack served by Trekker. | Per-pack tile cap; tile source configurable. (See note R2.) |
| 3a | Trail guides — web | httpx + BeautifulSoup scrape → LLM extraction. |
difficulty, obstacles, water crossings, gate codes, vehicle reqs, closures. |
| 3b | Trail guides — PDF | pypdf → OpenAI-compatible LLM extraction. |
Output requires manual review before saving. |
| 3c | LLM interface | OpenAI-compatible API; configurable base_url, model, api_key in trekker.yml. |
Supports local Ollama, Ollama Cloud, Gemini, Groq, OpenAI. |
| 4 | Weather | Open-Meteo — free, no API key, global. | 7/10-day, four granularity tiers, manual refresh. |
| 5 | Cell coverage | Not MVP — deferred. | — |
| 6 | Victron / battery | Not MVP — deferred. | — |
Implementation status
All phases below are built in this repo at MVP depth and covered by the
test suite (tests/, 31 tests).
Phase 1: GPX Ingestion Engine ✅
app/models/—TrekkerRoute(TrkPt/Wpt/Segment/RouteStats),TrailGuide,Trip,Asset,MapPack. Pydantic v2.app/ingest/gpx_parser.py— gpxpy parse, creator-first source detection (gaia/onx/manual),<rte>fallback for planned routes.app/ingest/segmenter.py— waypoint-driven segmentation (named waypoints snapped to track within 200 m define boundaries; no waypoints → one segment). Haversine distance, gain/loss, avg/max grade (10 m smoothing window against GPS jitter), duration.app/ingest/watchdog_daemon.py— dual watch folders; inotify observer for the internal path, PollingObserver for the remote mount (SMB/Nextcloud mounts don't deliver inotify events reliably). Size-stable check before ingesting.app/storage/— SQLite, JSON-blob tables (routes, trail_guides, assets, weather_cache, offline_map_packs), WAL mode, FK link route → guide.
Phase 2: Web UI Shell & Map ✅
- The map is the app: full-screen MapLibre GL JS; collapsible left sidebar (routes, layers, offline packs); slide-out route panel with summary, segments, guide link, trail card, export, delete.
- Grade-colored segment toggle (green→yellow→red by max grade) and weather overlay toggle (blue gradient by precip probability) — map layers, not sidebar charts.
app/services/elevation.py— Terrarium decodeele = (R*256 + G + B/256) − 32768, disk+memory tile cache; fills missingeleon upload so grade math works for onX exports that omit elevation.app/services/offline_maps.py+app/routes/offline_maps.py— estimate → create → background download → ready; tiles underdata/offline_maps/{pack}/{layer}/{z}/{x}/{y}.png; served at/api/offline-maps/tiles/...which the UI switches to when offline.
Phase 3: Trail Guide Module ✅
- Draft-first flow:
POST /api/trails/scrapeand/api/trails/upload-pdfreturn drafts; nothing is stored until the user reviews/edits andPOST /api/trailsconfirms (manual-review gate per 3c). app/services/llm.py— OpenAI-compatible adapter, JSON-mode with fallback, tolerant of code-fenced output from small local models. When no LLM is configured, drafts carry raw extracted text for manual entry.- Max 1 guide per route (
route.trail_guide_id), user-selectable link in the panel, generators degrade gracefully without one.
Phase 4: Weather Integration ✅
app/services/weather.py— Open-Meteo, 7-day default / 10-day toggle, per-segment midpoints (deduplicated on a 0.05° grid so short routes make one API call), aggregated to four tiers: hourly / 6 h / 12 h / daily.- Manual refresh only:
POST /api/routes/{id}/refresh-weather; cached in SQLite; compact forecast badge in the panel corner; wind advisory (≥40 km/h) and lightning risk (WMO codes 95/96/99) flags.
Phase 5: Asset Generator Pipeline ✅
app/generators/base.py—BaseGeneratorABC withrequires[]checks and user-facing errors (HTTP 409 when prerequisites are missing).trail_card(route + optional guide + optional weather → per-segment JSON card rendered in the panel),elevation_map(grade colors + summit/low callouts),weather_overlay(per-segment precip/wind/lightning).- Trigger is a user click; output stored as JSON in the assets table.
Phase 6: Export & Download ✅
app/export/gpx_writer.py— GPX 1.1 withtrekker:extensions: per-pointgrade_pct(+surface_typewhen known), per-track segment metadata and guide enrichment (difficulty, water crossing depth). Waypoints preserved — round-trips through the parser losslessly (tested).app/export/kml_writer.py— KML for onX; enrichment in the description.GET /api/routes/{id}/export?format=gpx|kml→ file download.
Phase 7: Dual-Host Sync & Deployment ✅
- Single Docker image;
docker/docker-compose.neutron.ymloverride adds the Tailscale sidecar (TAILSCALE_AUTH_KEY). - Manual sync: UI button →
POST /api/syncpulls the peer's bundle, merges, pushes local bundle back. Conflict policy: newestupdated_atwins, per record (see note R3). - UI offline state:
navigator.onLine→ offline badge, sync hidden, weather refresh disabled, basemap source switched to local pack tiles.
Phase 8: July 4 Trip — verification gate (next)
Deploy home + Neutron instances, run the full workflow with live Windrock routes before the June 27 departure: import GPX → map → link guide → trail card → weather → export → sync → offline mode in the driveway.
Verification Strategy
| Phase | What | How | Status |
|---|---|---|---|
| 1 | GPX parsing accuracy | Unit tests with Gaia/onX/minimal fixtures: source, segments, stats | ✅ automated |
| 2 | Map render + offline packs | Visual check + pack download, then disconnect and verify local tiles | manual (Phase 8) |
| 3 | LLM extraction | Compare output vs labeled guides, >80% field accuracy | manual, provider-dependent |
| 4 | Weather aggregation | Known-answer tier tests with synthetic Open-Meteo payloads | ✅ automated |
| 5 | Generator output | Known-answer tests incl. graceful degradation and 409s | ✅ automated |
| 6 | GPX round-trip | Export → re-import → segments/waypoints/elevations identical | ✅ automated |
| 7 | Sync | Two-store merge tests: newest-wins, idempotent re-merge | ✅ automated |
Run: pip install -r requirements.txt pytest && python -m pytest tests/
File Map (as built)
app/
├── main.py # FastAPI entry, SPA serving, watchdog lifespan
├── config.py # env + trekker.yml settings
├── deps.py # shared store/service singletons
├── models/ # route, trail_guide, trip, assets, map_pack
├── ingest/ # gpx_parser, segmenter, watchdog_daemon
├── services/ # elevation, offline_maps, weather, llm, scraper
├── generators/ # base, trail_card, elevation_map, weather_overlay
├── export/ # gpx_writer, kml_writer
├── storage/ # database (schema), route_store (all CRUD)
├── sync/sync_daemon.py # bundle build/merge + peer sync
└── routes/ # upload, trails, weather, export, offline_maps, sync
ui/ # index.html, app.js, styles.css (MapLibre)
tests/ # 31 tests + GPX fixtures
docker-compose.yml, docker/docker-compose.neutron.yml, Dockerfile
.env.example, trekker.yml.example
Revision notes (changes from the original v0 plan)
- R1 — Mapbox replaced with MapLibre + open data. The original plan's own Task 2.5 gate ("confirm the tile provider permits offline storage") fails for Mapbox: its ToS prohibits caching tiles outside the Mapbox SDK offline mechanism, and Terrain-RGB tiles count against a paid token. MapLibre GL JS is API-compatible for our usage and tokenless; AWS Terrain Tiles (Terrarium encoding, AWS Open Data) explicitly permit bulk download and use the near-identical decode formula. Net effect: no token to manage, offline packs are licence-clean for terrain.
- R2 — Offline basemap tiles are capped and source-configurable. OSM's
tile usage policy forbids bulk scraping. Packs default to the configured
basemap with a hard per-pack tile cap (
tiles.max_pack_tiles, default 4000) and a polite download delay; for big areas pointtiles.base_urlat a self-hosted or bulk-friendly tile source. Terrain layer has no such problem (R1). - R3 — Sync conflict policy simplified. "Home wins on structured data,
notes merge bidirectionally" was underspecified (no notes entity exists in
v0). MVP: per-record newest-
updated_atwins, both directions, idempotent. Field-level merge can come with the trip-notes feature. - R4 — shapely/geopandas dropped. Haversine + index math covers all v0 geometry; this keeps the Docker image slim (no GEOS/GDAL native deps).
- R5 — Stale deployment notes removed. The previous plan referenced a Gitea branch and a live Tailnet deployment that don't correspond to this repository; this repo is now the canonical source.
- R6 — Draft-then-confirm guide flow made explicit in the API. Scrape/PDF
endpoints never write to the DB; only the reviewed
POST /api/trailsdoes, which is what "manual review before saving" (3b) actually requires. - R7 — Firecrawl MCP dropped from MVP scrape path. Plain httpx + bs4 is dependency-free and works offline-ish (LAN sources); Firecrawl can be added later behind the same scraper interface.