Initial commit - Trekker V2 source

This commit is contained in:
Cortex committed 2026-06-10 11:03:37 -04:00
commit d71faf19e3
65 files changed
+3615

No files matched your search

+197
View File
@@ -0,0 +1,197 @@
# Trekker — v0 Implementation Plan (revised)
> Single canonical source of truth for Trekker product/architecture decisions.
> This revision supersedes the original v0 plan; the source tree in this
> repository is the implementation of it. Revision notes are at the bottom.
**Goal:** Build a vehicle-aware overland trip intelligence system that ingests
GPX files (manual + folder watch), trail guides (web scrape + PDF with LLM
extraction), weather data (Open-Meteo), and offline map areas, then generates
structured overland assets and exports enriched GPX/KML files for manual
import into onX and Gaia GPS.
**Architecture:** Dual-host FastAPI backend on home server (always-on,
Tailnet) + Neutron truck server (offline-first). GPX ingestion → canonical
`TrekkerRoute` model. Pluggable asset generators. Offline map packs provide
local basemap/terrain tiles for field use. Export layer → enriched GPX/KML
for onX/Gaia.
**Tech Stack:** FastAPI, gpxpy, openai Python SDK, pypdf, python-watchdog,
httpx → Open-Meteo, **MapLibre GL JS** + raster basemap, **AWS Terrain Tiles
(Terrarium)** for elevation, SQLite (JSON blobs), local offline tile packs,
Docker Compose.
---
## Scoped Items (locked with Peyo)
| # | Item | Decision | Notes |
| :-- | :--- | :------- | :---- |
| 1 | GPX input | **A + B**: manual upload via web UI + folder watch via `python-watchdog`. `gpxpy` → `TrekkerRoute`. | Handles Gaia and onX metadata. |
| 2 | Map / elevation | **MapLibre GL JS** for rendering (no token). **AWS Terrain Tiles (Terrarium encoding)** for elevation. | Powers UI display AND segment grade computation. *(Revised — see note R1.)* |
| 2b | Offline map areas | Save map areas for offline use: viewport/rectangle, zoom range → local tile pack served by Trekker. | Per-pack tile cap; tile source configurable. *(See note R2.)* |
| 3a | Trail guides — web | `httpx` + BeautifulSoup scrape → LLM extraction. | difficulty, obstacles, water crossings, gate codes, vehicle reqs, closures. |
| 3b | Trail guides — PDF | `pypdf` → OpenAI-compatible LLM extraction. | Output requires manual review before saving. |
| 3c | LLM interface | OpenAI-compatible API; configurable `base_url`, `model`, `api_key` in `trekker.yml`. | Supports local Ollama, Ollama Cloud, Gemini, Groq, OpenAI. |
| 4 | Weather | Open-Meteo — free, no API key, global. | 7/10-day, four granularity tiers, manual refresh. |
| 5 | Cell coverage | **Not MVP** — deferred. | — |
| 6 | Victron / battery | **Not MVP** — deferred. | — |
---
## Implementation status
All phases below are **built** in this repo at MVP depth and covered by the
test suite (`tests/`, 31 tests).
### Phase 1: GPX Ingestion Engine ✅
- `app/models/` — `TrekkerRoute` (TrkPt/Wpt/Segment/RouteStats), `TrailGuide`,
`Trip`, `Asset`, `MapPack`. Pydantic v2.
- `app/ingest/gpx_parser.py` — gpxpy parse, creator-first source detection
(`gaia` / `onx` / `manual`), `<rte>` fallback for planned routes.
- `app/ingest/segmenter.py` — waypoint-driven segmentation (named waypoints
snapped to track within 200 m define boundaries; no waypoints → one
segment). Haversine distance, gain/loss, avg/max grade (10 m smoothing
window against GPS jitter), duration.
- `app/ingest/watchdog_daemon.py` — dual watch folders; inotify observer for
the internal path, **PollingObserver** for the remote mount (SMB/Nextcloud
mounts don't deliver inotify events reliably). Size-stable check before
ingesting.
- `app/storage/` — SQLite, JSON-blob tables (routes, trail_guides, assets,
weather_cache, offline_map_packs), WAL mode, FK link route → guide.
### Phase 2: Web UI Shell & Map ✅
- The map is the app: full-screen MapLibre GL JS; collapsible left sidebar
(routes, layers, offline packs); slide-out route panel with summary,
segments, guide link, trail card, export, delete.
- Grade-colored segment toggle (green→yellow→red by max grade) and weather
overlay toggle (blue gradient by precip probability) — map layers, not
sidebar charts.
- `app/services/elevation.py` — Terrarium decode
`ele = (R*256 + G + B/256) − 32768`, disk+memory tile cache; fills missing
`ele` on upload so grade math works for onX exports that omit elevation.
- `app/services/offline_maps.py` + `app/routes/offline_maps.py` — estimate →
create → background download → ready; tiles under
`data/offline_maps/{pack}/{layer}/{z}/{x}/{y}.png`; served at
`/api/offline-maps/tiles/...` which the UI switches to when offline.
### Phase 3: Trail Guide Module ✅
- Draft-first flow: `POST /api/trails/scrape` and `/api/trails/upload-pdf`
return **drafts**; nothing is stored until the user reviews/edits and
`POST /api/trails` confirms (manual-review gate per 3c).
- `app/services/llm.py` — OpenAI-compatible adapter, JSON-mode with fallback,
tolerant of code-fenced output from small local models. When no LLM is
configured, drafts carry raw extracted text for manual entry.
- Max 1 guide per route (`route.trail_guide_id`), user-selectable link in the
panel, generators degrade gracefully without one.
### Phase 4: Weather Integration ✅
- `app/services/weather.py` — Open-Meteo, 7-day default / 10-day toggle,
per-segment midpoints (deduplicated on a 0.05° grid so short routes make
one API call), aggregated to four tiers: hourly / 6 h / 12 h / daily.
- Manual refresh only: `POST /api/routes/{id}/refresh-weather`; cached in
SQLite; compact forecast badge in the panel corner; wind advisory
(≥40 km/h) and lightning risk (WMO codes 95/96/99) flags.
### Phase 5: Asset Generator Pipeline ✅
- `app/generators/base.py` — `BaseGenerator` ABC with `requires[]` checks and
user-facing errors (HTTP 409 when prerequisites are missing).
- `trail_card` (route + optional guide + optional weather → per-segment JSON
card rendered in the panel), `elevation_map` (grade colors + summit/low
callouts), `weather_overlay` (per-segment precip/wind/lightning).
- Trigger is a user click; output stored as JSON in the assets table.
### Phase 6: Export & Download ✅
- `app/export/gpx_writer.py` — GPX 1.1 with `trekker:` extensions: per-point
`grade_pct` (+ `surface_type` when known), per-track segment metadata and
guide enrichment (difficulty, water crossing depth). Waypoints preserved —
round-trips through the parser losslessly (tested).
- `app/export/kml_writer.py` — KML for onX; enrichment in the description.
- `GET /api/routes/{id}/export?format=gpx|kml` → file download.
### Phase 7: Dual-Host Sync & Deployment ✅
- Single Docker image; `docker/docker-compose.neutron.yml` override adds the
Tailscale sidecar (`TAILSCALE_AUTH_KEY`).
- Manual sync: UI button → `POST /api/sync` pulls the peer's bundle, merges,
pushes local bundle back. Conflict policy: **newest `updated_at` wins, per
record** (see note R3).
- UI offline state: `navigator.onLine` → offline badge, sync hidden, weather
refresh disabled, basemap source switched to local pack tiles.
### Phase 8: July 4 Trip — verification gate (next)
Deploy home + Neutron instances, run the full workflow with live Windrock
routes before the June 27 departure: import GPX → map → link guide →
trail card → weather → export → sync → offline mode in the driveway.
---
## Verification Strategy
| Phase | What | How | Status |
| :---: | :--- | :-- | :----- |
| 1 | GPX parsing accuracy | Unit tests with Gaia/onX/minimal fixtures: source, segments, stats | ✅ automated |
| 2 | Map render + offline packs | Visual check + pack download, then disconnect and verify local tiles | manual (Phase 8) |
| 3 | LLM extraction | Compare output vs labeled guides, >80% field accuracy | manual, provider-dependent |
| 4 | Weather aggregation | Known-answer tier tests with synthetic Open-Meteo payloads | ✅ automated |
| 5 | Generator output | Known-answer tests incl. graceful degradation and 409s | ✅ automated |
| 6 | GPX round-trip | Export → re-import → segments/waypoints/elevations identical | ✅ automated |
| 7 | Sync | Two-store merge tests: newest-wins, idempotent re-merge | ✅ automated |
Run: `pip install -r requirements.txt pytest && python -m pytest tests/`
---
## File Map (as built)
```
app/
├── main.py # FastAPI entry, SPA serving, watchdog lifespan
├── config.py # env + trekker.yml settings
├── deps.py # shared store/service singletons
├── models/ # route, trail_guide, trip, assets, map_pack
├── ingest/ # gpx_parser, segmenter, watchdog_daemon
├── services/ # elevation, offline_maps, weather, llm, scraper
├── generators/ # base, trail_card, elevation_map, weather_overlay
├── export/ # gpx_writer, kml_writer
├── storage/ # database (schema), route_store (all CRUD)
├── sync/sync_daemon.py # bundle build/merge + peer sync
└── routes/ # upload, trails, weather, export, offline_maps, sync
ui/ # index.html, app.js, styles.css (MapLibre)
tests/ # 31 tests + GPX fixtures
docker-compose.yml, docker/docker-compose.neutron.yml, Dockerfile
.env.example, trekker.yml.example
```
---
## Revision notes (changes from the original v0 plan)
- **R1 — Mapbox replaced with MapLibre + open data.** The original plan's own
Task 2.5 gate ("confirm the tile provider permits offline storage") fails
for Mapbox: its ToS prohibits caching tiles outside the Mapbox SDK offline
mechanism, and Terrain-RGB tiles count against a paid token. MapLibre GL JS
is API-compatible for our usage and tokenless; **AWS Terrain Tiles**
(Terrarium encoding, AWS Open Data) explicitly permit bulk download and use
the near-identical decode formula. Net effect: no token to manage, offline
packs are licence-clean for terrain.
- **R2 — Offline basemap tiles are capped and source-configurable.** OSM's
tile usage policy forbids bulk scraping. Packs default to the configured
basemap with a hard per-pack tile cap (`tiles.max_pack_tiles`, default
4000) and a polite download delay; for big areas point `tiles.base_url` at
a self-hosted or bulk-friendly tile source. Terrain layer has no such
problem (R1).
- **R3 — Sync conflict policy simplified.** "Home wins on structured data,
notes merge bidirectionally" was underspecified (no notes entity exists in
v0). MVP: per-record newest-`updated_at` wins, both directions, idempotent.
Field-level merge can come with the trip-notes feature.
- **R4 — shapely/geopandas dropped.** Haversine + index math covers all v0
geometry; this keeps the Docker image slim (no GEOS/GDAL native deps).
- **R5 — Stale deployment notes removed.** The previous plan referenced a
Gitea branch and a live Tailnet deployment that don't correspond to this
repository; this repo is now the canonical source.
- **R6 — Draft-then-confirm guide flow made explicit in the API.** Scrape/PDF
endpoints never write to the DB; only the reviewed `POST /api/trails` does,
which is what "manual review before saving" (3b) actually requires.
- **R7 — Firecrawl MCP dropped from MVP scrape path.** Plain httpx + bs4 is
dependency-free and works offline-ish (LAN sources); Firecrawl can be added
later behind the same scraper interface.