Changelog

2026-03-31

Added GitHub canonical developer workflow manual:
docs/operations/GITHUB_CANONICAL_DEVELOPER_WORKFLOW.md
published as:
manuals/github-canonical-developer-workflow.html
explains:
GitHub as the canonical source,
Codex operating against the local clone,
local validation and rebuild flow,
how Docker/app/doc changes move from GitHub to the local server,
why runtime data remains outside GitHub.
Added repository split tooling so WIICCO Site can become its own dedicated GitHub repository:
scripts/ops/export_wiicco_site_repo.sh
docs/operations/REPO_SPLIT_RUNBOOK.md
make export-wiicco-site-repo
exports the projects/wiicco-site subtree into a clean standalone repository without interrupting the running Earthbond server.
Normalized Earthbond GitHub branch strategy to use main as the canonical deployment branch:
backup, deploy, and bootstrap scripts now default to main
docs and manuals now describe main as the Earthbond source-of-truth branch
this removes the long-term need for a separate earthbond deployment branch.
Normalized the local Git workflow around the Earthbond deployment branch:
local working branch is moving to main
scripts/ops/backup_to_github.sh now requires the checked-out branch to match the configured backup branch before pushing
added scripts/ops/bootstrap_earthbond_server_clone.sh to create or refresh a clean local server clone directly from git@github.com:robby1312/earthbond.git
updated Earthbond manuals and runbooks to treat the GitHub Earthbond repository as the canonical deployment source while the app continues to run on the local server.
Added GitHub-driven deploy workflow for the Earthbond deployment branch:
.github/workflows/deploy-earthbond.yml
scripts/ops/deploy_earthbond_from_github.sh
docs/operations/GITHUB_DEPLOY_RUNBOOK.md
defines GitHub as the canonical deployment trigger while the app continues to run on the existing server.
Added explicit workspace project separation:
projects/README.md
projects/earthbond/README.md
projects/wiicco-site/README.md
projects/earthbond/core
projects/earthbond/generated
projects/wiicco-site
WIICCO site/export assets were moved out of the repo root into projects/wiicco-site
Earthbond source and generated outputs were grouped under projects/earthbond as an organizational project view while preserving existing runtime compatibility.
Added repository classification inventory:
docs/operations/REPO_CLASSIFICATION_20260331.md
docs/operations/REPO_CLASSIFICATION_20260331.json
classifies the mixed repo into:
earthbond-core
earthbond-generated
wiicco-site
misc/unclassified
documents the recommended next physical separation for non-Earthbond website/export assets.
Added deployable source-backup artifact structure:
backups/README.md
backups/deployment/
backups/postgres/
backups/minio/
Added backup artifact export scripts:
scripts/ops/export_backup_artifacts.sh
scripts/ops/export_docker_backup_artifacts.sh
scripts/ops/export_postgres_schema_backup.sh
scripts/ops/export_minio_manifest_backup.sh
backup flow now exports:
redacted Docker config
PostgreSQL schema-only structure
MinIO bucket/config manifest without object data
no secrets, runtime data, MinIO objects, or PostgreSQL rows are included in the GitHub backup branch.
Added Earthbond build and access manual:
docs/operations/EARTHBOND_BUILD_AND_ACCESS_MANUAL.md
published as:
manuals/earthbond-build-and-access-manual.html
documents:
access URLs,
deployment shape,
proxy layers,
language stack,
PostgreSQL role,
MinIO role,
GitHub backup boundary,
backup branch and generated file structure.
Added GitHub source-backup workflow:
script:
scripts/ops/backup_to_github.sh
scripts/ops/install_backup_launchd.sh
scripts/ops/install_backup_cron.sh
scripts/ops/github-backup.env.example
runbook:
docs/operations/GITHUB_BACKUP_RUNBOOK.md
make target:
make backup-github
make install-backup-launchd
make install-backup-cron
behavior:
runs deployment healthcheck before backup by default,
stages repository changes,
commits with timestamped backup messages,
pushes to a configured GitHub remote,
supports first-run remote setup from GITHUB_REMOTE_URL,
supports tokenized HTTPS pushes via GITHUB_TOKEN without storing the token in git config,
supports secret-file driven automation from data/runtime/github-backup.env,
supports host-level scheduled execution through macOS launchd and Linux cron.
Hardened source backup exclusions in .gitignore so GitHub backup does not capture:
data/minio/
data/runtime/
data/auth-outbox/
data/external/
local Finder metadata under data/
Added focused ER-style database relationship schematic for colleague review:
generator:
scripts/dev/generate_database_relationship_schematic.py
outputs:
docs/architecture/DATABASE_RELATIONSHIP_SCHEMATIC_20260331.svg
docs/architecture/DATABASE_RELATIONSHIP_SCHEMATIC_MANUAL.html
published as:
manuals/database-relationship-schematic.html
groups the main relationship path by:
core
raw
ops
semantic
audit
shows key foreign-key paths for:
raw bundle/object/field extraction,
upload-to-log interpretation,
canonical well hub,
semantic projection,
audit lineage.
Added revised workflow comparison manual based directly on the new human workflow source:
source:
/Users/robertwilhelm/Documents/New project/New_human_input.drawio
generator:
scripts/dev/generate_revised_workflow_diagram.py
outputs:
docs/architecture/NEW_HUMAN_REFERENCE_VS_IMPLEMENTED_WORKFLOW_DETAILED.svg
docs/architecture/NEW_HUMAN_WORKFLOW_COMPARISON_MANUAL.html
published as:
manuals/new-human-workflow-comparison.html
shows:
the revised human workflow on the left,
the implemented build on the right,
explicit DB/storage references,
matched / partial / added / missing markers,
non-linear branch semantics preserved from the revised human input.
Added data platform storage and DB manual plus machine-readable audit export:
generator:
scripts/dev/generate_data_platform_storage_manual.py
outputs:
docs/architecture/DATA_PLATFORM_STORAGE_AND_DB_MANUAL.html
docs/operations/DATA_PLATFORM_STORAGE_AND_PROCESSING_AUDIT_20260331.json
docs/operations/DATA_PLATFORM_STORAGE_AND_PROCESSING_AUDIT_20260331.csv
published as:
manuals/data-platform-storage-and-db-manual.html
explains:
how MinIO is deployed and used,
how raw bytes are kept immutable,
how metadata is extracted and stored,
how APIs move data through the system,
exact key-table primary IDs and stored data points,
full audited table appendix for colleague review.
Logged revised Earthbond ingestion workflow as a new human-input revision:
source:
/Users/robertwilhelm/Documents/New project/New_human_input.drawio
documentation log:
docs/operations/HUMAN_INPUT_WORKFLOW_LOG_20260331_REVISION_2.md
captures the newly explicit human workflow requirements for:
core.tenant tenant anchoring,
flow_a.ingest_job_object,
flow_a.object_state_log,
typed LAS and WITSML extract tables,
input snapshots and lineage links,
ontology join presets,
outbox events and PG-A to PG-B event-driven handoff,
retained-object quarantine semantics.
Added revised human workflow alignment report:
docs/architecture/REVISED_HUMAN_WORKFLOW_ALIGNMENT_REPORT_MANUAL.html
published as:
manuals/revised-human-workflow-alignment-report.html
documents:
what the revised human workflow now requires,
where the current build matches strongly,
where the current build is partial or structurally different,
what is missing as an exact implementation of the revised flow_a schema,
what the build added beyond the revised raw-ingest workflow.
Logged uploaded Earthbond ingestion workflow as human-directed reference input:
docs/operations/HUMAN_INPUT_WORKFLOW_LOG_20260331.md
immutable audit event:
human_input.workflow_reference_logged
Added reference workflow alignment report:
docs/architecture/REFERENCE_WORKFLOW_ALIGNMENT_REPORT_MANUAL.html
published as:
manuals/reference-workflow-alignment-report.html
documents:
extracted reference workflow,
step-by-step current build alignment,
where the build expanded beyond the reference,
what is still missing or partial,
when those workflow expansions were introduced by migration phase.
Added side-by-side draw.io source for workflow review:
docs/architecture/EARTHBOND_REFERENCE_VS_IMPLEMENTED_WORKFLOW.drawio
includes:
human reference lane,
implemented lane,
DB table references,
status markers for matched, partial, added, and missing states.
Added rendered detailed workflow comparison diagram:
generator:
scripts/dev/generate_reference_workflow_diagram.py
SVG output:
docs/architecture/EARTHBOND_REFERENCE_VS_IMPLEMENTED_WORKFLOW_DETAILED.svg
HTML manual:
docs/architecture/REFERENCE_WORKFLOW_COMPARISON_DIAGRAM_MANUAL.html
published as:
manuals/reference-workflow-comparison-diagram.html
includes:
detailed DB references,
human intake logging references,
migration anchors,
explicit matched / partial / added / missing markers.
Updated the human workflow reference artifacts to follow the authoritative draw.io source:
/Users/robertwilhelm/Documents/New project/human_workflow.drawio
corrected exact branch semantics for:
invalid / missing from intake validation,
extract fail from metadata extraction,
retain raw object,
issue state / flags

2026-03-30

Added database schema defense manual:
generator:
scripts/dev/generate_database_schema_defense_manual.py
output:
docs/architecture/DATABASE_SCHEMA_DEFENSE_REPORT_MANUAL.html
published as:
manuals/database-schema-defense-report.html
explains:
why the DB exists,
schema roles,
what one row means by table family,
CRS/ECEF truth handling,
full live table-by-table walkthrough from the schema provenance audit.
Added single-page colleague-facing decision defense manual:
docs/architecture/DECISION_DEFENSE_REPORT_MANUAL.html
published as:
manuals/decision-defense-report.html
consolidates:
authorship and creator position,
system boundary,
end-to-end decision data flow,
DB role,
CRS/ECEF truth model,
schema-to-decision mapping,
human vs AI vs derived classes,
IP/copyright boundary,
colleague defense summary
intentionally omits roadmap/next-step language so it can be used as a review artifact.
Added colleague-facing HTML defense manual with navigation and schematics:
docs/architecture/BUILD_RATIONALE_SCHEMA_AND_IP_DEFENSE_MANUAL.html
published as:
manuals/build-rationale-schema-and-ip-defense.html
explains:
human-directed goals,
assistant-designed implementation decisions,
database role,
schema layers,
source vs deterministic vs heuristic outputs,
IP/copyright boundary,
how to defend the build to colleagues.
Added full schema provenance export:
scripts/dev/export_build_schema_audit.py
generates:
docs/operations/BUILD_SCHEMA_PROVENANCE_AUDIT_20260330.json
docs/operations/BUILD_SCHEMA_PROVENANCE_AUDIT_20260330.md
docs/operations/BUILD_SCHEMA_PROVENANCE_AUDIT_20260330.csv
includes:
live table inventory across core, raw, ops, semantic, and audit
table and column provenance classes
separation of direct human-authored source data vs deterministic transforms vs rule/heuristic outputs vs system/audit fields
explicit note that current runtime pipeline uses deterministic/rule-based logic rather than live LLM/ML inference
Added full build provenance manual:
docs/architecture/FULL_BUILD_EXPLANATION_AND_AI_DECISION_LOG.md
documents:
what was user-directed,
what the assistant chose,
what is directly loaded vs derived vs heuristic,
what still remains for deeper semantic processing.
Added IP and copyright compliance review:
docs/operations/IP_AND_COPYRIGHT_COMPLIANCE_REVIEW.md
documents:
current repo rights posture,
dataset/license boundaries,
AI authorship caveats,
required licensing/notice work still missing.
Hardened field-package profiling to stop embedding source-text excerpts into generated profile outputs:
libs/py/field_package/__init__.py
readme/license-like members now store redacted metadata instead of copied text snippets.
Added both new manuals to:
docs/INDEX.md
scripts/dev/build_ui_manuals.py
docs/operations/REPORT_INDEX.md

2026-03-23

Added layered validation-gate readiness reporting:
domain_coverage
parser_readiness
canonical_load_readiness
scoring_readiness
Expanded upload UI validation gate:
explicit phase-based gate table
fixed-size scrollable detail output
summary now distinguishes coverage from operational readiness
Added Volve domain export generator:
scripts/dev/export_volve_domain_files.py
writes one markdown file per domain plus:
INDEX.md
manifest.json
domain_summary.csv
generated current export set:
docs/operations/VOLVE_DATA_DOMAIN_EXPORT_20260323/
Added field-package export API and upload UI wiring:
POST /projects/{project_id}/field-package/export-domains
GET /projects/{project_id}/field-package/export-artifact
upload page can now:
generate the export set
load manifest/CSV artifacts
open per-domain markdown from validation-gate domain buttons
Added phase-2 Volve field-package ingestion:
new library:
libs/py/field_phase2/__init__.py
ingests interpreted Volve LAS files from Volve_Well_logs_pr_WELL.zip into:
ops.well_logs
ops.well_qc_cards
ops.well_interpretations
ops.well_pay_events
ops.well_bypassed_candidates
ingests Eclipse filename-derived dynamic support into:
ops.canonical_dynamic_reservoir_context
includes idempotent cleanup for repeat phase-2 log ingests
fixes reservoir-model well matching so F-1 no longer falsely matches F-10 references
Added forward-only migration 0020_field_package_phase2_context.py:
creates ops.canonical_dynamic_reservoir_context
Added field-package phase-2 API:
POST /projects/{project_id}/field-package/ingest-logs-phase2
POST /projects/{project_id}/field-package/ingest-reservoir-context
Expanded upload UI with field-package phase-2 controls:
3. Ingest Volve Logs
4. Ingest Reservoir Model
field-package workflow cards now show:
phase-2 log ingest
reservoir dynamic context
reopening scoring state
Added working Volve reopening-target scoring service:
new library:
libs/py/field_scoring/__init__.py
implements:
phase 1 scoring from canonical field-package data
phase 2 auto-enhancement when interpreted well-log outputs exist
ranked persistence into ops.well_reopening_targets
ranked event output for welllog.reopening_target_ranked
Added payload scoring inputs contract:
contracts/events/payload-scoring-inputs.schema.json
contracts/events/well-reopening-target-ranked.schema.json now types score_inputs
Added field-package scoring API:
POST /projects/{project_id}/field-package/run-reopening-scoring
GET /projects/{project_id}/field-package/reopening-targets
GET /projects/{project_id}/field-package/reopening-targets/event
Expanded upload UI with a payload scoring panel for ranked reopening targets:
apps/client-web/upload.html
apps/client-web/upload.js
Added unit coverage for phase-1 and phase-2 scoring logic:
tests/unit/test_field_scoring.py
Added reusable field-package profiling support for real field datasets:
new library:
libs/py/field_package/__init__.py
classifies local field packages into:
production
well completion
well tops
horizons/faults
reservoir model
reports
pending downloads
profiles ZIP archives and XLSX workbooks
produces ingest-priority recommendations for the current package state
Added Volve package profiling script:
scripts/dev/profile_volve_field_package.py
emits:
docs/operations/VOLVE_FIELD_PACKAGE_PROFILE_<YYYYMMDD>.json
docs/operations/VOLVE_FIELD_PACKAGE_PROFILE_<YYYYMMDD>.md
Generated first real Volve field-package report from the current local dataset root:
docs/operations/VOLVE_FIELD_PACKAGE_PROFILE_20260323.json
docs/operations/VOLVE_FIELD_PACKAGE_PROFILE_20260323.md
current report confirms:
production workbook is directly ingestible
geophysical interpretations package is directly ingestible
reservoir Eclipse model package is present and parser-worthy
multiple large downloads are still incomplete (*.crdownload)
Added unit coverage for field-package profiling:
tests/unit/test_field_package.py
Added forward-only migration 0018_reopening_target_scoring.py:
extends ops.well_data_gaps with:
gap_domain
promotion_action
adds ops.reopen_score_profiles
adds ops.well_reopening_targets
seeds default project scoring profile earthbond_reopen_v1
Added reopening-target JSON contract:
new event schema:
contracts/events/well-reopening-target-ranked.schema.json
defines portfolio-ready ranked reopening targets with:
component scores
classification
depletion and operability context
Archie assumptions
evidence references
Hardened existing bypassed-pay ranked event contract:
contracts/events/well-bypassed-pay-ranked.schema.json
candidate objects now declare:
recommended_action
completion_status
position/evidence fields
explicit candidate-level properties instead of opaque objects
Added reopening scoring and worklist design manual:
new architecture doc:
docs/architecture/WELL_REOPENING_SCORING_SCHEMA_AND_WORKLIST.md
includes:
exact PostgreSQL scoring schema,
ranking formula and gating rules,
JSON contract example,
scientist/engineer worklist UI design,
recommended next API endpoints for portfolio ranking.
Added the new worklist manual to:
docs/INDEX.md
HTML manual publisher manifest in scripts/dev/build_ui_manuals.py

2026-03-22

Added forward-only migration 0017_semantic_ontology_spine.py:
creates schema semantic
creates semantic ontology spine tables:
semantic.entity_types
semantic.relation_types
semantic.entities
semantic.entity_links
semantic.entity_source_links
semantic.query_profiles
seeds initial Earthbond entity model, relationship model, and query profiles for:
well_payload_review
spatial_truth_audit
Added Earthbond ontology blueprint manual:
new architecture doc:
docs/architecture/EARTHBOND_ONTOLOGY_BLUEPRINT_AND_AI_QUERY_LAYER.md
includes:
layered semantic architecture,
minimum canonical entity model,
relationship model,
PostgreSQL semantic schema explanation,
AI query contract,
example semantic retrieval patterns for workflows and models.
Added the ontology blueprint manual to:
docs/INDEX.md
HTML manual publisher manifest in scripts/dev/build_ui_manuals.py

2026-03-20

Added forward-only migration 0016_raw_layer_source_registry.py:
creates raw.source_bundles, raw.source_objects, and raw.extracted_fields
adds indexes for project/object lookup, source typing, and extracted field search
adds grants for app roles and client readonly access
Wired upload ingestion into the new raw layer:
direct uploads register raw bundle/object rows
ZIP uploads register one bundle with per-entry raw objects
upload metadata is mirrored into raw.extracted_fields
detect-kind now backfills raw object classification and extracted detection fields
Added upload-scoped workflow state API:
GET /projects/{project_id}/upload-workflow-state
returns uploads, raw overview, processing datasets, point-cloud metadata summaries, and well-job summaries for the upload UI
Expanded upload.html into a workflow-aware intake page:
added 12 module cards matching the well-log payload architecture
cards render processing state for:
source intake
metadata extraction
contract validation
gap classification
canonical well identity
spatial truth
trajectory/depth
curve normalization
deterministic interpretation
completion reconciliation
evidence pack
scientist review queue
added workflow manual shortcut from upload page
Added well-log data architecture manual:
new architecture doc:
docs/architecture/WELLLOG_PAYLOAD_ARCHITECTURE_AND_MODULE_DESIGN.md
includes:
high-level design architecture,
why each well-data class is needed,
what data matters most,
what the petrophysicist and scientist are looking for,
full staged workflow from intake through scientist review,
module-by-module architecture design,
recommended logical PostgreSQL layer separation (core, raw, ops, audit).
Added the new well-log architecture manual to:
docs/INDEX.md
HTML manual publisher manifest in scripts/dev/build_ui_manuals.py

2026-03-19

Added Wiicco / Toquis phased appliance and POV infrastructure manual:
new architecture doc:
docs/architecture/WIICCO_TOQUIS_PHASED_BOM_AND_POV_SCHEMA.md
includes:
phased bill of materials for Phase 0 through Phase 3
locked remote development topology
existing hardware role layout using current chassis
Wiicco appliance responsibilities vs Toquis operating responsibilities
exact PostgreSQL POV hardening delta
added published manual entry in HTML manuals generator:
wiicco-toquis-phased-bom-and-pov-schema.html
added architecture images published under:
apps/client-web/manuals/assets/pov_manual/
Added forward-only migration 0015_pov_schema_hardening.py:
hardens ops.well_logs with CRS / datum / transform / ECEF wellhead fields
hardens ops.well_pay_events with TVDSS + per-event ECEF + transform provenance fields
hardens ops.well_bypassed_candidates with completion status, evidence pack link, cone confidence, and processing version fields
adds interval ordering check and supporting indexes for project / ECEF / evidence lookups

2026-03-04

Rebuilt client web into dashboard-first operations flow:
top-right account menu now contains authentication controls (login/verify/logout)
converted /dashboard.html into a validator-only dashboard focused on CRS/ECEF proof, ordered validation flow, and DB ID chain traceability
added sticky validator progress header with stage bars that remain visible while scrolling
added inline (i) help popovers on validator actions showing what each button executes and dependencies
added ID execution order table (upload_id -> metadata_job_id -> register_job_id) to reduce DB trace confusion
added cross-reference checker for metadata detail vs optional external reference lat/lon/ECEF values
added sticky processing header in full console (index.html) with always-visible workflow/module progress bars
added upload quick-entry module on validator dashboard with auto-route processing:
upload file/zip from validator page
automatic kind detection
one-click start processing (point-cloud -> /jobs/ingest, well-log -> /well-logs/jobs/normalize)
added dedicated /viewer.html Toquis-style geodetic viewer:
left rail for WGS84 surface coordinates, local CRS, vertical datum, and ECEF validation proof
ECEF-relative 3D rendering for multi-layer alignment without local-grid distortion
inset Leaflet map with WGS84 footprint polygon or centroid marker
layer list with focus/hide/select actions and coloring modes aligned with geoscience viewer workflow
metadata detail API now returns viewer_model with:
WGS84 surface centroid
local CRS/source CRS/detection fields
ECEF centroid + roundtrip/formula/radius checks
WGS84 footprint polygon derived from native bounds
point preview extraction now carries RGB when LAS/LAZ color channels exist
summary dashboard links directly into full one-page module views using query routing (/index.html?layout=structured&module=...)
new **Operations Dashboard** with module cards (Workspace+Upload, Detect+Ingest, Scraper, Processing, Visualization, Outputs)
each module card now has run/open controls plus per-module progress bar and status
overall progress bar added for full workflow tracking
default layout now opens in structured mode focused on dashboard-led navigation
Added dashboard workflow orchestration with module-level progress bars and dependency-aware parallel execution:
new dashboard controls for single_page vs structured module layout
Run Dashboard Workflow executes module waves in parallel where safe and blocks dependent modules when prerequisites fail
continue-on-error support with show-stopper popup and auto-jump to the failing module for correction
Added validated output hub in client UI:
output prefix selector (derived, reports, open_data, raw) with structured listing
per-file download + page select-all + prefix bundle download
validated/deliverable tagging in output cards
Added storage download endpoints:
GET /storage/object-link (public/presigned single file link)
POST /storage/bundle-prefix (build zip bundle from a prefix and return download URL)
Added ESRI auth mode extensions for relevance plan:
supports public mode and optional token mode for protected ArcGIS layers
request fields include auth mode, token, or token URL + credentials
Hardened scraper workspace routing so each run is explicitly structured under new or existing project/customer context:
Data Scraper UI adds Scrape Routing (existing_project, new_project, auto) + workspace auto-save toggle
Search, ESRI Relevant Plan, and Download now always send workspace context (project_id, client_id, location_or_project_id)
POST /data-scraper/esri/relevance-plan adds routing_mode and returns effective project routing details
open-data context generator no longer falls back to shared public; it creates timestamped project context when project is not supplied
Added a unified **Seamless Workflow Runner** to drive end-to-end execution from one UI section:
mode selector: auto | pointcloud | welllog
run options:
auto 3D load
auto 2D terrain load
auto well output bundle
optional ECEF consistency
one-click Run Seamless now chains:
point-cloud: ingest -> metadata/3D -> export -> 2D map
well-log: normalize -> interpret -> bypassed-pay -> status/output
includes run-step timeline and workflow snapshot panel for debugging/traceability
Extended **Well Data Processing Visibility** to include well workflow jobs:
new API endpoint:
GET /projects/{project_id}/well-jobs
UI now shows normalize/interpret/bypassed-pay jobs with:
stage split + status cards
per-job detail panel (result + artifact pointers)
optional well_id filter
live API sync now refreshes both:
processing datasets list
well workflow jobs list
Added Upload-panel well-log stage controls to follow processing through deployment outputs:
Run Interpretation -> POST /well-logs/jobs/interpret
Run Bypassed Pay -> POST /well-logs/jobs/bypassed-pay
Download Output Bundle -> client-side JSON bundle with status-pack + job status payloads
added well workflow fields:
well_id, normalize_job_id, interpret_job_id, bypassed_pay_job_id
workflow next-step text now adapts for:
welllog_interpret
welllog_bypassed_pay
Fixed mixed LAS workflow routing so text well-log LAS files no longer fail in point-cloud ingest:
new API endpoint:
GET /uploads/detect-kind
detection reads object head bytes and classifies:
pointcloud_las_laz (LASF signature)
welllog_las_text (~Version marker)
Start Processing in client UI now auto-routes to the correct queue:
point cloud -> POST /jobs/ingest
well log -> POST /well-logs/jobs/normalize
added queue guardrails:
/jobs/ingest rejects well-log LAS text with actionable next route
/well-logs/jobs/normalize rejects binary LAS/LAZ point-cloud files with actionable next route
added unit tests:
tests/unit/test_upload_kind_detection.py
Added ZIP bundle upload workflow for structured well-log/data packages:
new API endpoint:
POST /uploads/direct-zip
server now unpacks .zip, preserves folder paths, uploads each file, and creates ops.upload_sessions rows per extracted file
safeguards included:
path traversal rejection
max file count and uncompressed-size limits
ignores system artifacts like __MACOSX/.DS_Store
client UI upload panel now includes:
Upload ZIP Bundle
uploaded ZIP entries automatically set selected_for_ingest to first LAS/LAZ entry when available
added upload-to-processing workflow shortcuts in client UI:
Locate Latest LAS/LAZ
Start Processing
hardened event binding for optional/new controls so a stale cached HTML page does not break all button handlers
added Upload panel workflow state tracker:
Upload / Selected / Queued / Processing bars
live next-action guidance text
automatic /jobs/status polling after queueing ingest
Added audited-source and direct URL download workflow in client web + data-plane API:
new API endpoints:
GET /data-scraper/audited-sources
POST /data-scraper/direct-download
audited-sources now supports manifest_object_key and Docker-safe local fallback manifest:
apps/data-plane-api/src/data_plane_api/audited_sources_manifest.json
client UI now includes:
Audited Sources list with Open Source and Direct Download row actions
Direct Download URL input/button for manual provider links
Added project-level ECEF distortion consistency audit:
new API endpoint:
POST /projects/{project_id}/ecef/consistency
validates cross-dataset geodetic-vs-ECEF distance consistency
returns flagged_jobs, worst_pairs, pass/fail counts, and recommendations
client UI controls:
ECEF Consistency Audit
Download ECEF JSON
Added ECEF consistency unit tests:
tests/unit/test_ecef_consistency.py
Added dedicated technical evaluation blueprint for Earthbond_POV_Technical_QA.pdf:
docs/architecture/POV_TECHNICAL_QA_EVALUATION_IMPLEMENTATION_V1.md
includes:
Q1-Q12 gap analysis against current implementation
formula appendix (MD->TVDSS, ECEF, petrophysics, Monte Carlo cone)
failure-mode matrix for ingest/storage/AI-readiness
phased monetization strategy before full AI
build-vs-buy benchmark framing for existing market tools
Added the new blueprint into UI manuals generation:
scripts/dev/build_ui_manuals.py now publishes:
apps/client-web/manuals/pov-technical-qa-evaluation-blueprint.html
Updated docs index with pointer to the new POV QA blueprint:
docs/INDEX.md
Added well-log traceability persistence and deterministic audit harness:
new migration: db/migrations/versions/0011_welllog_traceability_columns.py
stage traceability signatures now emitted in:
libs/py/welllog_core/__init__.py
worker persistence and status-pack exposure updated:
apps/worker-ingest/src/worker_ingest/main.py
apps/data-plane-api/src/data_plane_api/main.py
Added known-result self-audit mechanism:
fixture input: tests/fixtures/welllog/legacy17.las
golden expected snapshot: tests/fixtures/welllog/legacy17_expected.json
runner script: scripts/dev/run_welllog_known_result_audit.py
make target: run-welllog-known-result-audit
report outputs:
docs/operations/WELLLOG_KNOWN_RESULT_AUDIT_REPORT.json
docs/operations/WELLLOG_KNOWN_RESULT_AUDIT_REPORT.md
Added operations manual for root-cause traceability:
docs/operations/WELLLOG_TRACEABILITY_AND_AUDIT.md

2026-03-03

Added HTML docs/manuals publishing into client UI:
new generator script: scripts/dev/build_ui_manuals.py
generated static manual pages under apps/client-web/manuals/*
new UI panel **Docs + Manuals (HTML)** with:
manifest refresh
dropdown doc selection
inline frame viewer
open in new tab + index shortcuts
new build target: make build-ui-manuals
Added documentation stage-index structure for easier tracking of latest manuals and reports:
new root index: docs/INDEX.md
stage map: docs/stages/README.md
staged folders with readme indexes:
docs/stages/stage-01-foundation/README.md
docs/stages/stage-01b-welllog/README.md
docs/stages/stage-02-analog-risk/README.md
docs/stages/stage-03-digital-twin/README.md
report catalog: docs/operations/REPORT_INDEX.md
Added end-to-end interaction map doc:
docs/architecture/SYSTEM_INTERACTION_MAP.md
explicitly identifies what is integrated vs partial/not yet wired.
Added client-web collapsible section controls to reduce rigid long-form UI:
global controls: Expand All Sections, Collapse All Sections, Remember section state
per-section toggles on panel headers with persisted collapse state in local storage.
Added client-web Finder-level storage browsing controls in **Bucket + Jobs**:
Review Uploaded, Finder Root, Finder Up, Finder Open, Finder Refresh
new clickable folder/file list with breadcrumb + summary for project/raw object storage review
List Bucket now keeps Finder view in sync with the currently selected prefix
Added locked build contract for drilling risk POC:
docs/architecture/RISK_DRILLING_BUILD_CONTRACT_V1.md
captures client-approved scope: CRS/ECEF/well-match truth, protected access queue, risk engine outputs, 3D uncertainty cone, seismic extension, governance, and acceptance gates.
Added processing audit-validation API and UI flow:
new data-plane endpoint: POST /projects/{project_id}/processing/audit-validate
validates stage1 processed datasets against audit manifest (row_count + sha256)
aggregates stage2 download/OCR status in report output
writes report outputs in two formats: JSON and Markdown (stored in MinIO/object storage)
Added client-web controls in **Well Data Processing Visibility**:
Run Audit Validation
Download Audit JSON
Download Audit MD
manifest references via file path and object key (object key preferred for Docker)

2026-03-02

Completed 2D terrain map UI wiring in client web:
fixed all 2D map button handlers (Use Task ID, Load 2D Map, Clear 2D)
added live render-setting handlers (base-layer switch, hillshade toggle, contour toggle, opacity)
Added local Leaflet assets to avoid CDN dependency:
apps/client-web/vendor/leaflet.js
apps/client-web/vendor/leaflet.css
Finalized terrain preview API route usage:
GET /maps/terrain/preview exposed in contract and live through gateway (/data/maps/terrain/preview)
validated with ASC export job and CRS override for map overlay
Added unit tests for terrain preview math/parsing helpers:
tests/unit/test_terrain_preview.py
Updated operations docs and README with 2D hillshade/contour workflow and CRS notes.
Added real CA well-data test pack pipeline for drilling/oil-first POC testing:
script: scripts/dev/fetch_ca_well_test_pack.py
make target: make fetch-ca-well-pack
outputs normalized datasets + audit artifacts (manifest.json, summary.md, poc_ingest_manifest.json)
sources include CalGEM WellSTAR wells/WST, CA DWR WCR/geologic logs/report links, and USGS CA groundwater feeds
Added stage-2 queue + API workflow for CA pack operations:
scripts/dev/build_ca_well_stage2_queue.py (make build-ca-stage2-queue)
scripts/dev/run_ca_well_pack_api_workflow.py (make run-ca-pack-workflow)
workflow now performs upload + processing register-job execution + report output
Added new data-plane APIs for processing dataset tracking:
GET /projects/{project_id}/processing-datasets
POST /processing-datasets/jobs/register
upload endpoints now accept metadata tags: source_tag, processing_stage, dataset_kind
Added client UI visibility panel:
**Well Data Processing Visibility** in client web
stage filter (all/stage1/stage2), dataset filter, and per-dataset profile detail

2026-03-01

Reviewed and integrated updated client workflow document Earthbond_POV_v0.2.docx.
Added architecture delta document:
docs/architecture/WELLLOG_POV_V0_2_INTEGRATION.md
includes module-level additions for:
well-log normalization (LAS/DLIS), QC cards, MD->TVDSS handling
deterministic petrophysical interpretation (Vsh/PHIE/Rt, optional Sw)
bypassed-pay classification/ranking and status-pack outputs
required DB, API, worker, and UI expansion steps
Updated README.md with pointer to the new well-log integration design doc.
Implemented first operational well-log workflow slice:
new migration 0010_welllog_pov_tables.py with well-log/QC/interpretation/bypass/data-gap tables
new data-plane endpoints:
POST /well-logs/jobs/normalize
POST /well-logs/jobs/interpret
POST /well-logs/jobs/bypassed-pay
GET /projects/{project_id}/wells/{well_id}/qc-card
GET /projects/{project_id}/wells/{well_id}/status-pack
GET /projects/{project_id}/portfolio/bypassed-pay-summary
worker-ingest support for welllog_normalize, welllog_interpret, welllog_bypassed_pay job types
added libs/py/welllog_core with LAS parsing, normalization/QC, deterministic interpretation, and bypass classification logic
Added operational/manual docs:
docs/operations/WELLLOG_POC_OPERATOR_MANUAL.md
docs/architecture/VISION_PHASE_STATUS.md
Added real-file LAZ workflow regression script:
scripts/dev/run_real_laz_workflow.py
exercises upload -> listing -> ingest -> metadata -> export, and writes JSON report
Client-web hardening for upload/listing flow:
explicit required-field validation for project/object identifiers
clearer API URL missing error
automatic API Gateway compatibility (http://localhost:8080 -> /data/... fallback)
browser upload path now prefers POST /uploads/direct to avoid MinIO CORS failures on presigned PUT
presigned PUT retained as fallback path
Declared CRS input now defaults to blank (auto-detect), reducing accidental CRS override errors
Fixed client upload/list UI runtime regression:
added missing backend dependency python-multipart so POST /uploads/direct no longer crashes data-plane-api
stale/invalid bearer tokens now fall back to trusted proxy subject for UI requests (x-proxy-subject)
client default Data API URL switched to gateway (http://localhost:8080) for stable browser pathing
validated live workflow:
direct upload succeeds
List Uploads returns latest records
folder tree view returns nested uploaded paths
Added Autodesk 2D terrain export for LAZ/LAS:
new export format asc (ESRI ASCII DEM, Civil 3D friendly)
new export request fields: grid_cell_size, elevation_stat=max|mean|min
worker now writes terrain sidecar metadata:
<source>.terrain-map.json
client export UI now exposes ASC DEM option plus terrain controls
Improved class/layer isolation UX:
added export class checkboxes for ground(2), building(6), vegetation(5), water(9)
class-checkbox selections sync to Class Filter (codes csv) for export payload
viewer class checkbox toggles now rerender immediately (points on/off without extra apply step)
Improved export discoverability in UI:
on successful Job Status, UI auto-sets Bucket Prefix to export folder and refreshes Folder Tree

2026-02-28

Upgraded WebGL visualizer for higher-density LiDAR inspection:
raised point budget cap from 200k to 1,000k (1M)
new quality presets tuned for larger renders (performance, balanced, high, max)
viewer canvas height increased for closer visual inspection
Added production-style inspection controls:
single-click point inspection panel (class, xyz, elevation, intensity, source)
Focus Dense camera action to jump to densest local point cluster
improved close zoom radius so dense mesh/noise inspection is practical
fast zoom modifier (Shift + wheel)
Improved class/color workflow:
quick class preset buttons now rerender immediately when viewer is active
Reset Colors now rerenders when viewer is active
richer viewer note/help text for inspection workflow
Updated client UI for new visualizer controls and point inspection output.
Rebuilt client-web container and validated runtime wiring.
Implemented Open Data Auto-Search Architecture baseline:
canonical artifact pipeline: queries.json, results_normalized.json, shortlist.json, download_queue.json
weighted ranking model: overlap, resolution, recency, coverage, license openness, format readiness
canonical storage paths under open_data/client/<client>/<location>/...
analytics artifacts: summary.json, risk_report.json
Added data-plane API enhancements:
POST /data-scraper/auto-search-from-upload (derive AOI from processed upload metadata)
enriched POST /data-scraper/search output with ranked/normalized artifacts
upgraded POST /data-scraper/download background job to use ranked queue and persist artifacts
Added ingest-worker auto-discovery:
after successful pointcloud processing, auto-runs open-data search from metadata centroid AOI
writes standard search/analytics JSON artifacts to object storage
Added S3 JSON upload helper and open-data pipeline unit tests.

2026-02-27

Added strict pointcloud CRS validation integration in ingest worker output path.
Added exact CRS->ECEF conversion and fixed-anchor local ENU coordinates.
Added LAZ backend hardening (lazrs) for reliable .laz decompression.
Added Autodesk/ML export pipeline:
POST /jobs/export/autodesk
output formats: PTS, CSV, JSONL
classification summary artifact output
Added metadata browsing API:
GET /projects/{project_id}/metadata
GET /projects/{project_id}/metadata/{job_id}
Added client UI features:
metadata browser
WebGL 3D viewer (Three.js)
classification filtering in viewer
Added multi-file grid vector indexing:
migration 0009_pointcloud_grids_vector_index
PostGIS tables: ops.pointcloud_grids, ops.pointcloud_tiles
grid APIs:
POST /projects/{project_id}/grids
GET /projects/{project_id}/grids
GET /projects/{project_id}/grids/{grid_id}/tiles
GET /projects/{project_id}/grids/{grid_id}/summary
ingest flow now supports grid_id and auto-indexes tiles in the grid
Added deployment/access docs:
docs/SITE_ACCESS.json
docs/OPERATIONS_AND_OUTPUT_GUIDE.md
Added client upload UX improvements:
folder batch upload (webkitdirectory) with per-file status summary
uploaded files panel + folder-style uploaded tree view
click-to-select upload rows to populate upload_id/object_key
Added drag-and-drop upload support with per-file progress bars in client UI.
Increased point-vector preview sampling for metadata output to support denser WebGL rendering.
Added WebGL viewer controls for max rendered points and point size.
Added WebGL elevation/intensity filters.
Added mouse 360 camera orbit + wheel zoom controls in WebGL viewer.
Added multi-file pointcloud rendering from checked metadata rows (Render Checked Files).
Improved close zoom and focus behavior:
robust outlier-trimmed camera framing for dense-point regions
tighter minimum zoom distance
double-click point to focus camera on local area
Added CRS location validation metadata:
canonical centroid schema:
centroid_native (source CRS)
centroid_geodetic (EPSG:4326 lat/lon + height)
centroid_status (VALID/INVALID)
Google Maps URL for visual location validation
bounds validity status
Added strict geodetic validation standard implementation:
hard fail rules for invalid lat/lon range, unknown CRS, transform failures
heuristic detection for projected values mislabeled as lat/lon (abs(value)>1000)
longitude normalization to [-180, 180]
Upgraded ECEF validation:
pyproj CRS->ECEF conversion with roundtrip geodetic checks
independent WGS84 formula cross-check (EPSG method 9602 equations)
ECEF radius plausibility validation
Added optional PDAL CRS metadata extraction (pdal info --metadata) as secondary CRS detector.
Added viewer RGB color mode based on XYZ coordinates.
Added viewer filter help panel with practical high/low examples.
Added freer 3D mouse controls:
left-drag orbit
right/shift-drag pan
wheel zoom
Switched WebGL dependency from remote CDN to local static vendor file:
apps/client-web/vendor/three.min.js
avoids runtime breakage when external CDN is blocked
Added location/map UI controls in metadata browser:
Location Validation toggle (on/off)
Force Map Link fallback mode
Open Map action for current metadata selection
Added point-detection controls in viewer:
Detect Points reports preview/raw point estimates and filter impact
Set Max auto-sets max rendered points from detected filtered preview counts
Added spatial tile safety checks for wrong-location files:
automatic dominant-cluster outlier detection from per-tile geodetic centroids
Detect Tile Issues report in UI
Fix Tile Set removes flagged outlier tiles from current selection
Include Outlier Tiles yes/no override; default behavior auto-excludes outliers from render
explicit render-time yes/no prompt to fix or keep detected outlier tiles
viewer now treats invalid_range CRS validation status as outlier-equivalent for auto exclusion/fix
Added storage management API for MinIO data operations:
GET /storage/objects (paginated object listing by project/prefix)
DELETE /storage/object (single object delete + optional DB status cleanup)
DELETE /storage/prefix (dry-run + bulk prefix delete + optional DB status cleanup)
Expanded Admin Web UI with Storage Manager:
list/paginate objects
select all on current page
select all across current query scope (all pages up to configured cap)
clear selection
delete single object
delete selected objects
dry-run and execute prefix delete with safety controls
Added **Beta Production Control** panel in Client Web:
guided pipeline board with channel switch (beta, production-sim)
computed step-state cards and missing-item cards
Run Next deterministic action runner for point-cloud and well-log paths
Run Full Pipeline orchestration (seamless + audit validation)
quick navigation to output hub and manuals hub
Hardened button action reliability:
added data-action attributes for key upload/workflow buttons
fallback delegated action dispatcher for buttons without bound listeners
Added **Project Workspace** profiles:
save/load/delete workspace profiles with customer + description metadata
auto-generated project IDs for new runs
optional full module sync on workspace load
Added upload-first automation controls in Upload panel:
auto-run pipeline after upload (Run Next or Run Full Pipeline)
manual Run Next Now, Run Full Now, and Sync Context controls
automatic cross-module sync after selection and job success to reduce manual re-selection across sections
Added **Resource Link Library** for scraper source management:
persistent local source catalog
open selected source URL
push selected source URL directly into scraper direct-download input
Added ESRI/ArcGIS relevance planning pipeline:
backend endpoint: POST /data-scraper/esri/relevance-plan
ESRI layer discovery by AOI with count/sample queries
automatic relevance classification (well, surface, survey, water, reference)
merged queue generation with project/customer-bound storage context
optional large WCR CSV classification from object key or URL for well-vs-water filtering
Added scraper UI controls for ESRI relevance planning:
ESRI Relevant Plan action
optional extra ESRI layer URLs
WCR object key / URL inputs for tabular domain classification
Dashboard + workflow UX update:
moved authentication into top-right account menu in header
added Operations Dashboard card grid with per-module run/open controls
added module and overall progress bars for dashboard execution
added dependency-aware dashboard workflow runner with parallel waves
added continue-on-error toggle and show-stopper popup with jump-to-module
added workflow tracker Download lane so upload and downstream download progress are visible together