Well Log Payload Architecture And Module Design

Source: docs/architecture/WELLLOG_PAYLOAD_ARCHITECTURE_AND_MODULE_DESIGN.md

Manual Index Client UI

Well Log Payload Architecture And Module Design

Purpose

This document defines the data-first Earthbond architecture for:

  1. abstracting data from well logs and associated drilling records,
  2. identifying what is present, what is missing, and what can be normalized,
  3. separating deterministic outputs from assumptions,
  4. ranking review payloads for scientists and engineers.

The goal is not to make every well look usable.

The goal is to make each well land in the correct confidence state with a full evidence trail.

High-Level Design Architecture

The platform should be built around canonical truth generation, not around file upload or 3D display.

flowchart TD A["Source Intake"] --> B["Raw Vault + Source Bundle Registry"] B --> C["Metadata Extraction"] C --> D["Contract Validation"] D --> E["Gap Classification"] D --> F["Canonical Well Identity"] D --> G["Spatial Truth Engine"] D --> H["Log Normalization Engine"] G --> I["Trajectory / TVDSS / WGS84 / ECEF"] H --> J["Petrophysical Interpretation"] I --> K["Canonical Event Builder"] J --> K K --> L["Bypassed-Pay Candidate Builder"] E --> L L --> M["Scientist Review Queue"] K --> N["Evidence Pack"] N --> O["Audit Ledger"] I --> P["2D / 3D Publication"] L --> P

Why We Need The Data

The system is trying to answer six different questions, and each requires different data quality:

  1. Is this the correct well?
  2. Where is it really located in space?
  3. What depth reference is the data using?
  4. Do the logs show a technically interesting interval?
  5. Was that interval already completed, tested, or depleted?
  6. Is the interval strong enough to send to a scientist or engineer for deeper review?

Without the underlying data, the platform cannot do any of the following safely:

  1. compare wells across a field,
  2. align surface and subsurface layers,
  3. rank bypassed-pay opportunities,
  4. evaluate uncertainty,
  5. produce defensible outputs for technical or commercial decisions.

What Data Is Important

1. Well identity data

This prevents mismatching logs, completions, and trajectories.

Important fields:

  1. API number
  2. UWI
  3. permit number
  4. well name
  5. operator
  6. field / basin / lease
  7. county / state
  8. spud / completion / abandonment date

2. Surface location and spatial reference data

This determines whether the well can be correctly placed in 2D and 3D.

Important fields:

  1. latitude / longitude
  2. easting / northing
  3. source CRS
  4. resolved EPSG
  5. local grid name if used
  6. coordinate epoch if applicable
  7. vertical datum
  8. KB / GL / DF / RT elevation
  9. WGS84 geodetic output
  10. ECEF coordinates

3. Depth reference and survey data

This determines whether interval depths can be compared across wells.

Important fields:

  1. measured depth (MD)
  2. true vertical depth (TVD)
  3. true vertical depth subsea (TVDSS)
  4. deviation survey stations
  5. inclination
  6. azimuth
  7. survey method
  8. tie points
  9. reference point used for subsea conversion

4. Log curve data

This is the primary petrophysical input.

Important fields:

  1. Gamma ray (GR)
  2. Bulk density (RHOB)
  3. Neutron porosity (NPHI)
  4. Resistivity (RT, ILD, LLD, similar families)
  5. Caliper (CAL)
  6. Sonic (DT)
  7. Spontaneous potential (SP)
  8. Photoelectric factor (PEF)
  9. environmental corrections
  10. sampling interval
  11. curve units
  12. null handling

5. Completion and operational data

This determines whether a good interval is actually actionable.

Important fields:

  1. perforated intervals
  2. tested intervals
  3. workovers
  4. stimulation history
  5. production history
  6. shut-in status
  7. abandonment status
  8. casing and cement notes
  9. pressure or test data

6. Geologic and drilling context

This helps separate a local log anomaly from a real reservoir interval.

Important fields:

  1. formation tops
  2. markers
  3. cuttings or lithology descriptions
  4. mud logs
  5. offset wells
  6. structural maps
  7. seismic / horizons later
  8. DEM / LiDAR / survey control points for surface truth

What The Petrophysicist Is After

The petrophysicist is not after the raw LAS file.

They are after a technically defensible answer to these questions:

  1. Is this interval reservoir or non-reservoir?
  2. How shaly is it?
  3. How much effective porosity is present?
  4. Is there evidence of hydrocarbons rather than water?
  5. How thick is the net interval?
  6. Was it already completed or properly tested?
  7. Is the data quality high enough to trust the conclusion?

For a bypassed-pay workflow, the scientist usually wants:

  1. top/base of interval,
  2. gross thickness,
  3. net pay,
  4. Vsh,
  5. PHIE,
  6. Rt,
  7. optional Sw,
  8. QC state,
  9. completion overlap result,
  10. recommendation,
  11. confidence and uncertainty.

What The Scientist Looks For To Find Payload

The platform should not declare "payload found" from one curve.

It should combine geology, petrophysics, completion history, and spatial context.

The scientist is typically looking for:

  1. intervals with low-to-moderate shale,
  2. adequate porosity,
  3. supportive resistivity,
  4. enough net thickness,
  5. no convincing evidence the interval was already properly perforated or tested,
  6. acceptable log quality,
  7. acceptable spatial and depth-reference confidence.

The review output is therefore a ranked list of:

  1. wells,
  2. intervals,
  3. reasons they are interesting,
  4. reasons confidence is high or low,
  5. what additional data is still needed.

What We Normally Expect From Well Log Data

Modern digital wells

Usually available:

  1. LAS or DLIS file,
  2. run metadata,
  3. curve inventory,
  4. clearer header data,
  5. better deviation surveys,
  6. cleaner unit conventions,
  7. more complete completion context.

Older or legacy wells

Often missing or ambiguous:

  1. CRS,
  2. vertical datum,
  3. deviation survey,
  4. machine-readable completions,
  5. reliable identifiers,
  6. consistent mnemonics,
  7. exact unit definitions,
  8. clean digital source.

Paper-only wells

Usually require:

  1. OCR,
  2. header extraction,
  3. table extraction,
  4. image segmentation,
  5. manual review,
  6. confidence tagging before promotion.

What Is Missing Most Often

The most common historical gaps are:

  1. missing well identifier or conflicting aliases,
  2. missing source CRS,
  3. missing vertical datum,
  4. missing KB/RT elevation,
  5. missing deviation survey,
  6. missing completion/perforation intervals,
  7. missing caliper,
  8. unclear curve units,
  9. scanned rather than digital curves,
  10. multiple contradictory versions of the same record.

The system should explicitly capture these as structured gaps, not narrative notes.

Can We Adjust Missing Data

Yes, but only under controlled rules.

Allowed adjustments:

  1. mnemonic alias mapping,
  2. unit normalization,
  3. null cleanup,
  4. depth resampling,
  5. minimum-curvature TVD/TVDSS when a valid survey exists,
  6. vertical-well assumption only with downgrade,
  7. CRS transformation when the source definition is sufficiently known,
  8. authority ranking across multiple sources.

Not allowed as silent automation:

  1. inventing CRS,
  2. inventing vertical datum,
  3. silently converting MD to TVDSS,
  4. inventing completion intervals from vague narratives,
  5. merging wells purely by name similarity.

Entire Workflow

Stage 1: Intake

Inputs:

  1. LAS / DLIS / LIS,
  2. PDF / scanned reports,
  3. CSV / XLSX completion tables,
  4. survey files,
  5. spatial files,
  6. manual metadata entry.

Outputs:

  1. source bundle,
  2. raw object registry,
  3. upload session record.

Stage 2: Metadata Extraction

Tasks:

  1. detect file type,
  2. parse headers,
  3. extract identifiers,
  4. extract location fields,
  5. extract curve inventory,
  6. extract depth range,
  7. identify likely completion tables,
  8. assign preliminary authority ranking.

Outputs:

  1. extracted metadata fields,
  2. file profile,
  3. parse status.

Stage 3: Contract Validation

Validate:

  1. Contract A: identity + spatial anchor,
  2. Contract B: log run + curves,
  3. Contract C: completion + survey + context.

Outputs:

  1. pass / partial / fail per contract,
  2. promotion recommendation,
  3. initial gap list.

Stage 4: Gap Classification

Classify all gaps into domains:

  1. identity,
  2. location,
  3. CRS,
  4. depth,
  5. curve,
  6. completion,
  7. QC,
  8. chronology.

Each gap gets:

  1. severity,
  2. promotion action,
  3. source object link,
  4. recommended next step.

Stage 5: Canonical Well Resolution

Tasks:

  1. resolve aliases,
  2. match source records into one canonical well,
  3. assign preferred source by authority,
  4. retain conflicting alternatives for audit.

Outputs:

  1. canonical well record,
  2. alias records,
  3. source-to-canonical lineage.

Stage 6: Spatial Truth Normalization

Tasks:

  1. resolve source CRS,
  2. validate coordinate plausibility,
  3. standardize vertical datum,
  4. convert to WGS84,
  5. convert to ECEF,
  6. compute transform status and notes.

Outputs:

  1. preferred location record,
  2. transform provenance,
  3. WGS84 and ECEF values,
  4. spatial confidence state.

Stage 7: Depth And Trajectory Normalization

Tasks:

  1. parse deviation survey,
  2. compute TVD and TVDSS,
  3. mark whether minimum curvature was used,
  4. downgrade if only vertical assumption is possible.

Outputs:

  1. trajectory header,
  2. station table,
  3. depth reference status,
  4. TVDSS confidence state.

Stage 8: Curve Normalization

Tasks:

  1. standard mnemonic mapping,
  2. unit conversion,
  3. resampling,
  4. null cleanup,
  5. presence/absence analysis,
  6. QC summaries.

Outputs:

  1. normalized curve catalog,
  2. sample storage,
  3. curve-level QC.

Stage 9: Deterministic Interpretation

Tasks:

  1. compute Vsh,
  2. compute porosity metrics,
  3. compute or estimate Sw where justified,
  4. identify candidate intervals,
  5. assign interval-level QC.

Outputs:

  1. pay-event records,
  2. derived-preview curves,
  3. interpretation traceability.

Stage 10: Completion Reconciliation

Tasks:

  1. map perforated intervals,
  2. compare candidate intervals to tested intervals,
  3. remove clearly exhausted/already tested events,
  4. classify remaining intervals.

Outputs:

  1. bypassed-pay candidates,
  2. recommended next actions,
  3. action confidence.

Stage 11: Evidence And Audit

Tasks:

  1. collect source links,
  2. collect transform chain,
  3. collect formulas and cutoffs,
  4. collect output signatures,
  5. publish evidence pack.

Outputs:

  1. evidence pack,
  2. immutable audit entries,
  3. reproducibility record.

Stage 12: Scientist Review Queue

Tasks:

  1. rank wells by score and confidence,
  2. separate blocked wells from review-ready wells,
  3. expose top intervals and missing-data reasons,
  4. drive manual confirmation workflow.

Outputs:

  1. ranked review queue,
  2. promoted candidate list,
  3. blocked/quarantined list.

Module Architecture Design

Module 1: Source Intake Module

Purpose:

  1. accept files and bundles,
  2. land them in raw storage,
  3. preserve source integrity.

Why it exists:

  1. raw data must never be overwritten,
  2. uploads often contain mixed file types,
  3. paper and digital sources must coexist.

Primary storage targets:

  1. MinIO raw vault,
  2. raw.source_bundles,
  3. raw.source_objects.

Module 2: Metadata Extraction Module

Purpose:

  1. classify the source,
  2. extract all detectable header and structural information,
  3. prepare the source for contract validation.

Why it exists:

  1. every downstream module depends on knowing what the file actually is,
  2. partial metadata is still useful,
  3. OCR and digital parse can be handled through the same field registry.

Primary outputs:

  1. extracted metadata fields,
  2. curve inventory,
  3. document profile,
  4. parse confidence.

Module 3: Contract Validation Module

Purpose:

  1. separate usable, partial, and unusable data,
  2. prevent silent promotion of bad wells.

Why it exists:

  1. most failures happen before interpretation,
  2. missing CRS or completions must change behavior,
  3. well identity errors are fatal.

Primary outputs:

  1. contract status by domain,
  2. validation errors,
  3. promotion recommendation.

Module 4: Gap Engine

Purpose:

  1. store what is missing or ambiguous in structured form,
  2. drive quarantine and review rules.

Why it exists:

  1. old wells are rarely complete,
  2. data gaps must be queryable,
  3. review queues depend on gap severity.

Primary outputs:

  1. ops.well_data_gaps,
  2. domain, severity, action,
  3. gap summary for UI and reports.

Module 5: Canonical Well Resolver

Purpose:

  1. unify aliases and repeated sources into one canonical well.

Why it exists:

  1. multiple files often refer to the same well differently,
  2. completion history and logs must be linked to the same well,
  3. the scientist should review one well record, not twenty fragments.

Primary outputs:

  1. canonical well row,
  2. alias table,
  3. source lineage map.

Module 6: Spatial Truth Module

Purpose:

  1. normalize horizontal and vertical references,
  2. compute WGS84 and ECEF,
  3. quarantine bad transforms.

Why it exists:

  1. 2D/3D alignment depends on reliable spatial anchoring,
  2. field-scale comparison fails if datum handling is wrong,
  3. old wells often have projection mistakes.

Primary outputs:

  1. preferred location record,
  2. transform provenance,
  3. geodetic and ECEF coordinates.

Module 7: Trajectory And Depth Module

Purpose:

  1. transform MD into TVD/TVDSS when valid,
  2. represent the well path in canonical depth terms.

Why it exists:

  1. bypassed-pay comparisons require interval consistency,
  2. deviated wells cannot be treated as vertical,
  3. many legacy wells have incomplete survey support.

Primary outputs:

  1. trajectory record,
  2. survey stations,
  3. depth reference status,
  4. uncertainty notes.

Module 8: Curve Normalization Module

Purpose:

  1. convert raw log curves into a deterministic canonical set.

Why it exists:

  1. mnemonics vary,
  2. units vary,
  3. null handling varies,
  4. interpretation requires consistent sampling.

Primary outputs:

  1. curve catalog,
  2. normalized curve values,
  3. curve-level QC summary.

Module 9: Interpretation Module

Purpose:

  1. run deterministic petrophysical workflows.

Why it exists:

  1. first-pass payload detection must not depend on black-box AI,
  2. scientists need explainable outputs,
  3. formulas and cutoffs must be auditable.

Primary outputs:

  1. pay-event rows,
  2. interpretation summaries,
  3. trace signatures.

Module 10: Completion Reconciliation Module

Purpose:

  1. classify whether a technically good interval is actually bypassed.

Why it exists:

  1. a promising interval is not payload if it was already completed and tested,
  2. completion history changes actionability.

Primary outputs:

  1. bypassed-pay candidates,
  2. scores,
  3. recommendations,
  4. next-action list.

Module 11: Evidence Pack Module

Purpose:

  1. assemble the full chain from raw source to output.

Why it exists:

  1. outputs must be reproducible,
  2. disagreements must be traceable,
  3. commercial trust depends on auditability.

Primary outputs:

  1. evidence pack record,
  2. hash-linked references,
  3. delivery-ready JSON / markdown packages.

Module 12: Scientist Work Queue Module

Purpose:

  1. route the best opportunities to experts.

Why it exists:

  1. not every well deserves the same human attention,
  2. the system must reduce workload, not increase it,
  3. payload identification is a ranking problem, not only a detection problem.

Primary outputs:

  1. ranked wells,
  2. ranked intervals,
  3. blocked wells,
  4. required follow-up actions.

Recommended Data Model Separation

Use four logical layers:

  1. core
  1. raw
  1. ops
  1. audit

This is the correct shape for PostgreSQL because it keeps:

  1. raw immutable,
  2. canonical normalized,
  3. derived outputs explicit,
  4. audit separate and defensible.

Bulletproof Operating Rule

There is no bulletproof geology.

There is a bulletproof workflow.

That workflow is:

  1. preserve raw,
  2. extract and classify,
  3. validate contracts,
  4. quantify gaps,
  5. normalize spatial truth,
  6. normalize curves,
  7. compute deterministic intervals,
  8. reconcile with completion history,
  9. rank review payload,
  10. publish evidence-backed outputs.

The system should never silently upgrade a well from uncertain to decision-grade.