Skip to main content

LOE COE (Likelihood of Event & Consequence of Event)

Complete Documentation​

Document Version: 2.0 Last Updated: October 27, 2025 Location: General Repository / dev / AI Compiled From: Projects and General repository documentation, developer logs, code analysis

Documentation Creation Process​

This comprehensive documentation was created using Cursor AI (an AI-powered code editor) to systematically analyze and consolidate all LOE COE related information across two repositories:

Source Repositories Analyzed​

  1. Projects Repository (C:\Git\gqc\projects)

    • Primary source: internal\Pipe Breaks\ directory
    • Contains: Technical documentation, setup guides, algorithm details, database schemas
    • Key files: 02_loe-coe-update_7_29_2024.md, readme.md, training-and-predictions.md, 01_setup.md

  2. General Repository (C:\Git\gqc\general)

    • Primary source: Developer logs, daily updates, troubleshooting guides
    • Contains: people\Vishwanatha\Loe-Coe\, people\Srujana\2023\august\, people\Pavan\Daily Updates\
    • Key files: loe-coe-comprehensive.md, how to run loe-coe.md, pipe-breaks-in-colab.md

Creation Methodology​

Step 1: Comprehensive Search

  • Used Cursor's AI to search both repositories for all LOE COE related content
  • Identified 20+ relevant files across multiple directories
  • Analyzed developer logs, setup guides, troubleshooting notes, and technical documentation

Step 2: Content Analysis

  • Extracted key information from each source file
  • Identified overlapping content and unique insights
  • Categorized information by topic (setup, algorithms, issues, history, etc.)

Step 3: Intelligent Consolidation

  • Combined related topics from both repositories
  • Eliminated duplication while preserving all unique information
  • Organized content into logical sections with clear hierarchy

Step 4: Enhanced Organization

  • Created 5 major parts with 20 detailed sections
  • Added visual elements (Mermaid diagrams, tables, status indicators)
  • Included practical quick reference cards and troubleshooting guides

Tools and Techniques Used​

  • Cursor AI: Primary tool for content analysis and document creation
  • Semantic Search: Found relevant content across both repositories
  • Code Analysis: Analyzed actual code files and configuration
  • Documentation Synthesis: Combined technical depth with practical guidance
  • Visual Enhancement: Added diagrams, tables, and formatting for clarity

Result​

This document represents the most comprehensive LOE COE documentation available, combining:

  • Technical depth from the Projects repository
  • Practical insights and troubleshooting from developer logs
  • Complete setup and usage instructions
  • Historical development context
  • Current issues and future roadmap

The documentation serves as a single source of truth for the entire LOE COE project ecosystem.

Table of Contents​

Part I: Project Overview

  1. Introduction & Definitions
  2. System Architecture
  3. Repository Structure

Part II: Getting Started 4. Environment Setup 5. Installation Guide 6. Running the Application

Part III: Application Details 7. Application Features & UI 8. Machine Learning Algorithms 9. Data Processing Workflow 10. Databases

Part IV: Development 11. Dependency Management 12. Development Workflow 13. Technical Implementation 14. MSI Server & Infrastructure 15. Google Colab Integration

Part V: Reference 16. Known Issues & Solutions 17. Development History 18. Team Contributions 19. Future Improvements 20. Appendices

Part I: Project Overview

1. Introduction & Definitions​

What is LOE COE?​

LOE COE is a comprehensive machine learning system for predicting water main pipe breaks and assessing their impact on infrastructure.

LOE (Likelihood of Event)​

Definition: The probability that a pipe break will occur.

Methodology: Uses survival analysis and machine learning to analyze:

  • Historical pipe break data
  • Pipe characteristics (age, material, diameter, installation date)
  • Environmental factors
  • Temporal and spatial patterns

Output: Partial hazards or risk scores for each pipe segment indicating relative likelihood of failure.

COE (Consequence of Event)​

Definition: The impact or severity if a pipe break were to occur.

Methodology: Assesses consequences by:

  • Identifying affected buildings and infrastructure
  • Calculating properties that would lose water service
  • Evaluating critical infrastructure dependencies
  • Assessing service disruption impacts

Output: List of affected buildings and infrastructure for any water main segment.

Project Purpose​

  • Predict pipe breaks using historical data and infrastructure characteristics
  • Analyze likelihood-of-event scenarios for water main failures
  • Evaluate consequence-of-event impacts on surrounding infrastructure
  • Implement transfer learning to apply models across utilities

2. System Architecture​

Three Main Components​

Component 1: pipe-breaks​

Purpose: Foundational ML project for training and testing

  • Repository: github.com/gqc/pipe_breaks
  • Function: Algorithm development, model training, data processing
  • Command: python script_runner.py -h
  • Key Responsibility: Initial data processing and model training

Component 2: loe-coe-app​

Purpose: Streamlit-based user interface

  • Repository: github.com/gqc/loe_coe_app (Development: deven-gqc/loe-coe-app)
  • Function: User-friendly interface for LOE/COE analysis
  • Command: streamlit run app.py
  • Key Responsibility: User interaction and predictions
  • Features: Model training, prediction, visualization, scenario management

Component 3: pipe-breaks-transfer-learning​

Purpose: Transfer learning implementation

  • Repository: github.com/gqc/pipe-breaks-transfer-learning
  • Function: Cross-utility model application
  • Command: streamlit run app.py
  • Key Responsibility: Transfer learning and enhanced testing

Application Flow​

3. Repository Structure​

Main Repositories​

RepositoryDescriptionDevelopment MachineEnvironmentCommand
pipe-breaksOriginal pipe breaks projectHP Laptop / MSIpipe_breaks (Py 3.8.10)
pipe-breaks (Py 3.9.16)		python script_runner.py -h
loe-coe-app_deven-forkLikelihood-of-Event app forkHP Laptop / MSIpipe_breaks (Py 3.8.10)
pipe-breaks (Py 3.9.16)		streamlit run app.py
pipe-breaks-transfer-learningTransfer learning implementationHP Laptop / MSIpipe_breaks_transfer_learning (Py 3.8.10)streamlit run app.py

Code Structure​

pipe_breaks/
├── algorithms/
│   ├── cox_ph.py
│   ├── weibull_ph.py
│   ├── kaplanmeier.py
│   ├── random_survival_forest_GQC.py
│   ├── xg_boost.py
│   ├── xgbse_gqc.py
│   ├── lightgbm.py
│   └── test_processing.py
├── common/
│   ├── buildings_affected.py (COE)
│   ├── create_and_clean.py
│   ├── db_connection.py
│   ├── functions.py
│   ├── plotting.py
│   ├── processing.py (LOE)
│   ├── spacetime_processing.py
│   └── sqlconnection.py
├── script_runner.py (Entry Point)
├── settings.py (Global Config)
├── constants_gqc.py
└── [algorithm]_settings.py

Documentation Locations​

Local Documentation:

  • Projects Repo: ./internal/Pipe Breaks/
    • readme.md, training-and-predictions.md
    • 02_loe-coe-update_7_29_2024.md
    • 01_setup.md, dnv.md, task_list.md
  • General Repo: people/Vishwanatha/Loe-Coe/
    • Comprehensive setup and troubleshooting guides

Remote Documentation:

  • GitHub: https://github.com/gqc/pipe_breaks/tree/master/docs
  • Sphinx Documentation: loe-coe-app (gh-pages branch)

Part II: Getting Started

4. Environment Setup​

Prerequisites​

  • Python Version: 3.8.10 recommended (3.8.19 on MSI)
  • Operating System: Ubuntu/Linux (WSL on Windows)
  • Git Access: SSH keys configured for GitHub
  • Database Access: SSH keys for MSI server

Python Virtual Environments​

Environment NamePython VersionPrimary UseKey Packages
pipe-breaks383.8.10 / 3.8.19LOE-COE app, pipe-breaksscikit-learn=1.2.2
pipe_breaks3.8.10Original pipe breaksscikit-learn=1.2.2
pipe_breaks_transfer_learning3.8.10Transfer learningscikit-learn=0.24.2

Note: Python 3.8.10 recommended for maximum compatibility.

Creating Virtual Environment​

# Using virtualenvwrapper
mkvirtualenv pipe-breaks38

# Activate environment
workon pipe-breaks38

5. Installation Guide​

Step 1: Clone Repository​

# Navigate to git directory
cd /home/gqc/git/gqc

# Clone the loe_coe_app repository
git clone git@github.com:gqc/loe_coe_app.git

# Note: Main development in fork: deven-gqc/loe-coe-app

Step 2: Switch to Development Branch​

cd loe_coe_app
git checkout dev

Step 3: Install Requirements​

Important: The repository has TWO requirements files

# Install main requirements
pip install -r requirements.txt

# Install additional requirements
pip install -r reqs.txt

Known Issues:

  • reqs2.txt exists but purpose unclear
  • requirements.txt contains Windows-specific packages (remove on Linux):
    • pywin32==227
    • pywinpty==2.0.5

Step 4: Download Databases​

# Ensure WSL has SSH key to MSI server
scp msi:/home/gqc/git/gqc/loe_coe_app_deven_fork/data .

Downloads:

  • dnv_coe.db (LOE database - should rename to dnv_loe.db)
  • dnv_coe_buildings.db (COE buildings database)

Step 5: Run Application​

streamlit run app.py

Access at http://localhost:8501

6. Running the Application​

Method 1: Streamlit UI (app.py)​

Quick Start​

# 1. Activate environment
workon pipe-breaks38

# 2. Navigate to project
cd /home/gqc/git/gqc/loe_coe_app_deven_fork

# 3. Run Streamlit
streamlit run app.py

UI Components​

Sidebar Options:

  • Algorithm Selection: cox_ph, weibull_ph, kaplanmeier, xgbse_gqc, lightgbm, random_survival_forest_GQC, xg_boost
  • Utility Selection: dnv, cov, cgy, eve
  • Method Selection: Train, Predict

Available Buttons (9 total):

  1. Create Shell Script
  2. Find buildings affected
  3. Check db connection
  4. Clean data
  5. View Params
  6. Calculate results
  7. View results
  8. Scenario Table
  9. View Scenario Table

Method 2: Script Runner (script_runner.py)​

Command Line Execution​

# 1. Activate environment
workon pipe-breaks38

# 2. Navigate to project
cd /home/gqc/git/gqc/pipe_breaks

# 3. View help
python script_runner.py -h

# 4. Run specific algorithm and utility
python script_runner.py -s cox_ph -u dnv
python script_runner.py -s xgbse_gqc -u dnv

Script Runner Arguments​

  • -s, --scripts: Algorithm to run (required)
  • -d, --directory: Directory (optional, default=algorithms)
  • -u, --utility: Utility name (optional, default=settings.UTILITY)

Method 3: VSCode Debugging​

Launch Configuration (.vscode/launch.json)​

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "script_runner.py",
            "cwd": "${workspaceFolder}",
            "console": "integratedTerminal",
            "args": ["-s", "cox_ph", "-u", "dnv"],
            "justMyCode": true
        }
    ]
}

To Debug:

  1. Open script_runner.py in VSCode
  2. Set breakpoints
  3. Press F5 to start debugging

Note: Debugging in pipe-breaks repo is easier than Streamlit due to multithreading issues.

Part III: Application Details

7. Application Features & UI​

Feature 1: Create Shell Script (Cluster Computing)​

Purpose: Generate SLURM scripts for training on compute clusters

Function: create_shell_script()

Inputs:

  • Nodes, GPUs, tasks
  • Account credentials
  • Mail type and email
  • Algorithm selection
  • Utility selection

Output: Shell script file with SLURM directives

Example SLURM Script:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=v100l:4
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24
#SBATCH --account=def-blence
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@email.com
nvidia-smi
python script_runner.py -s xgbse_gqc -u dnv

Feature 2: Find Buildings Affected (COE Analysis)​

Purpose: Identify buildings impacted by a water main break

Function: water_main() → buildings_affected_by_watermain(asset_id)

Database: dnv_coe_buildings.db

Process:

  1. User inputs Asset ID (water main identifier)
  2. SQL queries retrieve buildings served by water main
  3. Results merged and displayed
  4. Shows list of affected properties and infrastructure

Feature 3: Database Connection Check (LOE)​

Purpose: Verify database connectivity

Function: check()

Database: dnv_coe.db

Output: Success or failure message

Feature 4: Create and Clean Dataset (LOE)​

Purpose: Prepare data for survival analysis

Function: dataset_for_surv_analysis()

Database: dnv_coe.db

Process:

  1. Check if survival table exists
  2. Load data from table
  3. Clean and process dataset
  4. Handle missing values
  5. Calculate time-to-event
  6. Merge replacement dates from CSV
  7. Calculate duration using study period settings
  8. Write processed data to survival table
  9. Optionally save to CSV

Feature 5: View Model Parameters​

Purpose: Display current model configuration

Function: view_params()

Output: Predefined model parameters in markdown format

Feature 6: Calculate Results (Model Execution)​

Purpose: Execute model training/prediction

Function: config(args)

Status: ⚠️ Currently not working (requires debugging)

Process:

  • Display processing spinner
  • Load configuration file (.toml) for utility
  • Execute model calculations

Workaround: Use script_runner.py directly

Feature 7: View Results (Visualization)​

Purpose: Generate and display scatter plots

Function: make_scatter(args)

Output: Visualizations of pipe break predictions and partial hazards

Feature 8: Scenario Table (Run Management)​

Purpose: Create new analysis scenario entries

Function: scenarios(args)

Process:

  1. Retrieve current run ID from database
  2. Initialize to 1 if none exists, otherwise increment
  3. Insert new scenario entry with run ID, status, parameters

Feature 9: View Scenario Table​

Purpose: Display recent analysis scenarios

Function: view_scenarios(args)

Output: Last 5 entries from scenarios table

8. Machine Learning Algorithms​

Supported Algorithms​

The LOE COE system supports 8 survival analysis and ML algorithms:

1. Cox Proportional Hazards (Cox-PH)​

  • Module: pipe_breaks.algorithms.cox_ph
  • Type: Survival regression
  • Library: lifelines
  • Output: Partial hazards for each pipe
  • Reference: Lifelines Cox PH

2. Weibull Proportional Hazards​

  • Module: pipe_breaks.algorithms.weibull_ph
  • Type: Parametric survival model
  • Library: lifelines

3. Kaplan-Meier​

  • Module: pipe_breaks.algorithms.kaplanmeier
  • Type: Non-parametric survival analysis
  • Library: lifelines
  • Use: Basic survival curve estimation

4. Random Survival Forest​

  • Module: pipe_breaks.algorithms.random_survival_forest_GQC
  • Type: Ensemble survival model
  • Use: Tree-based survival analysis

5. XGBoost​

  • Module: pipe_breaks.algorithms.xg_boost
  • Type: Gradient boosting
  • Best Practices:
    • Add W&B (Weights & Biases) for tracking
    • Plot losses with validation set
    • Configure W&B sweep for hyperparameter tuning
    • Use native saving (don't pickle)

6. LightGBM​

  • Module: pipe_breaks.algorithms.lightgbm
  • Type: Gradient boosting
  • Applications: Concordance indices across utilities
  • Best Practices: Same as XGBoost

7. XGBSE (XGBoost Survival Embeddings)​

  • Module: pipe_breaks.algorithms.xgbse_gqc
  • Type: Gradient boosting for survival analysis
  • Status: ⚠️ No longer maintained (no activity since Mar 2022)

8. NMTLR (Neural Multi-Task Logistic Regression)​

  • Status: ⚠️ No longer maintained (no updates since Apr 2019)
  • Note: Need alternatives

XGBSE Algorithm Data Flow​

XGBSE Process:

  1. Import: Data from URI_SQLITE_DB = "data/dnv_coe.db"
  2. Select: Necessary columns based on settings
  3. Split: Features (X) and Target (y: duration + event)
  4. Split: Training and validation sets
  5. Convert: To DMatrix format (Dtrain, Dval)
  6. Train: Model using training data
  7. Predict: Survival probabilities on validation
  8. Evaluate: Calculate concordance index

Performance Metrics​

Concordance Index (C-Index):

  • Measures how well predictions rank actual outcomes
  • Also called Harrell's C-index
  • Higher values indicate better discrimination
  • Primary metric for survival analysis evaluation

9. Data Processing Workflow​

Input Data Requirements​

Two Primary Sources:

  1. Breaks Data: Historical record of pipe failures
  2. Mains Data: Pipe infrastructure characteristics

Format: Standardized format per "Standard Table Creation" document

Processing Pipeline​

Step 1: Data Loading​

  • Function: processing.load_dataset()
  • Source: Databases or CSV files

Step 2: Data Cleaning​

  • Function: processing.dataset_for_surv_analysis()
  • Actions:
    • Remove duplicates (drop_dup_keep_highest())
    • Handle missing values
    • Convert data types
    • Standardize formats (diameter_conversion())

Step 3: Time-to-Event Calculation​

  • Function: calculate_days_num()
  • Purpose: Duration from installation to break/censoring
  • Function: days_between() - Calculate time differences

Step 4: Feature Engineering​

  • dataset_for_break_date_greater_0_cluster() - Spatial clustering
  • dataset_for_num_prev_breaks_greater_0() - Historical break counting
  • dataset_for_num_prev_breaks_greater_0_cluster() - Clustered break history

Step 5: Survival Data Creation​

  • Function: process_full_dataset()
  • Output:
    • Time-to-event (duration)
    • Event indicator (break occurred: yes/no)
    • Covariates (features)
    • Pipe identifiers

Train-Test Split Methodology​

Critical: Temporal nature of data

❌ Original Approach (Problematic):

  • Random split of dataset
  • Issue: Can use future data to predict past events

✅ Recommended Approach:

  1. Sort dataset by install_date
  2. Take first 70-80% for training
  3. Use remaining 20-30% for testing
  4. Ensure test period includes breaks
  5. Check distribution of breaks vs. non-breaks
  6. Apply SMOTE if class imbalance significant

Imbalance Handling:

  • SMOTE: Synthetic Minority Over-sampling Technique
  • Alternatives: AdaSyn, Border SMOTE
  • Modern: Adjust loss function weights

Data Files Generated​

From Processing:

  1. {utility}_mains_w_survival_data.csv
  2. {utility}_output_dataframe.csv
  3. {utility}_output_dataframe_filtered.csv

After Algorithm Execution: 4. {utility}_{algorithm}_partial_hazards.csv

After Comparison: 5. {utility}_{algorithm}_partial_hazards_main_id.csv

10. Databases​

Database 1: LOE Database (dnv_coe.db)​

⚠️ Note: Should be renamed to dnv_loe.db

Purpose: Stores pipe infrastructure and survival analysis results

Connection: URI_SQLITE_DB = "data/dnv_coe.db"

Key Tables:

survival Table:

  • Pipe characteristics (material, diameter, length, installation date)
  • Break history and time-to-event data
  • Covariates and features for model training
  • Duration calculations for survival analysis

scenarios Table:

  • Run ID for each analysis execution
  • Parameters used in each run
  • Status indicators
  • Timestamp information

Database 2: COE Database (dnv_coe_buildings.db)​

Purpose: Maps water mains to affected buildings

Connection: URI_BLGS_DB = "data/dnv_coe_buildings.db"

Key Tables:

  • Buildings information
  • Water main relationships
  • Service connections
  • Spatial relationships between pipes and buildings

Database ERD Diagrams​

LOE Database Schema (image-8.png):

  • Survival table with pipe and break data
  • Scenarios table for run tracking

COE Buildings Database Schema (image-7.png):

  • Buildings and properties
  • Water main service relationships

Part IV: Development

11. Dependency Management​

Critical Issue: scikit-survival​

The project faces significant dependency conflicts:

Issue 1: Version 0.13.0 (Required for Transfer Learning)​

  • ❌ Fails to install on any Python version
  • ❌ Causes build errors
  • ❌ Requires scikit-learn <= 0.24.0

Issue 2: scikit-learn Compatibility​

  • scikit-learn <= 0.24.0 requires Python < 3.10
  • Creates conflicts with other packages
  • Incompatible with newer Python versions

Issue 3: Manual Downgrade​

  • Downgrade to scikit-survival == 0.20.0 + scikit-learn <= 0.24.0
  • Results in missing methods and API changes
  • Transfer learning breaks

Issue 4: Modern Version Trade-offs​

  • scikit-learn > 1.0.0 allows Streamlit app to run
  • But completely disables transfer learning

Dependency States Comparison​

ConfigurationStreamlit StatusTransfer LearningNotes
scikit-survival 0.13.0 + sklearn ≤0.24.0CrashesN/ABuild errors
scikit-survival 0.20.0 + sklearn ≤0.24.0Runs with errorsBrokenAPI changes
sklearn >1.0.0 (no scikit-survival)✅ Runs❌ Not supportedRecommended

For LOE-COE App (Current Best):

Python==3.8.10
scikit-learn==1.2.2
# Transfer learning not available

For Transfer Learning (If Needed):

Python==3.8.10
scikit-learn==0.24.2
scikit-survival==0.20.0
# Expect API compatibility issues

Requirements Files​

Three Files:

  1. requirements.txt: Main dependencies (comprehensive, may include unnecessary packages)
  2. reqs.txt: Additional requirements
  3. reqs2.txt: Unknown purpose (in uncommitted changes)

Known Problems:

  • Windows-specific packages fail on Linux: pywin32==227, pywinpty==2.0.5
  • May contain unnecessary packages
  • Likely created by exporting all system packages

Version Compatibility Best Practices​

  1. ✅ Always use Python 3.8.10 for maximum compatibility
  2. ✅ Match Python versions between training and testing
  3. ✅ Verify environment selection in VSCode (bottom right)
  4. ✅ Document any version changes
  5. ✅ Test in clean virtual environment before deployment

12. Development Workflow​

Repository Connection Structure​

github.com/gqc/loe_coe_app  (Main repository)
    ↑
    | (merge from)
    |
deven-gqc/loe-coe-app  (Development fork)
    ↑
    | (local clone)
    |
loe_coe_app_deven_fork  (MSI machine: master branch)

Development Practice:

  1. Changes made in Deven's fork: deven-gqc/loe-coe-app
  2. Changes merged to main: github.com/gqc/loe_coe_app
  3. Local work on MSI: loe_coe_app_deven_fork

Development Standards​

Code Import Standards​

✅ Accepted:

import settings

❌ Not Accepted:

import settings as S
from settings import *

Exception (if nesting > 3 levels):

import settings as ST

Path Handling​

❌ BAD (causes cross-OS issues):

path = "data/" + filename

✅ GOOD:

from pathlib import Path
path = Path("data") / filename

# OR
import os.path
path = os.path.join("data", filename)

Best Practices for Algorithm Development​

  1. ✅ Use processing.dataset_for_surv_analysis() as input
  2. ✅ Sort dataset by install_date before splitting
  3. ✅ Temporal split: first 70-80% train, rest test
  4. ✅ Verify test period contains breaks
  5. ✅ Handle class imbalance with SMOTE
  6. ✅ Set random seeds for reproducibility
  7. ✅ Document parameters in settings files

13. Technical Implementation​

Key Functions and Modules​

Script Runner (script_runner.py)​

Central execution script for algorithms.

Functions:

  • parse_args(): Parse command line arguments
  • main(): Run specified script with parsed config

Arguments:

  • -s, --scripts: Script to run (required)
  • -d, --directory: Directory (optional, default=algorithms)
  • -u, --utility: Utility name (optional)

Processing Module (common/processing.py)​

Core data processing functions.

Key Functions:

  • dataset_for_surv_analysis(): Create survival analysis dataset
  • process_full_dataset(): Complete dataset processing
  • all_unbroken_pipes(): Filter for pipes without breaks
  • calculate_days_num(): Calculate time durations
  • diameter_conversion(): Standardize diameters
  • days_between(): Calculate date differences
  • load_dataset(): Load data from sources

Buildings Affected Module (common/buildings_affected.py)​

COE analysis functions.

Key Functions:

  • buildings_affected_by_watermain(asset_id): Find affected buildings
  • main(): Execute COE analysis

Database Connection Module (common/db_connection.py)​

Function: check() - Test database connection

Plotting Module (common/plotting.py)​

Functions:

  • make_scatter(args): Create scatter plots
  • make_barplot(): Create bar plots
  • make_heatmap(): Create heatmap visualizations

Functions Utilities (common/functions.py)​

Key Functions:

  • drop_dup_keep_highest(df): Remove duplicates, keep max scores
  • get_grid_row_col(gridnum, columns): Calculate grid positions
  • print_metrics(y, y_pred): Display performance metrics
  • read_data(csv_path, x_cols, y_col): Load data with column selection
  • write_csv_df(path, filename, df, index=False): Safe CSV writing

SQL Connection Module (common/sqlconnection.py)​

Function: get_connection(path: str) - Get cached database connection Note: Uses caching, handles thread-safety for Streamlit

Configuration Files​

  • .toml files: Utility-specific configurations (dnv.toml, cov.toml, cgy.toml, eve.toml)
  • settings.py: Global project settings
  • [algorithm]_settings.py: Algorithm-specific settings

Model Training Workflow​

Available Pre-trained Models​

Vannary provided models for multiple utilities:

  • Calgary (CGY)
  • Everett (Eve)
  • District of North Vancouver (DNV)
  • City of Vancouver (COV)

Two Configurations:

  1. Base covariates only
  2. Base + additional covariates

Location: Google Drive > VS Research > pipe_breaks > saved_models

Training Flow Evolution​

Old Flow:

New Flow (in development):

14. MSI Server & Infrastructure​

Server Overview​

The MSI server hosts the main LOE COE development environment.

Python Installations​

  • 3.8.19 ✅
  • 3.9.19
  • 3.10.12

Virtual Environments​

  • pipe-breaks38 (Used for LOE COE) ✅
  • cctv-apps
  • compute-msi
  • cuda
  • gqc-utility-notebooks
  • gqc-utility-python

Git Repositories on MSI​

Location: /home/gqc/git/gqc/

LOE COE Related:

  • loe_coe_app_deven_fork/ ✅
  • pipe_breaks/ ✅
  • pipe-breaks-transfer-learning/ ✅

Other:

  • basic_streamlit_app/
  • cctv-apps/
  • compute-canada/
  • compute-msi/
  • gqc-utility-notebooks/
  • gqc-utility-python/
  • simple-image-server/

Storage Configuration​

Internal Storage​

2 TB NVMe SSD - WD Green SN350:

  • Original SSD (reformatted from Windows)
  • ext4 filesystem
  • Mounted to /data0

2 TB NVMe SSD - WD_BLACK SN850X:

  • System drive (Ubuntu dual boot)

External Storage​

10 TB HDD (unionsine1): Contains VS_Research directory:

VS_Research/
├── loe_coe_app/
├── pipe_breaks/
├── CCTV/
├── 01 Flow path/
├── 04 COF for COV/  (Consequence of Failure)
├── 06 DNV/
└── [other research folders]

10 TB HDD (unionsine2):

  • Additional storage

5 TB HDD (WD_BLACK):

  • Contains calgary_pipe_breaks_models/

Accessing MSI Resources​

SSH Connection​

ssh gqc@msi
# OR
ssh msi

Downloading Data​

# Copy databases to local machine
scp msi:/home/gqc/git/gqc/loe_coe_app_deven_fork/data .

# Copy specific files
scp msi:/path/to/file /local/destination/

15. Google Colab Integration​

Prerequisites​

  1. Personal Access Token for GitHub

  2. Google Colab Pro Subscription

    • Required for Terminal access
    • Better compute resources

  3. Hardware Accelerator

    • Select GPU when starting notebook
    • Algorithms benefit from GPU acceleration

⚠️ Warning: Colab uses compute credits. Disconnect runtime when finished.

Setup Process​

1. Clone Repository to Google Drive​

# Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')
# In Colab Terminal
cd /content/drive/MyDrive/your_directory
git clone https://github.com/gqc/pipe_breaks.git

2. Install Dependencies​

cd pipe_breaks
pip install -e .

3. Configure Settings​

  • Set up input files in input_files/ folder
  • Configure settings.py (utility, algorithm)

4. Run Script Runner​

python script_runner.py -h
python script_runner.py -s cox_ph -u dnv

Colab Best Practices​

  1. ✅ Always mount Google Drive first
  2. ✅ Select GPU accelerator for compute-intensive algorithms
  3. ✅ Disconnect runtime when finished (save credits)
  4. ✅ Commit changes regularly (avoid data loss)
  5. ✅ Use Personal Access Token for Git operations

Advantages & Limitations​

Advantages:

  • Access to GPU resources
  • No local environment setup
  • Can run long-running algorithms
  • Easy sharing with team

Limitations:

  • Compute credit system
  • Session timeouts for inactive notebooks
  • Internet dependency
  • Limited terminal access (Pro only)

Part V: Reference

16. Known Issues & Solutions​

Issue 1: Streamlit App Buttons Not Working​

Problem: App loads but buttons non-functional

Solution:

  1. Load deven-loe-coe-fork repository (not main)
  2. Create virtual environment with Python 3.8
  3. Install both requirements:
    pip install -r requirements.txt
    pip install -r reqs.txt

Issue 2: Calculate Results Button Not Working​

Problem: "Calculate Results" button doesn't work

Status: Known issue, requires debugging

Trigger: config(args) function

Workaround: Use script_runner.py directly

Issue 3: Missing Columns Error (COV Utility)​

Problem: KeyError for missing columns:

  • within_peat, grid_num, grid_num_breaks
  • upper_ph, lower_ph, upper_cond, lower_cond

Solution:

necessary_columns = [
    'within_peat', 'grid_num', 'grid_num_breaks',
    'upper_ph', 'lower_ph', 'upper_cond', 'lower_cond'
]
for col in necessary_columns:
    if col not in data.columns:
        data[col] = 0  # or appropriate default

Issue 4: Database Naming Inconsistency​

Problem: dnv_coe.db contains LOE data, not COE

Solution:

mv data/dnv_coe.db data/dnv_loe.db

Update all references:

URI_SQLITE_DB = "data/dnv_loe.db"  # was dnv_coe.db

Issue 5: scikit-survival Installation Failures​

Problem: Multiple dependency conflicts

Solution Options:

Option A: LOE-COE App Only (No Transfer Learning):

pip install scikit-learn==1.2.2
# Skip scikit-survival

Option B: Transfer Learning (Limited Functionality):

pip install scikit-learn==0.24.2
pip install scikit-survival==0.20.0
# Expect API compatibility issues

Issue 6: Windows-Specific Package Failures​

Problem: Installation fails with Windows packages

Solution: Remove from requirements.txt:

  • pywin32==227
  • pywinpty==2.0.5

Issue 7: Shell Script New Line Bug​

Problem: Shell scripts have formatting issues

Status: Known, documented in TODO

Workaround: Manually fix after creation

Issue 8: TOML File Reference Issues​

Problem: Config files not found/incorrectly referenced

Workaround: Verify .toml file paths in settings

Issue 9: Transfer Learning Functionality​

Problem: Broken due to dependency conflicts

Current Situation:

  • Modern scikit-learn: Transfer learning not supported
  • Old scikit-learn: API compatibility issues

Status: Needs architectural refactoring

Issue 10: Environment Selection in VSCode​

Problem: VSCode may use wrong Python environment

Solution:

  1. Check bottom right corner of VSCode
  2. Click on Python version
  3. Select correct environment (e.g., pipe-breaks38)
  4. Reload window if necessary

Issue 11: Multithreading Debug Difficulty​

Problem: Streamlit multithreading makes debugging difficult

Solution: Debug in pipe-breaks repository:

  1. Clone pipe-breaks repository
  2. Run script_runner.py directly
  3. Use VSCode debugger with launch configuration
  4. Easier execution flow tracing

Active Issues from GitHub (as of May 5, 2025)​

Core Functionality (loe_coe_app repository):

  • #1: Algorithm Migration - Need to verify and complete migration
  • #2: Results Display - Showing incorrect algorithm outputs
  • #8: Dependencies Conflicts - Package version conflicts
  • #13: Database Connections - Need proper connection management

Architecture:

  • #14: Directory Structure - Needs reorganization for Sphinx docs
  • #15: Database ORM - Need to implement SQLAlchemy
  • #10: Output Management - Rework logic, create run_id-based folders

Environment:

  • #6: Colab Integration - Issues running in Google Colab
  • #7: Database Settings - Configuration management needed
  • #11: Environment Detection - Detect Compute Canada environment

Code Quality:

  • #20: Code Formatting - Indentation needs change (4→2 spaces)
  • #23: Deprecated Algorithms - xgbse, NMTLR need alternatives
  • #12: Execution Flow Documentation - Need IDEF0 diagram
  • #16: API Issues - Missing files/attributes, need requirements.txt

17. Development History​

December 2022: Initial Development​

Week 4 (Dec 19-23):

  • Streamlit app under development
  • Algorithm migration initiated
  • Shell script creation focus

Week 5 (Dec 26-30):

  • Created loe_utils package
  • XGBSE_GQC notebook debugging (settings.py issues)
  • LightGBM standalone notebook (runs with/without SMOTE)
  • Shell script generation prioritized
  • Development standards established
  • Figures display and disk writing implemented

July 2023: Development Continues​

Week 3-5:

  • Pipe breaks app meetings and discussions
  • UI refinements
  • Testing with multiple team members

August 2023: Team Onboarding​

Week 1 (July 31 - Aug 4):

  • Srujana onboarded
  • Understanding loe-coe, pipe breaks, transfer learning flow
  • Models trained on Calgary examined
  • Environment setup (HP laptop, WSL, MSI)
  • Cox-CGY and Cox-DNV models tested
  • Models from Vannary obtained
  • Documentation updates to projects site

Week 2:

  • XGBSE algorithm debugging
  • Environment synchronization planned
  • Data format standardization
  • pipe-breaks-transfer-learning environment created
  • Models successfully used between repositories

Week 3:

  • Added to JIRA for tracking

October 2023: Maintenance​

Week 3-4:

  • Pipe-breaks maintenance and updates
  • Chart cleanup with correct labels

November 2023: Documentation Push​

Week 1: Documentation tasks identified

Week 4 (Nov 27 - Nov 30):

  • Comprehensive documentation of uncommitted changes
  • Three repositories analyzed:
    • pipe-breaks
    • loe-coe-app_deven-fork
    • pipe-breaks-transfer-learning
  • Transfer learning notebooks discovered
  • New streamlit_helper.py found (incomplete)
  • Documentation requirements identified

July 2024: Major Documentation Effort​

Week 3 (July 15-19):

  • Repository relationships documented
  • Clear responsibility delineation

Week 4 (July 22-26):

  • VolView, VTK.js, GQC-VTK documentation
  • Climate model integration concepts
  • July 25: Major documentation gaps identified
    • Jake notes lack of LOE/COE definitions
    • Call for comprehensive app documentation
    • Current state: Only DNV utility supported, SQLite storage
  • July 26: Comprehensive UI documentation created
    • All 9 buttons documented
    • Input/Process/Output flows defined

Week 5 (July 29-31):

  • July 29: LOE COE deep dive review
    • Full application workflow documented
    • All 9 button functions described
    • Database ERD diagrams created
    • COV utility KeyError issue identified and resolved
  • July 30: Documentation consolidated and pushed
  • July 31: Final documentation edits

Key Milestones​

  1. 2022: Core algorithm implementation
  2. Late 2022 - Early 2023: Streamlit integration and UI development
  3. Mid 2023: Environment stabilization and dependency management
  4. Mid 2023: Transfer learning and cross-utility model application
  5. July 2024: Comprehensive documentation effort

18. Team Contributions​

Core Development Team​

Vishwanatha (Developer, Documentation)​

Contributions:

  • Comprehensive LOE COE deep dive documentation (July 2024)
  • XGBSE algorithm flow documentation
  • Script runner workflow diagrams
  • How-to guides for running application
  • Initial review and issue identification
  • COV utility debugging and fixes
  • Database ERD documentation

Key Documents:

  • loe-coe-initial-review.md
  • loe-coe-comprehensive.md
  • how to run loe-coe.md
  • script_runner.md
  • xgbse.md

Deven (Lead Developer, Architecture)​

Contributions:

  • Fork maintenance (deven-gqc/loe-coe-app)
  • Streamlit app architecture
  • Shell script generation functionality
  • loe_utils package creation
  • Development standards establishment
  • Code refactoring and optimization
  • Google Colab integration documentation

Repository: Primary development in deven-gqc/loe-coe-app

Srujana (Developer, Documentation)​

Contributions:

  • Environment setup documentation
  • Testing across multiple environments (HP laptop, WSL, MSI)
  • Model portability verification
  • Transfer learning testing
  • Requirements synchronization
  • XGBSE debugging assistance
  • Projects documentation site updates

Period: August 2023 onward

Pavan (Code Review, Documentation)​

Contributions:

  • Uncommitted changes documentation (November 2023)
  • Code review and analysis
  • Repository state documentation
  • Change tracking across three repositories
  • Sphinx documentation review

Key Document: Week 4 November 2023 documentation

Vannary (Research, Model Training)​

Contributions:

  • Model training on Compute Canada
  • Models for multiple utilities: Calgary (CGY), Everett (Eve), DNV, City of Vancouver (COV)
  • Base and additional covariate configurations
  • SLURM script examples
  • Research foundation for algorithms

Model Location: Google Drive > VS_Research > pipe_breaks > saved_models

Jake (Project Management, Infrastructure)​

Contributions:

  • MSI server configuration documentation
  • Storage management
  • SSH and access management
  • Gap identification in documentation
  • Requirement for clear LOE/COE definitions

Dhananjay (Developer, Setup)​

Contributions:

  • LOE-COE app setup on MSI 3 laptop
  • Environment configuration
  • Application testing
  • Troubleshooting assistance

Sudhir (Technical Direction)​

Contributions:

  • Model usage explanation
  • Architecture decisions
  • Development standards
  • Code review
  • Technical guidance

Additional Contributors​

  • Eric: Early algorithm development
  • Sudharshan: Neural network models, algorithm development

Collaboration Structure​

Development Fork Flow:

  • Main development: deven-gqc/loe-coe-app
  • Production: github.com/gqc/loe_coe_app
  • Merge process: Fork → Main repository

Communication Channels:

  • Documentation sites (general.gqc.com, projects.gqc.com)
  • JIRA for task tracking
  • Daily updates and logs
  • Meeting notes

19. Future Improvements​

1. Documentation​

  • ✅ Complete IDEF0 execution flow diagram
  • ✅ Expand algorithm best practices documentation
  • ✅ Create feature/covariate descriptions
  • ✅ Document all configuration parameters

2. Code Refactoring​

  • Implement SQLAlchemy ORM
  • Reorganize directory structure (Sphinx compatibility)
  • Apply consistent code formatting (2-space indentation)
  • Add comprehensive unit tests
  • Improve logging and error handling

3. Algorithm Maintenance​

  • Replace/update deprecated algorithms:
    • xgbse (no activity since Mar 2022)
    • NMTLR (no updates since Apr 2019)
  • Standardize algorithm interfaces
  • Implement W&B tracking for all algorithms
  • Add hyperparameter tuning pipelines

4. Environment Management​

  • Standardize Python version (3.8.10)
  • Create comprehensive requirements.txt
  • Implement environment detection (Compute Canada)
  • Support multiple execution environments

5. Database Improvements​

  • ✅ Rename dnv_coe.db to dnv_loe.db
  • Implement proper connection pooling
  • Add database migration scripts
  • Create backup and recovery procedures
  • Implement SQLAlchemy ORM

6. UI/UX Enhancements​

  • Fix "Calculate Results" functionality
  • Improve results display accuracy
  • Add better visualization options
  • Implement run comparison features
  • Support additional utilities beyond DNV

7. Output Management​

  • Implement run_id-based folder structure
  • Add output versioning
  • Improve scenario tracking
  • Create automated report generation

8. Model Improvements​

  • Implement fine-tuning capabilities
  • Add model versioning
  • Create model performance tracking
  • Support ensemble predictions

9. Reproducibility​

  • Set random seeds throughout
  • Version control datasets
  • Track all hyperparameters
  • Implement experiment tracking (W&B, MLflow)

10. Testing​

  • Create unit tests for all modules
  • Add integration tests
  • Implement continuous integration
  • Add data validation tests

Transfer Learning Refactoring​

  • Resolve dependency conflicts
  • Update to modern scikit-survival (if possible)
  • Complete streamlit_helper.py page
  • Fix API compatibility issues

Bug Fixes Priority​

  1. Fix "Calculate Results" button
  2. Resolve shell script formatting issues
  3. Fix TOML file references
  4. Fix Windows-specific package issues
  5. Resolve COV utility missing columns

20. Appendices​

Appendix A: Utilities Analyzed​

  1. DNV: District of North Vancouver (Primary development utility)
  2. COV: City of Vancouver
  3. CGY: Calgary
  4. Eve: Everett
  5. MASC: (Additional utility)

Appendix B: Configuration Files​

settings.py Structure​

Global configuration for pipe breaks project:

  • Utility selection (DNV, COV, CGY, EVE)
  • Study period (start and end dates)
  • Input file paths
  • Output file paths
  • Algorithm-specific parameters
  • Database connections

Algorithm-Specific Settings​

Each algorithm has its own settings file:

  • cox_ph_settings.py
  • weibull_ph_settings.py
  • kaplanmeier_settings.py
  • xgbse_settings.py
  • etc.

TOML Configuration Files​

Utility-specific configurations:

  • dnv.toml
  • cov.toml
  • cgy.toml
  • eve.toml

Appendix C: Common Error Messages​

Error: "ModuleNotFoundError: No module named 'sklearn.neighbors._dist_metrics'"​

Cause: scikit-learn version mismatch Solution: pip install scikit-learn==0.24.2

Error: "KeyError: 'within_peat'" (and other columns)​

Cause: Missing columns in dataset for specific utility Solution: See Issue 3 - Add default values for missing columns

Error: "Unable to install scikit-survival==0.13.0"​

Cause: Build failures on Python versions Solution: Use scikit-learn > 1.0.0 and skip transfer learning

Error: "Connection to database failed"​

Cause: Database files not present or incorrectly pathed Solution:

  1. Verify files exist in data/ directory
  2. Check URI_SQLITE_DB and URI_BLGS_DB variables
  3. Download from MSI server if missing

Appendix D: External Resources​

GitHub Repositories:

  • Main: github.com/gqc/pipe_breaks
  • Fork: deven-gqc/loe-coe-app
  • Transfer Learning: github.com/gqc/pipe-breaks-transfer-learning

Documentation Sites:

  • General: https://general.gqc.com
  • Projects: https://projects.gqc.com
  • CCTV: https://cctv.gqc.com

Model Storage:

  • Google Drive: VS_Research/pipe_breaks/saved_models/

External Libraries:

Related Projects:

  • HydroTrek: Water distribution modeling platform
  • CCTV: Sewer inspection analysis
  • DeepVibe: Vibration analysis for infrastructure
  • COF for COV: Consequence of Failure for City of Vancouver

Appendix E: Contact Information​

For questions or issues related to the Pipe Breaks LOE COE project, refer to:

  • contacts.md in projects repository
  • Development team members listed in Team Contributions

Quick Reference Card​

Common Commands​

# Activate environment
workon pipe-breaks38

# Run Streamlit app
streamlit run app.py

# Run script directly
python script_runner.py -s cox_ph -u dnv

# Download databases from MSI
scp msi:/home/gqc/git/gqc/loe_coe_app_deven_fork/data .

# SSH to MSI
ssh msi

Key Directories​

/home/gqc/git/gqc/
├── loe_coe_app_deven_fork/    # Streamlit app
├── pipe_breaks/                # Original project
└── pipe-breaks-transfer-learning/  # Transfer learning

Database Files​

data/
├── dnv_coe.db                  # LOE database (rename to dnv_loe.db)
└── dnv_coe_buildings.db        # COE buildings database

Configuration Files​

.
├── settings.py                 # Global settings
├── dnv.toml                    # DNV utility config
├── requirements.txt            # Main dependencies
└── reqs.txt                    # Additional dependencies

End of Documentation

For updates or corrections, please contact the development team or update this documentation file.

Last updated on by Viraj