Model Calibration Skill

Overview

Model calibration involves adjusting parameters to minimize differences between simulations and observations. RMSE (Root Mean Squared Error) is a standard metric for measuring model performance.

RMSE Calculation

Basic Formula

RMSE = sqrt(mean((simulated - observed)^2))

Implementation

import numpy as np
import pandas as pd

def calculate_rmse(simulated, observed):
    """Calculate RMSE between matched pairs"""
    if len(simulated) == 0:
        return np.nan
    residuals = simulated - observed
    rmse = np.sqrt(np.mean(residuals**2))
    return rmse

def calculate_metrics(sim_temps, obs_temps, depths=None, dates=None):
    """
    Calculate multiple RMSE metrics

    Parameters:
    - sim_temps: matched simulated temperatures
    - obs_temps: matched observed temperatures
    - depths: depth of each match (for deep water filtering)
    - dates: datetime of each match (for seasonal filtering)

    Returns:
    - overall_rmse: RMSE of all matched pairs
    - annual_deep_rmse: RMSE of pairs at depths >= 13m
    - summer_deep_rmse: RMSE of summer (Jun-Sep) pairs at depths >= 13m
    """

    overall_rmse = calculate_rmse(sim_temps, obs_temps)

    # Annual deep water (depths >= 13m)
    if depths is not None:
        deep_mask = np.array(depths) >= 13
        annual_deep_rmse = calculate_rmse(
            sim_temps[deep_mask],
            obs_temps[deep_mask]
        )
    else:
        annual_deep_rmse = np.nan

    # Summer deep water (Jun-Sep, depths >= 13m)
    if dates is not None and depths is not None:
        summer_mask = (np.array([d.month for d in dates]) >= 6) & \
                     (np.array([d.month for d in dates]) <= 9)
        deep_mask = np.array(depths) >= 13
        combined_mask = summer_mask & deep_mask
        summer_deep_rmse = calculate_rmse(
            sim_temps[combined_mask],
            obs_temps[combined_mask]
        )
    else:
        summer_deep_rmse = np.nan

    return {
        'overall_rmse': overall_rmse,
        'annual_deep_rmse': annual_deep_rmse,
        'summer_deep_rmse': summer_deep_rmse
    }

Matching Observations to Simulation

Critical: Use exact datetime + rounded-depth matching (no interpolation)

def match_obs_to_sim(obs_df, sim_temps, sim_z, sim_dates, round_depth=1):
    """
    Match observations to simulation output

    obs_df: DataFrame with columns [datetime, depth, temp]
    sim_temps: [time, depth] array
    sim_z: depth values
    sim_dates: datetime for each time step
    round_depth: rounding for depth matching

    Returns: matched simulation temps, observed temps, and metadata
    """
    import pandas as pd

    obs_df['depth_rounded'] = (obs_df['depth'] / round_depth).round() * round_depth

    matched_sim = []
    matched_obs = []
    matched_depths = []
    matched_dates = []

    for idx, row in obs_df.iterrows():
        obs_datetime = pd.Timestamp(row['datetime'])
        obs_depth = row['depth_rounded']
        obs_temp = row['temp']

        # Find exact datetime match
        time_matches = [i for i, d in enumerate(sim_dates)
                       if d == obs_datetime]

        # Find exact depth match
        depth_matches = [i for i, z in enumerate(sim_z)
                        if z == obs_depth]

        # Need both to match
        if time_matches and depth_matches:
            t_idx = time_matches[0]
            z_idx = depth_matches[0]
            sim_temp = sim_temps[t_idx, z_idx]

            matched_sim.append(sim_temp)
            matched_obs.append(obs_temp)
            matched_depths.append(obs_depth)
            matched_dates.append(obs_datetime)

    return (np.array(matched_sim), np.array(matched_obs),
            matched_depths, matched_dates)

Parameter Tuning Strategy

1. Sensitivity Analysis

Run model with one parameter varied at a time to identify most influential parameters.

2. Systematic Search

For 5 parameters with small ranges, can do grid search:

• Test 3-5 values per parameter
• Evaluate all combinations
• Select best performers

3. Iterative Refinement

• Start with literature values
• Adjust parameters that have largest impact on target RMSE
• Focus on worst-performing seasonal/depth bins

4. Constraint Checking

Always verify final parameters are within published ranges:

def validate_parameters(params):
    ranges = {
        'Kw': (0.1, 0.5),
        'coef_mix_hyp': (0.3, 0.7),
        'wind_factor': (0.7, 1.3),
        'lw_factor': (0.7, 1.3),
        'ch': (0.0005, 0.002)
    }
    for param, (min_val, max_val) in ranges.items():
        if not (min_val <= params[param] <= max_val):
            raise ValueError(f"{param} out of range: {params[param]}")
    return True

Evaluation Thresholds

For Lake Mendota:

• Overall RMSE < 1.60°C
• Annual deep (≥13m) RMSE < 1.55°C
• Summer deep (Jun-Sep, ≥13m) RMSE < 1.70°C

Model is successful when ALL three thresholds are met.