Code map¶
Eight stages, from command to report. Click a box to open the code; arrow labels name what is passed. Stage 0 is offline; stage 3 is DAM 1; stage 7 shows what DAM 2/3 change.
Stages¶
flowchart LR
st_prep["<b>0 · Before training (optional)</b><br/>run_whisper_asr.py<br/><i>optional</i>"]
class st_prep opt
click st_prep "code-map/#stage-prep"
st_launch["<b>1 · Launch</b><br/>launcher.py ·<br/>launcher_utils.py ·<br/>utils.py<br/><i>steps 1–14</i>"]
class st_launch main
click st_launch "code-map/#stage-launch"
st_data["<b>2 · Data</b><br/>datasets_and_dataloaders.py<br/>· dataset_utils.py ·<br/>augmentations.py ·<br/>constants.py<br/><i>steps 15–24</i>"]
class st_data main
st_launch ==> st_data
click st_data "code-map/#stage-data"
st_model["<b>3 · Model (DAM 1)</b><br/>mtl_mar_2024/model.py ·<br/>feature_extraction_whisper.py<br/>· base_models.py ·<br/>constants.py<br/><i>steps 25–31</i>"]
class st_model main
st_data ==> st_model
click st_model "code-map/#stage-model"
st_loss["<b>4 · Loss</b><br/>base_models.py · losses.py<br/>· utils.py<br/><i>steps 32–35</i>"]
class st_loss main
st_model ==> st_loss
click st_loss "code-map/#stage-loss"
st_metrics["<b>5 · Thresholds and metrics</b><br/>base_models.py ·<br/>ordinal_thresholding.py ·<br/>metrics.py<br/><i>steps 36–41</i>"]
class st_metrics main
st_loss ==> st_metrics
click st_metrics "code-map/#stage-metrics"
st_outputs["<b>6 · Outputs and analysis</b><br/>launcher_utils.py ·<br/>launcher.py · utils.py ·<br/>plotting.py<br/><i>steps 42–44</i>"]
class st_outputs main
st_metrics ==> st_outputs
click st_outputs "code-map/#stage-outputs"
st_variants["<b>7 · The other versions (DAM 2, DAM 3)</b><br/>mtl_jun_2024/model.py ·<br/>mtl_aug_2024/model.py ·<br/>base_models.py<br/><i>optional</i>"]
class st_variants opt
st_outputs -.-> st_variants
click st_variants "code-map/#stage-variants"
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
0 · Before training (optional)¶
Transcripts are produced once, outside training, and only matter for runs that read text: DAM 2, and config_llma_dam3.0.yaml. Current DAM 3 uses them only indirectly, through the cached DAM 2 embeddings.
flowchart LR
subgraph scripts_run_whisper_asr_py___box["run_whisper_asr.py"]
direction TB
scripts_run_whisper_asr_py__run_whisper_asr["<b>run_whisper_asr()</b><br/>Transcribe a 16 kHz WAV (optional<br/>start/end in s) with Whisper<br/>large-v3 to JSON."]
class scripts_run_whisper_asr_py__run_whisper_asr opt
click scripts_run_whisper_asr_py__run_whisper_asr "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/scripts/run_whisper_asr.py#L18" _blank
end
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| — | run_whisper_asr |
scripts/run_whisper_asr.py:18 |
Transcribe a 16 kHz WAV (optional start/end in s) with Whisper large-v3 to JSON. |
1 · Launch¶
One command reads a YAML config, validates it, builds the model and the data loaders, and hands everything to a PyTorch Lightning Trainer.
Main path: steps 1–14 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph scripts_launcher_py___box["launcher.py"]
direction TB
scripts_launcher_py__launcher["<b>1 · launcher()</b><br/>Run one mode<br/>(train/evaluate/cache/compile) end<br/>to end from a YAML config."]
class scripts_launcher_py__launcher main
click scripts_launcher_py__launcher "../walkthrough/#step-1"
scripts_launcher_py___setup_launcher["<b>2 · _setup_launcher()</b><br/>Load the YAML into LauncherConfig<br/>and create the W&B logger on rank<br/>0 only."]
class scripts_launcher_py___setup_launcher main
click scripts_launcher_py___setup_launcher "../walkthrough/#step-2"
scripts_launcher_py___setup_callbacks["<b>10 · _setup_callbacks()</b><br/>Best-checkpoint saver, early<br/>stopping and LR monitor on<br/>'mode:metric' monitor_loss."]
class scripts_launcher_py___setup_callbacks main
click scripts_launcher_py___setup_callbacks "../walkthrough/#step-10"
scripts_launcher_py__run_training["<b>12 · run_training()</b><br/>Fit on train/val, reload the best<br/>checkpoint, then test it on the<br/>test split."]
class scripts_launcher_py__run_training main
click scripts_launcher_py__run_training "../walkthrough/#step-12"
scripts_launcher_py__run_evaluation["<b>run_evaluation()</b><br/>Test a trained checkpoint<br/>(model_params.ckpt_path) on the<br/>test split."]
class scripts_launcher_py__run_evaluation opt
click scripts_launcher_py__run_evaluation "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/scripts/launcher.py#L141" _blank
scripts_launcher_py__run_caching["<b>run_caching()</b><br/>Run the backbone over<br/>train/val/test and store its<br/>outputs in W&B (cache mode)."]
class scripts_launcher_py__run_caching opt
click scripts_launcher_py__run_caching "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/scripts/launcher.py#L299" _blank
scripts_launcher_py__run_compilation["<b>run_compilation()</b><br/>Export the model with torch.export<br/>on one test window, check outputs,<br/>log .pt2 to W&B."]
class scripts_launcher_py__run_compilation opt
click scripts_launcher_py__run_compilation "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/scripts/launcher.py#L350" _blank
scripts_launcher_py___log_to_wandb["<b>11 · _log_to_wandb()</b><br/>Upload experiment + kirad source,<br/>package versions and ideal log-mel<br/>energies to W&B."]
class scripts_launcher_py___log_to_wandb main
click scripts_launcher_py___log_to_wandb "../walkthrough/#step-11"
end
subgraph kirad_launcher_utils_py___box["launcher_utils.py"]
direction TB
kirad_launcher_utils_py__LauncherConfig___init__["<b>3 · LauncherConfig.__init__()</b><br/>Read YAML, parse<br/>training/data/model params and<br/>instantiate the model."]
class kirad_launcher_utils_py__LauncherConfig___init__ main
click kirad_launcher_utils_py__LauncherConfig___init__ "../walkthrough/#step-3"
kirad_launcher_utils_py__LauncherConfig__init_data_params["<b>5 · LauncherConfig._init_data_params()</b><br/>Parse data_params keys<br/>'‹modality›=‹class.path›' into<br/>factories + per-split params."]
class kirad_launcher_utils_py__LauncherConfig__init_data_params main
click kirad_launcher_utils_py__LauncherConfig__init_data_params "../walkthrough/#step-5"
kirad_launcher_utils_py__LauncherConfig_get_params_by_split["<b>6 · LauncherConfig.get_params_by_split()</b><br/>Merge config params for one split:<br/>default ‹ val_test (val/test) ‹<br/>split."]
class kirad_launcher_utils_py__LauncherConfig_get_params_by_split main
click kirad_launcher_utils_py__LauncherConfig_get_params_by_split "../walkthrough/#step-6"
kirad_launcher_utils_py__LauncherConfig__init_model["<b>7 · LauncherConfig._init_model()</b><br/>Import model_path, build the model<br/>and load ckpt weights (or defer<br/>full restore)."]
class kirad_launcher_utils_py__LauncherConfig__init_model main
click kirad_launcher_utils_py__LauncherConfig__init_model "../walkthrough/#step-7"
kirad_launcher_utils_py__LauncherConfig_get_dataset["<b>14 · LauncherConfig.get_dataset()</b><br/>StackDataset of metadata + each<br/>modality dataset for a split<br/>(built once)."]
class kirad_launcher_utils_py__LauncherConfig_get_dataset main
click kirad_launcher_utils_py__LauncherConfig_get_dataset "../walkthrough/#step-14"
kirad_launcher_utils_py__LauncherConfig_get_data_loader["<b>13 · LauncherConfig.get_data_loader()</b><br/>DataLoader for a split (cached per<br/>split and mode); batch_size =<br/>recordings."]
class kirad_launcher_utils_py__LauncherConfig_get_data_loader main
click kirad_launcher_utils_py__LauncherConfig_get_data_loader "../walkthrough/#step-13"
kirad_launcher_utils_py__TrainingParams["<b>4 · TrainingParams()</b><br/>Validated training_params: batch<br/>sizes, epochs, monitor metric,<br/>optimizer, seed."]
class kirad_launcher_utils_py__TrainingParams main
click kirad_launcher_utils_py__TrainingParams "../walkthrough/#step-4"
kirad_launcher_utils_py__ModelParams["<b>ModelParams()</b><br/>Validated model_params: model<br/>class path, config kwargs and<br/>checkpoint to load."]
class kirad_launcher_utils_py__ModelParams opt
click kirad_launcher_utils_py__ModelParams "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/launcher_utils.py#L151" _blank
end
subgraph kirad_utils_py___box["utils.py"]
direction TB
kirad_utils_py__import_object["<b>import_object()</b><br/>Import a class/function from a<br/>dotted path, e.g.<br/>'model.DepAnxClassifier'."]
class kirad_utils_py__import_object opt
click kirad_utils_py__import_object "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L792" _blank
kirad_utils_py__localize_wandb_artifact["<b>localize_wandb_artifact()</b><br/>Download a W&B artifact once to<br/>~/.cache/kintsugi and return its<br/>local path."]
class kirad_utils_py__localize_wandb_artifact opt
click kirad_utils_py__localize_wandb_artifact "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L152" _blank
end
subgraph research_stable_ordinal_regression_mtl_mar_2024_model_py___box["mtl_mar_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier["<b>8 · DepAnxClassifier()</b><br/>DAM 1: log-mel -› Whisper small.en<br/>encoder (LoRA) -› mean pool -›<br/>multitask CORAL head."]
class research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier main
click research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier "../walkthrough/#step-8"
end
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__KintsugiBackboneBase_apply_lora["<b>9 · KintsugiBackboneBase.apply_lora()</b><br/>Wrap self.backbone with PEFT LoRA<br/>from lora_params; no-op (full<br/>fine-tune) if empty."]
class kirad_base_models_py__KintsugiBackboneBase_apply_lora main
click kirad_base_models_py__KintsugiBackboneBase_apply_lora "../walkthrough/#step-9"
end
scripts_launcher_py__launcher ==> scripts_launcher_py___setup_launcher
scripts_launcher_py___setup_launcher ==> kirad_launcher_utils_py__LauncherConfig___init__
kirad_launcher_utils_py__LauncherConfig___init__ ==> kirad_launcher_utils_py__TrainingParams
kirad_launcher_utils_py__TrainingParams ==> kirad_launcher_utils_py__LauncherConfig__init_data_params
kirad_launcher_utils_py__LauncherConfig__init_data_params ==> kirad_launcher_utils_py__LauncherConfig_get_params_by_split
kirad_launcher_utils_py__LauncherConfig_get_params_by_split ==> kirad_launcher_utils_py__LauncherConfig__init_model
kirad_launcher_utils_py__LauncherConfig__init_model ==> research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier ==> kirad_base_models_py__KintsugiBackboneBase_apply_lora
kirad_base_models_py__KintsugiBackboneBase_apply_lora ==> scripts_launcher_py___setup_callbacks
scripts_launcher_py___setup_callbacks ==> scripts_launcher_py___log_to_wandb
scripts_launcher_py___log_to_wandb ==> scripts_launcher_py__run_training
scripts_launcher_py__run_training ==> kirad_launcher_utils_py__LauncherConfig_get_data_loader
kirad_launcher_utils_py__LauncherConfig_get_data_loader ==> kirad_launcher_utils_py__LauncherConfig_get_dataset
to_next(["▶ next, step 15: KintsugiMetadata()"])
class to_next edge
kirad_launcher_utils_py__LauncherConfig_get_dataset ==> to_next
click to_next "#stage-data"
kirad_launcher_utils_py__LauncherConfig___init__ -.-> kirad_launcher_utils_py__LauncherConfig__init_data_params
kirad_launcher_utils_py__LauncherConfig___init__ -.-> kirad_launcher_utils_py__ModelParams
kirad_launcher_utils_py__ModelParams -.->|"W&B ckpt"| kirad_utils_py__localize_wandb_artifact
kirad_launcher_utils_py__LauncherConfig___init__ -.-> kirad_launcher_utils_py__LauncherConfig__init_model
kirad_launcher_utils_py__LauncherConfig__init_model -.->|"model.py"| kirad_utils_py__import_object
scripts_launcher_py__launcher -.-> scripts_launcher_py___setup_callbacks
scripts_launcher_py__launcher -.->|"mode"| scripts_launcher_py__run_training
scripts_launcher_py__launcher -.-> scripts_launcher_py__run_evaluation
scripts_launcher_py__launcher -.-> scripts_launcher_py__run_caching
scripts_launcher_py__launcher -.-> scripts_launcher_py__run_compilation
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| 1 | launcher |
scripts/launcher.py:43 |
Run one mode (train/evaluate/cache/compile) end to end from a YAML config. |
| 2 | _setup_launcher |
scripts/launcher.py:155 |
Load the YAML into LauncherConfig and create the W&B logger on rank 0 only. |
| 10 | _setup_callbacks |
scripts/launcher.py:191 |
Best-checkpoint saver, early stopping and LR monitor on "mode:metric" monitor_loss. |
| 12 | run_training |
scripts/launcher.py:119 |
Fit on train/val, reload the best checkpoint, then test it on the test split. |
| — | run_evaluation |
scripts/launcher.py:141 |
Test a trained checkpoint (model_params.ckpt_path) on the test split. |
| — | run_caching |
scripts/launcher.py:299 |
Run the backbone over train/val/test and store its outputs in W&B (cache mode). |
| — | run_compilation |
scripts/launcher.py:350 |
Export the model with torch.export on one test window, check outputs, log .pt2 to W&B. |
| 11 | _log_to_wandb |
scripts/launcher.py:232 |
Upload experiment + kirad source, package versions and ideal log-mel energies to W&B. |
| 3 | LauncherConfig.__init__ |
kirad/launcher_utils.py:234 |
Read YAML, parse training/data/model params and instantiate the model. |
| 5 | LauncherConfig._init_data_params |
kirad/launcher_utils.py:331 |
Parse data_params keys "<modality>=<class.path>" into factories + per-split params. |
| 6 | LauncherConfig.get_params_by_split |
kirad/launcher_utils.py:533 |
Merge config params for one split: default < val_test (val/test) < split. |
| 7 | LauncherConfig._init_model |
kirad/launcher_utils.py:372 |
Import model_path, build the model and load ckpt weights (or defer full restore). |
| 14 | LauncherConfig.get_dataset |
kirad/launcher_utils.py:418 |
StackDataset of metadata + each modality dataset for a split (built once). |
| 13 | LauncherConfig.get_data_loader |
kirad/launcher_utils.py:464 |
DataLoader for a split (cached per split and mode); batch_size = recordings. |
| 4 | TrainingParams |
kirad/launcher_utils.py:101 |
Validated training_params: batch sizes, epochs, monitor metric, optimizer, seed. |
| — | ModelParams |
kirad/launcher_utils.py:151 |
Validated model_params: model class path, config kwargs and checkpoint to load. |
| — | import_object |
kirad/utils.py:792 |
Import a class/function from a dotted path, e.g. "model.DepAnxClassifier". |
| — | localize_wandb_artifact |
kirad/utils.py:152 |
Download a W&B artifact once to ~/.cache/kintsugi and return its local path. |
| 8 | DepAnxClassifier |
research/stable/ordinal_regression_mtl_mar_2024/model.py:25 |
DAM 1: log-mel -> Whisper small.en encoder (LoRA) -> mean pool -> multitask CORAL head. |
| 9 | KintsugiBackboneBase.apply_lora |
kirad/base_models.py:818 |
Wrap self.backbone with PEFT LoRA from lora_params; no-op (full fine-tune) if empty. |
2 · Data¶
The metadata CSV says which WAV files to open and what their labels are. Audio is cut into 30 s windows and turned into features; windows of all recordings in a batch are concatenated and length remembers where each recording ends.
Main path: steps 15–24 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph kirad_datasets_and_dataloaders_py___box["datasets_and_dataloaders.py"]
direction TB
kirad_datasets_and_dataloaders_py__KintsugiMetadata["<b>15 · KintsugiMetadata()</b><br/>Metadata table indexed by UUID:<br/>loads CSV, merges KD scores,<br/>filters by duration."]
class kirad_datasets_and_dataloaders_py__KintsugiMetadata main
click kirad_datasets_and_dataloaders_py__KintsugiMetadata "../walkthrough/#step-15"
kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata["<b>17 · KintsugiMetadata.preprocess_metadata()</b><br/>Join KD score targets, optionally<br/>merge rows per subject, count<br/>windows per recording."]
class kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata main
click kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata "../walkthrough/#step-17"
kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata["<b>18 · KintsugiMetadata.filter_metadata()</b><br/>Subsample `fraction` of rows, then<br/>drop recordings outside min/max<br/>duration (s)."]
class kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata main
click kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata "../walkthrough/#step-18"
kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__["<b>21 · KintsugiMetadata.__getitem__()</b><br/>Return {uuid, label: {task:<br/>{primary: raw score, kd: teacher<br/>score}}} for one UUID."]
class kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__ main
click kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__ "../walkthrough/#step-21"
kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__["<b>22 · KintsugiAudioDataset.__getitem__()</b><br/>Load, concatenate (channel 0),<br/>augment and window one recording's<br/>audio."]
class kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ main
click kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ "../walkthrough/#step-22"
kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization["<b>19 · get_preprocessor_with_audio_normalization()</b><br/>Build the function audio [T] -›<br/>windows -› features used by<br/>KintsugiAudioDataset."]
class kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization main
click kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization "../walkthrough/#step-19"
kirad_datasets_and_dataloaders_py__get_window_starts["<b>23 · get_window_starts()</b><br/>Start sample(s) of the windows cut<br/>from one recording, by<br/>window_method."]
class kirad_datasets_and_dataloaders_py__get_window_starts main
click kirad_datasets_and_dataloaders_py__get_window_starts "../walkthrough/#step-23"
kirad_datasets_and_dataloaders_py__KintsugiTextDataset___getitem__["<b>KintsugiTextDataset.__getitem__()</b><br/>Read the transcript JSON(s) of one<br/>recording and tokenize them."]
class kirad_datasets_and_dataloaders_py__KintsugiTextDataset___getitem__ opt
click kirad_datasets_and_dataloaders_py__KintsugiTextDataset___getitem__ "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/datasets_and_dataloaders.py#L572" _blank
kirad_datasets_and_dataloaders_py__KintsugiCacheDataset["<b>KintsugiCacheDataset()</b><br/>Precomputed backbone features by<br/>UUID (DAM 3 teacher cache),<br/>optional dim slice."]
class kirad_datasets_and_dataloaders_py__KintsugiCacheDataset opt
click kirad_datasets_and_dataloaders_py__KintsugiCacheDataset "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/datasets_and_dataloaders.py#L628" _blank
kirad_datasets_and_dataloaders_py__UuidSampler["<b>20 · UuidSampler()</b><br/>Yields metadata UUIDs (shuffled<br/>for train, sequential for<br/>val/test) for the loader."]
class kirad_datasets_and_dataloaders_py__UuidSampler main
click kirad_datasets_and_dataloaders_py__UuidSampler "../walkthrough/#step-20"
end
subgraph kirad_dataset_utils_py___box["dataset_utils.py"]
direction TB
kirad_dataset_utils_py__load_relative_dataset["<b>16 · load_relative_dataset()</b><br/>Load metadata and add<br/>audio_filename / text_filename<br/>paths from the filename column."]
class kirad_dataset_utils_py__load_relative_dataset main
click kirad_dataset_utils_py__load_relative_dataset "../walkthrough/#step-16"
kirad_dataset_utils_py__load_dataset["<b>load_dataset()</b><br/>Return metadata as a DataFrame:<br/>pass-through, read a CSV path, or<br/>a Kintsugi version."]
class kirad_dataset_utils_py__load_dataset opt
click kirad_dataset_utils_py__load_dataset "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/dataset_utils.py#L49" _blank
kirad_dataset_utils_py__merge_df_rows_by_subject["<b>merge_df_rows_by_subject()</b><br/>Merge rows sharing user + labels +<br/>demographics into one row per<br/>subject."]
class kirad_dataset_utils_py__merge_df_rows_by_subject opt
click kirad_dataset_utils_py__merge_df_rows_by_subject "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/dataset_utils.py#L651" _blank
kirad_dataset_utils_py__broadcast_collate["<b>24 · broadcast_collate()</b><br/>DataLoader collate: concat windows<br/>of all recordings; length =<br/>windows per recording."]
class kirad_dataset_utils_py__broadcast_collate main
click kirad_dataset_utils_py__broadcast_collate "../walkthrough/#step-24"
end
subgraph kirad_augmentations_py___box["augmentations.py"]
direction TB
kirad_augmentations_py__RawBoost["<b>RawBoost()</b><br/>RawBoost waveform augmentation<br/>(LnL/ISD/SSI noise); default algo<br/>4 = all three."]
class kirad_augmentations_py__RawBoost opt
click kirad_augmentations_py__RawBoost "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/augmentations.py#L166" _blank
end
subgraph kirad_constants_py___box["constants.py"]
direction TB
kirad_constants_py__DatasetFields["<b>DatasetFields()</b><br/>Key names of a batch dict: uuid,<br/>features, label.‹task›.primary/kd,<br/>length, loss."]
class kirad_constants_py__DatasetFields opt
click kirad_constants_py__DatasetFields "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/constants.py#L154" _blank
end
from_prev(["◀ from step 14: LauncherConfig.get_dataset()"])
class from_prev edge
from_prev ==> kirad_datasets_and_dataloaders_py__KintsugiMetadata
click from_prev "#stage-launch"
kirad_datasets_and_dataloaders_py__KintsugiMetadata ==> kirad_dataset_utils_py__load_relative_dataset
kirad_dataset_utils_py__load_relative_dataset ==> kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata
kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata ==> kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata
kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata ==> kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization
kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization ==> kirad_datasets_and_dataloaders_py__UuidSampler
kirad_datasets_and_dataloaders_py__UuidSampler ==> kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__
kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__ ==> kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__
kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ ==> kirad_datasets_and_dataloaders_py__get_window_starts
kirad_datasets_and_dataloaders_py__get_window_starts ==> kirad_dataset_utils_py__broadcast_collate
to_next(["▶ next, step 25: MultiTaskOrdinalRegressionPLModule.training_step()"])
class to_next edge
kirad_dataset_utils_py__broadcast_collate ==> to_next
click to_next "#stage-model"
kirad_dataset_utils_py__load_relative_dataset -.-> kirad_dataset_utils_py__load_dataset
kirad_datasets_and_dataloaders_py__KintsugiMetadata -.-> kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata
kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata -.->|"train only"| kirad_dataset_utils_py__merge_df_rows_by_subject
kirad_datasets_and_dataloaders_py__KintsugiMetadata_preprocess_metadata -.->|"count windows"| kirad_datasets_and_dataloaders_py__get_window_starts
kirad_datasets_and_dataloaders_py__KintsugiMetadata -.-> kirad_datasets_and_dataloaders_py__KintsugiMetadata_filter_metadata
kirad_datasets_and_dataloaders_py__UuidSampler -.->|"uuid"| kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__
kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ -.->|"optional"| kirad_augmentations_py__RawBoost
kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ -.-> kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization
kirad_datasets_and_dataloaders_py__get_preprocessor_with_audio_normalization -.->|"cut"| kirad_datasets_and_dataloaders_py__get_window_starts
kirad_datasets_and_dataloaders_py__KintsugiAudioDataset___getitem__ -.-> kirad_dataset_utils_py__broadcast_collate
kirad_datasets_and_dataloaders_py__KintsugiTextDataset___getitem__ -.-> kirad_dataset_utils_py__broadcast_collate
kirad_datasets_and_dataloaders_py__KintsugiMetadata___getitem__ -.->|"labels"| kirad_dataset_utils_py__broadcast_collate
kirad_dataset_utils_py__broadcast_collate -.->|"batch keys"| kirad_constants_py__DatasetFields
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| 15 | KintsugiMetadata |
kirad/datasets_and_dataloaders.py:71 |
Metadata table indexed by UUID: loads CSV, merges KD scores, filters by duration. |
| 17 | KintsugiMetadata.preprocess_metadata |
kirad/datasets_and_dataloaders.py:210 |
Join KD score targets, optionally merge rows per subject, count windows per recording. |
| 18 | KintsugiMetadata.filter_metadata |
kirad/datasets_and_dataloaders.py:257 |
Subsample fraction of rows, then drop recordings outside min/max duration (s). |
| 21 | KintsugiMetadata.__getitem__ |
kirad/datasets_and_dataloaders.py:293 |
Return {uuid, label: {task: {primary: raw score, kd: teacher score}}} for one UUID. |
| 22 | KintsugiAudioDataset.__getitem__ |
kirad/datasets_and_dataloaders.py:439 |
Load, concatenate (channel 0), augment and window one recording's audio. |
| 19 | get_preprocessor_with_audio_normalization |
kirad/datasets_and_dataloaders.py:790 |
Build the function audio [T] -> windows -> features used by KintsugiAudioDataset. |
| 23 | get_window_starts |
kirad/datasets_and_dataloaders.py:738 |
Start sample(s) of the windows cut from one recording, by window_method. |
| — | KintsugiTextDataset.__getitem__ |
kirad/datasets_and_dataloaders.py:572 |
Read the transcript JSON(s) of one recording and tokenize them. |
| — | KintsugiCacheDataset |
kirad/datasets_and_dataloaders.py:628 |
Precomputed backbone features by UUID (DAM 3 teacher cache), optional dim slice. |
| 20 | UuidSampler |
kirad/datasets_and_dataloaders.py:703 |
Yields metadata UUIDs (shuffled for train, sequential for val/test) for the loader. |
| 16 | load_relative_dataset |
kirad/dataset_utils.py:97 |
Load metadata and add audio_filename / text_filename paths from the filename column. |
| — | load_dataset |
kirad/dataset_utils.py:49 |
Return metadata as a DataFrame: pass-through, read a CSV path, or a Kintsugi version. |
| — | merge_df_rows_by_subject |
kirad/dataset_utils.py:651 |
Merge rows sharing user + labels + demographics into one row per subject. |
| 24 | broadcast_collate |
kirad/dataset_utils.py:830 |
DataLoader collate: concat windows of all recordings; length = windows per recording. |
| — | RawBoost |
kirad/augmentations.py:166 |
RawBoost waveform augmentation (LnL/ISD/SSI noise); default algo 4 = all three. |
| — | DatasetFields |
kirad/constants.py:154 |
Key names of a batch dict: uuid, features, label.<task>.primary/kd, length, loss. |
3 · Model (DAM 1)¶
Raw audio windows become log-mel images on the GPU, the Whisper encoder (with LoRA) turns each window into a 768-d vector, and a shared layer feeds one small head per task that outputs a single score.
Main path: steps 25–31 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph research_stable_ordinal_regression_mtl_mar_2024_model_py___box["mtl_mar_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier["<b>DepAnxClassifier()</b><br/>DAM 1: log-mel -› Whisper small.en<br/>encoder (LoRA) -› mean pool -›<br/>multitask CORAL head."]
class research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier opt
click research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_mar_2024/model.py#L25" _blank
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward["<b>27 · DepAnxClassifier.forward()</b><br/>Backbone then MultitaskHead -›<br/>({task: [ΣW, 1] scores}, lengths)."]
class research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward main
click research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward "../walkthrough/#step-27"
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone["<b>28 · DepAnxClassifier.forward_backbone()</b><br/>Pad raw audio to 480,000 samples,<br/>log-mel on GPU, Whisper encoder -›<br/>[ΣW, 768]."]
class research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone main
click research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone "../walkthrough/#step-28"
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_configure_optimizers["<b>DepAnxClassifier.configure_optimizers()</b><br/>AdamW with two groups (backbone,<br/>head + CORAL biases) and<br/>MultiStepLR."]
class research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_configure_optimizers opt
click research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_configure_optimizers "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_mar_2024/model.py#L109" _blank
end
subgraph kirad_feature_extraction_whisper_py___box["feature_extraction_whisper.py"]
direction TB
kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward["<b>29 · WhisperTorchFeatureExtractor.forward()</b><br/>Normalize each window, compute<br/>log-mel, shift bins to<br/>IDEAL_LOGMEL_ENERGIES."]
class kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward main
click kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward "../walkthrough/#step-29"
kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor__torch_extract_fbank_features["<b>WhisperTorchFeatureExtractor._torch_extract_fbank_features()</b><br/>Log10 mel spectrogram, clipped to<br/>8 below the max, scaled (x + 4) /<br/>4 as Whisper."]
class kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor__torch_extract_fbank_features opt
click kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor__torch_extract_fbank_features "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/feature_extraction_whisper.py#L97" _blank
end
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__WhisperBackbone_forward["<b>30 · WhisperBackbone.forward()</b><br/>Whisper encoder on log-mel<br/>windows, mean over time -› [ΣW, D]<br/>(768 for small.en)."]
class kirad_base_models_py__WhisperBackbone_forward main
click kirad_base_models_py__WhisperBackbone_forward "../walkthrough/#step-30"
kirad_base_models_py__KintsugiBackboneBase_apply_lora["<b>KintsugiBackboneBase.apply_lora()</b><br/>Wrap self.backbone with PEFT LoRA<br/>from lora_params; no-op (full<br/>fine-tune) if empty."]
class kirad_base_models_py__KintsugiBackboneBase_apply_lora opt
click kirad_base_models_py__KintsugiBackboneBase_apply_lora "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L818" _blank
kirad_base_models_py__MultitaskHead_forward["<b>31 · MultitaskHead.forward()</b><br/>Shared layers, then one TaskHead<br/>per task -› {task: [ΣW, 1] score}."]
class kirad_base_models_py__MultitaskHead_forward main
click kirad_base_models_py__MultitaskHead_forward "../walkthrough/#step-31"
kirad_base_models_py__MultitaskHead_SharedLayers["<b>MultitaskHead.SharedLayers()</b><br/>Shared MLP: Linear+Mish per<br/>proj_dim, last Linear without<br/>activation (e.g. D-›256-›64)."]
class kirad_base_models_py__MultitaskHead_SharedLayers opt
click kirad_base_models_py__MultitaskHead_SharedLayers "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L1002" _blank
kirad_base_models_py__MultitaskHead_TaskHead["<b>MultitaskHead.TaskHead()</b><br/>Per-task head: Linear(in,<br/>proj_dim) -› Mish -› Dropout -›<br/>Linear(proj_dim, 1, no bias)."]
class kirad_base_models_py__MultitaskHead_TaskHead opt
click kirad_base_models_py__MultitaskHead_TaskHead "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L1034" _blank
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_training_step["<b>25 · MultiTaskOrdinalRegressionPLModule.training_step()</b><br/>Lightning train step: returns the<br/>step dict with total_loss."]
class kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_training_step main
click kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_training_step "../walkthrough/#step-25"
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step["<b>26 · MultiTaskOrdinalRegressionPLModule.step()</b><br/>Run model once, sum each task's<br/>loss (task.step) into total_loss."]
class kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step main
click kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step "../walkthrough/#step-26"
end
from_prev(["◀ from step 24: broadcast_collate()"])
class from_prev edge
from_prev ==> kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_training_step
click from_prev "#stage-data"
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_training_step ==> kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step ==> research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward ==> research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone ==> kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward
kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward ==> kirad_base_models_py__WhisperBackbone_forward
kirad_base_models_py__WhisperBackbone_forward ==> kirad_base_models_py__MultitaskHead_forward
to_next(["▶ next, step 32: OrdinalRegressionPLModule.step()"])
class to_next edge
kirad_base_models_py__MultitaskHead_forward ==> to_next
click to_next "#stage-loss"
kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor_forward -.-> kirad_feature_extraction_whisper_py__WhisperTorchFeatureExtractor__torch_extract_fbank_features
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward_backbone -.->|"log-mel"| kirad_base_models_py__WhisperBackbone_forward
research_stable_ordinal_regression_mtl_mar_2024_model_py__DepAnxClassifier_forward -.->|"768-d"| kirad_base_models_py__MultitaskHead_forward
kirad_base_models_py__MultitaskHead_forward -.-> kirad_base_models_py__MultitaskHead_SharedLayers
kirad_base_models_py__MultitaskHead_SharedLayers -.->|"64-d"| kirad_base_models_py__MultitaskHead_TaskHead
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| — | DepAnxClassifier |
research/stable/ordinal_regression_mtl_mar_2024/model.py:25 |
DAM 1: log-mel -> Whisper small.en encoder (LoRA) -> mean pool -> multitask CORAL head. |
| 27 | DepAnxClassifier.forward |
research/stable/ordinal_regression_mtl_mar_2024/model.py:84 |
Backbone then MultitaskHead -> ({task: [ΣW, 1] scores}, lengths). |
| 28 | DepAnxClassifier.forward_backbone |
research/stable/ordinal_regression_mtl_mar_2024/model.py:89 |
Pad raw audio to 480,000 samples, log-mel on GPU, Whisper encoder -> [ΣW, 768]. |
| — | DepAnxClassifier.configure_optimizers |
research/stable/ordinal_regression_mtl_mar_2024/model.py:109 |
AdamW with two groups (backbone, head + CORAL biases) and MultiStepLR. |
| 29 | WhisperTorchFeatureExtractor.forward |
kirad/feature_extraction_whisper.py:138 |
Normalize each window, compute log-mel, shift bins to IDEAL_LOGMEL_ENERGIES. |
| — | WhisperTorchFeatureExtractor._torch_extract_fbank_features |
kirad/feature_extraction_whisper.py:97 |
Log10 mel spectrogram, clipped to 8 below the max, scaled (x + 4) / 4 as Whisper. |
| 30 | WhisperBackbone.forward |
kirad/base_models.py:877 |
Whisper encoder on log-mel windows, mean over time -> [ΣW, D] (768 for small.en). |
| — | KintsugiBackboneBase.apply_lora |
kirad/base_models.py:818 |
Wrap self.backbone with PEFT LoRA from lora_params; no-op (full fine-tune) if empty. |
| 31 | MultitaskHead.forward |
kirad/base_models.py:1075 |
Shared layers, then one TaskHead per task -> {task: [ΣW, 1] score}. |
| — | MultitaskHead.SharedLayers |
kirad/base_models.py:1002 |
Shared MLP: Linear+Mish per proj_dim, last Linear without activation (e.g. D->256->64). |
| — | MultitaskHead.TaskHead |
kirad/base_models.py:1034 |
Per-task head: Linear(in, proj_dim) -> Mish -> Dropout -> Linear(proj_dim, 1, no bias). |
| 25 | MultiTaskOrdinalRegressionPLModule.training_step |
kirad/base_models.py:722 |
Lightning train step: returns the step dict with total_loss. |
| 26 | MultiTaskOrdinalRegressionPLModule.step |
kirad/base_models.py:734 |
Run model once, sum each task's loss (task.step) into total_loss. |
4 · Loss¶
Each task turns its score into 27 (or 21) ordered yes/no questions with CORAL. Train: CORAL loss per window, averaged per recording; val/test: loss of the per-recording mean score. A variance penalty makes windows of the same recording agree.
Main path: steps 32–35 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step["<b>MultiTaskOrdinalRegressionPLModule.step()</b><br/>Run model once, sum each task's<br/>loss (task.step) into total_loss."]
class kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step opt
click kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L734" _blank
kirad_base_models_py__OrdinalRegressionPLModule_step["<b>32 · OrdinalRegressionPLModule.step()</b><br/>Accumulate per-recording mean/var<br/>scores for val/test, then return<br/>the loss."]
class kirad_base_models_py__OrdinalRegressionPLModule_step main
click kirad_base_models_py__OrdinalRegressionPLModule_step "../walkthrough/#step-32"
kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss["<b>33 · OrdinalRegressionPLModule.compute_and_log_loss()</b><br/>Loss = coral_w·CORAL +<br/>var_w·Var(window scores) +<br/>kd_w·MSE(teacher), logged."]
class kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss main
click kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss "../walkthrough/#step-33"
end
subgraph kirad_losses_py___box["losses.py"]
direction TB
kirad_losses_py__CORALLoss_forward["<b>35 · CORALLoss.forward()</b><br/>Weighted sum of BCE over cutoffs:<br/>sigmoid(s + b_k) vs [y › k] for<br/>each cutoff k."]
class kirad_losses_py__CORALLoss_forward main
click kirad_losses_py__CORALLoss_forward "../walkthrough/#step-35"
end
subgraph kirad_utils_py___box["utils.py"]
direction TB
kirad_utils_py__average_and_variance_tensor_in_segments["<b>34 · average_and_variance_tensor_in_segments()</b><br/>Per-recording mean and (biased)<br/>variance of window scores: [ΣW,<br/>...] -› two [B, ...]."]
class kirad_utils_py__average_and_variance_tensor_in_segments main
click kirad_utils_py__average_and_variance_tensor_in_segments "../walkthrough/#step-34"
kirad_utils_py__average_tensor_in_segments["<b>average_tensor_in_segments()</b><br/>Mean over each recording's<br/>windows: [ΣW, ...] -› [B, ...]<br/>using `length`."]
class kirad_utils_py__average_tensor_in_segments opt
click kirad_utils_py__average_tensor_in_segments "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L90" _blank
end
from_prev(["◀ from step 31: MultitaskHead.forward()"])
class from_prev edge
from_prev ==> kirad_base_models_py__OrdinalRegressionPLModule_step
click from_prev "#stage-model"
kirad_base_models_py__OrdinalRegressionPLModule_step ==> kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss
kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss ==> kirad_utils_py__average_and_variance_tensor_in_segments
kirad_utils_py__average_and_variance_tensor_in_segments ==> kirad_losses_py__CORALLoss_forward
to_next(["▶ next, step 36: MultiTaskOrdinalRegressionPLModule.on_validation_test_epoch_end()"])
class to_next edge
kirad_losses_py__CORALLoss_forward ==> to_next
click to_next "#stage-metrics"
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_step -.->|"per task"| kirad_base_models_py__OrdinalRegressionPLModule_step
kirad_utils_py__average_and_variance_tensor_in_segments -.-> kirad_utils_py__average_tensor_in_segments
kirad_base_models_py__OrdinalRegressionPLModule_compute_and_log_loss -.-> kirad_losses_py__CORALLoss_forward
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| — | MultiTaskOrdinalRegressionPLModule.step |
kirad/base_models.py:734 |
Run model once, sum each task's loss (task.step) into total_loss. |
| 32 | OrdinalRegressionPLModule.step |
kirad/base_models.py:572 |
Accumulate per-recording mean/var scores for val/test, then return the loss. |
| 33 | OrdinalRegressionPLModule.compute_and_log_loss |
kirad/base_models.py:504 |
Loss = coral_w·CORAL + var_w·Var(window scores) + kd_w·MSE(teacher), logged. |
| 35 | CORALLoss.forward |
kirad/losses.py:43 |
Weighted sum of BCE over cutoffs: sigmoid(s + b_k) vs [y > k] for each cutoff k. |
| 34 | average_and_variance_tensor_in_segments |
kirad/utils.py:122 |
Per-recording mean and (biased) variance of window scores: [ΣW, ...] -> two [B, ...]. |
| — | average_tensor_in_segments |
kirad/utils.py:90 |
Mean over each recording's windows: [ΣW, ...] -> [B, ...] using length. |
5 · Thresholds and metrics¶
After every validation, scores of all recordings are gathered, five threshold sets are tuned by dynamic programming, a sixth is read from the CORAL biases, and Sn/Sp with an indeterminate band are computed. Test reuses the validation thresholds.
Main path: steps 36–41 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_on_validation_test_epoch_end["<b>36 · MultiTaskOrdinalRegressionPLModule.on_validation_test_epoch_end()</b><br/>Run each task's epoch end, then<br/>log harmonic mean of shared<br/>metrics across tasks."]
class kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_on_validation_test_epoch_end main
click kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_on_validation_test_epoch_end "../walkthrough/#step-36"
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end["<b>37 · OrdinalRegressionPLModule.on_validation_test_epoch_end()</b><br/>Gather scores, log W&B table, tune<br/>thresholds on val, compute Sn/Sp<br/>metrics."]
class kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end main
click kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end "../walkthrough/#step-37"
kirad_base_models_py__OrdinalRegressionPLModule_quantize_labels["<b>38 · OrdinalRegressionPLModule.quantize_labels()</b><br/>Map input labels (e.g. PHQ-9 sum)<br/>to output labels (e.g. 0/1/2 for<br/>L/M/H)."]
class kirad_base_models_py__OrdinalRegressionPLModule_quantize_labels main
click kirad_base_models_py__OrdinalRegressionPLModule_quantize_labels "../walkthrough/#step-38"
kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics["<b>41 · OrdinalRegressionPLModule.log_ordinal_metrics()</b><br/>Log AUROC and min(Sn, Sp) per<br/>binary cut and indet budget, plus<br/>their mean."]
class kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics main
click kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics "../walkthrough/#step-41"
kirad_base_models_py__OrdinalRegressionPLModule_log_thresholding_metrics["<b>OrdinalRegressionPLModule.log_thresholding_metrics()</b><br/>Log each tuned method's cost when<br/>scored with every method's<br/>thresholds."]
class kirad_base_models_py__OrdinalRegressionPLModule_log_thresholding_metrics opt
click kirad_base_models_py__OrdinalRegressionPLModule_log_thresholding_metrics "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L315" _blank
end
subgraph kirad_ordinal_thresholding_py___box["ordinal_thresholding.py"]
direction TB
kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds["<b>40 · OptimalOrdinalThresholdingViaDynamicProgramming.tune_thresholds()</b><br/>Set thresholds by DP over<br/>midpoints of unique scores<br/>(+/-inf); return the best metric."]
class kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds main
click kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds "../walkthrough/#step-40"
kirad_ordinal_thresholding_py__OptimalCostPerSampleOrdinalThresholding_dp_step["<b>OptimalCostPerSampleOrdinalThresholding.dp_step()</b><br/>O(n) DP step: histogram per-sample<br/>costs over candidates, cumsum,<br/>running max."]
class kirad_ordinal_thresholding_py__OptimalCostPerSampleOrdinalThresholding_dp_step opt
click kirad_ordinal_thresholding_py__OptimalCostPerSampleOrdinalThresholding_dp_step "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/ordinal_thresholding.py#L291" _blank
kirad_ordinal_thresholding_py__OptimalCostPerClassOrdinalThresholding_dp_step["<b>OptimalCostPerClassOrdinalThresholding.dp_step()</b><br/>O(T^2) DP step: add prev_cost to<br/>each lower-threshold row and max<br/>over it."]
class kirad_ordinal_thresholding_py__OptimalCostPerClassOrdinalThresholding_dp_step opt
click kirad_ordinal_thresholding_py__OptimalCostPerClassOrdinalThresholding_dp_step "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/ordinal_thresholding.py#L450" _blank
kirad_ordinal_thresholding_py__CORALThresholding_tune_thresholds["<b>CORALThresholding.tune_thresholds()</b><br/>Copy -biases[threshold_indices]<br/>from the CORAL loss; scores and<br/>labels are ignored."]
class kirad_ordinal_thresholding_py__CORALThresholding_tune_thresholds opt
click kirad_ordinal_thresholding_py__CORALThresholding_tune_thresholds "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/ordinal_thresholding.py#L85" _blank
end
subgraph kirad_metrics_py___box["metrics.py"]
direction TB
kirad_metrics_py__MultiCatMetric["<b>MultiCatMetric()</b><br/>Accumulate parallel tensors over<br/>batches and ranks (uuids, scores,<br/>variances, labels)."]
class kirad_metrics_py__MultiCatMetric opt
click kirad_metrics_py__MultiCatMetric "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/metrics.py#L41" _blank
kirad_metrics_py__IndetSnSpArray_build["<b>39 · IndetSnSpArray.build()</b><br/>Sn/Sp/indet_frac for all threshold<br/>pairs (or the given ones) via<br/>histogram cumsums."]
class kirad_metrics_py__IndetSnSpArray_build main
click kirad_metrics_py__IndetSnSpArray_build "../walkthrough/#step-39"
kirad_metrics_py__IndetSnSpArray_roc_curve["<b>IndetSnSpArray.roc_curve()</b><br/>Pareto ROC curve of (Sn, Sp) using<br/>only pairs with indet_frac ‹=<br/>indet_budget."]
class kirad_metrics_py__IndetSnSpArray_roc_curve opt
click kirad_metrics_py__IndetSnSpArray_roc_curve "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/metrics.py#L291" _blank
kirad_metrics_py__IndetSnSpArray_sn_eq_sp_graph["<b>IndetSnSpArray.sn_eq_sp_graph()</b><br/>Best min(Sn, Sp) per indeterminate<br/>fraction; these thresholds are<br/>saved in the ckpt."]
class kirad_metrics_py__IndetSnSpArray_sn_eq_sp_graph opt
click kirad_metrics_py__IndetSnSpArray_sn_eq_sp_graph "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/metrics.py#L305" _blank
end
from_prev(["◀ from step 35: CORALLoss.forward()"])
class from_prev edge
from_prev ==> kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_on_validation_test_epoch_end
click from_prev "#stage-loss"
kirad_base_models_py__MultiTaskOrdinalRegressionPLModule_on_validation_test_epoch_end ==> kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end ==> kirad_base_models_py__OrdinalRegressionPLModule_quantize_labels
kirad_base_models_py__OrdinalRegressionPLModule_quantize_labels ==> kirad_metrics_py__IndetSnSpArray_build
kirad_metrics_py__IndetSnSpArray_build ==> kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds
kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds ==> kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics
to_next(["▶ next, step 42: DistributedModelCheckpoint.on_validation_end()"])
class to_next edge
kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics ==> to_next
click to_next "#stage-outputs"
kirad_metrics_py__MultiCatMetric -.->|"gathered scores"| kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.->|"val only"| kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds
kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds -.-> kirad_ordinal_thresholding_py__OptimalCostPerSampleOrdinalThresholding_dp_step
kirad_ordinal_thresholding_py__OptimalOrdinalThresholdingViaDynamicProgramming_tune_thresholds -.-> kirad_ordinal_thresholding_py__OptimalCostPerClassOrdinalThresholding_dp_step
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.-> kirad_ordinal_thresholding_py__CORALThresholding_tune_thresholds
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.-> kirad_metrics_py__IndetSnSpArray_build
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.->|"Sn=Sp ops (val)"| kirad_metrics_py__IndetSnSpArray_sn_eq_sp_graph
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.-> kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics
kirad_base_models_py__OrdinalRegressionPLModule_log_ordinal_metrics -.-> kirad_metrics_py__IndetSnSpArray_roc_curve
kirad_base_models_py__OrdinalRegressionPLModule_on_validation_test_epoch_end -.-> kirad_base_models_py__OrdinalRegressionPLModule_log_thresholding_metrics
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| 36 | MultiTaskOrdinalRegressionPLModule.on_validation_test_epoch_end |
kirad/base_models.py:680 |
Run each task's epoch end, then log harmonic mean of shared metrics across tasks. |
| 37 | OrdinalRegressionPLModule.on_validation_test_epoch_end |
kirad/base_models.py:423 |
Gather scores, log W&B table, tune thresholds on val, compute Sn/Sp metrics. |
| 38 | OrdinalRegressionPLModule.quantize_labels |
kirad/base_models.py:222 |
Map input labels (e.g. PHQ-9 sum) to output labels (e.g. 0/1/2 for L/M/H). |
| 41 | OrdinalRegressionPLModule.log_ordinal_metrics |
kirad/base_models.py:228 |
Log AUROC and min(Sn, Sp) per binary cut and indet budget, plus their mean. |
| — | OrdinalRegressionPLModule.log_thresholding_metrics |
kirad/base_models.py:315 |
Log each tuned method's cost when scored with every method's thresholds. |
| 40 | OptimalOrdinalThresholdingViaDynamicProgramming.tune_thresholds |
kirad/ordinal_thresholding.py:179 |
Set thresholds by DP over midpoints of unique scores (+/-inf); return the best metric. |
| — | OptimalCostPerSampleOrdinalThresholding.dp_step |
kirad/ordinal_thresholding.py:291 |
O(n) DP step: histogram per-sample costs over candidates, cumsum, running max. |
| — | OptimalCostPerClassOrdinalThresholding.dp_step |
kirad/ordinal_thresholding.py:450 |
O(T^2) DP step: add prev_cost to each lower-threshold row and max over it. |
| — | CORALThresholding.tune_thresholds |
kirad/ordinal_thresholding.py:85 |
Copy -biases[threshold_indices] from the CORAL loss; scores and labels are ignored. |
| — | MultiCatMetric |
kirad/metrics.py:41 |
Accumulate parallel tensors over batches and ranks (uuids, scores, variances, labels). |
| 39 | IndetSnSpArray.build |
kirad/metrics.py:176 |
Sn/Sp/indet_frac for all threshold pairs (or the given ones) via histogram cumsums. |
| — | IndetSnSpArray.roc_curve |
kirad/metrics.py:291 |
Pareto ROC curve of (Sn, Sp) using only pairs with indet_frac <= indet_budget. |
| — | IndetSnSpArray.sn_eq_sp_graph |
kirad/metrics.py:305 |
Best min(Sn, Sp) per indeterminate fraction; these thresholds are saved in the ckpt. |
6 · Outputs and analysis¶
The best checkpoint (with thresholds inside) is saved at the end of validation; scores, metrics, code and datasets go to W&B (_log_to_wandb runs at start-up, before training). After training, utils pulls the scores back for analysis and report cards. cache_backbone is the cache mode, called by run_caching.
Main path: steps 42–44 of a train run (walkthrough). Gold boxes open the step; grey boxes are side paths and open the code.
flowchart LR
subgraph kirad_launcher_utils_py___box["launcher_utils.py"]
direction TB
kirad_launcher_utils_py__DistributedModelCheckpoint_on_validation_end["<b>42 · DistributedModelCheckpoint.on_validation_end()</b><br/>Save best/last ckpt after<br/>validation (with tuned<br/>thresholds); replicate on nodes."]
class kirad_launcher_utils_py__DistributedModelCheckpoint_on_validation_end main
click kirad_launcher_utils_py__DistributedModelCheckpoint_on_validation_end "../walkthrough/#step-42"
kirad_launcher_utils_py__LauncherLogger["<b>LauncherLogger()</b><br/>W&B run for the experiment; logs<br/>code and pandas tables as<br/>artifacts."]
class kirad_launcher_utils_py__LauncherLogger opt
click kirad_launcher_utils_py__LauncherLogger "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/launcher_utils.py#L581" _blank
end
subgraph scripts_launcher_py___box["launcher.py"]
direction TB
scripts_launcher_py___log_to_wandb["<b>_log_to_wandb()</b><br/>Upload experiment + kirad source,<br/>package versions and ideal log-mel<br/>energies to W&B."]
class scripts_launcher_py___log_to_wandb opt
click scripts_launcher_py___log_to_wandb "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/scripts/launcher.py#L232" _blank
scripts_launcher_py___log_datasets_to_wandb["<b>43 · _log_datasets_to_wandb()</b><br/>Upload the metadata table actually<br/>used for each split as a W&B<br/>'dataset' artifact."]
class scripts_launcher_py___log_datasets_to_wandb main
click scripts_launcher_py___log_datasets_to_wandb "../walkthrough/#step-43"
end
subgraph kirad_utils_py___box["utils.py"]
direction TB
kirad_utils_py__cache_backbone["<b>cache_backbone()</b><br/>Run the backbone on all datasets,<br/>gather outputs by uuid across<br/>ranks, log to W&B."]
class kirad_utils_py__cache_backbone opt
click kirad_utils_py__cache_backbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L833" _blank
kirad_utils_py__get_dataframe_with_scores["<b>44 · get_dataframe_with_scores()</b><br/>Join a run's metadata CSVs with<br/>its W&B scores-‹task›-‹split›<br/>tables on uuid."]
class kirad_utils_py__get_dataframe_with_scores main
click kirad_utils_py__get_dataframe_with_scores "../walkthrough/#step-44"
kirad_utils_py__indet_analysis["<b>indet_analysis()</b><br/>Tune sn_eq_sp thresholds on val<br/>per budget and binary cut, then<br/>measure them on test."]
class kirad_utils_py__indet_analysis opt
click kirad_utils_py__indet_analysis "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L585" _blank
kirad_utils_py__analyze_dataframe_with_scores["<b>analyze_dataframe_with_scores()</b><br/>Mean AUROC and sn_eq_sp (0% indet)<br/>over binary cuts, per subgroup of<br/>the dataframe."]
class kirad_utils_py__analyze_dataframe_with_scores opt
click kirad_utils_py__analyze_dataframe_with_scores "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L646" _blank
kirad_utils_py__get_wandb_report_card_tsvs["<b>get_wandb_report_card_tsvs()</b><br/>Build a TSV report card per task<br/>from a W&B run and log it back as<br/>an artifact."]
class kirad_utils_py__get_wandb_report_card_tsvs opt
click kirad_utils_py__get_wandb_report_card_tsvs "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/utils.py#L253" _blank
end
subgraph kirad_plotting_py___box["plotting.py"]
direction TB
kirad_plotting_py__plot_indeterminate_analysis["<b>plot_indeterminate_analysis()</b><br/>Plot thresholds and Sn=Sp vs<br/>indeterminate budget, plus ROC<br/>curves per binary cut."]
class kirad_plotting_py__plot_indeterminate_analysis opt
click kirad_plotting_py__plot_indeterminate_analysis "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/plotting.py#L16" _blank
end
from_prev(["◀ from step 41: OrdinalRegressionPLModule.log_ordinal_metrics()"])
class from_prev edge
from_prev ==> kirad_launcher_utils_py__DistributedModelCheckpoint_on_validation_end
click from_prev "#stage-metrics"
kirad_launcher_utils_py__DistributedModelCheckpoint_on_validation_end ==> scripts_launcher_py___log_datasets_to_wandb
scripts_launcher_py___log_datasets_to_wandb ==> kirad_utils_py__get_dataframe_with_scores
scripts_launcher_py___log_to_wandb -.-> kirad_launcher_utils_py__LauncherLogger
scripts_launcher_py___log_datasets_to_wandb -.-> kirad_launcher_utils_py__LauncherLogger
kirad_utils_py__get_wandb_report_card_tsvs -.-> kirad_utils_py__indet_analysis
kirad_utils_py__indet_analysis -.-> kirad_utils_py__get_dataframe_with_scores
kirad_utils_py__get_dataframe_with_scores -.->|"df"| kirad_plotting_py__plot_indeterminate_analysis
kirad_utils_py__get_dataframe_with_scores -.->|"df"| kirad_utils_py__analyze_dataframe_with_scores
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| 42 | DistributedModelCheckpoint.on_validation_end |
kirad/launcher_utils.py:45 |
Save best/last ckpt after validation (with tuned thresholds); replicate on nodes. |
| — | LauncherLogger |
kirad/launcher_utils.py:581 |
W&B run for the experiment; logs code and pandas tables as artifacts. |
| — | _log_to_wandb |
scripts/launcher.py:232 |
Upload experiment + kirad source, package versions and ideal log-mel energies to W&B. |
| 43 | _log_datasets_to_wandb |
scripts/launcher.py:286 |
Upload the metadata table actually used for each split as a W&B "dataset" artifact. |
| — | cache_backbone |
kirad/utils.py:833 |
Run the backbone on all datasets, gather outputs by uuid across ranks, log to W&B. |
| 44 | get_dataframe_with_scores |
kirad/utils.py:442 |
Join a run's metadata CSVs with its W&B scores-<task>-<split> tables on uuid. |
| — | indet_analysis |
kirad/utils.py:585 |
Tune sn_eq_sp thresholds on val per budget and binary cut, then measure them on test. |
| — | analyze_dataframe_with_scores |
kirad/utils.py:646 |
Mean AUROC and sn_eq_sp (0% indet) over binary cuts, per subgroup of the dataframe. |
| — | get_wandb_report_card_tsvs |
kirad/utils.py:253 |
Build a TSV report card per task from a W&B run and log it back as an artifact. |
| — | plot_indeterminate_analysis |
kirad/plotting.py:16 |
Plot thresholds and Sn=Sp vs indeterminate budget, plus ROC curves per binary cut. |
7 · The other versions (DAM 2, DAM 3)¶
DAM 2 adds a text backbone over the transcript. DAM 3 first trains a Whisper to imitate BERT's vector from audio (WhisperLLMA), then freezes it and trains only the head.
flowchart LR
subgraph research_stable_ordinal_regression_mtl_jun_2024_model_py___box["mtl_jun_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase["<b>DepAnxClassifierBase()</b><br/>Concatenate embeddings of several<br/>backbones (ModuleDict order) and<br/>feed a MultitaskHead."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L55" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT["<b>DepAnxClassifierWhisperBERT()</b><br/>DAM 2: Whisper audio + BERT<br/>transcript embeddings (768 + 768)<br/>-› multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L142" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA["<b>DepAnxClassifierWhisperLLAMA()</b><br/>Whisper audio + LLaMA transcript<br/>embeddings concatenated -›<br/>multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L169" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA["<b>DepAnxClassifierLLAMA()</b><br/>Text only: LLaMA transcript<br/>embedding -› multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L189" _blank
end
subgraph research_stable_ordinal_regression_mtl_aug_2024_model_py___box["mtl_aug_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA["<b>WhisperLLMA()</b><br/>DAM 3 phase 1: Whisper encoder<br/>trained to predict a cached text<br/>embedding from audio."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L32" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step["<b>WhisperLLMA.step()</b><br/>MSE between per-recording Whisper<br/>embedding and the cached teacher<br/>embedding."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L102" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier["<b>DepAnxClassifier()</b><br/>DAM 3: frozen Whisper + frozen<br/>WhisperLLMA embeddings (768 + 768)<br/>-› trainable head."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L150" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone["<b>DepAnxClassifier.forward_backbone()</b><br/>No-grad Whisper + WhisperLLMA,<br/>averaged per recording -› [B,<br/>1536]; lengths set to 1."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L221" _blank
end
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__BERTBackbone["<b>BERTBackbone()</b><br/>BERT on transcript tokens; returns<br/>the pooler output [ΣW, 768] (DAM 2<br/>text branch)."]
class kirad_base_models_py__BERTBackbone opt
click kirad_base_models_py__BERTBackbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L918" _blank
kirad_base_models_py__LLAMABackbone["<b>LLAMABackbone()</b><br/>LLaMA cut to num_layers layers;<br/>embeds text as the last real<br/>token's hidden state."]
class kirad_base_models_py__LLAMABackbone opt
click kirad_base_models_py__LLAMABackbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L941" _blank
kirad_base_models_py__Wav2Vec2Backbone["<b>Wav2Vec2Backbone()</b><br/>wav2vec2 encoder on raw audio,<br/>mean-pooled (not used by the<br/>stable DAM configs)."]
class kirad_base_models_py__Wav2Vec2Backbone opt
click kirad_base_models_py__Wav2Vec2Backbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L888" _blank
end
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT -.-> research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT -.->|"text"| kirad_base_models_py__BERTBackbone
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA -.-> kirad_base_models_py__LLAMABackbone
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA -.-> kirad_base_models_py__LLAMABackbone
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA -.->|"phase 1: MSE"| research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA -.->|"frozen"| research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier -.-> research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| — | DepAnxClassifierBase |
research/stable/ordinal_regression_mtl_jun_2024/model.py:55 |
Concatenate embeddings of several backbones (ModuleDict order) and feed a MultitaskHead. |
| — | DepAnxClassifierWhisperBERT |
research/stable/ordinal_regression_mtl_jun_2024/model.py:142 |
DAM 2: Whisper audio + BERT transcript embeddings (768 + 768) -> multitask head. |
| — | DepAnxClassifierWhisperLLAMA |
research/stable/ordinal_regression_mtl_jun_2024/model.py:169 |
Whisper audio + LLaMA transcript embeddings concatenated -> multitask head. |
| — | DepAnxClassifierLLAMA |
research/stable/ordinal_regression_mtl_jun_2024/model.py:189 |
Text only: LLaMA transcript embedding -> multitask head. |
| — | WhisperLLMA |
research/stable/ordinal_regression_mtl_aug_2024/model.py:32 |
DAM 3 phase 1: Whisper encoder trained to predict a cached text embedding from audio. |
| — | WhisperLLMA.step |
research/stable/ordinal_regression_mtl_aug_2024/model.py:102 |
MSE between per-recording Whisper embedding and the cached teacher embedding. |
| — | DepAnxClassifier |
research/stable/ordinal_regression_mtl_aug_2024/model.py:150 |
DAM 3: frozen Whisper + frozen WhisperLLMA embeddings (768 + 768) -> trainable head. |
| — | DepAnxClassifier.forward_backbone |
research/stable/ordinal_regression_mtl_aug_2024/model.py:221 |
No-grad Whisper + WhisperLLMA, averaged per recording -> [B, 1536]; lengths set to 1. |
| — | BERTBackbone |
kirad/base_models.py:918 |
BERT on transcript tokens; returns the pooler output [ΣW, 768] (DAM 2 text branch). |
| — | LLAMABackbone |
kirad/base_models.py:941 |
LLaMA cut to num_layers layers; embeds text as the last real token's hidden state. |
| — | Wav2Vec2Backbone |
kirad/base_models.py:888 |
wav2vec2 encoder on raw audio, mean-pooled (not used by the stable DAM configs). |