Experiments: DAM 1, 2, 3¶
Each folder in research/stable/ has a model.py and YAML configs; model_path in the YAML
picks the class. config.yaml is the current config of each folder (dataset v2.11.4, 4
sources); config_dam*.yaml and config_llma_dam3.0.yaml are older snapshots (v2.9.4, 2 sources). Only DAM 1 config.yaml
validates as-is; see Limitations 5–8.
Comparison¶
| DAM 1 | DAM 2 | DAM 3 | |
|---|---|---|---|
| Folder | ordinal_regression_mtl_mar_2024 |
ordinal_regression_mtl_jun_2024 |
ordinal_regression_mtl_aug_2024 |
| Class | DepAnxClassifier |
DepAnxClassifierWhisperBERT |
phase 1 WhisperLLMA, phase 2 DepAnxClassifier |
| Input at inference | audio | audio + transcript | audio |
| Backbone(s) | Whisper small.en + LoRA | Whisper + LoRA, BERT base uncased + LoRA | frozen Whisper + frozen WhisperLLMA |
| Head input | 768 | 768 + 768 = 1536 | 768 + 768 = 1536 |
| Log-mel | in model, on GPU | HF extractor in dataset | HF extractor in dataset |
| Trained | encoder LoRA + conv1/conv2, head, CORAL biases | both LoRAs, head, biases | phase 1: Whisper (MSE to BERT half [768:1536] of a cached DAM 2); phase 2: head + biases only |
| Needs ASR | no | yes (scripts/run_whisper_asr.py) |
phase 1 only (via DAM 2 cache) |
Other classes in jun_2024/model.py: DepAnxClassifierWhisperLLAMA (Whisper + LLaMA 3.2 3B
cut to 15 layers, config_llama.yaml) and DepAnxClassifierLLAMA (text only).
Hyperparameters (current configs)¶
| Key | DAM 1 config.yaml |
DAM 2 config.yaml |
DAM 3 ph. 1 config_llma.yaml |
DAM 3 ph. 2 config.yaml |
|---|---|---|---|---|
| seed | 44 | 44 | 42 | 46 |
| batch_size / effective | 2 / 128 | 2 / 128 | 1 / 64 | 1 / 4 |
| max_epochs / patience | 32 / 10 | 32 / 10 | 18 / 10 | 5 / 10 |
| val every | 16,000 recordings | epoch | epoch | 16,000 recordings |
| monitor | max sn_eq_sp@0%i |
max sn_eq_sp@0%i |
min total_loss |
max sn_eq_sp@0%i |
| lr backbone / head | 1.6e-4 / 8e-4 | 1.6e-4 / 8e-4 | 4e-5 / — | unused (frozen) / 4e-4 |
| weight decay | 1e-3 | 1e-3 | 1e-3 | 1e-3 |
| MultiStepLR γ, milestones | 0.5, [20, 26] | 0.5, [20, 26] | 0.5, [14] | 0.5, [20, 26] |
| LoRA (Whisper) | r 32, α 64, drop 0.4 | same (+ BERT same) | none (full fine-tune) | r 32, α 64 (frozen) |
| shared / task proj / dropout | [256, 64] / 128 / 0.4 | same | — | same |
| targets | exact28 / exact22 | exact28 / exact22 | — | exact28 / exact22 |
| score_variance weight | 40 | 0 | — | 0 |
| train window_method | random_adjacent_pair |
single_random |
all |
all |
| val/test window_method | all |
all |
all |
all |
| merge_rows_by_subject (train) | many_to_one |
no | no | no |
audio preprocessor |
false |
openai/whisper-small.en |
same | same |
ideal_logmel_energies |
built-in constant | logmel_energies.npy |
same | same |
| other | text: bert-base-uncased |
max_duration 90 s (train), feat_start_idx 768 |
pad_last_chunk_to_full |
Phase 2 loads WhisperLLMA from llma_ckpt (a W&B URL). Its audio Whisper gets trained weights only
from ckpt_path, which is empty in config.yaml, so it runs as the pretrained HF encoder.
Tasks in every config: depression (label phq, 28 classes) and anxiety (label gad, 22
classes). Indeterminate budgets default to (0.0, 0.4) in every model.py.
7 · The other versions (DAM 2, DAM 3)¶
DAM 2 adds a text backbone over the transcript. DAM 3 first trains a Whisper to imitate BERT's vector from audio (WhisperLLMA), then freezes it and trains only the head.
flowchart LR
subgraph research_stable_ordinal_regression_mtl_jun_2024_model_py___box["mtl_jun_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase["<b>DepAnxClassifierBase()</b><br/>Concatenate embeddings of several<br/>backbones (ModuleDict order) and<br/>feed a MultitaskHead."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L55" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT["<b>DepAnxClassifierWhisperBERT()</b><br/>DAM 2: Whisper audio + BERT<br/>transcript embeddings (768 + 768)<br/>-› multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L142" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA["<b>DepAnxClassifierWhisperLLAMA()</b><br/>Whisper audio + LLaMA transcript<br/>embeddings concatenated -›<br/>multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L169" _blank
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA["<b>DepAnxClassifierLLAMA()</b><br/>Text only: LLaMA transcript<br/>embedding -› multitask head."]
class research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA opt
click research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_jun_2024/model.py#L189" _blank
end
subgraph research_stable_ordinal_regression_mtl_aug_2024_model_py___box["mtl_aug_2024/model.py"]
direction TB
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA["<b>WhisperLLMA()</b><br/>DAM 3 phase 1: Whisper encoder<br/>trained to predict a cached text<br/>embedding from audio."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L32" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step["<b>WhisperLLMA.step()</b><br/>MSE between per-recording Whisper<br/>embedding and the cached teacher<br/>embedding."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L102" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier["<b>DepAnxClassifier()</b><br/>DAM 3: frozen Whisper + frozen<br/>WhisperLLMA embeddings (768 + 768)<br/>-› trainable head."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L150" _blank
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone["<b>DepAnxClassifier.forward_backbone()</b><br/>No-grad Whisper + WhisperLLMA,<br/>averaged per recording -› [B,<br/>1536]; lengths set to 1."]
class research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone opt
click research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/research/stable/ordinal_regression_mtl_aug_2024/model.py#L221" _blank
end
subgraph kirad_base_models_py___box["base_models.py"]
direction TB
kirad_base_models_py__BERTBackbone["<b>BERTBackbone()</b><br/>BERT on transcript tokens; returns<br/>the pooler output [ΣW, 768] (DAM 2<br/>text branch)."]
class kirad_base_models_py__BERTBackbone opt
click kirad_base_models_py__BERTBackbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L918" _blank
kirad_base_models_py__LLAMABackbone["<b>LLAMABackbone()</b><br/>LLaMA cut to num_layers layers;<br/>embeds text as the last real<br/>token's hidden state."]
class kirad_base_models_py__LLAMABackbone opt
click kirad_base_models_py__LLAMABackbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L941" _blank
kirad_base_models_py__Wav2Vec2Backbone["<b>Wav2Vec2Backbone()</b><br/>wav2vec2 encoder on raw audio,<br/>mean-pooled (not used by the<br/>stable DAM configs)."]
class kirad_base_models_py__Wav2Vec2Backbone opt
click kirad_base_models_py__Wav2Vec2Backbone "https://github.com/Pipotheavenger/kintsugi-rad-docs/blob/main/kirad/base_models.py#L888" _blank
end
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT -.-> research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierBase
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperBERT -.->|"text"| kirad_base_models_py__BERTBackbone
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierWhisperLLAMA -.-> kirad_base_models_py__LLAMABackbone
research_stable_ordinal_regression_mtl_jun_2024_model_py__DepAnxClassifierLLAMA -.-> kirad_base_models_py__LLAMABackbone
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA -.->|"phase 1: MSE"| research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA_step
research_stable_ordinal_regression_mtl_aug_2024_model_py__WhisperLLMA -.->|"frozen"| research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier
research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier -.-> research_stable_ordinal_regression_mtl_aug_2024_model_py__DepAnxClassifier_forward_backbone
classDef main fill:#FFF4DC,stroke:#A67C1B,stroke-width:2px,color:#1A1A1A
classDef opt fill:#F3F4F6,stroke:#B8BEC7,color:#555,stroke-dasharray:4 3
classDef edge fill:#FFFFFF,stroke:#1F7A5C,color:#1F7A5C,stroke-width:1.5px
| Step | Function | File | What it does |
|---|---|---|---|
| — | DepAnxClassifierBase |
research/stable/ordinal_regression_mtl_jun_2024/model.py:55 |
Concatenate embeddings of several backbones (ModuleDict order) and feed a MultitaskHead. |
| — | DepAnxClassifierWhisperBERT |
research/stable/ordinal_regression_mtl_jun_2024/model.py:142 |
DAM 2: Whisper audio + BERT transcript embeddings (768 + 768) -> multitask head. |
| — | DepAnxClassifierWhisperLLAMA |
research/stable/ordinal_regression_mtl_jun_2024/model.py:169 |
Whisper audio + LLaMA transcript embeddings concatenated -> multitask head. |
| — | DepAnxClassifierLLAMA |
research/stable/ordinal_regression_mtl_jun_2024/model.py:189 |
Text only: LLaMA transcript embedding -> multitask head. |
| — | WhisperLLMA |
research/stable/ordinal_regression_mtl_aug_2024/model.py:32 |
DAM 3 phase 1: Whisper encoder trained to predict a cached text embedding from audio. |
| — | WhisperLLMA.step |
research/stable/ordinal_regression_mtl_aug_2024/model.py:102 |
MSE between per-recording Whisper embedding and the cached teacher embedding. |
| — | DepAnxClassifier |
research/stable/ordinal_regression_mtl_aug_2024/model.py:150 |
DAM 3: frozen Whisper + frozen WhisperLLMA embeddings (768 + 768) -> trainable head. |
| — | DepAnxClassifier.forward_backbone |
research/stable/ordinal_regression_mtl_aug_2024/model.py:221 |
No-grad Whisper + WhisperLLMA, averaged per recording -> [B, 1536]; lengths set to 1. |
| — | BERTBackbone |
kirad/base_models.py:918 |
BERT on transcript tokens; returns the pooler output [ΣW, 768] (DAM 2 text branch). |
| — | LLAMABackbone |
kirad/base_models.py:941 |
LLaMA cut to num_layers layers; embeds text as the last real token's hidden state. |
| — | Wav2Vec2Backbone |
kirad/base_models.py:888 |
wav2vec2 encoder on raw audio, mean-pooled (not used by the stable DAM configs). |