Principled Multimodal Risk Estimation for ICU Deterioration: Integrating Structured Sequences and Unstructured Notes with Robust Fusion

• Thanawat Boonmee

  Department of Computer Engineering, Uttaradit Rajabhat University, 27 Injaimee Road, Tha It, Mueang Uttaradit, Uttaradit 53000, Thailand

  Author

• Kittipong Saelim

  School of Information Technology, Sakon Nakhon Rajabhat University, 680 Nittayo Road, That Choeng Chum, Mueang Sakon Nakhon, Sakon Nakhon 47000, Thailand

  Author

Intensive care units continuously generate heterogeneous data streams that encode evolving physiologic state and clinician interpretation. Despite dense monitoring, clinically meaningful deterioration events often emerge from subtle, multivariate trajectories that are difficult to recognize early using threshold-based scoring rules. Computational approaches that treat prediction as a sequential inference problem can exploit temporal dependencies, but they are frequently constrained by irregular sampling, missingness, and the fact that many clinically salient cues appear only in free-text documentation. This paper develops a  for early deterioration forecasting that integrates structured time-series measurements with unstructured clinical text under a unified representation-learning perspective. We formalize ICU forecasting as conditional risk estimation over partially observed multichannel sequences, introduce a principled multimodal encoding strategy that maps both modalities into a shared latent space, and propose a fusion mechanism that remains well-defined when text is absent or temporally misaligned. The modeling components are analyzed in terms of identifiability under informative missingness, robustness to label sparsity, and calibration under extreme class imbalance. We emphasize objectives that control both ranking performance and probability quality, including imbalance-aware proper losses and post-hoc calibration operators that preserve discrimination. The resulting methodology is positioned as a general computational template rather than a dataset-specific recipe. We discuss design tradeoffs among recurrent, convolutional, and attention-based temporal encoders; contrast early, late, and gated fusion from an information bottleneck viewpoint; and derive operational metrics appropriate for low-prevalence event forecasting. The overall contribution is a cohesive technical treatment of multimodal ICU risk modeling with explicit attention to irregular sampling, fusion stability, and calibrated decision support.

• PDF