Physics Informed Viscous Value Representations

Offline goal-conditioned reinforcement learning (GCRL) learns goalconditioned policies from static pre-collected datasets. However, accurate value
estimation remains a challenge due to the limited coverage of the state-action
space. Recent physics-informed approaches have sought to address this by imposing physical and geometric constraints on the value function through regularization defined over first-order partial differential equations (PDEs), such as
the Eikonal equation. However, these formulations can often be ill-posed in
complex, high-dimensional environments. In this work, we propose a physicsinformed regularization derived from the viscosity solution of the HamiltonJacobi-Bellman (HJB) equation. By providing a physics-based inductive bias,
our approach grounds the learning process in optimal control theory, explicitly
regularizing and bounding updates during value iterations. Furthermore, we leverage the Feynman-Kac theorem to recast the PDE solution as an expectation, enabling a tractable Monte Carlo estimation of the objective that avoids numerical
instability in higher-order gradients. Experiments demonstrate that our method
improves geometric consistency, making it broadly applicable to navigation and
high-dimensional, complex manipulation tasks.