Researchers continue to search for modeling strategies that improve the representation of high-wavenumber content in practical computational fluid dynamics (CFD) applications across several applications. In this talk, we present novel modeling approaches that integrates data-driven methods with physical intuition in obtaining turbulence models that demonstrate a dynamic behavior for canonical unsteady decaying turbulence problems on coarse-grids. We demonstrate results from deployments which frame the turbulence-modeling problem as one given by regression, where an optimal closure is predicted, or as one of classification where an optimal closure model is flagged. Finally, we will discuss some promising avenues for the design and development of scalable machine learning based turbulence modeling approaches with domain-awareness, interpretability and robustness using leadership-class supercomputers.