Abstract: The intersection of physics and machine learning (ML) allows both fields to benefit from each other. By treating ML as a physical system, we study it as a phenomenon to be understood through controlled experiments, well-defined datasets, and falsifiable hypotheses. Conversely, by using ML as a tool, we can accelerate discoveries in physics across a variety of fields. This talk covers 3 projects spanning these topics. A large portion of the talk is dedicated to our work on developing physics-style methods using exact likelihoods, inductive biases, and information theory to probe what these models are really learning and explore the edge cases where they succeed or fail. Our first project models two hypotheses as Gaussian processes, allowing us to generate data for which we can compute the exact likelihood of a sample under both hypotheses. This allows the construction of a Bayes optimal classifier, an architecture-independent upper bound on performance, and the benchmarking of real classifiers against it to isolate the role of inductive bias. Our second project introduces a framework for generating highly multimodal datasets with explicitly calculable mutual information (MI) between modalities. This enables controlled experiments that systematically probe how mutual information and its distribution across modalities affect multimodal self-supervised learning and mutual information estimation. The remaining portion of this talk focuses on facilitating the use of ML in the designing of fusion devices. Our third project introduces Driftless Star, an open-source workflow that computes transport-consistent plasma profiles for a given stellarator boundary and plasma conditions in a unified, reproducible pipeline. The pipeline enables AI-accelerated fusion device design by allowing neural surrogates to replace costly physics stages and large language model agents to orchestrate the chain, make judgment calls between stages, and autonomously search for improved geometries.