Influencing generalization through the geometry of the loss
The minimal locus of the loss in models like neural networks is typically a complicated geometric object. Different regions in the minimal locus exhibit different generalization behavior. I'll introduce the technique of *patterning*, by which one may attempt to modify the optimization problem -- for instance by tuning the mixing parameters in a mixture of training data distributions -- in order to (dis)incentivize (un)desirable generalization behavior. This approach uses the program of singular learning theory first developed by Watanabe to understand the effect of the data distribution on the Bayesian posterior associated to the loss.

