History of Science and Mathematics
I dislike abbreviating "Mathematics" to "Math", in part because the abbreviation is dialect-specific (British English speakers use "Maths" rather than "Math") and in part because it seems a bit too casual. "Math History" sounds okayish to me, but "History of Math" has an odd contrast. In modern day English, prepositions (especially those which can be avoided by transposing word order) seem to increasingly indicate somewhat formal writing, in which I would not expect to see the abbreviation Mathematics -> Math(s). I guess it was intended to smooth this out somewhat by first combining Science and Mathematics to "Science and Math" (which still sounds problematic due to the mismatch of levels of formality, but significantly less so because we've swapped the preposition for a conjunction and because there's no equivalent abbreviation of the term "Science") and then prefixing "History of" to the name, but in my mind this naturally gets distributed out to "History of Science and History of Math", retaining the confounding subphrase.
Assuming it is necessary to explicitly include Mathematics in the title (and enough comments on Area 51 seem to indicate that it is for some users here), I think it should be written out completely.