The process of science and mathematics involves developing ideas and then proving them true. However, what is meant by a proof depends on what one is doing. In science, a proof is empirical. One starts with a hypothesis and then tests it experimentally or observationally. In pure math, a proof means that a given statement is consistent with a set of rules and axioms. There is a huge difference between these two approaches. Mathematics is completely internal. It simply strives for self-consistency. Science is external. It tries to impose some structure on an outside world. This is why mathematicians sometimes can’t relate to scientists and especially physicists and vice versa.
Theoretical physicists don’t need to always follow rules. What they can do is to make things up as they go along. To make a music analogy – physics is like jazz. There is a set of guidelines but one is free to improvise. If in the middle of a calculation one is stuck because they can’t solve a complicated equation, then they can assume something is small or big or slow or fast and replace the equation with a simpler one that can be solved. One doesn’t need to know if any particular step is justified because all that matters is that in the end, the prediction must match the data.
Math is more like composing western classical music. There are a strict set of rules that must be followed. All the notes must fall within the diatonic scale framework. The rhythm and meter is tightly regulated. There are a finite number of possible choices at each point in a musical piece just like a mathematical proof. However, there are a countably infinite number of possible musical pieces just as there are an infinite number of possible proofs. That doesn’t mean that rules can’t be broken, just that when they are broken a paradigm shift is required to maintain self-consistency in a new system. Whole new fields of mathematics and genres of music arise when the rules are violated.
The invention of the computer introduced a third means of proof. Prior to the computer, when making an approximation, one could either take the mathematics approach and try to justify the approximation by putting bounds on the error terms analytically or take the physicist approach and compare the end result with actual data. Now one can numerically solve the more complicated expression and compare it directly to the approximation. I would say that I have spent the bulk of my career doing just that. Although, I don’t think there is anything intrinsically wrong with proving my simulation, I do find it to be unsatisfying at times. Sometimes it is nice to know that something is true by proving it in the mathematical sense and other times it is gratifying to compare predictions directly with experiments. The most important thing is to always be aware of what mode of proof one is employing. It is not always clear-cut.