You can use non-deterministic simulations to make scientifically assisted predictions and the modelling would be valid at least in principle. Doing things right doesn't guarantee a correct result however.
In this case though they are describing dramatic differences in outcomes which is problematic.
It's not that it's technically non-deterministic but that it is greatly so.
My guess is they will say he ran it a thousand times and took the average from the results. Sometimes that's valid but more often people just do it with the assumption it fixes the problem. A rookie mistake. They assume consensus will find the right answer as is by magic.
These programs don't even replicate results when run from the same starting point.
This is not because of incompetence; it is by design. In fact, if they did not explicitly introduce chaos (in the form of one or more randomly-generated variables), they would produce the same result from the same seed each time, and that would defeat the purpose.
Yup, my experience with Monte Carlo simulations is in finance, specifically the Trinity Study. As I understand it, with these kinds of simulations, they say that xx% of the time it will yield a certain threshold result. For example, in the Trinity Study, it's safe to withdraw 4% of one's portfolio every year and there is a 95% chance the portfolio will survive one's entire retirement.
In the case of the IMHE model, it was framed as 100% chance and without any thresholds. That doesn't sound right to me.
You can use non-deterministic simulations to make scientifically assisted predictions and the modelling would be valid at least in principle. Doing things right doesn't guarantee a correct result however.
In this case though they are describing dramatic differences in outcomes which is problematic.
It's not that it's technically non-deterministic but that it is greatly so.
My guess is they will say he ran it a thousand times and took the average from the results. Sometimes that's valid but more often people just do it with the assumption it fixes the problem. A rookie mistake. They assume consensus will find the right answer as is by magic.
Correct.
Non-determinism in algorithms is okay; machine learning and Monte Carlo simulations do it all the time.
Forming conclusions and acting upon unstable (and buggy) non-determinism is not okay, and that's what happened here.
This is not because of incompetence; it is by design. In fact, if they did not explicitly introduce chaos (in the form of one or more randomly-generated variables), they would produce the same result from the same seed each time, and that would defeat the purpose.
Yup, my experience with Monte Carlo simulations is in finance, specifically the Trinity Study. As I understand it, with these kinds of simulations, they say that xx% of the time it will yield a certain threshold result. For example, in the Trinity Study, it's safe to withdraw 4% of one's portfolio every year and there is a 95% chance the portfolio will survive one's entire retirement.
In the case of the IMHE model, it was framed as 100% chance and without any thresholds. That doesn't sound right to me.
You are assuming altruistic motivations.
Do you think this code would have been any better if Ferguson weren't thinking with his dick while writing it? I have my doubts.