When the model is applied to a regulated part of the market, it's becoming apparent that the institutions using the model have to be able to defend it.
David Heinemeier Hansson (DHH) posted an interesting thread to Twitter after he was granted a credit limit 20x higher than his wife and the credit card company's initial response was it's just the algorithm.
Really? That's encouraging. I thought we're heading into some ironic dark enlightenment horseshoe of blackboxiness where the only artificial intelligence that works is as inscrutable as the natural intelligence it was meant to compete with.
Most models don't bother, because that isn't a necessary step to get reasonable answers. But it's possible.