Talking on a phone is more demanding than talking to someone in the car: the audio signal is degraded, you have no visual information to rely on, your conversational partner doesn’t stop talking if the road situation becomes complicated, you may try to imagine being with the other person, and so on. (It is true, however, that driving is somewhat worse with hand=held than hands-free devices; Backer-Grøndahl & Sagberg, 2011.)
So what's the solution? Presumably, not answering your phone.
Now new data show that even that carries a cost.
Seven dependent measures were collected, including crossings of the center line, collisions with other cars, and so on.
Ignoring the ringing mobile phone carried a cost. When their phone rang just prior to a hazard, subjects were more likely to hit pedestrians, to exceed the speed limit, and to cross the center line.
It is notable that these effects were more pronounced in subjects who said they usually answered their phone while driving than in subjects who said they did not. Hence, a significant contributor to the distraction may be the mental effort required to inhibit a habitual response.
What's the solution? The safest practice for drivers may be to turn the phone off, not just ignore it.
Backer-Grøndahl, A., & Sagberg, F. (2011). Driving and telephoning: Relative accident risk when using hand-held and hands-free mobile phones. Safety Science, 49, 324-330.
Holland, C., & Rathod, V. (2013). Influence of personal mobile phone ringing and usual intention to answer on driver error. Accident Analysis & Prevention, 50, 793-800.
Treffner, P. J., & Barrett, R. (2004). Hands-free mobile phone speech while driving degrades coordination and control. Transportation Research Part F: Traffic Psychology and Behaviour, 7, 229-246.