This would be detrimental to many careers, as there are several professional drivers (HGV, bus etc.) who would be at risk of losing their job.

Young people are reluctant to learn how to drive as it is, so implementation of this will wipe out the skill of driving within generations.

Also, I imagine this would be a danger, as many autonomous vehicles are designed to prioritise the drivers safety. For example, if someone was to walk in front of the car, the car would likely prioritise the drivers comfort by not giving them whiplash, and continue driving.