I figured this out. Don't go into Address Book.
Go into Contacts and do it. Somehow this stores it in an additional / duplicate / different location. When I get in the car and it fails out saying I have no addresses in my "Address Book" .... I can go to "Contacts" and boom - they all pop up.