The withdrawal, a rare retreat after AU troops seized swathes of territory from the radical Islamists, includes the towns of Qoryoley and Awdhegele, which have since been taken over by Shabaab in Somalia's southern Lower Shabelle region.
The Islamic militants have stepped up their attacks during Islam's holy fasting month of Ramadan.
"They pulled out of Qoryoley, there is not a single soldier left behind," local government official Mohamed Haji Osman said. Residents reported that the Shabaab had since moved in and hoisted their black flag over the town, some 80km southwest of the capital Mogadishu.
"Shabaab fighters entered the town and their flag is flying over the district headquarters. All Amisom and Somali troops pulled out," said resident Ahmed Hassan.
Awdhegele district commissioner Mohamed Aweys said he left the town after soldiers from the African Union Mission in Somalia (Amisom) retreated.
"This is a complete setback because al-Shabaab is gaining ground," he said.
The Shebab is fighting to overthrow Somalia's Western-backed government which is propped up and protected by the 22 000-strong Amisom force.
Last week, the Shebab attacked an Amisom base in Lego village, 100km northwest of the capital Mogadishu, manned by around 100 Burundian soldiers.
The Shabaab later boasted of having killed 80 soldiers and carried off the bodies of 60 of them, before setting the base on fire.
Video images taken by the Shabaab and seen by AFP show several bodies scattered around the ransacked base, with one shot showing around a dozen bodies lined up.
Shabaab commander Mohamed Abu-Abdallah said his fighters were celebrating their advance in Lower Shabelle.
"The enemy is fleeing from the region... this is the beginning of the fall of the Christian invaders," he said.