They criticised Kenyan authorities and warned people not to pass intelligence to the security forces.
The mosque is in Garissa county, 240km (150 miles) from Garissa town, where al-Shabab last month killed at least 148 people at a university.
Police commander Caleb Matoke told The Standard newspaper that 25 fighters were involved. He said he believed they were sent by Mohamed Kuno, the man Kenyan authorities have accused of being behind the Garissa University college attack.
One resident told the BBC: "They [militants] came to [Hulugho] town and proceeded to the mosque where they gathered people and started giving out sermons.
"They said that they are also Muslims and they are only targeting government officials whom they have a problem with."
Al-Shabab has been at war with Kenya ever since Kenyan forces entered Somalia in October 2011 in an effort to crush the militants.
It has staged numerous attacks in Kenya. As well as the attack on Garissa university, al-Shabab also attacked a shopping mall in 2013, leaving at least 67 dead. Westgate Mall is due to open again in July.
The BBC's Abdullahi Abdi in Nairobi says Kenya is building a wall between Somalia and Kenya in an effort to keep out al-Shabab militants but this shows that the militants are active deep inside Kenya.