The Lib Dem MP for Bristol West said pubs in parts of the city were closing because of the decline of the "white working class" and the arrival of migrants from Somalia and elsewhere.
He said reports of his remarks had been "twisted" to suggest a focus on race.
He had simply been proposing derelict premises should be converted, he said.
The comments were made as Mr Williams was outlining a government proposal to protect community pubs.
The plan involves changing the law to prevent places listed as assets of community value being demolished or having their use changed without planning permission.
Arguing against a blanket ban, Mr Williams told MPs: "In my own constituency, lots of pubs have closed - but it is usually because of demographic change.
"Some parts of my constituency, which had a 'white working-class community' 20 or 30 years ago, are now populated primarily by recently arrived Somalis and other people.
"Obviously the pubs in those areas have closed, and some have been converted to other uses, but some of them are still derelict.
'No value judgement'
"What the government is proposing is to look at the public houses that are genuinely popular and valued by the community now, giving them protection that is already allowed in the Localism Act and further enhancing that protection in terms of the planning laws, saying you cannot convert this pub into another use or demolish it without planning permission."
Some reports, including one in the Daily Telegraph newspaper, with the headline "'Somali immigrants' cause pub closures", had taken his comments out of context, Mr Williams said.
"There was no value judgement in what I said. It was simply an observation," he told the BBC.
"Some pubs in the Barton Hill and Redcliffe area of my constituency have closed because of the arrival of a non-drinking Muslim population to the area.
"I could have made a point about pubs closing because of the decline of a factory or a football stadium - both of which also apply to my constituency," he said.
"I was simply saying there was no point us giving blanket protection to all pubs, as some people suggest.
"But what we are announcing is powers for people across the country to protect their much-loved pubs by triggering a planning application if they are listed as a community asset."
Mr Williams rejected any suggestion his comments had been motivated by a negative view of migration.
"I have consistently argued that immigration is a good thing," he said. "That is my view as a Liberal."