The Liberal Democrat frontbencher made the remark as he explained the Government's proposal to change the law to prevent establishments which have been listed as assets of community value being demolished or having their use changed without planning permission.
It comes after comments made in December by Tory peer Lord Hodgson of Astley Abbotts, who said the growing Muslim population was one of the main reasons for pubs closing.
Today in the Commons, Labour's Andrew Gwynne (Denton and Reddish) described the process set out by the minister as "incredibly convoluted" and asked: "Why not just give the planning protections in the first place and allow the local authority to decide through the planning process whether the pub should be saved for the future or not?"
But Mr Williams (Bristol West) said "blanket protection" of every single pub in the country would mean protecting those no longer enjoying the patronage of the local community for a range of reasons such as the arrival of immigrants. He added:
"In my own constituency, lots of pubs have closed but it is usually because of demographic change. Particularly in some parts of my constituency, which used to have 'a white working class community' 20 or 30 years ago are now populated primarily by recently-arrived Somalis and other people.
"Obviously the pubs in that area have closed. Some have been converted to other uses, some of them actually are still derelict.
"What the Government is proposing today is to look at the public houses that are genuinely popular and valued by the community now, giving them protection that is already allowed in the Localism Act and further enhancing that protection in terms of the planning laws, saying you cannot convert this pub into another use or demolish it without planning permission.
"That should address all of the worries that people rightly have about the pubs that really are important to them."