Barclay was asked when he was made aware that there was a risk of Legionella on the barge, which asylum seekers moved onto last week as part of a government strategy to reduce costs of housing migrants in hotels, before being removed at the end of the week because of the dangerous bacteria in the water surrounding it.
He said he found out on Thursday evening but BBC host Sally Nugent said Dorset Council claimed they alerted a Home Office contractor on Monday evening.
"As health secretary, obviously I wouldn't be cited on whether someone in the Home Office had been alerted or not," he said.
"This is a Home Office lead," he added.
Nugent said: "Health is your brief, you would know how dangerous Legionella is."
Barclay said people were removed from the barge as soon as the government was aware of the situation.
People were unimpressed with his response and one said it was "insulting".