Parliament has its “sleazeballs” just like any other employer, but Grantham MP Nick Boles says sexual harassment is less of an issue today than it was 20 years ago when it was “very bad.”
The 51 year-old also said no-one had ever made any unwelcome advances on him in Westminster.
The MP’s comments came as national media reported 36 Conservative MPs being named in a “dirty dossier”.
Allegations have also been made against MPs from Labour and other parties.
Prime Minister Theresa May has said she will sack any ministers found guilty of being a sex pest.
Commons leader Andrea Leadsom is to make a statement on proposed changes to grievance procedures to tackle sexual harassment in Parliament.
Mr Boles said: “There are 650 MPs, 800 in the House of Lords, and another few hundred who work there as members of staff. In any industry and community there are going to be a few sleazeballs.”
The former minister said people expected higher standards of their MPs and sexual harassment was “not acceptable.”
He said: “I have been here seven years. I think most people think the problem is much less spread widespead now than 20 years ago. I think it was very bad 20 years ago.”
Asked what he knew, the MP continued: “There will be a few bad apples. We need to identify them and prevent them from behaving this way. There’s lots of tittle tattle in any community.”
“You do not know what is gossip and what is real.”
He called on Parliament to “set up a proper process” to deal with the issue.
“We must not fall into a kangaroo court. We need to be a bit careful not to rush into judgement.”
Mr Boles said such abuse derived from status and power.
While male MPs have been reported for making advances against both women and younger men, he said he was not aware of any women MPs or ministers behaving inappropriately.
Finally, when asked if anyone “hit on” him in Parliament, Mr Boles added: “I should be so lucky! No, no, no. I am 6ft 5”. You would have to be fairly confident.”