He believes the issue could erode the illusion that comedians are the audience's friends, the very everyman liveability which has helped fuel the current explosion in stand-up.
In a wide-ranging interview in the new edition of ScotsGay magazine, Lee said: †'I think this Jimmy Carr tax avoidance thing is pivotal.
'I think the public need to get fed up with stand-ups, and then the distinctive ones will be left standing. It might be a watershed moment.
'I think the assumption will be that any comedian on stage is a cunt and has stolen loads of money and therefore isnít entitled to talk about anything. It might make it difficult for everyone.'
Lee also said that Carr's position could lead to an end of 'ironically' nasty stand-up.
'The idea that youíre being ironic about the weaker members of society doesnít really work if youíve concealed millions of pounds a year which would have gone towards helping them.' he told interviewer Andrew Doyle.
'This perception that Jimmy Carr is in some way left-wing has always struck me as funny. There isnít any indication of him having any political affiliation at all, although his material would suggest if he did have one it would be right-wing anyway.
However, Lee was not too scathing about Carr's tax arrangements saying: 'In his defence, what heís done is a tiny drop in the ocean compared with what some of David Cameronís friends in business have done. Which is why itís so funny when Cameron chose to single out Jimmy Carr. Itís clear that no oneís advised him. He hasnít talked to anyone and heís just opened his mouth
'If handled properly, this could bring down the government. To pick on Jimmy Carr when youíre giving knighthoods to CEOs who are evading much more is absurd.
And Lee revealed he had been under investigation by the Inland Revenue himself... but their inquiry ended up with him receiving a rebate.
Click here to read the full interview.