A proposed $5 congestion tax for Melbourne's CBD has been shot down just hours after the government's hand-picked experts said it was a top priority.

Premier Daniel Andrews has ruled out putting tolls on existing roads, despite the recommendation being one of Infrastructure Victoria’s key planks for tackling congestion.

“We’ve had a very consistent policy about not tolling existing roads. That remains our policy, and that won’t be changing,” Mr Andrews told reporters on Tuesday.

Under one proposed model it would cost $5 to drive into inner suburbs like Clifton Hill and South Yarra and through the city.

“We are already paying the price of congestion,” chief executive Michel Masson told reporters.

“Twenty per cent of car trips in the morning are not related to work or education.”

But Mr Andrews said Infrastructure Victoria – which was set up to “take the politics out of infrastructure” – had provided interesting ideas for a debate.

Brian Negus, RACV general manager of public policy, said motorists were already paying fuel excise, sales tax, import duties and registration.

“Get rid of all those taxes and put in a proper road user price. It includes congestion but also includes how heavy you are – a truck – and how far you drive,” Mr Negus told ABC radio on Tuesday.

The premier was a lot stronger on the North East Link freeway, backing it to get built as a project that stacks up.

“It’s pretty farcical isn’t it that 20-30 years later we still don’t have the ring road acting as a ring around the city,” he said.

Infrastructure Victoria said the northeastern Melbourne road should be built within 15 years.

Opposition Leader Matthew Guy said a congestion charge would discourage people from living in downtown Melbourne.

“I don’t think that road user charge in terms of a congestion charge is a sensible idea,” Mr Guy told reporters.

Mr Guy backed the North East Link, but said the controversial axed East West Link also needed to be built.

The report didn’t recommend an East West Link timeline, but strongly encouraged the government to preserve the corridor for it to be built one day.

It also called for the closure of some police stations to create “supersites”, and opening up schools for community use.

The government will consider the draft report, but Police Minister Lisa Neville has already ruled out closing police stations.

“With the crime rate, we need to focus on giving more powers to the police and getting more police out on the streets,” she told ABC radio.