The Victorian government is refusing to return $1.5 billion in federal infrastructure funding and has an unexpected ally in its corner - the state opposition.

Both the Labor state government and Liberal opposition say the funds allocated to the scrapped East West Link tollway should stay in Victoria for spending on infrastructure.

Premier Daniel Andrews rejected the funding claw-back in Tuesday’s federal budget, saying it ran counter to the prime minister’s assurance he would not “rip off” the state and could support an alternative Western Distributor project.

“We’re not giving back any money,” Mr Andrews told reporters.

Victoria’s Treasurer Tim Pallas said the claw-back would punch a $730 million hole in Victoria’s 2015-16 budget and the state would ultimately lose $3 billion – the total federal allocation of the scrapped East West Link – across the forward estimates.

“It’s not their $1.5 billion. It is Victoria’s $1.5 billion for infrastructure that Victoria deserves,” Mr Pallas also told ABC Radio.

There were hopes the funding could be used as the required federal contribution to Transurban’s proposed $5.5 billion Western Distributor, should the project pass an assessment.

Mr Pallas said Victorians were being punished for ousting a one-term coalition government.

“That would have to be the inescapable conclusion of the behaviour of the federal government,” he said.

Liberal opposition leader Matthew Guy said the Andrews government’s first budget was unravelling and the premier was more interested in playing politics than being constructive.

“I’ll be seeing the prime minister in the next few days and I will take it upon myself to have a sensible and constructive dialogue … about why moneys should remain in Victoria for Victorian infrastructure projects,” Mr Guy told reporters.

“That is a hell of a lot more than it appears Daniel Andrews can do.”

Prime Minister Tony Abbott has also stated the $3 billion in federal funding should remain in a “locked box” awaiting a future Victorian government that would proceed on the East West Link.

Mr Truss said Premier Daniel Andrews should avoid getting into a silly argument over the issue.

“He’s got a clear obligation to return the money,” he said, adding the federal funds could not be used for other projects.

However, Mr Truss – who is also infrastructure minister – said he was prepared to have constructive talks with the Victorian government about a new program of works.