Plans to demolish Perth's infamously unfinished mansion, dubbed the Taj Mahal on the Swan, will be fought by the owners who still plan to live there one day.

The $70 million doomed eyesore regularly attracts trespassers and has been covered in graffiti since being abandoned by fertiliser magnates Pankaj and Radhika Oswal in 2011.

The Shire of Peppermint Grove voted unanimously on Tuesday night to demolish the derelict structure, which had been planned to include a temple, gym, observatory with a revolving roof and parking for 17 cars.

Taj Mahal on the Swan

When their empire collapsed in 2010, the Oswals left Australia and still owe millions of dollars in taxes.

Shire chief executive John Merrick told AAP that lawyers for the Oswals had 21 days to appeal the demolition order to the State Administrate Tribunal.

A lawyer for Ms Oswal, Rebekah Giles, said they had been in talks with the shire since November and were disappointed they had not yet been officially informed of the council’s resolution.

Ms Giles said her client’s efforts to resolve the council’s concerns were “frequently frustrated” by the shire’s media campaign against the structure.

“Any attempts to have the structure demolished will be resisted,” she said.

“It is Ms Oswal’s intention to finish the construction of her family home and reside in it.”

Ms Giles said her client would continue to engage contractors to deal with the health and safety concerns raised and talk with the council to resolve all outstanding issues.