Three former Cabinet ministers have led calls for the prime minister to make radical changes.

As Labour's conference got under way on Sunday, Tony Blair was tackled over trust, top up fees and his relationship with the Labour Party.

Former Cabinet minister and whip Nick Brown, who is close confident of chancellor Gordon Brown, warned re-establishing a clear direction and trust with voters now had to be the top priority for the government.

"There is a feeling now of disquiet, a sense that we have lost our way and in particular a feeling that we are losing the trust of people that ought to be on our side,'' he told the BBC1 Politics Show.

Brown also joined the ranks of MPs to criticise top-up fees.

"Increasing the amount of debt that students are expected to carry will be a barrier to students from poorer backgrounds and the Labour Party has no business putting barriers in the way of those that come from less privileged homes,'' he said.

Two former health secretaries also lobbied Tony Blair to change direction.

Frank Dobson, who was Tony Blair's first health secretary, warned that the prime minister was now "very unpopular" with both rank and file party members and the wider public.

"He has never really been loved by the party and I think that is a problem now,'' he told GMTV's Sunday programme.

"If your asset is that people in the party can realistically see that he does appeal over our heads to the public, if he ceases to appeal to the public, then most of his attractions are rather diminished."

Dobson also gave a characteristically blunt assessment of the party's current standing with the public.

"We are about as popular as the Tory Party led by Iain Duncan Smith and you can't get much worse than that."

He called for a u-turn on foundation hospitals and top-up fees.

"They are very bad and unpopular policies and very harmful,'' he said.

"The idea of going into the next general election being the only party that is going to put up fees for your children to go to university doesn't seem a very promising way to enter an election."

And his successor Alan Milburn, who quit to spend more time with his family, broke his summer of silence with a call for the prime minister to stop talking like a "competent administrator".

Writing in the Guardian, the former health secretary said there is a "sense that the government is drifting" and a "fog of doubt" hanging over Tony Blair.

Urging the government to adopt more "conviction politics", Milburn argued that Labour needs to get a sense of direction.

"Labour is in power to change things, not keep them the same, transformation not consolidation," he writes.

"Above all the public want to know where we are taking the country."

Milburn also warned Blair that being bold is not enough.

"Boldness is a means not an end. There must be a purpose to it. Faced with a tide of uncertainty, now, more than ever, Tony Blair has to spell out what the purpose of New Labour is," he said.

"He has to communicate it in terms of our values and vision for Britain. For me - and I suspect for him - it is simple enough. It is about creating a fair future for all."

Senior Labour backbencher David Hinchliffe, who is standing down as Wakefield's MP at the next election, also launched a broadside on the prime minister.

The health select committee chairman accused Blair to pursuing a right wing agenda that went against the values of the Labour Party.

"When the prime minister talks about modernisation, some of the ideas we're bringing forward are from the dustbin of Margaret Thatcher."

"We are losing direction...It worries me that we are departing from those principles, I believe at our peril."

But the government's record was defended by education secretary Charles Clarke.

He claimed the government appeared to the public to be "preoccupied" with the fallout from the Iraq conflict but this would be dispelled at the party's conference.

"The period over the summer on Iraq, on Hutton and all the rest of it has lead to a sense of preoccupation with those questions,'' he said.

"It is not true but it is a perception that is there. We are not drifting at all. There is, in fact, a very strong purpose which I think you will see this week.

But there also came a repeat that there would be no compromise on the issue of tuition fees, with Clarke ruling out a flat rate.

"Is it true that a degree from university A is identical to a degree from university B? Or course A compared to course B?" he said.

"It is not true. The fact is different universities, different courses, give better opportunities later in life. It is not unfair to have a fee system that reflects that. We are not talking about a flat rate, no. I know some of my colleagues support it. We are talking about a variable rate."