Britain needs to do more to help veterans who struggle to return civilian life, find it hard to secure work and risk ending up homeless, in prison or killing themselves, according to a top-level government review.

The 186-page report by the prime minister's special representative on veterans' affairs, Lord Ashcroft, found that those leaving the services after a relatively short time – four years – found it the hardest to cope.

But Ashcroft said it was a "misconception" that most veterans ended up in difficulty, and criticised charities and the media for perpetuating the view of veterans suffering from serious mental and physical problems, ending up homeless or in prison.

He said nine out of ten people thought it was common or very common "for former service personnel to be physically, mentally or emotionally damaged by their time in the forces. People think that have heard, or read, that veterans are unusually likely to be homeless, to suffer post-traumatic stress syndrome, to go to prison, even to commit suicide".

Ashcroft added: "It is easy to see where these ideas come from. Over the course of two unpopular wars we have become sadly accustomed to hearing reports of British military casualties. The media are naturally drawn to such news, while charities, wanting to raise money and draw attention to their cause, understandably highlight the most difficult cases."

The report says: "The MoD and the armed forces should be more proactive in changing perceptions of service-leavers. Lord Ashcroft's research has found that most people think it is common for those leaving the forces to have been physically, mentally or emotionally damaged by their service career. This view is mistaken and has the effect of restricting service-leavers' prospects in civilian life."

In spite of that, the report recommended help for some. "Those leaving with less than four years' service – who may have completed tours in places like Afghanistan – are the most likely to be unemployed and to have a range of problems."

It recommends that those leaving after four years should receive the same resettlement programme as those leaving after six or more and calls for an employers' council to help former service personnel find work.

Ashcroft said: "Those who serve four years or less get much more basic advice. Yet these early service-leavers are most likely to have problems when they leave. Only half of them currently manage to find a job within six months, compared to some 85% of those who receive career transition partnership support.

"Despite their relatively short service, these individuals have done their bit – often having completed operational tours in places like Afghanistan."

Ashcroft said he believed that work placement schemes would be more effective than some of the courses on offer. He questioned the value of someone who took a tiling course because he wanted to do up his house and others who took up courses in fly-fishing.

The report proposes a veterans' card that would provide a single number for forces' charities to replace what the report describes as "a confusing array of information that currently exists". He hoped that in time it would develop into a more substantial card, comparable to those in the US that offer a range of benefits, including price reductions in stores.

The initial reaction from the MoD was to accept the vast bulk of the recommendations, Lord Ashcroft said.