Hundreds of thousands of unpaid carers were hit with ruinous penalties for minor breaches of carer’s allowance rules in the five years after senior welfare officials promised to fix the scandal-hit benefit, an official audit has revealed.

Campaigners said the National Audit Office (NAO) report highlighted the scale of the misery and hardship inflicted on carers over the period as well as the extent of the failure by the Department for Work and Pensions (DWP) to tackle overpayments.

More than 262,000 repayable overpayments totalling in excess of £325m were clawed back from carers who had – mostly unwittingly – fallen foul of carer’s allowance rules, while 600 carers were prosecuted and received criminal records after DWP civil servants referred their cases to the Crown Prosecution Service.

Charities and MPs said the watchdog’s report showed that the DWP’s failure to properly staff its carer’s allowance department and address flaws in the benefit’s design had left thousands of carers in financial difficulties and emotional distress.

A Guardian investigation this year revealed how Conservative ministers allowed overpayments to grow almost unchecked for years, despite senior officials promising MPs in 2019 that new data-matching technology would eradicate the problem.

There are currently more than 134,500 unpaid carers repaying overpayment debts of more than £251m after falling foul of “cliff-edge” rules on earnings limits. A carer who earned £1 more than the £151 threshold for 52 weeks would have to pay back not £52 but £4,258.80.

The NAO report revealed that staffing levels in the carer’s allowance processing unit were designed to meet internal financial savings targets rather than to ensure carers were prevented from inadvertently accruing overpayments.

As a result, only half the electronic alerts identifying potential benefit overpayments were investigated annually. The number of staff assigned to check alerts and investigate cases was cut by 14% over the past two years.

The DWP has repeatedly refused to accept that it has any responsibility to identify all earnings breaches or alert individuals to potential overpayments, arguing that in law the onus is on claimants to inform it of any changes that could affect their eligibility for carer’s allowance.

On average, family carers – most of whom are in poverty – ended up repaying nearly £1,000 in earnings-related overpayments last year, although the Guardian has revealed that since April two cases had accrued more than £20,000 in overpayments, suggesting the DWP failed to spot the breaches for nearly five years.

The Liberal Democrat leader, Ed Davey, said: “It is appalling that so many family carers have been caught up in this scandal of the DWP’s own making. This has caused misery and distress on a staggering scale.”

Vivienne Groom, a former unpaid carer who was prosecuted for fraud and had a £16,000 inheritance from the mother she cared for seized by the DWP after unwittingly running up carer’s allowance overpayments, said it was disgusting that the DWP was still dragging carers through the courts.

“They need to stop,” she said. “I just feel sorry for all those people that are coming behind me who will end up losing their house or car.”

Helen Walker, the chief executive of Carers UK, said the NAO report showed a “serious failure” to act by the DWP which had “left thousands of carers experiencing emotional distress and financial hardship for years”.

Dominic Carter, the director of policy and public affairs at Carers Trust, said: “This vital report makes clear that far too many people are being allowed to rack up ruinous debts because of the broken carer’s allowance system.”

Stephen Timms, the minister for social security and disability, said: “This report sets out the scale of the challenge and underlines the importance of our independent review into overpayments so we can make the system fairer for thousands of selfless carers.

“Carers deserve to be supported, which is why we are boosting the earnings threshold, benefiting more than 60,000 people, while our review will get to the bottom of the problem so we can protect carers from unfair debt and protect taxpayers’ cash.”

The review, due to report next summer, will look at why carer’s allowance overpayments spiralled out of control, how to prevent them, and how best to support carers who fall foul of the system.