A culture of “blindness, ignorance and prejudice” led to repeated failures over decades to properly investigate cases in which children were abused by grooming gangs, a report has said.

As the government announced a public inquiry into the scandal, Louise Casey said for too long the authorities had shied away from the ethnicity of the people involved, adding it was “not racist to examine the ethnicity of the offenders”.

Lady Casey said she found evidence of “over-representation” of Asian and Pakistani heritage men among suspects in local data – collected in Greater Manchester, West and South Yorkshire – and criticised a continued failure to gather robust data at a national level.

The home secretary, Yvette Cooper, confirmed the government would accept all 12 recommendations of Casey’s rapid review, including setting up a statutory inquiry into institutional failures. This marked a significant reversal after months of pressure on Labour to act.

“While much more robust national data is needed, we cannot and must not shy away from these findings, because, as Baroness Casey says, ignoring the issues, not examining and exposing them to the light, allows the criminality and depravity of a minority of men to be used to marginalise whole communities,” Cooper said.

The number of cold cases to be reviewed again over child sexual abuse by grooming gangs is expected to rise to more than 1,000 in the coming weeks, she told the Commons.

Adult abusers targeted children, mainly girls, some as young as 10, and some of whom were in care, had physical or mental disabilities, or who had already suffered neglect or abuse.

According to Casey, the ethnicity of grooming gangs has been “shied away from” by authorities, allowing the continued abuse of hundreds of vulnerable girls, many of whom are now demanding justice.

Casey said there should be “a vigorous approach to righting the wrongs of the past” and state agencies should be held to account for any part they played in allowing these crimes to go undetected and unpunished.

“Blindness, ignorance, prejudice, defensiveness and even good but misdirected intentions, all play a part in a collective failure to properly deter and prosecute offenders or to protect children from harm,” she said.

In the report, Casey said: “We as a society owe these women a debt. They should never have been allowed to have suffered the appalling abuse and violence they went through as children.”

On the question of ethnicity, it said: “We found that the ethnicity of perpetrators is shied away from and is still not recorded for two-thirds of perpetrators, so we are unable to provide any accurate assessment from the nationally collected data.”

However, it added that at a local level for three police forces – Greater Manchester, South Yorkshire and West Yorkshire – there was enough evidence to show a “disproportionate numbers of men from Asian ethnic backgrounds amongst suspects for group-based child sexual exploitation”.

Asked if she was worried recording the data could lead to civil unrest, Casey said: “So let’s put it the other way around. If for a minute you had another report that ducked the issue, what do you think is going to happen? Do you think they’re not going to use that as well?”

She added: “If good people don’t grip difficult issues, in my experience bad people do.”

Casey also looked at about 12 live investigations and found that “a significant proportion appear to involve suspects who are non-UK nationals”, some of whom were claiming asylum in the UK.

Casey’s recommendations, which have been accepted in full, call for:

• Five existing local inquiries into grooming gangs to be coordinated by an independent commission with full statutory inquiry powers.

• The collection of ethnicity and nationality data for all suspects in child sexual abuse and criminal exploitation cases to be made mandatory.

• The law to be tightened to ensure there is no exception to those who sexually penetrate a child under 16 being charged with rape. Casey said she believed the public would be horrified to realise this was not the case already.

• Research into the drivers for group-based child sexual exploitation, including the role of social media, cultural factors and group dynamics.

• Every local police force in England and Wales to review records to identify cases of child sexual exploitation that have not been acted upon, including a review of cases that have been reported but have not resulted in prosecutions over the last 10 years. Convictions of the young victims, many of whom say they still face appalling discrimination, should be quashed.

Casey cited police figures from the 1990s which found almost 4,000 police cautions were given to children aged between 10 and 18 for offences relating to prostitution. It took until 2015 for the term “child prostitution” to be dropped and replaced with the term “child sexual exploitation”, when the legislation was changed in the Serious Crime Act.

She said that victims had regularly been retraumatised over the years from the shame of their convictions and the anger and at not being believed or living alongside their perpetrators.

“Sometimes they have criminal convictions for actions they took while under coercion,” Casey said. “They have to live with fear and the constant shadow over them of an injustice which has never been righted – the shame of not being believed.”

The report detailed how “group-based child sexual exploitation” is a “sanitised” way of talking about multiple sexual assaults against children by multiple men, including beatings and gang rapes.

Reacting to the report, the children’s commissioner for England, Rachel de Souza, said the failure to protect the girls was “a source of national shame”.

“This inquiry must be a wake-up call for how we respond to vulnerable children, especially violence against girls,” she said. “We cannot be more afraid of causing offence than we are of speaking out to protect children from exploitation and corruption.”

The Home Office said a nationwide policing operation to bring grooming gang members to justice would be led by the National Crime Agency.

Police have reopened more than 800 cases of child sexual abuse since Cooper asked them to review cases in January.