The settlement is with the US Federal Housing Finance Agency, which oversees the secondary mortgage market.
A separate deal with the Department of Justice is expected later this year.
The boss of RBS said the fine was a “stark” reminder of what happened to the bank before the financial crisis.
RBS has set aside a total of £6.7bn ($8.6bn) as a provision for both payments.
The bank’s chief executive Ross McEwan, said: “Today’s announcement is an important step forward in resolving one of the most significant legacy matters facing RBS.
“This settlement is a stark reminder of what happened to this bank before the financial crisis, and the heavy price paid for its pursuit of global ambitions.”
The UK government still owns 72% of RBS and has been waiting for its performance to recover so it can start selling some of that stake.
Analysis: Simon Jack, BBC Business Editor
Although the settlement is enormous – it is not unexpected and RBS had already set aside £6.7 billion to settle their final bill. This is not quite it.
A separate settlement with the Department of Justice is expected later this year which could match or exceed today’s settlement.
RBS today topped up the kitty for penalty payments by another £151m which will hit this year’s earnings.
Despite its core business producing reasonable profits, fines for historical conduct has meant RBS has lost money every year since a £45bn government bailout in 2008.
RBS chief executive Ross McEwan has said he is hopeful that by the end of this year, the legacy issues that have dogged this bank will be behind it and the government will be able to start selling chunks of the bank back to the private sector.
The shares are currently worth around half the price the government paid for them but Philip Hammond has indicated that the government may be prepared to start selling shares at a loss.
The settlement announced today relates to the role played by RBS in the mis-selling of products backed by home loans in the run up to the financial crisis.
During that period banks were buying up mortgages and repackaging them as investment products.
Those bundled up packages of mortgages were often marketed as safe investments, but when the financial crisis hit, many proved to be worthless.
Other banks, including Deutsche Bank, Citigroup, JP Morgan Chase, Bank of America and Goldman Sachs have already settled similar cases with US authorities.