Millennials, the poor and those with low education levels are the most likely to have fallen for it.
Greg McBride of Bankrate.com says it's a mistake more than a fifth of credit card users have made.
"There's a common misperception that you have to revolve a credit card balance, in other words carry it over from one month to the next and incur finance charges in order to actually boost your credit and that's not the case."
McBride says it's much better to pay your bills on time.
"You can build your credit very effectively by using your credit card and paying the balance, in full, at the end of the month."
McBride says there are two factors that are crucial in determining your credit score.
"The two biggest factors in your credit score are paying your bills on time and your utilization ratio, in other words, how much of your available credit you are using; the lower you keep that the better."
With interest rates at an all-time high, McBride says not only does carrying a balance not help your credit score, it takes an unnecessary bite out of your wallet.