I'm deleting Annotative scales from my text and it keeps showing them there

Make sure the add current scale to all annotative objects icon is off (I on to the left of the scale in the status bar)