Make it easier to change highlight color when working inline

When I got to highlight a selection, my method is to hover and then press one of the highlight numbers 1-6. Sometimes I type the wrong number and I want to correct it, but even though I am still hovering, typing a second (or third) number will not recolor the highlight. It should.