Why do you need time to refactor? It is just part of the work you need to do and should be accounted for when doing any other work. IMO a big mistake people make is thinking refactoring is some separate thing they need permission to do. You don’t, if you need to make a change in some area refactor it first to make it easier to accept your change, then add your change then refactor to clean up. This is not three separate tasks, just three steps in one task. You should be given enough time to do the whole task, not just part of it.
Who is in the wrong? Your manager, for not giving you time to refactor? Or you for giving him the option?
Why do you need time to refactor? It is just part of the work you need to do and should be accounted for when doing any other work. IMO a big mistake people make is thinking refactoring is some separate thing they need permission to do. You don’t, if you need to make a change in some area refactor it first to make it easier to accept your change, then add your change then refactor to clean up. This is not three separate tasks, just three steps in one task. You should be given enough time to do the whole task, not just part of it.