Should I just break up with him?

If you love him and want it to work desperately for the relationship to work, fix it as much as you can, like talking to him and asking about all the stuff you just wrote. If you really are annoyed and just want to escape everything, then threaten to break up with him, and if he doesn't do what you want him to do, then just do it.