Thank you for correcting me. I have a lot of trouble with these verbs and all the accompanying se, me, te and de mi that go along with it. I find it very confusing.
There really is a big difference between the two. There is no abiguity. In English you can say I forgot about it or you. It can be intentionally or not. You will never know unless you ask. In Spanish you know what the person means exactly by the word they choose. If you choose the wrong word (as an English speaker) you can really hurt someone's feelings unwittingly.
You can wind up saying I forgot to bring your book (intentionally) instead of saying I forgot your book (because I was so busy it slipped my mind)
It's not passive. It's overt.