This is an interesting question -- and yet when you think about it, it's horrifying that we'd have to consider even for an instant the propriety and legality of keeping our own annotated research materials. But, to provide a proper answer:
In principle, this may depend on the terms of the license under which you downloaded these papers. In practice, you are very likely to be in the right, both legally and morally. In general, I believe that the licenses allow you to reproduce (i.e., print or save) the document for research purposes -- without specifying a time limit or maximum duration of their subsequent use. See e.g. JSTOR's terms of service.
If you are asking about the ethical/moral propriety of what you are doing, you are clearly in the right to retain your collection of papers and notes on those papers. In fact, you may even have an ethical obligation to retain those materials if the notes therein are necessary to reproduce any of your present work.
As an analogy, think about what I did many years ago when moving from one university to another. I had an extensive paper collection of reprints, many photocopied from my institution's library and many extensively annotated. I boxed all of these up and brought them with me. No one in their right mind -- nor even most copyright lawyers, I'd imagine -- would have had me go through those files and discard the articles not in volumes held by my new institution.
Disclaimer: I am not a lawyer and this is not legal advice. It does seem to be good common sense, though, and sometimes that can be worth at least as much as anything a lawyer would tell you.