Whether it's ethical or not depends on your own system of ethics. But, a few things that come to mind. Full disclosure: I've never used these kinds of drugs and I'm not a medical doctor.
1) Just like steroids are considered unfair sportsmanship, cognition drugs will be viewed as unfair in academics by people who are concerned about fairness.
2) Taking drugs to artificially enhance your performance doesn't help you in the long run. Do you really want to take drugs your entire life to maintain an artificial level of performance? Things like better study skills, time management, and prioritization will enhance your performance more effectively and will help you through your whole life.
3) Are you going to feel OK with yourself knowing that you didn't live your life on your own terms? Later in life you're never really going to know whether you could have met your challenges on your own.
4) The people who don't want you to use adderall will tell you that there are a range of serious health consequences for abusing prescription drugs, including long-term cardiovascular, neurological, and mental issues. Sacrificing your health for success is viewed as unethical by some systems of reason.
5) Finally, these drugs are referred to as "cognition-enhancers", but the science is still out on whether or not they really assist in anything beyond rote memorization in people without ADHD. Other researchers have suggested that drug use might lead to deficiencies like decreased creativity.