I don't know the relevant laws, which may vary between U.S. states, but I wouldn't focus on the legal question. Regardless of whether you have the legal right to make a recording, you should not do so without the speaker's permission. If someone asked me, I would probably give permission, but making a recording without asking feels creepy and inappropriate. If you go around doing this in academia, it's likely to be viewed as unprofessional, and some speakers will become angry.
Even if you ask, people may wonder why you want a recording. (Are you stalking the speaker? Are you going to scrutinize the recording to try to find misstatements? Are you working on competing research and trying to document exactly what was said in your competitor's talk?) If you have a compelling reason to make a recording, it's worth explaining why. For example, perhaps it could help accommodate certain disabilities. Otherwise, you can certainly ask, but it may come across as a weird request.