I once read a description of the difference between Science Fiction and Fantasy which ran "In Science Fiction you can have mermaids if there is a good reason for your world to have mermaids. In Fantasy mermaids, bless 'em, just are."

I'm not sure the "reason" has to be totally scientifically plausible. But I think I prefer my science fiction to at least attempt to signal somehow whether it's extrapolating from known science or just making stuff up. It's when the latter is mistaken for the former that a lot of confusion arises that can be damaging to science. Hollywood films are usually clearly making stuff up, even the ones that don't call themselves Science Fiction.

There's also the "bounce you out of the fiction" problem. When something in a story is "just wrong" it can be very distracting and difficult to overlook especially if the bit that is wrong is related to something you are passionate about. People who are keen on history have similar problems with a lot of historical fiction that scientists do with a lot of mainstream SF.

I don't know quite what the answer is. It's difficult to know what someone is going to find distractingly out of place, or consider damagingly misleading.