Notice that the title of this post is not "Why Star Trek is better than Battlestar Galactica". I'm not an ill-tempered Internet fanboy who believes his opinions are the only correct ones. I'm just going to tell you what my personal preference is; your mileage may vary.
Despite the space travel element, the two franchises could not be more different. Star Trek has survived for over forty years, through good times and bad. It's told some truly great stories, lots of mediocre ones (see Voyager, entire run of), and even a few that are just plain awful, but almost all of its episodes and movies are united by Gene Roddenberry's idea of a future in which mankind has moved beyond its baser instincts. You could never imagine Captain Kirk or Captain Picard torturing an enemy for information, even in a "ticking time bomb" scenario, or executing an hostile alien, like a Klingon or Romulan, without a trial simply because they were a Klingon or Romulan. In one of Star Trek's more critically acclaimed episodes, Deep Space Nine's "In The Pale Moonlight", Captain Sisko's decision to resort to unethical, even illegal, means to turn the Dominion War in the Federation's favor clearly plagues his conscience. Even as he looks back on what he's done, and insists "I can live with it", the clear subtext is that he can't.
On the other hand, Battlestar Galactica's heroes don't behave like idealized 24th century humans, they behave like regular 21st century humans who haven't yet figured out how to get out of tight spots without selling out their most cherished beliefs. They claim to have morals, and yet they don't hesitate to engage in torture if the situation seems to call for it. President Roslin routinely airlocks Cylons without one shred of hesitation or remorse. And in the third season episode "Dirty Hands", Admiral Adama is prepared to execute Chief Tyrol's wife and baby son to end a strike of overworked menial laborers. Can you imagine any Star Trek Captain threatening the innocent family of an officer who disobeyed orders? For that matter, can you imagine that kind of behavior from a military commander in today's world, outside of places like North Korea or Zimbabwe? The human-Cylon conflict in Battlestar Galactica isn't good guys vs. bad guys; it's two equally amoral factions slugging it out in a mud pit. If Star Trek depicts humans as we ought to be, Battlestar Galactica depicts us as we truly are, and it's not pretty.
To be sure, there are things I like about BSG. I like the un-choreographed, documentary look of the special effects shots. I appreciate how the effects of 50,000 people being crammed onto a few spaceships for an extended period of time are realistically explored. BSG is well-shot, well-directed, and superbly acted, and I really wish I could enjoy it. In the end, though, it just doesn't give you any heroes to root for.
Star Trek has endured for forty years largely because its optimistic vision of the future inspired people. Kids have grown up to become doctors, engineers, scientists, and even astronauts because they grew up watching characters like Kirk, Spock, McCoy, and Scotty. Decades from now, will we be saying the same of BSG? Sure, someone may say Colonel Tigh inspired them to be an alcoholic, but that's not really the same thing.
Normally when I hear people comparing Star Trek and BSG, they're snidely asserting that BSG's grim-n'-gritty approach makes it automatically superior to the more optimistic Star Trek. It reminds me of the way high school jocks make fun of the "nerdy" kids who concentrate on their studies, not realizing that in a few years those same "nerds" will likely be their bosses. I guess I'm one of the nerds. If I want to watch something grim and depressing, there are four 24-hour news channels to choose from. Otherwise, I'll be watching Star Trek.