I recently attended a debate between libertarian students and conservative students. They were supposed to argue in general about why their respective sides had the better political philosophy, but they tended to get bogged down in the "weeds" of drug policy (pun intended) and other overly superficial issues. Not that drugs are unimportant, but you can find conservatives who are against the war on drugs (e.g. some folks at National Review). And I don't think the debaters did a good job explaining why this issue really illustrates a fundamental divide between their ideologies (if indeed they thought so). Litmus tests may not be the best way to look at this divide. I've trying to think of a way to understand the difference between the two philosophies, and it's not so easy to come up with a concise, satisfying answer. I haven't fully thought through this question, but I think a good starting point is to say that American conservatives and libertarians often do not come up with radically different policy proscription, but they have pretty different intellectual approaches when considering a problem. Libertarians tend to be very into casuistry: i.e. find a general principle and then apply it to a specific problem. So, there's a "correct" libertarian answer on drug policy, immigration, etc., and this answer would be correct in almost any time or place. [As a side note, I would say that I find it puzzling that, based on anecdotal evidence, that libertarians are in many cases atheistic/agnostic (and hence probably materialists.) So, it's not clear exactly how it is that so many of them believe in universal, objective truths. For that matter, how can they believe in liberty? After all, matter is not free. I will concede that there are some libertarians who will say things along the lines of: "I don't know what a good policy is and neither do you. So, let's just live and let live." But they seem to either be a minority of libertarians or a silent majority that is too lazy to leave the basement long enough to become political activists.]
In contrast, conservatives are less doctrinaire. Russell Kirk said that "conservatism is the negation of ideology." That might be too strong, but it does have a kernel of truth. Sometimes people say that conservatism is more of a temperament than a philosophy. There's also the famous Lionel Trilling quote that conservatives tend to express themselves with "irritable mental gestures which seek to resemble ideas." Again, I wouldn't fully endorse that notion, but it does strike a chord. If you're a conservative and it doesn't strike a chord, maybe you need to think about yourself and your friends a little more carefully. Now, most conservatives do embrace some doctrines, as any sane person would. But I think that, at least in the political realm, they do so about far fewer issues than libertarians do. For example, I can imagine circumstances in which I'd support fairly strict immigration laws. I could also imagine situations in which I basically support open borders. There are probably circumstances in which I'd take a laissez-faire political stance towards drugs, but there are also situations where I'd crack down hard. Conservatism is largely about prudence and judgment. We ought to seek to serve the common good, and there are many ways to do that.
Lastly, I'll say that it sometimes seems that American conservatives are not even all that conservative. I think they in many ways resemble some of the classical liberals and Whigs of old. After all, Edmund Burke was a Whig and Hayek called himself a Whig as well. Who is cited more often as an influence among conservatives in America? Burke, Locke, Smith, Madison, and Friedman? Or, say, Disraeli, Evelyn Waugh, T.S. Eliot, and Christopher Dawson? For better or worse, doesn't it sometimes seem that we are long on Whigs and short on Tories? Yes, some of this is semantics, but not all of it. Does this idiosyncrasy make us incoherent? Should we be citing Aristotle more and Jefferson less?