In this paper I argue that (at least many) philosophical thought experiments are unreliable. But I argue that this notion of unreliability has to be understood relative to the goal of thought experiments as knowledge producing. And relative to that goal many thought experiments in science are just as unreliable. But in fact thought experiments in science play a varied role and I will suggest that knowledge production is a goal only under quite limited circumstances. I defend the view that these circumstances can (sometimes) arise in philosophy as well.