Although the concepts of information and probability are prevalent in statistics, the better part of statistics can also be understood in terms of signal and noise.

Explicitly and implicitly, Tukey (1915-2000) has written a lot about this idea. I would also like to recommend Bookstein's more recent book "Measuring and Reasoning" (Cambridge Univ. Press, 2014).

Here is a recent application to a common illness: