In the mid-20th century, the concept of communication that was mostly tied to a source and a physical medium was challenged and ontologically shifted by a mathematician named Claude Shannon by the construction of a grand unified theory of communication. Shannon achieved this extraordinary goal by quantifying the information associated with a message, the capacity of a communication channel through which this message passes, the coding processes needed to make that channel efficient, study of noise that could arise in such a system, and noise’s effect on the accuracy of the received message by using a single concept from the physical sciences - entropy. Entropy which was already effective in the physical sciences to study the randomness in physical systems, ranging from decaying of the human body to the expansion of the universe, was appropriated and contextually transformed by Shannon to quantify information (giving it a physical sense) in a communication channel thus creating a shift in thinking by relating the physical world to the abstract world of mathematics.
In order to build an efficient and stable communication channel between a sender and a receiver using the tools of mathematics, Shannon excluded meaning or context from the message due to the inability of mathematical faculties to embed context in numbers. In fact, he considered the semantic meaning of the message unnecessary to build such a communication channel and thus introduced information not as “meaning” in a message but as a mathematical measure of one’s freedom of choice in selecting a message using the concept of probability [Weaver, 1953] [Shannon, 2001]. This arbitrariness in the selection of the message either by the sender or the receiver brings “uncertainty” or a sort of disorder in the system which was also being studied in the physical sciences using a tool called entropy. The concept of entropy which was used to measure the heat loss (or disorderness) in physical systems in thermodynamics, was contextualized by Shannon to “measure” the information associated with a message - bringing an ontological shift in the thinking by relating the physical world to the mathematical world.
While Shannon quantifies information mathematically by a slight transformation of the formula for entropy proposed by Boltzmann in the physical sciences, he also borrows the means from the physical sciences to connect micro to the macro like using knowledge of the micro (e.g. probabilistic dependency of symbols in a message) to understand the macro (the information associated with an entire communication procedure) in his proposed theory. Similar to the use of entropy to study the disorderness in a system like the movement of gaseous particles (micro) in a closed jar (macro), Shannon uses entropy to build a relationship between the sequence of symbols in a message by using Markov Models (micro) to the quantification of information in a message (macro). Just like the molecules in a closed jar would collide with each other and change the course of the direction and velocity of other molecules generating (dis)order in a physical system, the symbols in message or a series of messages would affect the amount of information sent from a sender to a receiver. As per the mathematical description of entropy by Shannon, information would be close to zero if there is order in the message sent, e.g. repeated blabbering of the same word by a child (such that the uncertainty of the next words in the sequence would be reduced) and would be high if the element of surprise or choice increases in selecting the message (entropy close to one), e.g. choosing one ball among thousand differently colored balls.
It is also interesting to note that physical systems with increasing entropy are irreversible in nature. To put it in a thermodynamic sense, the work done to move from a physical state A (analogous to the sender) to state B (analogous to the receiver) would not be equal to the work done from state B to A. While Shannon borrows entropy from the physical sciences to give information a physical sense, he also smuggles in this concept of irreversibility of a system (though unknowingly) as far as the transfer of the context or meaning in a message is concerned in the communication channel. Since Shannon was only interested in the flow of information between two poles (sender and the receiver), Shannon’s model fails as per the reversibility of the channel is concerned on a semantic level. This is where Shannon appropriates thermodynamic concept of entropy to describe information but fails to see the irreversibility he introduced in such a system (critiqued by Hall [Hall et al., 2001]) by only concentrating on the efficient transfer of a message mathematically devoid of any context or meaning. By joining the physical world to the mathematical world, Shannon brings an ontological shift in how one conceives about the entire process of communication. Shannon thus makes a huge contribution in bringing efficient communication systems to reality (by appropriating information using entropy) but also fails to bring into account the effects of socio-political and cultural differences on the two ends of a communication channel thus generating an asymmetry [Hall et al., 2001].
References