What is the relationship between physics and entropy? In-depth study of entropy in physics and Shannon’s theory. Meaning and characteristics.What is entropy? What does the term entropy mean? What is the formula for entropy change?
In this article we will analyze the phenomenon of entropy in physics, with definitions and formulas.
ENTROPY AND INFORMATION THEORY
Advanced course : Physics , Computer Science and Information: “ Entropy and information in computers ”.
Field : Physics, computer science and signal theory.
Subjects involved :
- Computer science : digital and analog information, the bit and the binary system.
- Physics : the concept of entropy in thermodynamics , order and disorder.
Starting point of the study : Today, information is a commodity like oil or metal, a public service like water and electricity . Politicians, stock market experts and lay people often say that we live in the “information age”. Even if not everyone knows what the information age actually is, its signs can be observed everywhere, from cell phones to emails exchanged. Our society is considered a global village in which almost every person has the possibility to access (for example via the Internet ) a lot of information and to exchange it with others. Behind all this there have been immense technological developments that have enabled this transmission of information. But what is information? What hinders its transmission? Why does the word “digital” appear so often in these new technologies (which means nothing else than that information is represented by discrete symbols such as numbers)? The answer to many of these questions can be found in the discoveries and research done by Claude Shannon .
CLAUDE SHANNON, ENTROPY
Claude Elwood Shannon was born on April 30, 1916, in Petoskey, Michigan. As a young man, he worked as a telegrapher for Western Union. Shannon began studying electrical engineering and mathematics at the University of Michigan in 1932 and received his bachelor’s degree in 1936. He attended college at the Massachusetts Institute of Technology (MIT), where he worked on the Vannevar Bush differential analyzer, an analog computer. In his master’s thesis at MIT, he proved several results relating Boolean algebra to electronic logic networks (such as relays and switches). Many consider his thesis to be among the most important and famous theses of the century. His doctoral thesis , written in 1940, is entitled ” An Algebra for Theoretical Genetics .”
He later began working at Bell Labs , before returning to MIT in the 1950s. In 1948, Shannon published what is perhaps his most famous work: “ A Mathematical Theory of Communication .” This work focuses on the problem of reconstructing with some degree of certainty the information transmitted by a sender. In this fundamental work, Shannon used tools such as chance analysis and large deviations, which were just being developed at the time.
Shannon defined the entropy of information as a measure of redundancy, laying the foundations for information theory .
His most important paper is the next one, written with Warren Weaver, The Mathematical Theory of Communication , which is short and surprisingly accessible to the non-specialist.
Another important article is “The theory of communication in cryptographic systems ”, with which Shannon practically founded the mathematical theory of cryptography . He also played an important role in introducing the sampling theorem, which studies the representation of a continuous (analog) signal by a discrete set of samples at regular intervals (digitization). Shannon was known for his lively intelligence and a very particular personality; it is known that he rarely used notes or sketches, and preferred to work only with his head. Outside of his academic interests, Shannon was a juggler, unicyclist and chess player. He died in February 2001 .
ENTROPY: THERMODYNAMICS
Entropy in physics : In thermodynamics, entropy is a state function that is studied following the second law of thermodynamics. It is a measure of the disorder of a physical system. Based on this definition, we can say that when a system passes from an ordered state to a disordered one, its entropy increases. In the International System, it is measured in joules per kelvin (J/K).
Entropy S as a state function was introduced in 1864 by Rudolf Clausius in the context of thermodynamics as DS=DQrev/T
where ΔQrev is the amount of heat reversibly absorbed by the system at temperature T. In one of its various formulations, the second law of thermodynamics states that in an isolated system entropy can only increase, or at most remain constant for reversible thermodynamic transformations.
ENTROPY, PHYSICAL FORMULA
There is also a definition of entropy that is closer to a “statistical” approach to possible configurations. Intuitively, one imagines that a certain macroscopic equilibrium condition of the system corresponds to a multitude of microscopic configurations. These microscopic configurations occupy a volume in a particular space (called phase space, which is indicated by Γ). Then we can define the “Boltzmann” entropy as S=k lnG where k is the Boltzmann constant. It is precisely this relationship, as we will see, that is analogous to the entropy introduced by Shannon.
INFORMATION AND ENTROPY: EXPLAINED
In information theory – and in relation to signal theory – entropy measures the amount of uncertainty or information present in a random signal. From another point of view, entropy is the minimum descriptive complexity of a random variable. Claude Shannon
is responsible for studying entropy in this context. In Shannon’s first theorem , or Shannon’s source coding theorem, he showed that a random source of information cannot be represented with a number of bits less than its entropy, that is, its average self-information. The mathematical relationship he indicated was the following:
S=-pLog(p)
where p means probability. This result recalled the statistical definition of entropy . As Shannon later recalled about his result: “ My biggest worry was what to call it. I thought of calling it information, but the word was overused, so I decided to call it uncertainty. When I discussed the matter with John Von Neumann, he had a better idea. He said I should call it entropy, for two reasons : “ First, your uncertainty function is already known in statistical mechanics by that name. Second, and more significantly, no one knows for sure what entropy is, so you will always have the upper hand in an argument .”
Is it possible to explain in simple words the correlation between Entropy and Information ? Entropy in thermodynamics is a quantity that is defined starting from the second law of thermodynamics. It represents a quantity that describes the irreversibility of processes and the inexorable dispersion in heat of other forms of energy present within a system.
For Shannon , information , in the sense of the theory of Information, is also a characteristic of systems , in particular of those systems that are used to communicate. The information of a message can never increase beyond the value it had at the time of transmission of the message. It can, however, decrease due to various processes that lead to a partial loss or deterioration of the message (just think of the disturbances in radio transmissions or in cell phones). Up to this point the relationship between the two quantities may not be evident. In fact, they appear as two quantities that present an opposite behavior (increase for entropy, decrease for information). But the analysis of these two quantities can be more detailed. If we look at a physical system from a microscopic point of view we can describe it with a probability distribution, which provides, for each microscopic constituent, the probability of observing it with a certain speed in a certain position.
Boltzmann established that there was a mathematical connection between this probability distribution and entropy , in particular that the entropy of a system is proportional to the mean value assumed by the logarithm of this probability distribution with its sign reversed. Shannon introduced a similar quantity to study information . At the most basic level, each message can be expressed as a sequence of characters. Depending on the message, each character can be predicted or not, with varying degrees of certainty starting from the preceding characters. The more predictable a character is, the less information it adds to the overall message, that is, it is redundant, it could be omitted and the message would be understood anyway. If I transmit the sequence “telefo” the person receiving it will know for sure that the letter that follows is “n”, because there are no words in Italian that begin with “telefo” and continue with other letters, so transmitting this “n” is redundant, but after the “n” he will have a certain degree of uncertainty about what follows, it could be “o”, and therefore have the word “telefono”, or an “a” if I am transmitting “telefonare” or other forms of this verb, or even other letters. He needs to know what letter it is to understand how to understand the message.
However, there are some letters that it can exclude (any consonant, for example) while the others can appear with different probabilities, also depending on other words that I may have transmitted before. This reasoning allows us to attribute a probability distribution to messages as well. Each character has a probability of being predicted based on the characters that follow.
Shannon introduced a quantity, called information entropy , which is related to this probability in exactly the same way that thermodynamic entropy is related to the probability of the state of a system.
This information entropy enjoys several properties similar to the thermodynamic one, including that of never decreasing over time, because as time passes a message can become less clear due to transmission disturbances. Transmission disturbances introduce characteristics that cannot be predicted by previous ones, so they have a flat probability distribution, the effect through the Boltzmann-Shannon formula is to increase the information entropy .The connection is therefore quite immediate, in both cases the entropy (thermodynamic or information) is a measure of how much the usable content of the system has degraded, an energy content in the thermodynamic case, an information content in the case of a message. Information plays the role of available energy, which decreases with the passage of time.
RELATIONSHIP BETWEEN PHYSICAL AND COMPUTER ENTROPY: EXAMPLE
Let us consider a physical system in given conditions of temperature , pressure and volume, and establish its entropy value ; in connection it is possible to establish the degree of order and therefore the amount of our information (in a microscopic sense).Let us now suppose that we lower the temperature while leaving the other parameters unchanged: we observe that its entropy decreases, but also that its degree of order increases, and with it our level of information. In the limit, at zero temperature, all the molecules are still, the entropy is zero and the order is the maximum possible, and with it the information is maximum; in fact there is no longer any alternative to choose from.