Chenguang Lu

Does semantic communication require a semantic information theory parallel to Shannon’s information theory, or can Shannon’s work be generalized for semantic communication? This paper advocates for the latter and introduces a semantic generalization of Shannon’s information theory (G theory for short). The core idea is to replace the distortion constraint with the semantic constraint, achieved by utilizing a set of truth functions as a semantic channel. These truth functions enable the expressions of semantic distortion, semantic information measures, and semantic information loss. Notably, the maximum semantic information criterion is equivalent to the maximum likelihood criterion and similar to the Regularized Least Squares criterion. This paper shows G theory’s applications to daily and electronic semantic communication, machine learning, constraint control, Bayesian confirmation, portfolio theory, and information value. The improvements in machine learning methods involve multi-label learning and classification, maximum mutual information classification, mixture models, and solving latent variables. Furthermore, insights from statistical physics are discussed: Shannon information is similar to free energy; semantic information to free energy in local equilibrium systems; and information efficiency to the efficiency of free energy in performing work. The paper also proposes refining Friston’s minimum free energy principle into the maximum information efficiency principle. Lastly, it compares G theory with other semantic information theories and discusses its limitation in representing the semantics of complex data.

Reflection Theory holds that our sensations reflect physical properties, whereas Empiricism believes that sense (data), presentations, and phenomena are the ultimate existence. Lenin adhered to Reflection Theory and criticized Helmholtz’s sensory symbolism for affirming the similarity between a sensation and a physical property. By using information and color vision theories, analyzing the ostensive definition with inverted qualia, and extending the relativity principle, this paper affirms the external world’s existence independent of personal sensations. Still, it denies the similarity between a sense and a physical property. Then, it improves symbolism to analog symbolism and Reflection Theory to Information Reflection Theory.

A new trend in deep learning, represented by Mutual Information Neural Estimation (MINE) and Information Noise Contrast Estimation (InfoNCE), is emerging. In this trend, similarity functions and Estimated Mutual Information (EMI) are used as learning and objective functions. Coincidentally, EMI is essentially the same as Semantic Mutual Information (SeMI) proposed by the author 30 years ago. This paper first reviews the evolutionary histories of semantic information measures and learning functions. Then, it briefly introduces the author’s semantic information G theory with the rate-fidelity function R(G) (G denotes SeMI, and R(G) extends R(D)) and its applications to multi-label learning, the maximum Mutual Information (MI) classification, and mixture models. Then it discusses how we should understand the relationship between SeMI and Shannon’s MI, two generalized entropies (fuzzy entropy and coverage entropy), Autoencoders, Gibbs distributions, and partition functions from the perspective of the R(G) function or the G theory. An important conclusion is that mixture models and Restricted Boltzmann Machines converge because SeMI is maximized, and Shannon’s MI is minimized, making information efficiency G/R close to 1. A potential opportunity is to simplify deep learning by using Gaussian channel mixture models for pre-training deep neural networks’ latent layers without considering gradients. It also discusses how the SeMI measure is used as the reward function (reflecting purposiveness) for reinforcement learning. The G theory helps interpret deep learning but is far from enough. Combining semantic information theory and deep learning will accelerate their development.

When we compare the influences of two causes on an outcome, if the conclusion from every group is against that from the conflation, we think there is Simpson’s Paradox. The Existing Causal Inference Theory (ECIT) can make the overall conclusion consistent with the grouping conclusion by removing the confounder’s influence to eliminate the paradox. The ECIT uses relative risk difference Pd = max(0, (R − 1)/R) (R denotes the risk ratio) as the probability of causation. In contrast, Philosopher Fitelson uses confirmation measure D (posterior probability minus prior probability) to measure the strength of causation. Fitelson concludes that from the perspective of Bayesian confirmation, we should directly accept the overall conclusion without considering the paradox. The author proposed a Bayesian confirmation measure b* similar to Pd before. To overcome the contradiction between the ECIT and Bayesian confirmation, the author uses the semantic information method with the minimum cross-entropy criterion to deduce causal confirmation measure Cc = (R − 1)/max(R, 1). Cc is like Pd but has normalizing property (between −1 and 1) and cause symmetry. It especially fits cases where a cause restrains an outcome, such as the COVID-19 vaccine controlling the infection. Some examples (about kidney stone treatments and COVID-19) reveal that Pd and Cc are more reasonable than D; Cc is more useful than Pd.

Many researchers want to unify probability and logic by defining logical probability or probabilistic logic reasonably. This paper tries to unify statistics and logic so that we can use both statistical probability and logical probability at the same time. For this purpose, this paper proposes the P–T probability framework, which is assembled with Shannon’s statistical probability framework for communication, Kolmogorov’s probability axioms for logical probability, and Zadeh’s membership functions used as truth functions. Two kinds of probabilities are connected by an extended Bayes’ theorem, with which we can convert a likelihood function and a truth function from one to another. Hence, we can train truth functions (in logic) by sampling distributions (in statistics). This probability framework was developed in the author’s long-term studies on semantic information, statistical learning, and color vision. This paper first proposes the P–T probability framework and explains different probabilities in it by its applications to semantic information theory. Then, this framework and the semantic information methods are applied to statistical learning, statistical mechanics, hypothesis evaluation (including falsification), confirmation, and Bayesian reasoning. Theoretical applications illustrate the reasonability and practicability of this framework. This framework is helpful for interpretable AI. To interpret neural networks, we need further study.

After long arguments between positivism and falsificationism, the verification of universal hypotheses was replaced with the confirmation of uncertain major premises. Unfortunately, Hemple proposed the Raven Paradox. Then, Carnap used the increment of logical probability as the confirmation measure. So far, many confirmation measures have been proposed. Measure F proposed by Kemeny and Oppenheim among them possesses symmetries and asymmetries proposed by Elles and Fitelson, monotonicity proposed by Greco et al., and normalizing property suggested by many researchers. Based on the semantic information theory, a measure b* similar to F is derived from the medical test. Like the likelihood ratio, measures b* and F can only indicate the quality of channels or the testing means instead of the quality of probability predictions. Furthermore, it is still not easy to use b*, F, or another measure to clarify the Raven Paradox. For this reason, measure c* similar to the correct rate is derived. Measure c* supports the Nicod Criterion and undermines the Equivalence Condition, and hence, can be used to eliminate the Raven Paradox. An example indicates that measures F and b* are helpful for diagnosing the infection of Novel Coronavirus, whereas most popular confirmation measures are not. Another example reveals that all popular confirmation measures cannot be used to explain that a black raven can confirm “Ravens are black” more strongly than a piece of chalk. Measures F, b*, and c* indicate that the existence of fewer counterexamples is more important than more positive examples’ existence, and hence, are compatible with Popper’s falsification thought.

Author’s Preface in English: My Two Discoveries and Their Philosophical Significance As long as a person opens his eyes to face this world, he will meet the problems discussed in this book. Is the red color of flowers and the green color of grass displayed in front of our eyes only sensations in our minds or objective existence as the same as we see? Is a color perception similar to or different from a natural light that objectively exists? The further question is: Are red flowers and green grass just our sensations in minds or what objectively exists in the way as we see, or what exists differently from our sensations? As early as ancient ages of Greece, atomists gave us a rather scientific answer. It is said that what objectively exists is only the material world made of atomics. The motion of atomics causes human sensations. Color, sound, smell… are all human sensations, and related to human sense organs’ conformations, and different from those atoms that cause sensations (I will explain why the answer is not entirely correct later). The answer from modern natural science is very similar. However, this answer brings a series of philosophical problems that puzzle us. If red and green colors are only sensations in human minds, why red flower and green grass are not? If a sensation is totally different from the external object or its property that causes a sensation, how can we know the appearance of the outside object itself? The problems brought by atomists are just revealed by the philosophically fundamental questions proposed by Engels, i.e., the question about the relationship between thinking and existence, which includes two questions: the question about the original existence of the world and the question about whether the world is knowable. I know there is a fashionable opinion: the fundamental question in philosophy, or the question about the original existence of the world, is a false question, and the argument between materialism and idealism is meaningless because there are no different sensations or experiences deduced from the affirmations about whether the world is made of matter or spirit. But, I still persist in discussing the fundamental questions in philosophy. This is not because I have some faith and fight for the faith, neither because I have encyclopedical knowledge so that I can provide better summing-up. This is only because I happened to have two discoveries, which force me and also force academe，to review the fundamental questions in philosophy from a new visual angle to get conclusions that accord with natural science better. My first discovery is that idealism, positivism，and materialism (mainly native materialism) all wrongly understand the relationship between language, sensations, and outside objects. In daily life, we define words “red”, “green”, “acid”, “salt”, “soft”, “hard”, “cold”, “hot” and son on actually according to outside objects instead of sensations. The conformability of people’s speaking language cannot ensure the conformability of their sensations (for example, it is possible that two persons conformably say “flower is red and the grass is green” and have opposite color perceptions ). It is unfeasible to reduce the world with red, green, acid, salt, cold, hot, and other material properties into sensations as idealism does, into world-elements as Machism does, or into sense-data as logical empiricism does. It is also unfeasible to affirm that a sensation is similar to a material property as inornate materialism does. This discovery forces me to believe that 1) The world described by daily language is not the natural world, phenomenological world, or presentational world proposed by Kant, neither neutral world or a world made of elements or sense data, but the material world that objectively exists as affirmed by materialism. 2) Atomists in ancient Greece and symbolists, such as Kant and Helmholtz of Germany, are correct in affirming the dissimilarity between a sensation and a material property that causes the sensation. Atomism and mechanical materialism represented by John Locke’s theory affirm that the motion of Atoms or matter causes human sensations; yet, a consciousness and a material property are totally dissimilar. However, symbolists represented by Kant and Helmholtz further affirm that the sensations in organs are only symbols, they are discretional, and there is no certain one-to-one relation between a sensation and a material property. This point of view is called agnosticism. So far, my above conclusion is probably also thought of as agnosticism. However, I happened to have a second discovery. My second discovery is that there exists a skillful mathematical model of color vision——the decoding model 【8～10】, whose principle is similar to the principle of 3-8 decoder in numeric circuits. The model can explain various phenomena of color vision, especially the evolution of color vision, better. We can conclude from the decoding model that 1) A color perception is only a symbol that can continuously vary. 2) A series of color perceptions contain information from external objects since different color perceptions reflect different natural lights, even if a color perception is dissimilar to a material property. 3) The mechanism of color vision was continuously evolving for the requirement of discerning outside objects. The presentational or phenomenological world was involving with the mechanism of color vision involving, by which the objectify theory developed by Feuerbach and Marx can be explained in the way of natural science. The generalized information theory I set up is tightly related to my second discovery. For measuring the information conveyed by color vision, I extended Shannon’s information theory to general information theory, which can measure and optimize sensitive information, as well as the information conveyed by language and forecasts. The results from the generalized information theory are consistent not only with common sense but also with Popper’s philosophical theory. For example, according to this theory, the more necessary an occasional event is predicted as, and the prediction can be proved correct by facts, the more information the prediction conveys; a proposition that is true in any case provides no information; a lie or bad prediction conveys negative information; it will decrease the information we have had to believe a fortune teller is running off at the mouth. There are mainly two kinds of agnosticism. One is Kant’s agnosticism, which affirms thing-in-itself (or objective thing) is unknowable. Another is Hume’s agnosticism, which affirms the relationship between a cause and a result (or objective rule) is unknowable. Marx’s practice-testing theory and Popper’s knowledge-evolving theory can resolve the problem with Hume’s agnosticism. But, about whether the problem with Kant’s agnosticism has been resolved, there are still arguments. Kant would have said that what you have resolved is only the problem about whether the phenomenological world is knowable; however, you still cannot resolve the problem about whether the thing-in-itself, which is totally different from sensations, is knowable. Now, with the help of my two discoveries and new information theory, we can have two ways to prove the thing-in-itself is knowable. 1) We know that the objects described by daily language are knowable. Yet, the objects pointed by daily language are just material objects themselves. So, material objects themselves are knowable, or say, the thing-in-itself is knowable. 2) Although sensations are dissimilar to material objects or their properties, they contain information about material objects. This information can be measured by the generalized information formula similar to Shannon’s information formula. So, the material object or the thing-in-itself is knowable from the point of view of natural science. Moreover, my research concerning lingual information also supports Mark’s practice-testing theory and Popper’s knowledge-evolving theory. So, agnosticism does not fit my standpoint at all. For explaining the relationship between sensations and outside objects and summarizing up my research, I proposed analog-symbolic theory or analog-symbolism, which is based on Darwin’s theory of evolution, the theories of color and color vision developed by Young and many later researchers, and also my theories on color vision and information. The analog-symbolism is also the extension of the relativistic principle from physics to philosophy. It accepts atomists and mechanical materialists’ viewpoint about the relation between the sensation and the outside object (the motion of matter results in the sensation, however a sensation and a material property are dissimilar), affirms and extends Kant and Helmholtz’s symbolism, carries on the relativity thought in Feuerbach and Marx’s objectify theory (for resolving the problem with Kant’s dualism), inherits Lenin and many materialists’ viewpoint that color, sound, smell… objectively exist（against idealism and positivism）. This book has both scientific explorations and philosophical analyses. However, what it provides is very different from a popular scientific philosophy. The popular scientific philosophy is supposed to sum up, and guide science. But my effort is the opposite. I always try to base my philosophy on science. For example, I use a mathematical model of color vision to explain that the sensations are analog symbols, use the information theory to resolve the problems with agnosticism, and extend relativity principle from physics to philosophy. I have unified many very extreme standpoints together in my theory. These standpoints look mutually repulsive to other people, yet from my viewpoint, they are not contradictive with each other but very compatible. The primary purpose of this book is to eliminate the people’s bepuzzlement about the fundamental questions in philosophy. Before writing this book, I wrote another book: “Mystery of Beauty-feeling and Evolution of Demand,” whose primary purpose is to eliminate the people’s bepuzzlement about problems with beauty-feeling and demand. It is also based on some discoveries — about the arising rule of pleasant feelings and avian esthetic interest. These two books respectively discuss the cognitive relationship and motivational relationship between humans and outside would. They are comprehensive summing-up of my many years’ researches. If my effort is not ineffectual, I will be incomparably gratified. Lu, Chenguang

An important problem with machine learning is that when label number n>2, it is very difficult to construct and optimize a group of learning functions, and we wish that optimized learning functions are still useful when prior distribution P(x) (where x is an instance) is changed. To resolve this problem, the semantic information G theory, Logical Bayesian Inference (LBI), and a group of Channel Matching (CM) algorithms together form a systematic solution. MultilabelMultilabel A semantic channel in the G theory consists of a group of truth functions or membership functions. In comparison with likelihood functions, Bayesian posteriors, and Logistic functions used by popular methods, membership functions can be more conveniently used as learning functions without the above problem. In Logical Bayesian Inference (LBI), every label’s learning is independent. For Multilabel learning, we can directly obtain a group of optimized membership functions from a big enough sample with labels, without preparing different samples for different labels. A group of Channel Matching (CM) algorithms are developed for machine learning. For the Maximum Mutual Information (MMI) classification of three classes with Gaussian distributions on a two-dimensional feature space, 2-3 iterations can make mutual information between three classes and three labels surpass 99% of the MMI for most initial partitions. For mixture models, the Expectation-Maxmization (EM) algorithm is improved and becomes the CM-EM algorithm, which can outperform the EM algorithm when mixture ratios are imbalanced, or local convergence exists. The CM iteration algorithm needs to combine neural networks for MMI classifications on high-dimensional feature spaces. LBI needs further studies for the unification of statistics and logic.

Semantic Information conveyed by daily language has been researched for many years; yet, we still need a practical formula to measure information of a simple sentence or prediction, such as “There will be heavy rain tomorrow”. For practical purpose, this paper introduces a new formula, Semantic Information Formula (SIF), which is based on L. A. Zadeh’s fuzzy set theory and P. Z. Wang’s random set falling shadow theory. It carries forward C. E. Shannon and K. Popper’s thought. The fuzzy set’s probability defined by Zadeh is treated as the logical probability sought by Popper, and the membership grade is treated as the truth-value of a proposition and also as the posterior logical probability. The classical relative information formula (Information=log(Posterior probability / Prior probability) is revised into SIF by replacing the posterior probability with the membership grade and the prior probability with the fuzzy set’s probability. The SIF can be explained as “Information=Testing severity – Relative square deviation” and hence can be used as Popper's information criterion to test scientific theories or propositions. The information measure defined by the SIF also means the spared codeword length as the classical information measure. This paper introduces the set-Bayes’ formula which establishes the relationship between statistical probability and logical probability, derives Fuzzy Information Criterion (FIC) for the optimization of semantic channel, and discusses applications of SIF and FIC in areas such as linguistic communication, prediction, estimation, test, GPS, translation, and fuzzy reasoning. Particularly, through a detailed example of reasoning, it is proved that we can improve semantic channel with proper fuzziness to increase average semantic information to reach its upper limit: Shannon mutual information.

Logical Probability (LP) is strictly distinguished from Statistical Probability (SP). To measure semantic information or confirm hypotheses, we need to use sampling distribution (conditional SP function) to test or confirm fuzzy truth function (conditional LP function). The Semantic Information Measure (SIM) proposed is compatible with Shannon’s information theory and Fisher’s likelihood method. It can ensure that the less the LP of a predicate is and the larger the true value of the proposition is, the more information there is. So the SIM can be used as Popper's information criterion for falsification or test. The SIM also allows us to optimize the true-value of counterexamples or degrees of disbelief in a hypothesis to get the optimized degree of belief, i. e. Degree of Confirmation (DOC). To explain confirmation, this paper 1) provides the calculation method of the DOC of universal hypotheses; 2) discusses how to resolve Raven Paradox with new DOC and its increment; 3) derives the DOC of rapid HIV tests: DOC of “+” =1-(1-specificity)/sensitivity, which is similar to Likelihood Ratio (=sensitivity/(1-specificity)) but has the upper limit 1; 4) discusses negative DOC for excessive affirmations, wrong hypotheses, or lies; and 5) discusses the DOC of general hypotheses with GPS as example.