Reliable estimation of prediction confidence remains a significant challenge in machine learning, as learning algorithms proliferate into difficult real-world pattern recognition applications. The Conformal Predictions (CP) framework is a recent development in machine learning to associate reliable measures of confidence with pattern recognition settings including classification, regression, and clustering. This framework is an outgrowth of earlier work on algorithmic randomness, transductive inference, and hypothesis testing, and has several desirable properties for potential use in various real-world applications. One of the desirable features of this frame.work is the calibration of the obtained confidence values in an online setting. While probability/confidence values generated by existing approaches can often be unreliable and difficult to interpret, the theory behind the CP framework guarantees that the confidence values obtained using this framework manifest themselves as the actual error frequencies in the online setting (i.e., they are well-calibrated).

Further, this framework can be applied across all existing classification and regression methods (such as neural networks, Support Vector Machines, k-Nearest Neighbors, ridge regression, etc.), thus making it a very generalizable approach. Over the last few years, there has been a growing interest in applying this framework to real-world problems such as clinical decision support, medical diagnosis, sea surveillance, net.work traffic classification, and face recognition. The promising results have resulted in further extensions of the framework to problem settings such as active learning, model selection, feature selection, change detection, outlier detection, and anomaly detection. This book captures the basic theory of the framework, demonstrates how the framework can be applied to real-world problems, and also presents several adap.tations of the framework including active learning, change detection, and anomaly detection.

The motivations for this book are as follows:

• The Conformal Predictions (CP) framework was proposed more than a decade ago, and numerous researchers have actively worked on this framework over the last few years. However, there is only one existing book on this topic, Algorithmic Learning in a Random World, which explains the theoretical foundations of this work and is intended for a researcher with an advanced mathematical and machine learning background. There is a need for a book that caters to a broader audience, presenting the framework from an applied researcher’s perspective. This book has been compiled to address this important need.

• While the CP framework has several desirable properties, the efforts of research.ers in applying or adapting this framework have been limited to a few groups of researchers scattered around the world. The framework has been slow in being adopted by the machine learning community due to its mathematical formulation .eral features that make it attractive from a machine learning perspective. This book has been written to bridge this gap, and to expose the framework to a large base of researchers in machine learning, pattern recognition, and artificial intelligence.

• As mentioned earlier, the last decade has seen several efforts from both
practitioners and researchers around the world in applying or adapting the
framework. Although there are several common trends in these efforts, there is no in-depth publication that captures these developments, nor is there a survey article that brings together these bodies of work for a new researcher to understand the theory through real-world applications so that new applications or adaptations can further be proposed. This book will provide a platform for researchers to understand the framework as well as the efforts so far in using the framework, thus providing a springboard for further research as well as a handbook for application in real-world problems.

Book Organization

The book is organized into three parts: Theory, Adaptations, and Applications. These three parts, each consisting of multiple chapters, reflect the categorization of the efforts that have been undertaken so far with respect to the CP framework. Part I (Theory), composed of Chapters 1 and 2, presents a summary of the theoretical foundations of conformal prediction that have been developed over the years. Part II (Adaptations), composed of Chapters 3 through 9, covers the different machine learning settings—such as active learning, feature selection, and change detection— in which the CP framework has been incorporated successfully, along with related theoretical and experimental results. Part III (Applications), composed of Chapters 10 through 13, describes the application domains in which the framework has been validated successfully—ranging from biometrics to medical diagnostics/prognostics to network intrusion detection, along with the corresponding experimental results and analysis. We hope that we will continue to expand each of these parts in the near future, to include additional efforts undertaken by researchers in the field, and to make this book a valuable resource for practitioners of conformal prediction.

Reprinted with permission from Morgan Kaufmann, an imprint of Elsevier.