Classification and clustering using Belief functions

Transcription

1 Classification and clustering using Belief functions Thierry Denœux 1 1 Université de Technologie de Compiègne HEUDIASYC (UMR CNRS 6599) tdenoeux Tongji University Shanghai, China July 7, 2016 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

2 Theories of uncertainty Fuzzy#sets#&# Possibility#theory# Imprecise## probability# Rough#sets# DS#theory# Probability## theory# Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

3 Focus of this talk Dempster-Shafer (DS) theory (evidence theory, theory of belief functions): A formal framework for reasoning with partial (uncertain, imprecise) information. Has been applied to statistical inference, expert systems, information fusion, classification, clustering, etc. Purpose of these talk: Brief introduction or reminder on DS theory; Review the application of belief functions to classification and clustering. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

4 Outline 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

5 Dempster-Shafer theory Outline 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

6 Outline Dempster-Shafer theory Mass, belief and plausibility functions 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

7 Dempster-Shafer theory Mass, belief and plausibility functions Mass function Let Ω be a finite set called a frame of discernment. A mass function is a function m : 2 Ω [0, 1] such that m(a) = 1. A Ω The subsets A of Ω such that m(a) 0 are called the focal sets of m. If m( ) = 0, m is said to be normalized (usually assumed). Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

8 Source Dempster-Shafer theory Mass, belief and plausibility functions A mass function is usually induced by a source, defined a 4-tuple (S, 2 S, P, Γ), where S is a finite set; P is a probability measure on (S, 2 S ); Γ is a multi-valued-mapping from S to 2 Ω. (S,$2 S,P)$ Ω" Γ $ s $ Γ(s)$ Γ carries P from S to 2 Ω : for all A Ω, m(a) = P({s S Γ(s) = A}). Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

9 Dempster-Shafer theory Mass, belief and plausibility functions Interpretation (S,$2 S,P)$ Ω" Γ $ s $ Γ(s)$ Ω is a set of possible states of the world, about which we collect some evidence. Let ω be the true state. S is a set of interpretations of the evidence. If s S holds, we know that ω belongs to the subset Γ(s) of Ω, and nothing more. m(a) is then the probability of knowing only that ω A. In particular, m(ω) is the probability of knowing nothing. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

10 Example Dempster-Shafer theory Mass, belief and plausibility functions A murder has been committed. There are three suspects: Ω = {Peter, John, Mary}. A witness saw the murderer going away, but he is short-sighted and he only saw that it was a man. We know that the witness is drunk 20 % of the time. (S,$2 S,P)$ Ω" drunk$(0.2) $ Γ $ Peter$ Mary$ not$drunk$(0.8) $ John$ We have Γ( drunk) = {Peter, John} and Γ(drunk) = Ω, hence m({peter, John}) = 0.8, m(ω) = 0.2 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

11 Special cases Dempster-Shafer theory Mass, belief and plausibility functions A mass function m is said to be: logical if it has only one focal set; it is then equivalent to a set. Bayesian if all focal sets are singletons; it is equivalent to a probability distribution. A mass function can thus be seen as a generalized set, or as a generalized probability distribution. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

12 Belief function Degrees of support and consistency Dempster-Shafer theory Mass, belief and plausibility functions Let m be a normalized mass function on Ω induced by a source (S, 2 S, P, Γ). Let A be a subset of Ω. One may ask: 1 To what extent does the evidence support the proposition ω A? 2 To what extent is the evidence consistent with this proposition? (S,2 S,P)! Γ! s 3! A! B 3! Ω! s 2! s 1! s4! B 1! B 2! B 4! Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

13 Dempster-Shafer theory Mass, belief and plausibility functions Belief function Definition and interpretation For any A Ω, the probability that the evidence implies (supports) the proposition ω A is Bel(A) = P({s S Γ(s) A}) = B A m(b). (S,2 S,P)! Γ! B 3! Ω! s 3! A! s 2! s 1! s4! B 1! B 2! B 4! The function Bel : A Bel(A) is called a belief function. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

14 Dempster-Shafer theory Mass, belief and plausibility functions Belief function Characterization Function Bel : 2 Ω [0, 1] is a completely monotone capacity: it verifies Bel( ) = 0, Bel(Ω) = 1 and ( k ) Bel A i i=1 =I {1,...,k} for any k 2 and for any family A 1,..., A k in 2 Ω. ( ) ( 1) I +1 Bel A i. Conversely, to any completely monotone capacity Bel corresponds a unique mass function m such that: m(a) = ( 1) A B Bel(B), A Ω. =B A i I Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

15 Plausibility function Dempster-Shafer theory Mass, belief and plausibility functions The probability that the evidence is consistent with (does not contradict) the proposition ω A Pl(A) = P({s S Γ(s) A }) = 1 Bel(A) (S,2 S,P)! Γ! s 3! A! B 3! Ω! s 2! s 1! s 4! B 1! B 2! B 4! The function Pl : A Pl(A) is called a plausibility function. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

16 Dempster-Shafer theory Mass, belief and plausibility functions Special cases If m is Bayesian, then Bel = Pl and it is a probability measure. If the focal sets of m are nested (A 1 A 2... A n ), m is said to be consonant. Pl is then a possibility measure: Pl(A B) = max (Pl(A), Pl(B)) for all A, B Ω and Bel is the dual necessity measure. DS theory thus subsumes both probability theory and possibility theory. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

17 Dempster-Shafer theory Mass, belief and plausibility functions Summary A probability measure is precise, in so far as it represents the uncertainty of the proposition ω A by a single number P(A). In contrast, a mass function is imprecise (it assigns probabilities to subsets). As a result, in DS theory, the uncertainty about a subset A is represented by two numbers (Bel(A), Pl(A)), with Bel(A) Pl(A). This model has some connections with rough set theory, in which a set is approximated by lower and upper approximations, due to coarseness of a knowledge base. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

18 Outline Dempster-Shafer theory Dempster s rule 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

19 Dempster-Shafer theory Dempster s rule Dempster s rule Murder example continued The first item of evidence gave us: m 1 ({Peter, John}) = 0.8, m 1 (Ω) = 0.2. New piece of evidence: a blond hair has been found. There is a probability 0.6 that the room has been cleaned before the crime: m 2 ({John, Mary}) = 0.6, m 2 (Ω) = 0.4. How to combine these two pieces of evidence? Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

20 Dempster-Shafer theory Dempster s rule Dempster s rule Justification (S 1, P 1 ) drunk (0.2) not drunk (0.8) Γ 1 Peter Ω Mary If interpretations s 1 S 1 and s 2 S 2 both hold, then X Γ 1 (s 1 ) Γ 2 (s 2 ). If the two pieces of evidence are independent, then the probability that s 1 and s 2 both hold is P 1 ({s 1 })P 2 ({s 2 }). cleaned (0.6) (S 2, P 2 ) John If Γ 1 (s 1 ) Γ 2 (s 2 ) =, we know that s 1 and s 2 cannot hold simultaneously. not cleaned (0.4) Γ 2 The joint probability distribution on S 1 S 2 must be conditioned to eliminate such pairs. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

21 Dempster s rule Definition Dempster-Shafer theory Dempster s rule Let m 1 and m 2 be two mass functions and κ = m 1 (B)m 2 (C) their degree of conflict. B C= If K < 1, then m 1 and m 2 can be combined as (m 1 m 2 )(A) = 1 1 κ and (m 1 m 2 )( ) = 0. B C=A m 1 (B)m 2 (C), A, Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

22 Dempster-Shafer theory Dempster s rule Dempster s rule Properties Commutativity, associativity. Neutral element: m Ω. Generalization of intersection: if m A and m B are categorical mass functions and A B, then m A m B = m A B Generalization of probabilistic conditioning: if m is a Bayesian mass function and m A is a logical mass function, then m m A is a Bayesian mass function corresponding to the conditioning of m by A. Notation for conditioning (special case): m m A = m( A). Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

23 Outline Dempster-Shafer theory Decision analysis 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

24 Dempster-Shafer theory Decision analysis Problem formulation A decision problem can be formalized by defining: A set of acts A = {a 1,..., a s}; A set of states of the world Ω; A loss function L : A Ω R, such that L(a, ω) is the loss incurred if we select act a and the true state is ω. Bayesian framework Uncertainty on Ω is described by a probability measure P; Define the risk of each act a as the expected loss if a is selected: R P (a) = E P [L(a, )] = ω Ω L(a, ω)p({ω}). Select an act with minimal risk. Extension when uncertainty on Ω is described by a belief function? Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

25 Dempster-Shafer theory Lower and upper expected risk Decision analysis Let m be a normalized mass function, and P(m) its credal set, defined as the set of probability measures on Ω such that Bel(A) P(A) Pl(A), A Ω. The lower and upper risk of each act a are defined, respectively, as: R(a) = E m [L(a, )] = inf R P(a) = m(a) min L(a, ω) P P(m) ω A A Ω R(a) = E m [L(a, )] = sup R P (a) = m(a) max L(a, ω) P P(m) ω A A Ω Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

26 Decision strategies Dempster-Shafer theory Decision analysis For each act a we have a risk interval [R(a), R(a)]. How to compare these intervals? Three strategies: 1 a is preferred to a iff R(a) R(a ) (optimistic strategy) 2 a is preferred to a iff R(a) R(a ) (pessimistic strategy) 3 a is preferred to a iff R(a) R(a ) (interval dominance); The interval dominance strategy yields only a partial preorder: a and a are not comparable if R(a) > R(a ) and R(a ) > R(a) We can consider the set of non dominated acts (the set of acts a such that no act is strictly preferred to a) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

27 Dempster-Shafer theory Decision analysis Other decision strategies How to find a compromise between the pessimistic and optimistic strategies? Two approaches: 1 Hurwicz criterion: a is preferred to a iff R ρ (a) R ρ (a ) with R ρ (a) = (1 ρ)r(a) + ρr(a). and ρ [0, 1] is a pessimism index describing the attitude of the decision maker in the face of ambiguity. 2 Minimize the risk with respect to the pignistic probability measure P m, defined from m by the probability mass function p m (ω) = B ω m(b), ω Ω. B It can be shown that P m P(m). Consequently, R(a) R Pm (a) R(a), a A. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

28 Dempster-Shafer theory Decision analysis Decision making Example Let m({john}) = 0.48, m({john, Mary}) = 0.12, m({peter, John}) = 0.32, m(ω) = We have p m (John) = , p m (Peter) = p m (Mary) = Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

29 Evidential classification Outline 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

30 Classification problem Evidential classification? A population is assumed to be partitioned in c groups or classes Let Ω = {ω 1,..., ω c } denote the set of classes Each instance is described by A feature vector x R p A class label y Ω Problem: given a learning set L = {(x 1, y 1 ),..., (x n, y n )}, predict the class label of a new instance described by x Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

31 Outline Evidential classification Evidential K -NN rule 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

32 Evidential classification Evidential K -NN rule Principle X d i X i Let N K (x) L denote the set of the K nearest neighbors of x in L, based on some distance measure Each x i N K (x) can be considered as a piece of evidence regarding the class of x The strength of this evidence decreases with the distance d i between x and x i Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

33 Definition Evidential classification Evidential K -NN rule If y i = ω k, the evidence of (x i, y i ) can be represented by m i ({ω k }) = ϕ k (d i ) m i ({ω l }) = 0, l k m i (Ω) = 1 ϕ (d i ) where ϕ k, k = 1,..., c are decreasing functions from [0, + ) to [0, 1] such that lim d + ϕ k (d) = 0 The evidence of the K nearest neighbors of x is pooled using Dempster s rule of combination m = x i N K (x) Decision: any of the decision rules mentioned in the first part. With 0-1 losses and no rejection, the optimistic, pessimistic and pignistic rules yield the same decisions. m i Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

34 Learning Evidential classification Evidential K -NN rule Choice of functions ϕ k : for instance, ϕ k (d) = α exp( γ k d 2 ). Parameters γ 1,..., γ c can be optimized (see below). Parameter γ = (γ 1,..., γ c ) can be learnt from the data by minimizing the following cost function C(γ) = n c (pl ( i) (ω k ) t ik ) 2, i=1 k=1 where pl ( i) is the contour function obtained by classifying x i using its K nearest neighbors in the learning set. t ik = 1 is y i = k, t ik = 0 otherwise. Function C(γ) can be minimized by an iterative nonlinear optimization algorithm. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

35 Evidential classification Evidential K -NN rule Computation of pl ( i) Contour function from each neighbor x j N K (x i ): { 1 if y j = ω k pl j (ω k ) = 1 ϕ k (d ij ) otherwise, k = 1,..., c Contour function of the combined mass function pl ( i) (ω k ) (1 ϕ k (d ij )) 1 t jk x j N K (x i ) where t jk = 1 if y j = ω k and t jk = 0 otherwise It can be computed in time proportional to K Ω Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

36 Evidential classification Evidential K -NN rule Example 1: Vehicles dataset The data were used to distinguish 3D objects within a 2-D silhouette of the objects. Four classes: bus, Chevrolet van, Saab 9000 and Opel Manta. 846 instances, 18 numeric attributes. The first 564 objects are training data, the rest are test data. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

37 Evidential classification Vehicles datasets: result Evidential K -NN rule Vehicles data test error rate EK NN voting K NN Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

38 Evidential classification Evidential K -NN rule Example 2: Ionosphere dataset This dataset was collected by a radar system and consists of phased array of 16 high-frequency antennas with a total transmitted power of the order of 6.4 kilowatts. The targets were free electrons in the ionosphere. "Good" radar returns are those showing evidence of some type of structure in the ionosphere. "Bad" returns are those that do not. There are 351 instances and 34 numeric attributes. The first 175 instances are training data, the rest are test data. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

39 Evidential classification Ionosphere datasets: result Evidential K -NN rule Ionosphere data test error rate EK NN voting K NN Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

40 Implementation in R Evidential classification Evidential K -NN rule library("evclass") data("ionosphere") xapp<-ionosphere$x[1:176,] yapp<-ionosphere$y[1:176] xtst<-ionosphere$x[177:351,] ytst<-ionosphere$y[177:351] opt<-eknnfit(xapp,yapp,k=10) class<-eknnval(xapp,yapp,xtst,k=10,ytst,opt$param) > class$err > table(ytst,class$ypred) ytst Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

41 Evidential classification Partially supervised data Evidential K -NN rule We now consider a learning set of the form L = {(x i, m i ), i = 1,..., n} where x i is the attribute vector for instance i, and m i is a mass function representing uncertain expert knowledge about the class y i of instance i Special cases: m i ({ω k }) = 1 for all i: supervised learning m i (Ω) = 1 for all i: unsupervised learning Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

42 Evidential classification Evidential K -NN rule Evidential k-nn rule for partially supervised data Each mass function m i is discounted (weakened) with a rate depending on the distance d i (X i, m i ) m i (A) = ϕ (d i ) m i (A), A Ω d i m i (Ω) = 1 A Ω m i (A) X The K mass functions m i are combined using Dempster s rule m = x i N K (x) m i Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

43 Example: EEG data Evidential classification Evidential K -NN rule EEG signals encoded as 64-D patterns, 50 % positive (K-complexes), 50 % negative (delta waves), 5 experts A/D converter output A/D converter output time Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

44 Results on EEG data (Denoeux and Zouhal, 2001) Evidential classification Evidential K -NN rule c = 2 classes, p = 64 For each learning instance x i, the expert opinions were modeled as a mass function m i. n = 200 learning patterns, 300 test patterns K K -NN w K -NN Ev. K -NN Ev. K -NN (crisp labels) (uncert. labels) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

45 Outline Evidential classification Evidential neural network classifier 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

46 Evidential classification Evidential neural network classifier Principle The learning set is summarized by r prototypes. X d i p i Each prototype p i has membership degree u ik to each class ω k, with c k=1 u ik = 1. Each prototype p i is a piece of evidence about the class of x, whose reliability decreases with the distance d i between x and p i. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

47 Evidential classification Evidential neural network classifier Propagation equations Mass function induced by prototype p i : m i ({ω k }) = α i u ik exp( γ i d 2 i ), m i (Ω) = 1 α i exp( γ i d 2 i ) k = 1,..., c Combination: m = r i=1 The computation of m i requires O(rp) arithmetic operations (where p denotes the number of inputs), and the combination can be performed in O(rc) operations. Hence, the overall complexity is O(r(p + c)) operations to compute the output for one input pattern. The combined mass function m has as focal sets the singletons {ω k }, k = 1,..., c and Ω. m i Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

48 Evidential classification Neural network implementation Evidential neural network classifier x j p ij u ik -1 1 m i m 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

49 Learning Evidential classification Evidential neural network classifier The parameters are the The prototypes p i, i = 1,..., r (rp parameters) The membership degrees u ik, i = 1,..., r, k = 1..., c (rc parameters) The α i and γ i, i = 1..., r (2r parameters). Let θ denote the vector of all parameters. It can be estimated by minimizing a cost function such as C(θ) = n (pl ik t ik ) 2 + µ r i=1 i=1 α i where pl ik is the output plausibility for instance i and class k, t ik = 1 if y i = k and t ik = 0 otherwise, and µ is a regularization coefficient (hyperparameter). The hyperparameter µ can be optimized by cross-validation. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

50 Implementation in R Evidential classification Evidential neural network classifier library("evclass") data(glass) xtr<-glass$x[1:89,] ytr<-glass$y[1:89] xtst<-glass$x[90:185,] ytst<-glass$y[90:185] param0<-prodsinit(xtr,ytr,nproto=7) fit<-prodsfit(x=xtr,y=ytr,param=param0) val<-prodsval(xtst,fit$param,ytst) > print(val$err) > table(ytst,val$ypred) ytst Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

51 Evidential classification Evidential neural network classifier Results on the Iris data Mass on {ω 1 } m({ω 1 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

52 Evidential classification Evidential neural network classifier Results on the Iris data Mass on {ω 2 } m({ω 2 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

53 Evidential classification Evidential neural network classifier Results on the Iris data Mass on {ω 3 } m({ω 3 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

54 Evidential classification Evidential neural network classifier Results on the Iris data Mass on Ω m(ω) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

55 Evidential classification Evidential neural network classifier Results on the Iris data Plausibility of {ω 1 } Pl({ω 1 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

56 Evidential classification Evidential neural network classifier Results on the Iris data Plausibility of {ω 2 } Pl({ω 2 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

57 Evidential classification Evidential neural network classifier Results on the Iris data Plausibility of {ω 3 } Pl({ω 3 }) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

58 Outline Evidential classification Decision analysis 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

59 Evidential classification Decision analysis Simple decision setting To formalize the decision problem, we need to define: The acts The loss matrix For instance, let the acts be a k = assignment to class ω k, k = 1,..., c And the loss matrix (for c = 3) a 1 a 2 a 3 ω ω ω R(a i ) = 1 Pl({ω i }) and R(a i ) = 1 Bel({ω i }). The optimistic, pessimistic and pignistic decision rules yield the same result Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

60 Implementation in R Evidential classification Decision analysis param0<-prodsinit(x,y,6) fit<-prodsfit(x,y,param0) val<-prodsval(xtst,fit$param) L<-1-diag(c) D<-decision(val$m,L=L,rule= upper ) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

61 Evidential classification Decision regions (Iris data) Decision analysis Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

62 Evidential classification Decision analysis Decision with rejection Let the acts now be a k = assignment to class ω k, k = 1,..., c a 0 = rejection And the loss matrix (for c = 3) a 1 a 2 a 3 a 0 ω λ 0 ω λ 0 ω λ 0 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

63 Implementation in R Evidential classification Decision analysis param0<-prodsinit(x,y,6) fit<-prodsfit(x,y,param0) val<-prodsval(xtst,fit$param) L<-cbind(1-diag(c),rep(0.3,c)) D1<-decision(val$m,L=L,rule= upper ) D2<-decision(val$m,L=L,rule= lower ) D3<-decision(val$m,L=L,rule= pignistic ) D4<-decision(val$m,L=L,rule= hurwicz,rho=0.5) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

64 Evidential classification Decision analysis Decision regions (Iris data) Lower risk Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

65 Evidential classification Decision analysis Decision regions (Iris data) Upper risk Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

66 Evidential classification Decision analysis Decision regions (Iris data) Pignistic risk Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

67 Evidential classification Decision analysis Decision regions (Iris data) Hurwicz strategy (ρ = 0.5) Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

68 Evidential classification Decision analysis Decision with rejection and novelty detection Assume that there exists an unknown class ω u, not represented in the learning set Let the acts now be a k = assignment to class ω k, k = 1,..., c a u = assignment to class ω u a 0 = rejection And the loss matrix a 1 a 2 a 3 a 0 a u ω λ 0 λ u ω λ 0 λ u ω λ 0 λ u ω u λ 0 0 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

69 Implementation in R Evidential classification Decision analysis param0<-prodsinit(x,y,6) fit<-prodsfit(x,y,param0) val<-prodsval(xtst,fit$param) L<-cbind(1-diag(c),rep(0.3,c),rep(0.32,c)) L<-rbind(L,c(1,1,1,0.3,0)) D1<-decision(val$m,L=L,rule= lower ) D2<-decision(val$m,L=L,rule= pignistic ) D3<-decision(val$m,L=L,rule= hurwicz,rho=0.5) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

70 Evidential classification Decision regions (Iris data) Decision analysis Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

71 Evidential classification Decision regions (Iris data) Decision analysis Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

72 Evidential classification Decision regions (Iris data) Decision analysis Petal.Width Petal.Length Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

73 Evidential classification Decision analysis References on classification I cf. tdenoeux T. Denœux. A k-nearest neighbor classification rule based on Dempster-Shafer theory. IEEE Transactions on SMC, 25(05): , T. Denœux. A neural network classifier based on Dempster-Shafer theory. IEEE transactions on SMC A, 30(2): , T. Denœux. Analysis of evidence-theoretic decision rules for pattern classification. Pattern Recognition, 30(7): , C. Lian, S. Ruan and T. Denœux. An evidential classifier based on feature selection and two-step classification strategy. Pattern Recognition, 48: , Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

74 Evidential classification Decision analysis References on classification II cf. tdenoeux C. Lian, S. Ruan and T. Denœux. Dissimilarity metric learning in the belief function framework. IEEE Transactions on Fuzzy Systems (to appear), Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

75 Application to clustering Outline 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

76 Application to clustering Clustering n objects described by Attribute vectors x 1,..., x n (attribute data) or Dissimilarities (proximity data). Goal: find a meaningful structure in the data set, usually a partition into c crisp or fuzzy subsets. Belief functions may allow us to express richer information about the data structure. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

77 Outline Application to clustering credal partition 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

78 Application to clustering credal partition Clustering concepts Hard and fuzzy clustering Hard clustering: each object belongs to one and only one group. Group membership is expressed by binary variables u ik such that u ik = 1 if object i belongs to group k and u ik = 0 otherwise Fuzzy clustering: each object has a degree of membership u ik [0, 1] to each group, with c k=1 u ik = 1 Fuzzy clustering with noise cluster: each object has a degree of membership u ik [0, 1] to each group and a degree of membership u i [0, 1] to a noise cluster, with c k=1 u ik + u i = 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

79 Application to clustering credal partition Clustering concepts Possibilistic, rough, credal clustering Possibilistic clustering: the condition c k=1 u ik = 1 is relaxed. Each number u ik can be interpreted as a degree of possibility that object i belongs to cluster k Rough clustering: the membership of object i to cluster k is described by a pair (u ik, u ik ) {0, 1} 2, with u ik u ik, indicating its membership to the lower and upper approximations of cluster k Evidential clustering: based on Dempster-Shafer (DS) theory (the topic of this talk) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

80 Application to clustering credal partition Evidential clustering In evidential clustering, the cluster membership of each object is considered to be uncertain and is described by a (not necessarily normalized) mass function m i over Ω The n-tuple M = (m 1,..., m n ) is called a credal partition Example: Butterfly data x Credal partition {ω 1 } {ω 2 } {ω 1, ω 2 } m m m m x 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

81 Application to clustering credal partition Relationship with other clustering structures More%general% m i %unormalized%% Bayesian% Credal%par''on% m i %general% Fuzzy%par''on% with%a%noise%cluster% Fuzzy%par''on% m i %Bayesian% Possibilis'c%par''on% m i %consonant% Rough%par''on% m i %logical% Hard%par''on% m i %certain% Less%general% Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

82 Application to clustering credal partition Rough clustering as a special case m({ω 1 })=1( m({ω 1, ω 2 })=1( m({ω 2 })=1( Lower( approxima4ons( Upper( approxima4ons( ω 1L ( ω 2L ( ω 2U ( ω 1U ( Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

83 Application to clustering credal partition Summarization of a credal partition More complex unnormalized pignis'c/plausibility transforma'on Fuzzy par''on with a noise cluster Less complex normaliza'on Fuzzy par''on maximum probability Credal par''on contour func'on Possibilis'c par''on maximum plausibility Hard par''on interval dominance or maximum mass Rough par''on Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

84 Application to clustering credal partition Algorithms 1 Evidential c-means (ECM): (Masson and Denoeux, 2008): Attribute data, HCM, FCM family (alternate optimization of a cost function). 2 EVCLUS (Denoeux and Masson, 2004; Denoeux et al., 2016): Proximity (possibly non metric) data, Multidimensional scaling approach. 3 EK-NNclus (Denoeux et al, 2015) Attribute or proximity data Decision-directed clustering algorithm based on the evidential K-NN classifier Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

85 Outline Application to clustering Evidential c-means 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

86 Principle Application to clustering Evidential c-means Problem: generate a credal partition M = (m 1,..., m n ) from attribute data X = (x 1,..., x n ), x i R p. Generalization of hard and fuzzy c-means algorithms: Each cluster is represented by a prototype; Cyclic coordinate descent algorithm: optimization of a cost function with respect to the prototypes and to the credal partition. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

87 Application to clustering Fuzzy c-means (FCM) Evidential c-means Minimize J FCM (U, V ) = n c i=1 k=1 u β ik d 2 ik with d ik = x i v k under the constraints k u ik = 1, i. Alternate optimization algorithm: v k = n i=1 uβ ik x i n i=1 uβ ik k = 1,..., c, u ik = d 2/(β 1) ik c l=1 d 2/(β 1) il. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

88 ECM algorithm Principle Application to clustering Evidential c-means v 2 v 1 v 2 v 4 v 3 v 1 v 3 Each cluster ω k represented by a prototype v k. Each non empty set of clusters A j represented by a prototype v j defined as the center of mass of the v k for all ω k A j. Basic ideas: For each non empty A j Ω, m ij = m i (A j ) should be high if x i is close to v j. The distance to the empty set is defined as a fixed value δ. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

89 Application to clustering Evidential c-means ECM algorithm: objective criterion Criterion to be minimized: J ECM (M, V ) = n i=1 {j/a j,a j Ω} A j α m β ij d 2 ij + n δ 2 m β i i=1 subject to {j/a j Ω,A j } m ij + m i = 1, i {1,..., n}, Parameters: α controls the specificity of mass functions (default: 1) β controls the hardness of the credal partition (default: 2) δ controls the proportion of data considered as outliers J ECM (M, V ) can be iteratively minimized with respect to M and V using a cyclic coordinate descent algorithm. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

90 Application to clustering Evidential c-means ECM algorithm: update equations Optimization of J ECM (M, V ) w.r.t. M for fixed V : m ij = c α/(β 1) j d 2/(β 1) ij A k c α/(β 1) k d 2/(β 1) ik + δ, 2/(β 1) for i = 1,..., n and for all j such that A j, and m i = 1 A j m ij, i = 1,..., n Optimization of J ECM (M, V ) w.r.t. V for fixed M: solving a system of the form HV = B, where B is the matrix of size c p and H the matrix of size c c Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

91 Implementation in R Application to clustering Evidential c-means library(evclust) data( butterfly ) n<-nrow(butterfly) clus<-ecm(butterfly[,1:2],c=2,delta=sqrt(20)) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

92 Application to clustering Evidential c-means Butterfly dataset x Butterfly data x 1 masses m( ) m(ω 1 ) m(ω 2 ) m(ω) objects Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

93 Four-class dataset Application to clustering Evidential c-means data("fourclass") clus<-ecm(fourclass[,1:2],c=4,type= pairs,delta=5) plot(clus,x=fourclass[,1:2],ytrue=fourclass[,3],outliers = TRUE, approx=2) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

94 4-class data set Application to clustering Evidential c-means x x 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

95 Application to clustering Evidential c-means Determining the number of groups If a proper number of groups is chosen, the prototypes will cover the clusters and most of the mass will be allocated to singletons of Ω. On the contrary, if c is too small or too high, the mass will be distributed to subsets with higher cardinality or to. Nonspecificity of a mass function: N(m) m(a) log 2 A + m( ) log 2 Ω A 2 Ω \ Proposed validity index of a credal partition: N 1 n (c) n log 2 (c) m i (A) log 2 A + m i ( ) log 2 (c) i=1 A 2 Ω \ Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

96 Application to clustering Example (Four-class dataset) Evidential c-means C<-2:7 N<-rep(0,length(C)) for(k in 1:length(C)){ clus<-ecm(fourclass[,1:2],c=c[k],type= pairs,alpha=2, delta=5,disp=false) N[k]<-clus$N } plot(c,n,type= b,xlab= c,ylab= nonspecificity ) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

97 Results Application to clustering Evidential c-means nonspecificity c Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

98 Outline Application to clustering EVCLUS 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

99 Application to clustering EVCLUS Learning a Credal Partition from proximity data Problem: given the dissimilarity matrix D = (d ij ), how to build a reasonable credal partition? We need a model that relates cluster membership to dissimilarities. Basic idea: The more similar two objects, the more plausible it is that they belong to the same group. How to formalize this idea? Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

100 Application to clustering EVCLUS Formalization Let m i and m j be mass functions regarding the group membership of objects o i and o j. The plausibility of the proposition S ij : objects o i and o j belong to the same group can be shown to be equal to: pl(s ij ) = m i (A)m j (B) = 1 κ ij A B where κ ij = degree of conflict between m i and m j. Problem: find a credal partition M = (m 1,..., m n ) such that larger degrees of conflict κ ij correspond to larger dissimilarities d ij. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

101 Application to clustering EVCLUS Cost function Approach: minimize the discrepancy between the dissimilarities d ij and the degrees of conflict κ ij. Example of a cost (stress) function: J(M) = η i<j (κ ij ϕ(d ij )) 2 where η = ( i<j ϕ(d ij) 2 ) 1 is a normalizing constant, and ϕ is an increasing function from [0, + ) to [0, 1]. For instance: ϕ(d) = 1 exp( γd 2 ) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

102 Application to clustering EVCLUS Butterfly example Data and dissimilarities Butterfly data x ϕ(d ij) α d x 1 d ij Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

103 Application to clustering EVCLUS Butterfly example Credal partition Butterfly data x masses m( ) m(ω 1) m(ω 2) m(ω) x 1 objects Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

104 Application to clustering EVCLUS Butterfly example Shepard diagram κ ij δ ij Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

105 Application to clustering EVCLUS Optimization algorithm How to minimize J(M)? Two methods: 1 Using a gradient or quasi-newton algorithm (slow). 2 Using a cyclic coordinate descent algorithm minimizing J(M) with respect to each m i at a time. The latter approach exploits the particular approach of the problem (a quadratic programming problem is solved at each step), and it is thus much more efficient. This algorithm is called Iterative Row-wise Quadratic Programming (IRQP). Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

106 Application to clustering EVCLUS IRQP algorithm Vector representation of the cost function The stress function can be written as J(M) = η i<j (m T i Cm j δ ij ) 2. where δ ij = ϕ(d ij ) are the scaled dissimilarities m i and m j are vectors encoding mass functions m i and m j C is a square matrix, with general term C kl = 1 if F k F l = and C kl = 0 otherwise. Fixing all mass functions except m i, the stress function becomes quadratic. Minimizing J w.r.t. m i is a linearly constrained positive least-squares problem, which can be solved using efficient algorithms. By iteratively updating each m i, the algorithm converges to a local minimum of the cost function. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

107 Application to clustering EVCLUS Reducing the number of parameters If the mass functions have a general form, the number of parameters to estimate of n(2 c 1). It grows exponentially with c. To reduce the complexity, focal sets can be reduced to {ω k } c k=1,, and Ω. A more sophisticated strategy will be described later. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

108 Application to clustering EVCLUS Proteins example Dissimilarity matrix derived from the structural comparison of 213 protein sequences. Each of these proteins is known to belong to one of four classes of globins: hemoglobin-α (HA), hemoglobin-β (HB), myoglobin (M) and heterogeneous globins (G). The next figure displays a two-dimensional MDS configuration of the data with the true partition, as well as the clustering result obtained by EVCLUS, with c = 4 and d 0 = max i,j d ij. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

109 Implementation in R Application to clustering EVCLUS library(evclust) data(protein) clus <- kevclus(d=protein$d,c=4,type= simple,d0=max(protein$d)) z<- cmdscale(protein$d,k=2) plot(clus,x=z,mfrow=c(2,2),ytrue=protein$y, Outliers=FALSE,approx=1) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

110 Application to clustering Proteins example: partition EVCLUS axis axis axis axis 1 axis axis axis axis 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

111 Application to clustering EVCLUS Proteins example: Shepard diagram κij Shepard diagram δij Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

112 Application to clustering EVCLUS Proteins example: learning curves Gradient, Protein data IRQP, Protein data stress stress time (s) time (s) Stress vs. time (in seconds) for 20 runs of the Gradient (a) and IRQP (b) algorithms on the Protein data. Note the different scales on the x-axes. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

113 Application to clustering EVCLUS Proteins example: learning curves Protein Protein time stress gradient IRQP gradient IRQP Boxplots of computing time (a) and stress value at convergence (b) for 20 runs of the Gradient and IRQP algorithms on the Protein data. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

114 Application to clustering EVCLUS Example with a four-class dataset (2000 objects) x[, 2] x[, 2] x[, 1] x[, 1] x[, 2] x[, 2] x[, 1] x[, 1] Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

115 Application to clustering Handling large datasets EVCLUS EVCLUS requires to store the whole dissimilarity matrix: it inapplicable to large dissimilarity data. Idea: compute the differences between degrees of conflict and dissimilarities, for only a subset of randomly sampled dissimilarities. Let j 1 (i),..., j k (i) be k integers sampled at random from the set {1,..., i 1, i + 1,..., n}, for i = 1,..., n. Let J k the following stress criterion, n k J k (M) = η (κ i,jr (i) δ i,jr (i)) 2, i=1 r=1 The calculation of J k (M) requires only O(nk) operations. If k can be kept constant as n increases, or, at least, if k increases slower than linearly with n, then significant gains in computing time and storage requirement could be achieved. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

116 Application to clustering EVCLUS Zongker Digit dissimilarity data Similarities between 2000 handwritten digits in 10 classes, based on deformable template matching. As the dissimilarity matrix was initially non symmetric, we symmetrized it by the transformation d ij (d ij + d ji )/2. The k-evclus algorithm was run with c = 10 and the following values of k: 30, 50,100, 200, 300, 400, 500, 1000 and Parameter d 0 was fixed to the 0.3-quantile of the dissimilarities. For each value of k, k-evclus was run 10 times with random initializations. Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

117 Implementation in R Application to clustering EVCLUS load( zongker.rdata ) n<-nrow(zongker$d) k=200 D<-matrix(0,n,k) J<-matrix(0,n,k) for(i in 1:n){ ii<-sample((1:n)[-i],k) J[i,]<-ii D[i,]<-zongker$D[i,ii] } clus<-kevclus(d=d,j=j,c=10,type= simple,d0=quantile(d,0.3)) library(mclust) adjustedrandindex(zongker$y,clus$y.pl) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

118 Application to clustering EVCLUS Zongker Digit dissimilarity data Results ARI time (s) k k Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

119 Outline Application to clustering EK-NNclus 1 Dempster-Shafer theory Mass, belief and plausibility functions Dempster s rule Decision analysis 2 Evidential classification Evidential K -NN rule Evidential neural network classifier Decision analysis 3 Application to clustering credal partition Evidential c-means EVCLUS EK-NNclus Handling a large number of clusters Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

120 Application to clustering EK-NNclus Decision-directed clustering Decision-directed approach to clustering: Prior knowledge is used to design a classifier, which is used to label the samples The classifier is then updated, and the process is repeated until no changes occur in the labels The c-means algorithm is based on this principle: here, the nearest-prototype classifier is used to label the samples, and it is updated by taking as prototypes the centers of each cluster Idea: apply this principle using the evidential K -NN rule as the base classifier Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

121 Application to clustering EK-NNclus Example Toy dataset Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

122 Application to clustering EK-NNclus Example Iteration 1 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

123 Application to clustering EK-NNclus Example Iteration 1 (continued) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

124 Application to clustering EK-NNclus Example Iteration 2 Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149

125 Application to clustering EK-NNclus Example Iteration 2 (continued) Thierry Denœux (UTC/HEUDIASYC) Classification and clustering using Belief functions Tongji University, July 7, / 149