JL i> pdZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z ejeZGdd Zed k(rddlZddlZdd lmZej*Zej/d d dddej/dddddej/ddddgdej/ddddgd ej/d!d"d#d$%ej/d&d'd(d)d*ej/d+d,d-d.d/ej/d0d1d2d3d4ej/d5d6d7dd8ej/d9d:d;d?ej1\ZZej6sej9eej<d@dAeej@zz BgZ!e"ej65Z#e#D]Z$e$jKejLZ'e'de(e'dCdDe)e'dDjUjKejVcZ,Z-Z.ej^ej`k(sHe1ej^dkDre,ej^vs%e1ej`dkDse,ej`vse!jee,e-e.f dddejfr+ee!e4eej(ejfZ5nee!e4eej(Z5ejlrne7e4e5ejpejryy#1swYxYw)EaC Implementations of inter-annotator agreement coefficients surveyed by Artstein and Poesio (2007), Inter-Coder Agreement for Computational Linguistics. An agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance. In defining the AnnotationTask class, we use naming conventions similar to the paper's terminology. There are three types of objects in an annotation task: the coders (variables "c" and "C") the items to be annotated (variables "i" and "I") the potential categories to be assigned (variables "k" and "K") Additionally, it is often the case that we don't want to treat two different labels as complete disagreement, and so the AnnotationTask constructor can also take a distance metric as a final argument. Distance metrics are simply functions that take two arguments, and return a value between 0.0 and 1.0 indicating the distance between them. If not supplied, the default is binary comparison between the arguments. The simplest way to initialize an AnnotationTask is with a list of triples, each containing a coder's assignment for one object in the task: task = AnnotationTask(data=[('c1', '1', 'v1'),('c2', '1', 'v1'),...]) Note that the data list needs to contain the same number of triples for each individual coder, containing category values for the same set of items. Alpha (Krippendorff 1980) Kappa (Cohen 1960) S (Bennet, Albert and Goldstein 1954) Pi (Scott 1955) TODO: Describe handling of multiple coders and missing data Expected results from the Artstein and Poesio survey paper: >>> from nltk.metrics.agreement import AnnotationTask >>> import os.path >>> t = AnnotationTask(data=[x.split() for x in open(os.path.join(os.path.dirname(__file__), "artstein_poesio_example.txt"))]) >>> t.avg_Ao() 0.88 >>> round(t.pi(), 5) 0.79953 >>> round(t.S(), 2) 0.82 This would have returned a wrong value (0.0) in @785fb79 as coders are in the wrong order. Subsequently, all values for pi(), S(), and kappa() would have been wrong as they are computed with avg_Ao(). >>> t2 = AnnotationTask(data=[('b','1','stat'),('a','1','stat')]) >>> t2.avg_Ao() 1.0 The following, of course, also works. >>> t3 = AnnotationTask(data=[('a','1','othr'),('b','1','othr')]) >>> t3.avg_Ao() 1.0 N)groupby) itemgetter) deprecated)binary_distance)ConditionalFreqDistFreqDistceZdZdZdefdZdZdZddZdZ dZ d Z e d dd Z dd Zd ZdZdZddZddZdZdZdZdZdZdZdZdZddZddZy)AnnotationTaska/Represents an annotation task, i.e. people assign labels to items. Notation tries to match notation in Artstein and Poesio (2007). In general, coders and items can be represented as any hashable object. Integers, for example, are fine, though strings are more readable. Labels must support the distance functions applied to them, so e.g. a string-edit-distance makes no sense if your labels are integers, whereas interval distance needs numeric values. A notable case of this is the MASI metric, which requires Python sets. Nc||_t|_t|_t|_g|_||j |yy)aInitialize an annotation task. The data argument can be None (to create an empty annotation task) or a sequence of 3-tuples, each representing a coder's labeling of an item: ``(coder,item,label)`` The distance argument is a function taking two arguments (labels) and producing a numerical distance. The distance from a label to itself should be zero: ``distance(l,l) = 0`` N)distancesetIKCdata load_array)selfrr s \/mnt/ssd/data/python-lab/Trading/venv/lib/python3.12/site-packages/nltk/metrics/agreement.py__init__zAnnotationTask.__init__`sD!    OOD ! cNdjtd|jS)Nz cf|dd|djddddj|dS)Ncoder item_,labels)replacejoin)xs rz(AnnotationTask.__str__..vs3W:qy00d;SXXak=RTr)r maprrs r__str__zAnnotationTask.__str__ss({{ T    rc|D]v\}}}|jj||jj||jj||jj |||dxy)zLoad an sequence of annotation results, appending to any data already loaded. The argument is a sequence of 3-tuples, each representing a coder's labeling of an item: (coder,item,label) )rrrN)raddrrrappend)rarrayrrrs rrzAnnotationTask.load_array|se $) O E4 FFJJu  FFJJv  FFJJt  II  uM N  Orcz|xs |j}tfd|D}|dk(rtfd|D}ntfd|D}dt|j|d|dz }tj d|tj d|d|dd|z |S) z,Agreement between two coders on a given itemc3HK|]}|dfvs |dk(s|ywrrN).0r!cAcBis r z%AnnotationTask.agr..s*OQwZB8%;& Q!Os """rc3FK|]}|dk(s |dk(s|ywr,r-)r.r!r0r1s rr2z%AnnotationTask.agr..&MA7r)9ai1naM !!!c3FK|]}|dk(s |dk(s|ywr,r-)r.r!r/r1s rr2z%AnnotationTask.agr..r4r5?rz.Observed agreement between %s and %s on %s: %fz"Distance between "%r" and "%r": %f)rnextfloatr logdebug)rr/r0r1rk1k2rets ``` ragrzAnnotationTask.agrs tyyOTO O g;" MMMBMMMBE$--8 blCDD BBAsS 0"X,8 cTWi  rcRttfd|jDS)Nc34K|]}|dk(s dyw)rNr-)r.r!ks rr2z$AnnotationTask.Nk..sBq81ABs r9sumr)rrCs `rNkzAnnotationTask.NksSBDIIBBCCrcVttfd|jDS)Nc3FK|]}|dk(s |dk(sdyw)rrrBNr-)r.r!r1rCs rr2z%AnnotationTask.Nik..s'Uq6aAhKSTDTUr5rD)rr1rCs ``rNikzAnnotationTask.NiksSUDIIUUVVrcVttfd|jDS)Nc3FK|]}|dk(s |dk(sdyw)rrrBNr-)r.r!crCs rr2z%AnnotationTask.Nck..s'Vq7qQx[TUEUVr5rD)rrLrCs ``rNckzAnnotationTask.NcksSVDIIVVWWrzUse Nk, Nik or Nck insteadc ||||j|}nG||||j||}n.||||j||}ntd|d|d|dtj d|||||S)z>Implements the "n-notation" used in Artstein and Poesio (2007)z*You must pass either i or c, not both! (k=z,i=z,c=)zCount on N[%s,%s,%s]: %d)rFrIrM ValueErrorr:r;)rrCr1rLr>s rNzAnnotationTask.Ns =QY19''!*C ]q}((1a.C ]q}((1a.C.s D1QwZB8-CQDc3LK|]\}}j||ywN)r?)r.r item_datar/r0rs rr2z$AnnotationTask.Ao..s$Pi$((2r43Ps!$z(Observed agreement between %s and %s: %f)rVrrElenrr:r;)rr/r0rr>s``` rAozAnnotationTask.Aosc!! D D P4PPSV FFT   s r_pairwise_averagez AnnotationTask._pairwise_averagest  FFKKM&& B HHRL "b))Q   ai rch|j|j}tjd||S)z7Average observed agreement across all coders and items.zAverage observed agreement: %f)rhr`r:r;)rr>s ravg_AozAnnotationTask.avg_Aos*$$TWW- 2C8 rc.d}fd|jD}|jd|D]2\}}||jt|dt|dz }4|t |j |zz }t jd||S)z=The observed disagreement for the weighted kappa coefficient.c36K|]}|dfvs |ywrYr-rZs rr2z0AnnotationTask.Do_Kw_pairwise..s ?a' r2h(>?r[rrz+Observed disagreement between %s and %s: %f)rrVr r8r_rr:r;) rr/r0 max_distancererr1itemdatar>s `` rDo_Kw_pairwisezAnnotationTask.Do_Kw_pairwises?499?--fd; WKAx T]]4>(#;T(^H=UV VE Ws466{\12 ?RM rc`jfd}tjd||S)zAveraged over all labelersc*j||Sr])rpr/r0rnrs rr"z&AnnotationTask.Do_Kw..s4..r2|DrzObserved disagreement: %f)rhr:r;)rrnr>s`` rDo_KwzAnnotationTask.Do_Kws,$$ D  -s3 rchdt|jz }|j|z d|z z }|S)z"Bennett, Albert and Goldstein 1954r7)r_rrj)rAer>s rSzAnnotationTask.Ss2 3tvv; {{}r!cBh/ rcd}td|jD}|jD] \}}||dzz }|t|jt|j zdzz }|j |z d|z z S)z_Scott 1955; here, multi-pi. Equivalent to K from Siegel and Castellan (1988). rlc3&K|] }|d ywrNr-r.r!s rr2z$AnnotationTask.pi..s>qq{>rB)rritemsr_rrrj)rre label_freqsrCfrvs rpizAnnotationTask.pis >DII>> %%' DAq QTME  s466{S[0Q6 7 "q2v..rcd}tt|j}td|jD}|j D]}|||||z ||||z zz }|S)Nrlc30K|]}|d|dfyw)rrNr-r{s rr2z*AnnotationTask.Ae_kappa..s)W1X;' *C)Ws)r9r_rrr conditions)rr/r0rvnitemsrrCs rAe_kappazAnnotationTask.Ae_kappasw s466{#))WTYY)WW '') PA ;q>"%.;q>"3E3NO OB P rc|j||}|j|||z d|z z }tjd||||S) r7z(Expected agreement between %s and %s: %f)rr`r:r;)rr/r0rvr>s rkappa_pairwisezAnnotationTask.kappa_pairwisesG ]]2r "wwr2#b1 EAr$**, >2rBw$--1*=== > >U{llQ.>?@@rct|jdk(r tdt|jdk(rtj dyt|j dk(r#t|j dk(r tdd}d}tg}d}|jdD]R\}}td|D}t|j}|d kr7||z }||j||zz }Tt|jdk(rtj d y|t|jz } |j|} d | | z z } | S) zKrippendorff 1980rz(Cannot calculate alpha, no data present!rBz-Only one annotation value, alpha returning 1.z8Cannot calculate alpha, only one coder and item present!rlrc3&K|] }|d ywrzr-r{s rr2z'AnnotationTask.alpha..5s"A11X;"Ar|r}z3Only one valid annotation value, alpha returning 1.r7) r_rrPr:r;rrrrVrErrkeys) rtotal_disagreement total_ratingsall_valid_labels_freqtotal_dor1ror labels_countdodek_alphas ralphazAnnotationTask.alpha%s] tvv;! GH H tvv;!  IIE F tvv;! DFF q 0WX X  ( --f5 FKAx""A"AAK{1134La ![ 0 ! ))+6E EH F $))+ , 1 IIK L 188:; ;   4 5R-rcd}tfd|jD}|jD]:}|jD])}|||||z|j||zz }+<||t t |j dzz }tjd||j} d| |z z } | S) Cohen 1968rlc3FK|]}|dfvs |d|dfyw)rrNr-rZs rr2z9AnnotationTask.weighted_kappa_pairwise..Ks2* *+AgJ2r(s `` rweighted_kappa_pairwisez&AnnotationTask.weighted_kappa_pairwiseHs)* /3yy*   WAVV WR+k"oa.@@4==QRTUCVVV W WlSTVVa%88 9 ?RL  R (R"Wo rc0jfdS)rc*j||Sr])rrss rr"z/AnnotationTask.weighted_kappa..Zs477B Mr)rh)rrns``rweighted_kappazAnnotationTask.weighted_kappaWs%% M  rr])NNN)r7)__name__ __module__ __qualname____doc__rrr%rr?rFrIrMrrQrVr`rhrjrprtrwrrrrrrrrrr-rrr r Ss !?"&  O&DWX,- . O    /;1A!F  rr __main__)r z-dz --distancer rzdistance metric to use)destdefaulthelpz-az --agreement agreementrz"agreement coefficient to calculatez-ez --excludeexcluder(z8coder names to exclude (may be specified multiple times))ractionrrz-iz --includeincludez.coder names to include, same format as excludez-fz--filefilezPfile to read labelings from, each line with three columns: 'labeler item labels')rrz-vz --verboseverbose0z+how much debugging to print on stderr (0-4)z-cz --columnsep columnseprzIchar/string that separates the three columns in the file, defaults to tabz-lz --labelseplabelseprz[char/string that separates labels (if labelers can assign more than one), defaults to commaz-pz --presencepresencez=convert each labeling into 1 or 0, based on presence of LABELz-Tz --thoroughthoroughF store_truez6calculate agreement for every subset of the annotators)rrrr2 )levelrB):rlogging itertoolsroperatorrnltk.internalsrnltk.metrics.distancernltk.probabilityrr getLoggerrr:r optparsere nltk.metricsr OptionParserparser add_option parse_argsoptions remainderr print_helpexit basicConfigintrropeninfilersplitrtoksstr frozensetstriprrobject_rrrr_r(rgetattrtaskrprintrshutdownr-rrrs=~%1:g!H H V z %#X " " $F   ! %     1     G     =     _      :     X     j     L     E ",,.Wi << Gb2GOO(<#<<= D gll  6v 6A777,,-DQD2J$r(..*001A1ABC #E7F GOO3(1,'//1I(1,goo1M UGV45 6 6 5'(G$4$45g6F6FG dGHg6F6F$GH  .gdG--.01Gkt 6 6s9B5L,/L,>L,,L5