From 8fe6da08d4c634bf6126ae8e7caa30f557137582 Mon Sep 17 00:00:00 2001 From: Skye Wanderman-Milne Date: Mon, 22 Jun 2020 17:33:23 -0700 Subject: [PATCH] Add instructions for how to use the TensorBoard profiler to the profiling docs. (#3481) --- docs/_static/tensorboard_profiler.png | Bin 0 -> 29822 bytes docs/profiling.md | 139 +++++++++++++++++++++++++- 2 files changed, 134 insertions(+), 5 deletions(-) create mode 100644 docs/_static/tensorboard_profiler.png diff --git a/docs/_static/tensorboard_profiler.png b/docs/_static/tensorboard_profiler.png new file mode 100644 index 0000000000000000000000000000000000000000..9b276b41d9b43cd90261b8f58736be626a386faa GIT binary patch literal 29822 zcmcG$by!zzw=KK~6$MEtNohnvkd{zM=@6trN<={NMr`53njd#nnl z)u_A1mua!y&|>{1Wg;OV;o~bw#Km}kkNLodwMUii;(v(9zkK;JCMM<))n)kC8yq!x zx=;#>4545x* zeSMpDm-|`8BAietrMUZt^?pQ(ii(Gahe=6E7{zzsBAAEzrPi%Ut1e1wTTZ549v{TjEmD(Q6WHjdV9;t%WvMi8TIvRjG+BxNSS$OT&lZ*jLcVl8*1dKmR6P9J}&b8`*-;DIoa8BZIRm>lXY`* zX2ir_f`V}2i8muf-1tNjKNAuV-1Rteu(7d$3k?hmgolS~6`Co;i;8e@y=n{~H!?C35fNe2%)Rfn zmz|tUZ(7pV*LUmIt=`^VLPA1LgKGD^mB@mEyQW&i#KbBpDp#*w)z#HKKReFz^_5mp z3HkPIso1iAXJ@CSrDbSnNYwp6Sxv33t?j;98<_*{$BR`kI~4;NLg8#33u6#9|yCo5zI<)n6EoJ z6z|@>yY$}9&Q4oLhnblQ)?NPkgq#QB# zEIxn!tm^D6;A12&A8)SnfX4i1jo++4W`j`rqe z8AZjkfCjDr2|9&nK6VBMOiqXe0*Lc__Z)0($wgduW`2jlBgx6ht&dieS`G94E9x=) z(K0$n7ZWn24qazkh_7D{WRwe+hp27&!p1;&6>)HI*yzhGD{~pEbSW+_77`LFC@6q+ zNlrH2oN9pQzPC*y?3DfPoy7C!3lQlLy_pf*E?YDHq@1E|dpLFZ+w+~m!ooy;67Z7s z^~Vkl4p`OF7M7Og=H^O@ida}!cFgr)8{(&&dv2f}xBP)O-`(B)>zB6-!wXBx{{DWH zlhb8P%$LI@?JX@I<(cyItF$5Vxh_8b)YiU{pj%vC?u_Ei+JC60t9xHWB$!4_QRG@( z{^oSk@zGJPepQq>ZGK^)w3HM#H}^9#Geg5yD+6~WB_%Ikz6=`(1LN&k-bEO{*&({c z#AG&9Wbt1aAS0OiV!I|@DGs~%j)!>vkp!s!Gi79?N?hE#t7po%=06+TEy4cr|My3T zkmi6d6iZzAGnyYn+?%_*x1)O@Qcq4!%sOJq92Wmx!X|lcX-RSOW?D+hD|`F%H?#!r zrba7J5X-x}y9k%p*~#i~X)v{jYsT9nio4ex5iD-^(P|Ij+qXZ8(`IC4jaRy;85qzY z@Et|Yo6jUA8zMLjI${J1%gc}2qxs3n$-BC`1_uZA^zg!+G+EzJl(%rplJKqrtks1&X@Q9RqtI5v6 z;lAt6LL`rQIJ=HdjXyfS^~jHbfu~GNOd=vxDJdzCd^w{%7@A z5O21&o2546sx1~JQi!3sIV{w6wpO7=uI}LIsK~v^Gc9E~KUxD`quO`T{5H?XoVMrM z{fL zLlX}<&46Tv2Io`}t~a~9ygWJifPoMe@9^MYK{Nc*r>~+OLNqi6Ue-4sep;_>Y%H;z zmLeh|x_sFuA%XTt^8m_2{C$^ahK4_DYrk9e=U!Yb0H276$E~CCqELChU$5HVC4Kjf z)b3BIo$(EhFjR*}RJDJaxMt9BLdkqaJ~10LV;~Tv&0IxR!%*ss z7L3WZ5Fc5bN7l`u@LkA%M6#&EI@=cNDi+p4=dBk;Mw6UaKDg-KTt#R&zGOc4ksBD- zCe#o}drNwFZ69lXAMuNSY+wCB`d3Ni1$t0 zMz3#Z!S?_9*4m$|ahK_9$LV~ZTT+%Bo(S_ChPoVY+CS#!FKr5@QHzjy{J5l~WUugj za8S@#zENH9B|ROT<;BI!#Kgpm3<1-Yus?s43H{y~4L}WRXlQ5(Am=e_`&wY!1ho@N z5P4mGObjK|sijm-t?x5`5+B`DfocM&Y-wTPosZAw-zEoJA2QR^&CgB_+uPec%*=kh zxddsrhjDgxR!U0hUhlPQ*Klxf(&Qtj*4EC>PTg?opaxH`u8zhBbkX28t}V_dCz0d& zz%v=5l-?3d+JrnR;>8_yKZ}h#`CYl&rP>D-Q8_a!D>)!(MWVc-BJ10?nb}!48m*ne z_b59%$X(rdn0f{VHfCn9tHR^s&!K*+r7M033xk4(5nq^;G*t*iU&L*1XHiX4Qxg^k z67E^O)8gOMwzjs)%1S^fTU%Q@I}TC19|Hnn05)-Q+QfVH!;g^Ny!rQhn4ga?SErPN zk8clZ1f*2R6qx8nQ}uo{qV6ml9MnVfX`;SjL`v~>zqlrBaaYzXxadyU?LL$=8b$xd z3l!CE$dey2b8S*0 zKilpU1E@mgV`^?bJTUO;)vNscd;wwM^pq5&mjFuqFk}l#Dk?NuAt6^Nkk}-5o)$gt zN|N;7C~SkwJ3l{B!Ut;)A{Z5Z@m6Y^meV^vOGd=-9$k(r5qobEsS0NiegZMKypHE3)K%QG! zv9q!k6c?wa)B&!Ll$_m~ZDnO;l@6kGIX&8)YmZJ>jE#zpc7rg68uacRCc5{lvlAC8 zDk_}1me$tmI5-)akEEm;+uH+$0Q~|CZVF{q3L?Av3}UQmIvoez7rYuQ%!lxs&Ybws zy=kFT#mlPAX{oAm10-vqxMoo4Oh85D0fiV4-BRumWOb!jp)db@NS=Osq6@DmLnV2m z6T%UG@%nhp6@;BV#}|)Am2KY8-xE{?sElbMkb+G_}oDv#s7v9Yl~fBv+#W~Ha6r=>Lx4hr({Y)>}@ z0}zKL?BQ%+XJ?P)F~{{Wa(8!!Ya~kr-oA6^WIo;t=lXTn_2+$2qA3uIHZVa$^@dEv|bTIWKBN4NryrLq2RW1q&3JeU4_RULv61C%| zoed3)uwX`Hk@6Rq4u?ig@R{r8)cYSl1Q^ZW`%a96>FEIs(mi&uwJitI9>Bhj((NS2DU*;JJ37;Y&qzga8n!@W6FFwx7%Bu6SO$qzp z=Qo*0(9{HLr`kA0^EEQk;_M^xr@Q;nc9K8~3lr1O$VWfF(Q-%Z^OL<*9YDk(^wLv2 z=FgthF5~qQbmx)pl@*+(&~%W+%1?PopCFb3GgkRLf@Fe}XLa7~w_0Z6V}Dn#dpuEB zzsJMFbMKzIib^6!Q|RZ=Q0ULz7!9PzhQfVYkCwkTdiu{EVn!xN72`!ct|5zyi;z1d z=m-$VMG&U|j+qpqBxE@NQs(Ba4HlZ;#j&)ouyt^Nb}E+J^s$4(y-4}Br6oICTUpro zTr0g!`W~Qc$Y;1rI&rc4w zPfpz37kg|~eX_H&nH6JLZr_gLGJ5mv+Y5L@y1JBSp++xWe5;5cdrN<-TeHo zg7#_9Zxt5uIL2J`v?m9fuzxr5;D!2=ut$k{xGE_HM?{dCmcjd_rly8h4o!bm=`A!g zZ;UIrcXUeRH8o2i?b=zNuGgGH##HTi=Jis5tRnsYl*`+7N#wKMBaY4#G9g_dID9jlBb$xyP18p75Ln?(x?pY`p z*RNmy2S6Gd8FlsdH&rA``QTr4Uwbhw7v0y%wE|x}cHJKv(Y?@C3=9ss?NfQv){yxa zLBIq4d_*PVCE`lJ#wHjM5^@^NJ3JK6Uy~PhqjSN>G9jxrX>&Yqc{RTSQD@MrLo zNLn+A-P+=QEUTzk5*t_?P!yJtp8#K@FSzT>|L9RhRh37i?=cjgYRhlgZ8#i?PVl*e z>+>qkvj3IscTPtX#SgM8mqTRgjT_H5Hv`l^QB=5RN;^98Iu=843~KAh%6|AQBUjeu zD1QZ%Exh=_I~T7q@)DiYx#b^eudL+bC>a3RB}K61+whP z4D}DcbLeex74xB#>(RH~ELKJnZb2S;CL<9$53fH@Pfb;|@tA~@lP#Lh__Dqt1EH}ouFdX#-S}3Q z&Vcu+&3Iqe4L*K;Umr~K`MF60p1Jar2KzP5ly>GisRK zW!U|Aq;GmGs{{>gfjZ=Ojf*(?CaCyS|}5(;&iVeIg%TRxW`qLi@9+ zduXV)MIEY_A}OWF!RC zqOy+^UI&|pefU(B(+*&4L#0EpkT2U{etn*ek(J~2orQ7}72`*BL%dlH(tgb{a;pQ? zIjHQY(4I1ZLA{7O-!gs#8|b%oc5ZKPIu(^-ytdTLvmPhOZx}YOlgbvT3x%tv(b?bl=&SymQ5k_3GVC6}s*D zb^w*U`qVC5&%T#vJbU=&79>VO6iCKYUK*g?b3r+MivYST4{X;)Ph)m+GU*D#~-eDdvSz4N! zdJ35n(=dMLzgOS#S6Ak>lI#QP^D__U!yeYqO)<*ctSmWs^@CSCiK3bhWVHs{N9o1H z^3vw6BS3cL8q}cKFrkU7ssh3YJ^C_w!2~X-$yu1Me~OPUH99S);BVQ~#4pUxfA?;! zH}XC?lKW+JVroje*wu0F@MWdvM35B;N%)wkr}oegFRx7i&aQz*RY1T;+|i2jmM^Np z6w7gO)QCX7*SLn)c$xh?b9&5h74yQ{XhoK0PQo4GwS$7AQX5W^8$Qw!OiT>&Q7Ik2 zetBjtY%TFxzJ0r=dz&S}k9V)RtMUZ>l4JwqikVqbN@348;#mMZrrB6 zzoT0k=k2|BlQmLazq-x1Yl<-{^qz%OtVpzpk;Qv2PgoHb7hoJOwLj5-X5lNnbdJ?a zvZ&joSF53X%l)`-nl@&F!ktc0Pc$?x8>D{|5*58gfCM$g;tW(%A?PJX z$A_*uTEj82GCjSqLSADd0Xs!`3Zll1@&FLC+vHE!e2mB!7faOVn-~axk99EdGP<;q ztWVoAlaZ0h2L@(%4LP86zBe?@FYVeH5ri=MlR(Ssv>%sH>3@l4Xjn5xH>iJbrP!js zLwtUa!<>+iSw2Pp71Rn{<)(^oi46crPp#7F>Uduk`;{&Gp z_^#-?dJpA$`i7QdUIOMI6@>{vxVQHMadBlSsmq*QJ%5KvJeuG&GB7Ye-YBmq=Pv9R z9vl|#jPu|S*&g>=kC9Wj^g3NB-ZfXJ#%y(noX4Ew>5u0P-=6g6>Zexz`T91L=fp4t zu+wz5*4DLa_eVyAQL7=Lp*j_PYEPf0?2ocdH!1z;`Rn>b&&t>`i^qiX|dAy8<94zFFmJ~u-d z65{x4sL`p!U(R*zu)Mr{;A`E%z}ngx#^mIS7fVr5+8TX*>r2bV&w`=VfI&u7RHRB? z*O22f0A)b|fxg_`p1*yf8|OSZqQFYaK9)s${VwI*hY!!|Do_NIMr@inLj*s%2L?>W z&oE$xbxu#K`2?}C{WBxA8qaBd-v-!d>6T}+acF8JI7pgnP~jx7_o1Rm)gHUG$9@rd z5!m)&*Vr_2`FMF@x7yopJ3|VYYainQ>XyqW$+zh*P>&(MXVTKr;NB_tU0jY2ZjzBn zNm2ccyk9nBcUQEmy80K?Ke$_9K_l{>mU)fq6Z+k=3@1WT(qrqE8JL)El9OX&UGubJ ze*=Yztqd-K=OF{Xx9=b zwr~x;`g0yWJ^-t%vqGjqtR$Z*n8`H!k|m!-UHqOFv!u-<%1C{%r|k> z4{TL#^*+v2*0_c=22x0C!7wZi+D+hgKglrI`zr-Y0sXTQbH7FMRi_Qs$kIl-1PKJoR(8R|Zd&(1BR#YHazQTy^T;j^55z48y<6E$PCp@IgjSuBD~5 zyR!?QrR!LgFkGz4+1-2nI%A=(ikXOzB+=da#fwU>b860g(%=`5Wgnx7&%V84*_U1J zxq1`gFDZ$TACF&IPc664ATbDm8 zHyWrQ5BqDZI>*xb#pTz~f%u@-F)=TVmsVdvaD$ZE$ABtovMRVZT&C{6Oecjc1CJ^hsz zL$I~&CEDLq>Kz_Vg1iELe{4k1W~{QK{kM_1IX`c5zuI|pe;yj*mBo_#?ORJ<{hd=czo6K7xffAmWBJki z^QXAW*ZHnG-*A$-hYugJ4hquIB{?u%R18@hBq1ZCo|6`zkWyl_S6oi6H7A;XW@dIH z4>FL!GOd)fxt(N#$E8b>yu6O^DJhvsHC*x~BBILk^au!v+%;e*ZXW0r!w_bJh^n3C zHY+b5AJDTv3t(emIXOCNX=?+kJ>5C5>lYgupuytsb7iQvRaya@SvPBoQu`mar9UQXygCO%+c4${#M%J|#gN`?=Un zI6+bWsRfv|m7;?ciWBu{bU?^{38J5QZDAs%)}t#63qQqpxVg}gI6(z=c7PU|y}i!0 zy*)i_w{JVGkNo~Ap4Y@^X*vAH|IqL1Ri?DGdw72r7UVsTOM>1?yJ z6iN|7OEc?R40mu4s7^UKcT!DL?6HLa4UN$e&=yQgDEBk^_b>&NmJ|>W6MGzP znOZA4>F7j9^pjLl^Yf!L%+m5dG>?=O#n$I)El0ZZ{T}-3-q7;oPIh+46?*IO@?vI| zr|@c(Wfq>l%-(qpgWMte6g(ZNXdh~5O zz0v0~qWt6uKp$e{;NU?0NmE2bgrK1CG;wye74W+??1q*`z&{f4T)teJ`sPh#r7mfN zU|~!QVp!*+5J^t1L1LDrk+-<8FjB>)EF3JKa2T}(HbwCac_)+0ccw|98rY@g>1L)$ji$J zI0*RI)h=q#~7>#BGtNa}tsUPR7wHnXuH;_eSW z_cvbObK)Brxqma1DK*txTSp5z@yMvCQ}Jv^$6|%Rz{f2ui1OjYuOGIz#j3~7%}uRh z@x6S?8Mc&*I>K0Oo6fFYzq+)z81m(de}K?E)R?<_cG726C+jf{FEI!}82UnuLSe2Y zoplw&KYX}P?H73cDsB{)?bP84<$XOEM%gkeS?ShzHf*KB`cycr`<v$*Xc+ZMvXvoBLHi%M0O3-y7Jz)ah)AKA@`WX=3Fp^SJ?FLA(sgl*7 zeL8F`}y-qsL*!-yggmV|;U>|EEIY$k@3` zZYH?Cv$MvocWL9*P_>(cs3td9@!=s&^xW-=vKb+_VZiJvtRYU`NX+sGCs0(NL8;L=eKvzWV;BghQO|E`=f8@wd z)hATrePe>o+vz*W_v9;c6IZ`brwcp2P@uB%6^4pghyQ{#qy~Ge9p3|{`XC@Vm=rF8 zLG>q@6qs?Iy)3GGhr6EU**7!$Bm#3cFX+<7jiaIFF#$Az8oex+v->L)S*Ks+r3!B>H2khfBSHLp{re|@`?HrfB%ntdD~FPbgRs+i(YicqsFYr4!+;6 z&2mR-T2$oCcp;C*zb!YBG{tBpIy#U~RXgR(E9YWB3c^3^A)~~>i_ld5q3inoZ_b{) zhzF{a@$k?^s!FJbTKhk^uko&@`Wk#zjX0GtG7OE4lw(Wl{l|BIeEfLry5sfW)K=Hd zoU$^(dkg}?{K2Y8sj1LOYrs0mi-#3hjc;=DX~|`0*IWz) z1EUT)I+#s|h1-RF9r*Yqi)FXe`S|9|g&gf-Ri*LQp`k>`{eGML!UFB0=Xd#0UT)7W zaEY;d`O(ndzjPc-il=*5-LCkiwuajWPM{-bqHzFdbIv?lK5ikjbPV{^Bx$*P4gh@)O)64NbVU2d#CEb{nTse{_>LX2gBy;oghlkF> zL&Io4i?cLKZLkNXRtt@iN!#B+Rm}*Z!9!q1buStCy#L_59{YX$@uR#PDy>jz%H(JT zTWTs_Nl6dU-0ZAkcP!n{rexrX_j6;zcskDVG#gHjo%s5aCEmgB59i!{elh!l+Vg^_ zi6#ImB7zFa)N?C~&6!POyNtQr_2Y=^3Ktd0>m02J=c(ohbkLY+K49WEuszE($1 zM?^%(q@X%KVOXQ?;IbQ4{Of@m2l0(R^*7ySp6h#hze$=Z-=)~v!VaPmgE>8SA;${V z$W;WC5yW`Zz)N)g+>o4~fTY$LFOrp=t*#M9Ix(#h7NPJVI6VNghret8el&Xk%15op z$S}Ddk&)yrEz^wwha>(def`|zs@Naz>VJ2ehvz*lhF1H8SLxIs5Bg*JN2ee-%XrjP zfx0p~KVQM2|4!*_Ww9GDm6=|1Ziah(d!3cpi!BgkczSx$8;8ym%7&#fFfwI)3(}s)~8X$mYY!u?d@o1o+jLp-ri`4 zfPi2fo)pZqnp;{fUs`1fl>z$Z7CsipxvtdJK}VLDo~J~$46sIq;cwQ`3yKQ?L$9iu zs>$q(SzC->F!7D0iuL4{mMaW|i@!sg(1xAz^SRFLcQT@vpE)`4SyBxDi_PKN6S`7d zQnIdb&>u2%hi<(wwJ@@j#CoF-R4tQG2dje!oWCByX+Z+0YHjV-MozE zw}3e;$RiJS+X(0dJSs~k=BG@!4-HMiva|0X5|ja|VG$yZTlHJSZ1ZiAJi~)eIO+|e0w8!dKpa3zJjR;{XiGyc2?r8Gwqt?evnoDS)#Pc#7?_KqVt8AJ z8u?~*qX4!2i4#GTPru!)W>@nVE}Qv(GG}J0tV~MnfQ4T9``G zg={?G~&wr2nK2LHqDjg9BerB!m;wvvWdZ1R?k|ovrP&GYjJ8w_#HXySaIyp`EZ15rrluPS*K;K{l#)MUYpo zcDe{kiUSqQq%I>M?FF`K%G~+fV4MlT8dy@&G8nwejVSMz&aByb zwOv$n4+DX5a`N<@%2fHs@?Qnjlq4j=RINUzrM*uj-W7{FM1GF_`9$0r?rJqQwLCvQ zpTr89D~g+AfuMg-t^M^I4ar#_?~ca@ywq8TiJ;a-;?jf}XLWDxTIIsD_~^0^q{wA^ zZglrE_E!x2r5`^okzQ%$6!ey5I zipYj;DJ~Ll{rVmimYRmf++5oJ!;OzH20U>yhxZZod2KEDM=%vwr&ascBnVI#>_N=z>2U7kX{qskhSQTY!OK_IbQh{2}umnX} z;Fr@KqD3I6>$-eF3GgK})`Rv=(z!ndlBYnCj3@biZPZr`|Ka#-YXeYoLh7V8I#D{v z>)_z^3KGDOTUQJbL{gH`6U`gigQeWd+S=OIqm{Rq&lmad0o&y5b|okClqF47zj>1s z6;%$ONlAZ$tcC&8U2G>H{xnf2N5|_YxkDgoIq<9YyzwOrG*c_E7Wh7*UvVxQ=??WvptQEh#9Vg93r>+?)x>HC$Y*el6SD z+nbx3UGx9Y7AqWVMgrXxQ@q0Wl6wX)S;S_uocZ-(xYT;}>NU`j;@=^I;~1_z(!=ojwQ`R-YvooV#(fsn$bOc0qUH?jS>x0-z?8z$(1&<>Zh zQntL~;IM~*OxcgNU-1qr!K9q}D?{WNNl8aQL^QvR@08FR9RWHOEn)6_R)Q|Gq=boH zBnNbw@(<~D<1<%|92MAe&Ny$t77N~79P8IoRD=$2a?*(P_wV+0h002Io?SERu_+#> zmGos#iYh*v&7qm9!Peh?B`P5yA-~%z7S`u0dg?%kY>^bDJ$Hx=gAD{f{?kmc?c1u@ z>-_w|jVZDgi;;aAB(t-oX=!E~r+^otZ$g9m1uX&SD}cgBJjBe*=6#3Qp4FoI(R`i< zf6_8jzr;pMAC7jl3<=#jlaD9T9v*myd1Y{5V2|3=(%dU5)YFnjV&v9C%TivVdPS=k9m0X(!$b`=N|X9Yq*JVg^3|u z(8j}j>C`|2f}AeoV7&%)cDnD68#Lb*>)Uw>uvliDi6ROoZvSgDEHp!hC-ri7-|~4<)q2D;nAP3w@}<7-9w271ZFxh# z{GQiX4A7(JYr!AtsYPcN4_8GGL5&Y&q~}4)l{Yvy2>o_|sb^&!8q4t#Igz9rswDCi z`h=#@y}S2f+vL{P2gHUY>Uw361>m#C?r zt`omt^?v_;c4=2>bQ7nH7-ndzf0qGYE>T2X;LA~wbxtmi#(>Xknu`t|zL2ZTA61C` z)83v{8nD)WZ;W#j*3s#_QNrMQFbg6?{ujn_%vIdX8pmK)9GID$0GYj z^Z(%_z(CYBHG>HWz87&NuT{jzGet*72573BZA?BgeuP2FEbJJOSY>Kz2C@+0f!(fs zid?XL_=m>&hn~W=zn2WC4I1Jv#IrU0NAI!cBBRK<^3?S88ykLmu8)nb#|mCJ8jvwn z&>gOg5vi*`b%y?!`x#78HS=@@T~`i15YQj-B=eYe0KuzQ`4!j~OQBv+nVGo?DHv&M zFYQYQUGM`nwdq-T*d!d>e0-bRyDn!Zg2+gvi?R6xt__XB^+4IvNI4jr01PfI#pB28 z%nU6`59lc>D(2L?eXuc!k(vU+Do*~=_|aH;xLPyk-XW^xt%?c+JX<&?cne_6>ijuv z?&vzjv0y~lOd%%v1EukweSK>)egZ-^2_VX24*d;4BJA@?r7ksfoNQE7PcO9B3+-Py z=op~a3dq|hX{jQHjv<=Q`V%!ftLS4Bou3*@m#%JXDjU$84ZR-(&bG}iA6mj2G>lV| zb3+3IA0qnu`r2Zr=|CL8@YCER!)v$Yi)XGdEZUa&@rdfj44m-Zy<)Nn!RQ(+8)X~@%B!1vt(X-f{ zZYDU;+S*DZW@`t`F4&j0=RI8?OsnFRhB8l8R`St&6xG$X16j;;MPEtUTwqAzS?jlm z5S6Y(w|#>$)feW*qut#Yh;^=<{u52Gufzr^O&HGSmllGA^e8v-Vm-Z5IO7N-XJ^8` z;>SlsP!N`DK75!IU+8?avlvJ*dzGRS{WYZ0uXl~7rh*n1m%VbTiv!!u@)V${g*O0y z3JwltT99v=D^45V#Tll?H#D=hU3i6q05N{;>eaFe_A=Mmj|&+pkZp}734jzt{}~YJ zI_^fK?ELg8N>kIU&&9}y08#D-?R@kr4qnIb(C%H!e)JNn()=Ry*PGv_@hAsits9$~ z_Vu=CVp1MfZvUJirI=)Ij-CP~5E67#|r) zN=}~sn~IM7aK0PwZwx1-&JS{{w6ar)=R+W{`&ASaL5M-JODgK^_@ga2XWh;EoyFpY?-j^-7FSt-_b ze*;wX+C{I<$Shb<{fx1(vE#h<7db0<=WuEH*M|?aauFSADKHNw(QU)S9Ws9TjN3Np z=INrG++zsJJsPag2+f2WM@P<04O9kYFLwLWzL%6-#Tfw)P-Lvft;Ltuc3JfGMUjlW zy!0^llY@e?Ouyjx_~2j~8b%I|px_rTPu%Ao)8d1|aMp78xMZxA>UVoiQc`(ClW+Ma ze@g~rV{2<|h^ojEr3`b;t5-KKsISln;1!5Yd;fXI&hZ_rY-{7=Boj4;lRycmAp@mi z0TB_MqwTxVRIUy2g6u2|Mb9@sXLhHloV7;^@53+)yom41d$ErVTMw#$1t zW39bik-93j1`i&nC@AQ`r;3V2Q;8K+_~K&TYj(!=_V$Rltd`4hkT~d8L4?|6fVx;? zo1d5nq!PLCJA}lf8DSJ$ZNJIp`@ zE!hbRy8&394x&juL;D={;X?*Y6D6Dn2sQbaxG(=tftEO9*8p1O>|$9;xKy#2dJRal!p$Kn#3UB!q;}8vL#wkd7$%AE6-$ zF%{#jTa#sG#li7486TeO>s!MFp}4qxfA)gjJtVq|_8Nu|$-a1;CiAMOCSz+OLAR3D zUmvh|c(B21g*3OcHMg8hW#ZsE1pxpa9vbRuQx{iOHW;VnWfWu)c<1k*)p-dl54tr@ z?Tu-{W9xw5I$LVXQ8qHFs!0h63DFh4rNNwI36MBvCnx;^K0LQMH(Z|(aLQgumd4x6 zL`SV1{TZXq?~S*w$AS?U*nhOOAM%)!lShtJ`k#4L%ZZFce*MbyEz>lvhBhw&f!z?l zxx2ZyziVY-5gr0a7H375u2z@$xxc(2#{i%lFfA{b$M3Bv_*S5AEw+q3W2QD3)V&|7OX|bpn2s1%$ zDn_x&Rj_tUH;YUx*kf|OE;WodKB~X^^H2}_?6%khrecTw|6A7o+r?2AY`V$on3gxW z)ifb1b?T0vXh!GBo=JBPf37=?x!x&YuXwljj3X>}wbMrH>SKio!UR_7e}fY0xB|G} z{5P23HuDQhh^M8cZDPs-CSrFu%!!c0u&} zT*E-X3iWS6fqX!6=Uxo7QYxqKk@%~ZJH>QkT5iyL#G;givkS1xY&a^L(o zE1r~O!T0aEPpgZnsz$ATPXP>A=6|^|udDjGlMoA2!~;V|rvl1qM&6WaQBhHSWBncN z%KDL!H{~O?eRr76=b>2}8X7`=4qAl!MlZJzw?YC>WvCD5)cF{JA#P>WJ|f7W`g!Uh zx;wC+0Fe%VeUZ8-2cG{H4Ul0x@><{b`|o&QNpBwdS%@PIjlL>r!A_^em6b|e-FIFJ zG9N&q{%Zg1?TLSc^$3eWk)f#m+Fbd}6GulAlb*!`Ok5{iH{ocI60L(!`eIG>4T|0a z-3PoDC?S@omzi(f>gv?G51N?9z^y?J4kRHbM+6*dKYkcjWj)b+g7)*d#o}0UW6+mN zuc3uGVZ-CJF;Tw<_QaS<_vZ`MBO@`u9-#e%`b=rw{yo2l``*2eej?wWZ$4{TWV*nM zIG;JS!9X1aQ!qsQB$7vht{}cp!2ak49$s*Gng+0zz%qX+R?sjr1CXk~d-C~Z{Q3zP zB~s3WT()k26eSi-F^>DaIPCj@eH%+y9p; z>zG*O_@7#Ue^YVq@iL~SAPGyetz$ecn{>a%(d~*pl8d;iChesXAs~1k!67Lj)e&FwvtvwD;CW}PaA}R8hlQ*x z4A9Roj7Y!Rvz`O>Fg+F2-4i3~aamaSc(72m%KUR!SaV||pGTwo$ucl0EXfdFpfG`4 ziuO7p0@c5kAgvS^{$*UZ?1kgQhxbpV?_ZoBFf;G*;(w})#aUVYXFGt!u&-TZ*RC`H zXGWNp{alB~&W4ee7P(_P%viIZmMsm4NrPNDqZR&`3Zk*MN6`q{G3cvm+mVa;VZQDO zGeN%cc}Q7V1-=;>7#X2NlvOpxs_B(gRkuLif$<6BpWqT=LO|1@{oN!aBV!>hK}k~= z77ytAm{_yb0qT7Fe(K8nqM|UfXAy$-;K!yk7re{ux%ntLHnyriEztHKLzGH{&DfM4 zjHhAm%#nnHTM~H$k1f|^EZ?e<W8^-!axbeQYgyN|u{4XW8TIi$jott1k-@Lv^*{#e;T1m# zSLkEc9YV!iBLJ;K-FkoFIEKIO;8gGnGBfOLl3o30<9#ve|K}8nnyfOC@!W1ac}n+Z zjXIZ2G{zeYq^dgY{#U#2#S;B%N{gB3dliv-m4Drl3~P&RXWr6C$9f&b-zxZ2vx zOju~>+g+h&M+ck9@835-o3@eH6hsv*5`@gxJJc`1=UB#;oUM7}g7U2+3s($-+8wLPTTW5F%Rj%OLk|g>A9-`Hi z6)>WLx!o4b=`LU@kLF*$m{6(%LF@v)a!|epQ3~2k{Pc#Y$e3`k=yaLqsVi7rgMt*} zLj(i`!5-(A7UL?~1BZ&Ra5jL#P#eYps;a7Bkj>5I7#jhTYXLx$A3wgHIdP2yeFZ0^ zbIe;KF7B%srEWI^u)O>!lX7ea=C;?{W|Zb~g2U~W`*K1v4vvmgrBs~g;PeuZ+d3P8 z)oN$48$87qXOVy!a`Wiu2nJ1npp@c88@N`wlBHoR3?-?tu@Q{?&@TcH59cd@$4}75 z2*f-Q9C~#47fyRzTwFMg0uH)B3;~`~QsPtvVZ-9LOhfzG(^CYPv#^i|fwNHH!um(Uzg`JHTe24b&#}+1W(J&*e``E`tfZv0 zM1N_CVi_#tBO|)tRI7Xi9X(tW?m!r|2G;gU+v!H|cEQmjTie?({wC$nm7xp4r4llM zBo!O$v9r(xXO-lC|GqX*@CZ7oe-2`Z5%bcALqzKGK^sddww?U$xrGHx$K!2}*#U1><)6Vpa-=bcsYFi0*dD+@Ag0xV4ToN6b^6z~Rt zG%O{h2y}()IwfGzp4|Oy!kx+lCRdxabcy4WLwkrEXrC%2Kx3(>h>H{z7e_}$L3H`$ zNlHsggMR=HgYhwfoDWYY?pVJA<5{q0!%%VwMFOt-Y~3;Fh`o1HA>6M z$N)wP>-cA5REKYSQ~l5JP`$pORpG>5U6bIVO2Jv_aSTVaOmCK4K>$Q#qBxu@BuC@m z=n@z?ft3yx3QlQ}gQGUY_LrzWMMVw4p)(nBpwa_BXkceLFe=}_Z^52{SURv02E!rj z2XL^59YNXvXZ<~CqI(3YL2p6c-(0^VNwXPN;2bTT335tpJ-yY52~cI#1DOk+$5>$$ z$ZZ!E7l}zp;p71qpmRWw2!eEH53uz=_B^`xf8X=#;i@ob_~&}cpa3tY9QT`s7T`1>}5DkW1(0;;!39uOOsy;|e!6T@usqOCWW(G4s2YBY<06P9|j}gU%sg1MMOqw!D&Kp6%Ym+ zTUeAC)m;W48x?g^ShxaCOTwwcr4)cv6IdDq#VajMN?SYXUvK>XoC;U7Y)0X512B1d zt&TcARkt8THh6w5` z?ho{S^n)G7s;Y$hYX47X-x<_Y6mEM!5d@Tuh#-RW-g}XV^xiv4lU{|;0w}0-q?bSt zkX{9(cSJf!ml9BVliusw-gkezJMYbXGj}FGGGr3=IXU_E`qsDBIz668XEk5!=c~c; zfgu?rC5U_NKH=srM=J?8Yd(7x2)^->5{_s=grp+k2=C`#pM`IH6!_q3dP3-ee=%fX zMxgoe@OzOpZweT&%T+kj1cdk=0v|sc3k$l|HQV0XQ_;zZGg5ZZgs}x3bJ^R4`bqNi zF8U;Vn&hEBiA9K1?6w5`XBn<&v<_5@{(>-1p|r&i{tpQfxr;PN~3kw&h!KIWsaSr=n{|jy)1nH|$z&sz01W0X@RX z`T6D1WO`Xobr3G@zMM!zd@LcKAms>N!*iTW8iUabrJ1<6&Gk2fz?6J6!VvyS#`0W~uYZa}gYSWZ z4$RMQyUF(t5gVAp`o4nWc+B=0snw{O2)S?0W-g}YKGb)tbot#)%URUuV-x)#ZfIh1 zggPiqSWQS&&Zjh;p@gx)`H!d1y>wF*n)R!=w;ewnO*P0edYSqC$>UgZSjDV5`85o_ zK32Socx_mRG_kV63hxA32M~GWx5G4MU%IsP?@*OQ^n`PU+g^=vFJ|t!&S-jWZETKB zgK)0gJfSDd!nNQF%qOcz-LK1Ya%WveHgMhjKA3JX&*4PV%vj1SLffBUYEt8=T>G{| z9ZIhEubx&{foc<6t((Q(+TF4rtfKeOy~_ zargkVMBuN7R~ITDA1Wz@8KolAMch*>d!~CwJi_>D7}8bsfZRzr#F>pWsI8n|i0|j* zVt+7*8?tc{?C))F1beJEpCGcyqc8ClqpT*eIj$S87NeL%z;)wOt_pzTQ5j`fjm2TH zFnJfkbX9DK_El!x!DVD5a^?QhxE>G*2Ow)&9*ER{!`%~-5+)``MSPZA8X9U57iw&H z?oc_+tJPILY-q{yOSJqSarR`Ad zfLu7$lL4flK`OE&)R^Nb1JCS8)>Di3KPR;)J{!m0)HQ#&55c47`B6551{sxnY)-Oo z7K)Sm#0GCY`>{WUmcF~L1n*Qe${>TYT6o(<^f_C-XMjGDWz-F@*;UT$h%{8NhO_y* z>&G^~(K{b8VVa<}to#+@8E)5S5B)Zr&cRc7Z8*?2JMe?Yix=V_6Cmv2Q2LfqbkwCs zW7k{g>wy0g{vHSV>gwwGR90W#aIOhHKY(8BpE`ibZ4VL;rXEcHZDHP;oWIe(;4Et$ z=;#64f2FVq9%DkF{x4Dm_y9>;b#1T}nAAMrU}dpgefL&6=+2Is6a+PB zo)jB5rB=GBEEVo8GO?SGJ6Ssoik%^BeerN{RntWg8;4V%h<&(&jqwav$u{GAK$L;) z#i)?u-1% zv264^TUhEL8Ga;qKA1ie^A(nFL?Mcgh)?gcMb@{%<+|h2 z%Dp}^Fb#I7AaGHhnPhsreqQjdNSc>Art`ZL*@3(vvwF_s z=uPze>%=9pmZqED?>!511yZ1&8-A2Ks0+Cqv?fFj#-x=+58*+9@^rB*OZC0e@pWHG z9@Kr4kAC(9M+sLiN%!cu_Omm|Lx_|t;WpO$Zu*L-Qcft}KL1wWyR0mr1!%Ymdj0@- z_b>y0|Ng|6$0yY3Fq`4u3c{VvDssA_snZ2!9zg(!aQhspX^xDDFW^7q<>zMh((0Fu zd(|N*BgbJq{?uuXbqDKKYg^Dm&_^~LOiGwYNI+ocruxerd%~HoT^Lr#3NaTk z_w@)2(3#s=SKE15%Edk)CZ9u9#JL-^Q_-}L5%r}guYmj-Fo?Vr^iJf+NL6*a{k;QY zTmfjW=;&a}d2Y>S_BW?ZC{-zMR-iKik|Pym<%kbd5^moI(A58uJIu`mjSLO;25B<4 zt@wwxPIs_v=rDZ*Ra&}fOi6W#KxeH7+iL!Wfxuprz>Gh=CD;5|8`+a8B z)i5K3mB2Dy#>y=S*{Qle_dVbFayP3yW)6a!-OYMz*+SUlM`UgfI*ZwETcdS96%HjR z%pr~S)PztkkC!LxeYpw=#upy^zL}m3^~ZM{RGX3pE#$0~hQ`P^Kp~U?)GZ1xYQ%X4ZhYn@;k7dAOD#%myw!01r~-KE-YJCb05W+#D~Yi%Tuf z&+899YG)@`($+rQsw>XV_h``4vZ!$aJrBWM6GGzp&z+`VsQXyLhnbUg)-R_pjP*6} zIzQ9bry$($2nyi6zGz2myCO%P=|~uL)UIndkQ*639~|kZE{WO~4Z9_WABH3vu>Km! zm>hFn0*L_JmMcN<(Et_M|B;lG}8-4X^qpg52-BZ8Q=t>8;g zbQBr`(P0bKoxlX_6gIWCvYlB{g7yQp+w8vh9!<@Wq%vb_ej9F(1PJq^ie`HvmQ`Or z?Y;lywOR&B*kc+n^bMvk*ust+0kX#7#JyLSC~!BShv4Vr{8zq&wwJSwu1`6gjATyT z@RW*H3NQmh^7AwQd}ioN6fk39XCH}w9*u6N$keq>*#r-a)@~6C*};dmAX$?B2*zZL zz^3Tzrs$_NIWwslB6y4B{OOR5kUjtFwmEol(~kb2o0aSPkcfOZ?Abw6Ifb&jo;_V? z_mswgN~T;pMk|Pm?CZTTPFiSzzk)Z(9%kG6oFsrq#J?-NOloPr*|{kqNV=kDrUkAt zj-A+MucBPq>AwC zb7PRP*Svrdb`YGgFI?T!D;s)kVx!>;6w)BLF%Ib)eQ*e{6Nr%ddxxG#0s3htYN8~0 zFs5+NW4xk*mp8@1(W24`K?*J1+-w)e2p8>l8W7Uss}o ziOyxyjRV#c?W)mS*%zWjhuJowxuj%dzyfJ#4r@d<$mN|ohfQ@44N3iMed((SI%y64 zU*_AnU|gN!pr)&#vEIE-F9!c&+~l?l@W8uw?*PoClp1vPbsM3gF>|^FX4ydH9KU1< zqFabVtKd7R(Ud&sb9Hs1Y_nH&KxQx3!QM?RdVr*h!;gvwBGmY-BsHi;1O^(!( znAIx{1AgzABE9{Cnj7yA`NeGkDS>s}j&Ll-zO!_&p|+;w#Y zQZK;HaHL{keWVgi=CN%TzB$7n^}hyf!VbEr!4rM18iftJiVvw`UgeIlhv_4kMxKt5 zq(-qIt}bueR4w-^Nr4{iE(1F820N`L{}U4|gLXBGly8X{a*ej@l4l3bfMF{ZTwk0Hgfci-UeH2jziW z5~ww%dDlM(>Jd)yKJU{C0fB&n=|ez(4YWyndngrpVl``&({4X9iTS#h0pG6v zrek1GSD0HmuF_Fbf+t1(O@CdYt809qY5bTJ*Q{t$>>WH7FvBKMmpPAp?&8n}pYxd} z#46?hb%*C93NR<%zt;yuA-7dxYryOILLN&QOrq(W^z`2)IK0DZpY9yv91$|crDSBx z+XpQ8^OL`vZA(OOKe`VCr{!LoVklV8-n#^1+F-(k9Jx|r=`|#yw_&WNJO)&Lr`*1B z3OQYm#-j0MV>!I#yO&$WcRKushXK6#yOLC4yh-z<2zWl`USFc=6pm(P7(QZS1`HDB z$^~4BIhgP~NAY41Fx} z@0gmJo+mXptEraejW)171RpTGgl;P08<~?loE-G@^A`Bn`7$4A)Gy*!%tU-~|BX=& zmDH<+)4L=MY?R>JF}fZU_da=6Ut6KLegRMIC}EQ#_4hj;vqrT9hi$}Q(WcS8;TCRn zJEnTkAw3|*yRoUpZlqjRTwGLKMR28KyZgx^oP1(ZTWDlGv9i%aHp0*UY~*N3#-Enf zU%D-{rNq%=N2l`z{P?e9et`xzw_l8V&&%1{_mql~^xRO^ucxqX|Gl=V-&bF(4|BtC z4c#lEQ<`rXGJdE^7x(b!Xi=}s+vN}kIiM*hDAMeedD50W-r1SL*3ol$8S&x6S0c7E ztoMd*b|QzjUK~xx1CIcz7n);uPPZo1+TR<8xHBs~B zz~-c=XnjDj$?J<7?}vbip#J8B1}@l;kPtvZaYXA&=V)TIcA!-NDXXt<1HvwJ5)y!& znFm<|5;*)ie?JLUXHtJ5!+3q%Rf#uMqY1xm6yxAJnae0>Fcx&A)hg)ZGQ&|&N?Kk1 z#vBK^2t74X*08lad>32eY-Z9B8z7};Xf!h+^|`rT^wo&h_^aGJy^>uq|1efMHvyjO zl|fRTPF==B!o)-#&l6@YW(w}}yayslh7KhhY>yhu$wHM?llhzLYlZEvXmsZ!DHC;v zxJ=mlhG@j6QaWF{kcd&&jr@>$Hg2hJy4}=I_GfUIhoc@3!E-VXFH0C=2{5Yk>3$MK zZpKEJ2xoO#+R(|Cy*fNbT<3BaWlxl#t61&;$K4Ug1zA)@UapUJrKR-&-RJ+=00Kb~ z1_Z)I-(g}lHm%arGx8bq7ofHQBL{s-teynzEy%4ZMf+Z^V$(A*4eX}=K(7PM0-*9i zsHUXH`Bvp?wdX-W_2Qi17i17?w*j@vWry~%K@d$suT4HSWFZuNA$MO&)Z6?geIA!@ zLZd%No)jTOKR%}1WKG)QS0Fxy1D(DPmRVZDeJnv5laLb@0tJrzrhUO>`bL~akUyP~ zg|+M%T^vRrD|^39K9~1_p(+T6ej^KHnUr%`dV75j6rX=^^VGZCUrs`c2{F&V*JZW( zmBj$mE@za$B;&O=qT2bb29AS+PfAEgA#tAPeSKgd==72|+BLT)Z>)VnU`f;JNSWT$d!3+Vi#sD&gY zd2IVBx<3E@t%9sUiMXHtIND4LgtfQNXnAfBP}x{p3wCvJiHU9Y1z7|WZdjdLP_YZJ zZ$S{r-+%WA0GHAE(q)}4TTXo9L>E&ukZ6x~J#G0QZ*R9ORZ9RJjgK#%dGFskECObW zq9TkZH*JW}`;BU{xx0E~WEiboOlrdRIJesaj`IsTrw-{iXES^?@7^{)U?cb<@D2B| z4)l|WZ&d!7jRR_EYLBTYvdnzFDsf;<`U_}M=$?+_lvw|S6U50LN`djXLDM$~g ztK8}B?&ePEKR>70o2Lm@Z-K3>>QP&>GPArOaa=C;f2wz~|L*e2-#zO6mEDFwhr^Gd zPA=5W!IA30wbfx(4nGf1*Dl$`>iHM%Ay&V`pCdDQX!Jb~kA^xv2FJnTuy^ltA4M3} zu37gUA9^>}eY{T@fqYVwcQSch^*|4(@7WDApLnl5kT;*}yE9~vfH6``mW*n4~PD=_Tgh>p55*p-60zK;( zQS2DOg_iED`F>|`tFB1Kw}40Mip8>!uIArdGNe_wx*z0;K9${wpy)wNZ@guJ@Liom z!#+*Ytidm}4^-x`k+~$OI=Iy9SJTJTvnB6{i7{5Wpup-#YDm7YKy5yzx8Wi>_eik- z43-x*QeA&TkaF4*|IgOXm9F98H|ZJspgo43Fn~y2-inIj}v0TJ!Jib@Ri&ZXAi!t`tiI3|vX(-W$J0)4UhaA_nQn#Vo;1Ir^EURzkd zC@Cr{T#(Gc!HL*j0{Y^`=|)A%;f0)!_a9qUPMHXQz8wgxhl2z-OYAfve)0fcf^dj~ zG4~Dr8*l%e`KE*2v|dLr*qd&Wiv|OgckdQ}<7svlnX4t~w?>E@5EQCegM9!sM9lij z_A=+Ay84H8^$0}22Hpb%f^baQQCE?gI-P}+^XTXv_|=+(_#XVYCzT-rVvv%!OhiAS z9%m$fzf$&^J>6@Q_5-<-+jGbtjvy=&2o}D55B91NoC5+pPz}WC%uyLT#%44?L$x}D zy16!50%0=)9OasH;3Tj#Zj1jb7@#bacHYCs1*wFchqhV?I@+jnmPZ(3zt*oZByxH+5THp}jk))^pg0{ZLa!+`Z z92Qcfx0=Te2CpH}4Q>n{Wf&Xk>rPfm0glToT_0Ce$kB|8B`&zU1QFuETIAzC3GkKr z+AoJ?0OYFg%x(%dbRB*)W0 z&N*kS+K8b`N#H#8I&Kr+M6wMB$uXf1(lPL?2U<4)qdyUP+v04ewY~1`dfVH+?*W7y z6Oxx%$fTcd9?S|*NERDcRI7Y+%SjxCw#$)82N*NcR8z;ey2HM1Bm6Vp0IO8D2rw5zEM?n7q{e$ z_(AJRsdyc)snh|_a!lyv`(dG|XsEpM@yWtA&+*~%DEx*oF64gDA!b*!gV;A3JYB_L zA2mnsqb~vn$@d}ylCK)KoG4S;A2@DG@cy*urk}xPDoa(PLHZuX>Mc|>t5&*CI1Hu< zbxe4D+PDhX*X>N3ek_3)Sm}Qr+DzA8W%5u+l3dUKRQs#INKkZimEYCgBlwNPPE3z^ z-H^%(`)IyjgSoXVxF3m!e{X`6nU^w^;4>phZaVa+ZE{K7AX;u`vMVDz%EgwbF51*p z%$$#FldX@-!?n0L8_8*G&!;ZN%&@pp&+RJxb|8vT+0 zFKA{`%C&_%ve#g4TMF$Qp>o`jS6RL7WXXHwUlA%6_mq}BnSZFLFiO%-;ShaK$xy~A zCN!yJ${Bm8I7N`>!~>O7Qji@ZJ-1rnYHu!*;I>Wih=gN&o*-R89Ij((ObR`2VgoEh z8Cwen=t&nR9Na0ypP3mn*L)~3>rf0;-yzca#He)|LZ1~_i$jxu38i^sD9_66WVEQF z9X^*=)qc`u)1o+j%{5O(FrkMTD1EM2Hb;f~?w&~a_9#g83!Lo_O2T6n7 z^duZQ5cjaAgdbm$77NWPg1NNVGg9$RAf?aMOcrr*JeAir8w{bQES<_o_l8BV*jN(_8Ztf3p26K<>6!CC zBZpWw*PJC2*4TO`CxqfB$1t9qHj@~fWE6|JZL22_p?;fvkqQ1KF8Aea+oi{3CO8oo zJ?r=Ib4#;|v%j`wa%WuOS*uY9%0Jhw_g<39Xvxt$l}ywFtA^jCbcmemSEpw>on!7y z-PcXK65vg724c>@fVlu}ZJj0lQJr|LghXM|6?f+Tu2E}>Ld# z_OTL>80*Rer2`|ZQk?xN@e50D=v`cO=<}E2A8g#q{ixPrHZ?aV2{RHw+rQP7qLdYb zQO|vSl>O%#!(}s}z;S3`97S?Te5_)MR5#UXaQb2SiqfN~ros0OZzN13hM0y&eo{th z+}2V2y|Q)X6pbQVYS&))e3!(DN5pVzq3F53#Vu#-B7z*UzL6+7)%4#P$%CQd<3Hk~ zmF*sTCs*5b%lPI=5o_l#1Q)0{|Dh~otuIzf75d?6Fuk{ht#%3k;Imsn3XCyXQC38E zgU@AUi=(^#AnPD*8I$$A%=4%iBLnv8sCN-HLE~Sgb~5Tt)D~uKXTg1gh|$-p6{V}Z{zX$l-S3gwgY+#5F&ZBVTr)b)QYtp4NA9oX!0CD%~BWi8aY zOWm;(&^Ygz=RDW>aY#7M_f3kK-!yo--R#39u&5XQVpGJ-vrot$;RJScv+Ikqq)ook zd6PuX%RZ`R|LA$P%QV3HoMV9+SxJO}rr5o_>cVI21Dn-VZV1!A6El`RID6sSvFGrL zQYTqfWf#fXs%8*yc-bj8L`nHji~YlS^f#lbi?xe|k+5k|GM2sIy3y${Ed#k4Q{r@T z(Qx@E%JWqRBdI@q%jU`?+*E^Q$w9O9~Yy$>!Y|m_@oH zJI#&O>>AJY8*etu=;<PISJwJ4OAV)M(vSFuMPIXKUcyB`KGY)}1)3tQ?=S zoz;hUtDxA6{k!>`m!snTe@bI4VsRo3pF7)Fa@QduTf9y@k`oK_pG3kK33Rp2^>sB&O~iy=(LiHklgLe* z##8k%aX0Cn783cdH%0<4uxy<)dSKeU$SmTSx1yHB!BN1D-!wVT!7i0do%+ z(;8A^5kUjilYA5sg7Wi^|Kmz=TJr3y-jv6OsP|s3XJJ->(jhOC6_M|9 z&~3EpMXICy_16|CMqpQhz{U-Re_sl%(J}677J4xuxAVT;O_PvhJG~8g+=~PszQ3U2 z?!Lj=dq=IjW@6IENZalDIu7-?$QcJI=sVB^>x5EqlKxJ)Kl$!fcihvI zAwDg}oM055Ks?W-)fyKf-FnglaoKJ@>@&DO82FH<(%58zL6gOgAD6*)j$Z>KH9My5 zSGPVn@~^d(1Hd=i%29pi^yJ7AW?Ef+!?wtS$v*h3FjPJ@<7jnW5FSJ8K<+E!6%m@d zLGBE%$+z^E+pzYbr~M^~GC6k! zNy$D(;Y6SLD#`5JbGmu*X6Sdzu(Fxa%?F&L)+8x>{b1Re{*!oS@2kRLVs z_GE3HGV7h_l-pL)W(u5(XI^|ol41Q}3_d%>&8)RV-KZT^Y$2zu{)`mHtncBkyKiWF ze&9Xas#xHfjN|x0NG5f8qsP;7*ylmG6GZe~n?2U&YW`h@f9rqdp->q=fV#t2dDZ8H%vpPo-Sk%!Y_ zS6-WlsrBPrii8g-{@xd(;6#J`QMubVd7aBroKE?8nmnYp6pK_qzf3#E_&;9C|M-pn zc?>gxe!s$0zX|ib-uUGxAGl%#15DasySrrVu35khE uVywXprzFzv!l) +can be used to profile JAX programs. Tensorboard is a great way to acquire and +visualize performance traces and profiles of your program, including activity on +GPU and TPU. The end result looks something like this: -Nvidia's `nvprof` tool can be used to trace and profile JAX code on GPU. For -details, see the `nvprof` documentation. +![TensorBoard profiler example](_static/tensorboard_profiler.png) + +### Installation + +```shell +# Requires TensorFlow and TensorBoard version >= 2.2 +pip install --upgrade tensorflow tensorboard_plugin_profile +``` + +### Usage + +The following are instructions for capturing a manually-triggered N-second trace +from a running program. + +1. Start a TensorBoard server: + + ```shell + tensorboard --logdir /tmp/tensorboard/ + ``` + + You should be able to load TensorBoard at . You can + specify a different port with the `--port` flag. See {ref}`remote_profiling` + below if running JAX on a remote server.

+ +1. In the Python program or process you'd like to profile, add the following + somewhere near the beginning: + + ```python + import jax.profiler + jax.profiler.start_server(9999) + ``` + + This starts the profiler server that TensorBoard connects to. The profiler + server must be running before you move on to the next step. + + If you'd like to profile a snippet of a long-running program (e.g. a long + training loop), you can put this at the beginning of the program and start + your program as usual. If you'd like to profile a short program (e.g. a + microbenchmark), one option is to start the profiler server in an IPython + shell, and run the short program with `%run` after starting the capture in + the next step. Another option is to start the profiler server at the + beginning of the program and use `time.sleep()` to give you enough time to + start the capture.

+ +1. Open , and click the "CAPTURE PROFILE" button + in the upper left. Enter "localhost:9999" as the profile service URL (this is + the address of the profiler server you started in the previous step). Enter + the number of milliseconds you'd like to profile for, and click "CAPTURE".

+ +1. If the code you'd like to profile isn't already running (e.g. if you started + the profiler server in a Python shell), run it while the capture is + running.

+ +1. After the capture finishes, TensorBoard should automatically refresh. (Not + all of the TensorBoard profiling features are hooked up with JAX, so it may + initially look like nothing was captured.) On the left under "Tools", select + "trace_viewer". + + You should now see a timeline of the execution. You can use the WASD keys to + navigate the trace, and click or drag to select events to see more details at + the bottom. See [these TensorFlow + docs](https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras#use_the_tensorflow_profiler_to_profile_model_training_performance) + for more details on using the trace viewer.

+ +1. By default, the events in the trace viewer are mostly low-level internal JAX + functions. You can add your own events and functions by using + {func}`jax.profiler.TraceContext` and {func}`jax.profiler.trace_function` in + your code and capturing a new profile. + +### Troubleshooting + +#### GPU profiling + +Programs running on GPU should produce traces for the GPU streams near the top +of the trace viewer. If you're only seeing the host traces, check your program +logs and/or output for the following error messages. + +**If you get an error like: `Could not load dynamic library 'libcupti.so.10.1'`**
+Full error: +``` +W external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcupti.so.10.1'; dlerror: libcupti.so.10.1: cannot open shared object file: No such file or directory +2020-06-12 13:19:59.822799: E external/org_tensorflow/tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1422] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found. +``` + +Add the path to `libcupti.so` to the environment variable `LD_LIBRARY_PATH`. +(Try `locate libcupti.so` to find the path.) For example: +```shell +export LD_LIBRARY_PATH=/usr/local/cuda-10.1/extras/CUPTI/lib64/:$LD_LIBRARY_PATH +``` + +**If you get an error like: `failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES`**
+Full error: +```shell +E external/org_tensorflow/tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1445] function cupti_interface_->EnableCallback( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES +2020-06-12 14:31:54.097791: E external/org_tensorflow/tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] function cupti_interface_->ActivityDisable(activity)failed with error CUPTI_ERROR_NOT_INITIALIZED +``` + +Run the following commands (note this requires a reboot): +```shell +echo 'options nvidia "NVreg_RestrictProfilingToAdminUsers=0"' | sudo tee -a /etc/modprobe.d/nvidia-kernel-common.conf +sudo update-initramfs -u +sudo reboot now +``` + +See [Nvidia's documentation on this +error](https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti) +for more information. + +(remote_profiling)= +#### Profiling on a remote machine + +If the JAX program you'd like to profile is running on a remote machine, one +option is to run all the instructions above on the remote machine (in +particular, start the TensorBoard server on the remote machine), then use SSH +local port forwarding to access the TensorBoard web UI from your local +machine. Use the following SSH command to forward the default TensorBoard port +6006 from the local to the remote machine: + +```shell +ssh -L 6006:localhost:6006 +``` + +## Nsight + +Nvidia's `Nsight` tools can be used to trace and profile JAX code on GPU. For +details, see the [`Nsight` +documentation](https://developer.nvidia.com/tools-overview). ## XLA profiling