ó w®¸Qc@sTdgZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z m Z dZ dZdZejƒZd„Zd efd „ƒYZefed „Zdefd „ƒYZd efd„ƒYZeZdefd„ƒYZdefd„ƒYZdefd„ƒYZdefd„ƒYZdS(tPooliÿÿÿÿN(tProcesst cpu_countt TimeoutError(tFinalizetdebugiiicCs t|ŒS(N(tmap(targs((s*/sys/lib/python2.7/multiprocessing/pool.pytmapstar@stMaybeEncodingErrorcBs)eZdZd„Zd„Zd„ZRS(sVWraps possible unpickleable errors, so they can be safely sent through the socket.cCsAt|ƒ|_t|ƒ|_tt|ƒj|j|jƒdS(N(treprtexctvaluetsuperR t__init__(tselfR R ((s*/sys/lib/python2.7/multiprocessing/pool.pyRKscCsd|j|jfS(Ns(Error sending result: '%s'. Reason: '%s'(R R (R((s*/sys/lib/python2.7/multiprocessing/pool.pyt__str__Ps cCsdt|ƒS(Ns(tstr(R((s*/sys/lib/python2.7/multiprocessing/pool.pyt__repr__Ts(t__name__t __module__t__doc__RRR(((s*/sys/lib/python2.7/multiprocessing/pool.pyR Gs  cCs¡|j}|j}t|dƒr>|jjƒ|jjƒn|dk rW||Œnd}x/|dks~|rŽ||krŽy |ƒ}Wn"ttfk r¯t dƒPnX|dkrÊt dƒPn|\} } } } } yt | | | Žf}Wnt k r}t |f}nXy|| | |fƒWnMt k r€}t ||dƒ}t d|ƒ|| | t |ffƒnX|d7}q`Wt d|ƒdS(Nt_writeris)worker got EOFError or IOError -- exitingsworker got sentinel -- exitingis0Possible encoding error while sending result: %ssworker exiting after %d tasks(tputtgetthasattrRtcloset_readertNonetEOFErrortIOErrorRtTruet ExceptiontFalseR (tinqueuetoutqueuet initializertinitargstmaxtasksRRt completedttasktjobtitfuncRtkwdstresulttetwrapped((s*/sys/lib/python2.7/multiprocessing/pool.pytworkerXs>     !    cBseZdZeZddddd„Zd„Zd„Zd„Zd„Z did„Z dd„Z dd „Z dd „Z didd „Zddd „Zed „ƒZed„ƒZed„ƒZed„ƒZd„Zd„Zd„Zd„Zed„ƒZed„ƒZRS(sH Class which supports an async version of the `apply()` builtin c Cs)|jƒtjƒ|_i|_t|_||_||_||_|dkr|y t ƒ}Wq|t k rxd}q|Xn|dkr—t dƒ‚n|dk rÂt |dƒ rÂtdƒ‚n||_g|_|jƒtjdtjd|fƒ|_t|j_t|j_|jjƒtjdtjd|j|j|j|jfƒ|_t|j_t|j_|jjƒtjdtjd|j|j|jfƒ|_ t|j _t|j _|j jƒt!||j"d|j|j#|j|j|j|j|j |jfddƒ|_$dS( Nis&Number of processes must be at least 1t__call__sinitializer must be a callablettargetRt exitpriorityi(%t _setup_queuestQueuet _taskqueuet_cachetRUNt_statet_maxtasksperchildt _initializert _initargsRRtNotImplementedErrort ValueErrorRt TypeErrort _processest_poolt_repopulate_poolt threadingtThreadRt_handle_workerst_worker_handlerRtdaemontstartt _handle_taskst _quick_putt _outqueuet _task_handlert_handle_resultst _quick_gett_result_handlerRt_terminate_poolt_inqueuet _terminate(Rt processesR$R%tmaxtasksperchild((s*/sys/lib/python2.7/multiprocessing/pool.pyRˆsV                     $          cCswt}xjttt|jƒƒƒD]M}|j|}|jdk r"td|ƒ|jƒt }|j|=q"q"W|S(sCleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. scleaning up worker %dN( R!treversedtrangetlenRAtexitcodeRRtjoinR(RtcleanedR*R0((s*/sys/lib/python2.7/multiprocessing/pool.pyt_join_exited_workersÂs"  c Cs¦xŸt|jt|jƒƒD]}|jdtd|j|j|j|j |j fƒ}|jj |ƒ|j j ddƒ|_ t|_|jƒtdƒqWdS(s€Bring the number of pool processes up to the specified number, for use after reaping workers which have exited. R2RRt PoolWorkers added workerN(RVR@RWRARR0RQRKR;R<R:tappendtnametreplaceRRGRHR(RR*tw((s*/sys/lib/python2.7/multiprocessing/pool.pyRBÑs#   cCs|jƒr|jƒndS(sEClean up any exited workers and start replacements for them. N(R[RB(R((s*/sys/lib/python2.7/multiprocessing/pool.pyt_maintain_poolás cCsPddlm}|ƒ|_|ƒ|_|jjj|_|jjj|_ dS(Ni(t SimpleQueue( tqueuesRbRQRKRtsendRJRtrecvRN(RRb((s*/sys/lib/python2.7/multiprocessing/pool.pyR4çs   cCs|j|||ƒjƒS(s1 Equivalent of `apply()` builtin (t apply_asyncR(RR+RR,((s*/sys/lib/python2.7/multiprocessing/pool.pytapplyîscCs|j|||ƒjƒS(s/ Equivalent of `map()` builtin (t map_asyncR(RR+titerablet chunksize((s*/sys/lib/python2.7/multiprocessing/pool.pyRõsics¶|dkrQt|jƒ‰|jj‡‡fd†t|ƒDƒˆjfƒˆStjˆ||ƒ}t|jƒ‰|jj‡fd†t|ƒDƒˆjfƒd„ˆDƒSdS(sZ Equivalent of `itertools.imap()` -- can be MUCH slower than `Pool.map()` ic3s0|]&\}}ˆj|ˆ|fifVqdS(N(t_job(t.0R*tx(R+R-(s*/sys/lib/python2.7/multiprocessing/pool.pys sc3s0|]&\}}ˆj|t|fifVqdS(N(RkR(RlR*Rm(R-(s*/sys/lib/python2.7/multiprocessing/pool.pys scss"|]}|D] }|Vq qdS(N((Rltchunktitem((s*/sys/lib/python2.7/multiprocessing/pool.pys sN(t IMapIteratorR7R6Rt enumeratet _set_lengthRt _get_tasks(RR+RiRjt task_batches((R+R-s*/sys/lib/python2.7/multiprocessing/pool.pytimapüs cs¶|dkrQt|jƒ‰|jj‡‡fd†t|ƒDƒˆjfƒˆStjˆ||ƒ}t|jƒ‰|jj‡fd†t|ƒDƒˆjfƒd„ˆDƒSdS(sK Like `imap()` method but ordering of results is arbitrary ic3s0|]&\}}ˆj|ˆ|fifVqdS(N(Rk(RlR*Rm(R+R-(s*/sys/lib/python2.7/multiprocessing/pool.pys sc3s0|]&\}}ˆj|t|fifVqdS(N(RkR(RlR*Rm(R-(s*/sys/lib/python2.7/multiprocessing/pool.pys scss"|]}|D] }|Vq qdS(N((RlRnRo((s*/sys/lib/python2.7/multiprocessing/pool.pys sN(tIMapUnorderedIteratorR7R6RRqRrRRs(RR+RiRjRt((R+R-s*/sys/lib/python2.7/multiprocessing/pool.pytimap_unordereds cCsAt|j|ƒ}|jj|jd|||fgdfƒ|S(s> Asynchronous equivalent of `apply()` builtin N(t ApplyResultR7R6RRkR(RR+RR,tcallbackR-((s*/sys/lib/python2.7/multiprocessing/pool.pyRf s+csæt|dƒst|ƒ}n|dkrhtt|ƒt|jƒdƒ\}}|rh|d7}qhnt|ƒdkrƒd}ntj|||ƒ}t|j |t|ƒ|ƒ‰|j j ‡fd†t |ƒDƒdfƒˆS(s< Asynchronous equivalent of `map()` builtin t__len__iiic3s0|]&\}}ˆj|t|fifVqdS(N(RkR(RlR*Rm(R-(s*/sys/lib/python2.7/multiprocessing/pool.pys :sN( RtlistRtdivmodRWRARRst MapResultR7R6RRq(RR+RiRjRytextraRt((R-s*/sys/lib/python2.7/multiprocessing/pool.pyRh)s ( cCsotjƒ}xB|jtks6|jrP|jtkrP|jƒtjdƒqW|j j dƒt dƒdS(Ng™™™™™™¹?sworker handler exiting( RCtcurrent_threadR9R8R7t TERMINATERattimetsleepR6RRR(tpooltthread((s*/sys/lib/python2.7/multiprocessing/pool.pyRE>s  * c Cs1tjƒ}x¶t|jdƒD]˜\}}d}x‚t|ƒD]P\}}|jratdƒPny||ƒWq>tk rtdƒPq>Xq>W|rtdƒ||dƒqqPqWtdƒy@tdƒ|j dƒtdƒx|D]} |dƒqðWWntk r"td ƒnXtd ƒdS( Niÿÿÿÿs'task handler found thread._state != RUNscould not put task on queuesdoing set_length()istask handler got sentinels/task handler sending sentinel to result handlers(task handler sending sentinel to workerss/task handler got IOError when sending sentinelsstask handler exiting( RCRtiterRRRqR9RRR( t taskqueueRR#RƒR„ttaskseqt set_lengthR*R(tp((s*/sys/lib/python2.7/multiprocessing/pool.pyRIKs6             cCsÝtjƒ}x¡y |ƒ}Wn"ttfk r@tdƒdSX|jrXtdƒPn|dkrrtdƒPn|\}}}y||j||ƒWqtk r¬qXqx¢|rT|jt krTy |ƒ}Wn"ttfk rùtdƒdSX|dkrtdƒq³n|\}}}y||j||ƒWq³tk rPq³Xq³Wt |dƒrÀtdƒy5x.t dƒD] }|j j ƒs—Pn|ƒq~WWqÀttfk r¼qÀXntdt|ƒ|jƒdS( Ns.result handler got EOFError/IOError -- exitings,result handler found thread._state=TERMINATEsresult handler got sentinels&result handler ignoring extra sentinelRs"ensuring that outqueue is not fulli s7result handler exiting: len(cache)=%s, thread._state=%s(RCRRRRR9Rt_settKeyErrorR€RRVRtpollRW(R#RtcacheR„R(R)R*tobj((s*/sys/lib/python2.7/multiprocessing/pool.pyRMrsV              ccsCt|ƒ}x0ttj||ƒƒ}|s1dS||fVqdS(N(R…ttuplet itertoolstislice(R+tittsizeRm((s*/sys/lib/python2.7/multiprocessing/pool.pyRs¬s  cCstdƒ‚dS(Ns:pool objects cannot be passed between processes or pickled(R=(R((s*/sys/lib/python2.7/multiprocessing/pool.pyt __reduce__µscCs5tdƒ|jtkr1t|_t|j_ndS(Ns closing pool(RR9R8tCLOSERF(R((s*/sys/lib/python2.7/multiprocessing/pool.pyRºs  cCs-tdƒt|_t|j_|jƒdS(Nsterminating pool(RR€R9RFRR(R((s*/sys/lib/python2.7/multiprocessing/pool.pyt terminateÀs   cCsStdƒ|jjƒ|jjƒ|jjƒx|jD]}|jƒq;WdS(Ns joining pool(RRFRYRLRORA(RR‰((s*/sys/lib/python2.7/multiprocessing/pool.pyRYÆs     cCsWtdƒ|jjƒx9|jƒrR|jjƒrR|jjƒtjdƒqWdS(Ns7removing tasks from inqueue until task handler finishedi( Rt_rlocktacquiretis_aliveRRŒReRR‚(R"t task_handlerR“((s*/sys/lib/python2.7/multiprocessing/pool.pyt_help_stuff_finishÏs    c Cs‘tdƒt|_t|_tdƒ|j||t|ƒƒt|_|jdƒtdƒtjƒ|k r|j dƒn|rÔt |ddƒrÔtdƒx-|D]"} | j dkr«| j ƒq«q«Wntdƒtjƒ|k r|j dƒntd ƒtjƒ|k r,|j dƒn|rt |ddƒrtd ƒx;|D]0} | j ƒrVtd | jƒ| j ƒqVqVWndS( Nsfinalizing pools&helping task handler/workers to finishsjoining worker handlerg}Ô%­I²TiR–sterminating workerssjoining task handlersjoining result handlersjoining pool workersscleaning up worker %d(RR€R9R›RWRRRCRRYRRXR–R™tpid( tclsR†R"R#Rƒtworker_handlerRštresult_handlerRR‰((s*/sys/lib/python2.7/multiprocessing/pool.pyRPØs6              N((((RRRRRRR[RBRaR4RgRRuRwRfRht staticmethodRERIRMRsR”RR–RYR›t classmethodRP(((s*/sys/lib/python2.7/multiprocessing/pool.pyR‚s0  9         ':     RxcBsDeZd„Zd„Zd„Zdd„Zdd„Zd„ZRS(cCsStjtjƒƒ|_tjƒ|_||_t|_ ||_ |||js RpcBs>eZd„Zd„Zdd„ZeZd„Zd„ZRS(cCsktjtjƒƒ|_tjƒ|_||_tj ƒ|_ d|_ d|_ i|_|||j#s,        *ÿ‰.-I