o ¢ÄiÑgã@s–ddlZddlZddlmZmZddlZddlZddlZddl Z ddl Z ddl m Z m Z ddl mZddlZddlmZmZmZmZGdd„dƒZdS)éN)Ú AudioFeaturesÚre_arg)ÚdequeÚ defaultdict)Úpartial)ÚListÚUnionÚ DefaultDictÚDictc@säeZdZdZeddiƒggddiddfdeedeed ed e d ed e d efdd„ƒZ dd„Z dd„Z iiddfde jdedede def dd„Zd-deee jfdefd d!„Z " #d.d$ede d%efd&d'„Zd/de jd)efd*d+„Zd,S)0ÚModelz­ The main model class for openWakeWord. Creates a model object with the shared audio pre-processer and for arbitrarily many custom wake word/wake phrase models. Zwakeword_model_pathsÚwakeword_modelsFrgš™™™™™¹?ÚtfliteÚclass_mapping_dictsÚenable_speex_noise_suppressionÚ vad_thresholdÚcustom_verifier_modelsÚcustom_verifier_thresholdÚinference_frameworkc s\t |¡} g} |gkr| }ttj ¡ƒ} nDt|ƒdkrYt|ƒD]9\} ‰tj  ˆ¡r9|   tj  tj  ˆ¡¡d¡q‡fdd„| Dƒ} | gkrMt d ˆ¡ƒ‚| d|| <|   ˆ¡qi|_i|_i|_i|_i|_i|_||_|dkr¸z ddlm} dd „}Wn9ty·t d ¡|gkrštd d„|Dƒƒršd }n|gkr±td d„|Dƒƒr±d }dd„|Dƒ}nt dƒ‚Ynw|d krÒz ddl}dd„}Wn tyÑt dƒ‚wt|| ƒD]\}}|d kr+d|vrét dƒ‚| ¡}d|_d|_ |j!||dgd|j|<|j| "¡dj#d|j|<|j| $¡dj#d|j|<t% &||j|¡}||j|<|dkrd|vr9t dƒ‚| j'|dd|j|<|j| (¡|j| )¡ddd|j|<|j| *¡ddd|j|<|j| )¡dd}|j| *¡dd}t% &||j|||¡}||j|<|rª|| +|¡ ,|d¡rª|| +|¡|j|<n!tj- ,|d¡r»tj-||j|<ndd„t.d|j|ƒDƒ|j|<t/|t0ƒrå| ,|d¡råt1 2t3||dƒ¡|j|<t|j ¡ƒt| ¡ƒkr÷t d ƒ‚q×t4t&t5d!d"ƒ|_6|rdd#l7m8}| 9d$d%¡|_:nd|_:||_;|dkr"t <¡|_=t>d'd&|i|¤Ž|_?dS)(aQ Initialize the openWakeWord model object. Args: wakeword_models (List[str]): A list of paths of ONNX/tflite models to load into the openWakeWord model object. If not provided, will load all of the pre-trained models. Alternatively, just the names of pre-trained models can be provided to select a subset of models. class_mapping_dicts (List[dict]): A list of dictionaries with integer to string class mappings for each model in the `wakeword_models` arguments (e.g., {"0": "class_1", "1": "class_2"}) enable_speex_noise_suppression (bool): Whether to use the noise suppresion from the SpeexDSP library to pre-process all incoming audio. May increase model performance when reasonably stationary background noise is present in the environment where openWakeWord will be used. It is very lightweight, so enabling it doesn't significantly impact efficiency. vad_threshold (float): Whether to use a voice activity detection model (VAD) from Silero (https://github.com/snakers4/silero-vad) to filter predictions. For every input audio frame, a VAD score is obtained and only those model predictions with VAD scores above the threshold will be returned. The default value (0), disables voice activity detection entirely. custom_verifier_models (dict): A dictionary of paths to custom verifier models, where the keys are the model names (corresponding to the openwakeword.MODELS attribute) and the values are the filepaths of the custom verifier models. custom_verifier_threshold (float): The score threshold to use a custom verifier model. If the score from a model for a given frame is greater than this value, the associated custom verifier model will also predict on that frame, and the verifier score will be returned. inference_framework (str): The inference framework to use when for model prediction. Options are "tflite" or "onnx". The default is "tflite" as this results in better efficiency on common platforms (x86, ARM64), but in some deployment scenarios ONNX models may be preferable. kwargs (dict): Any other keyword arguments to pass the the preprocessor instance ércs.g|]}ˆ dd¡| tjj¡dvr|‘qS)ú Ú_éÿÿÿÿ)ÚreplaceÚsplitÚosÚpathÚsep)Ú.0Új©Úi©úL/home/kim/smarthome/.venv/lib/python3.10/site-packages/openwakeword/model.pyÚ _s.z"Model.__init__..z3Could not find pretrained model for model name '{}'r NcSs"| ||¡| ¡| |¡dS)N©N)Z set_tensorZinvokeZ get_tensor)Ztflite_interpreterZ input_indexZ output_indexÚxr!r!r"Útflite_predictts z&Model.__init__..tflite_predictzŠTried to import the tflite runtime, but it was not found. Trying to switching to onnxruntime instead, if appropriate models are available.cSsg|]}d|v‘qS)ú.onnxr!©rr r!r!r"r#|óZonnxcSs g|] }tj | dd¡¡‘qS©ú.tfliter')rrÚexistsrr(r!r!r"r#~s cSsg|]}| dd¡‘qSr*)rr(r!r!r"r#€sz‰Tried to import the tflite runtime for provided tflite models, but it was not found. Please install it using `pip install tflite-runtime`cSs| d| ¡dj|i¡S)Nr)ÚrunÚ get_inputsÚname)Z onnx_modelr%r!r!r"Ú onnx_predict‰sz$Model.__init__..onnx_predictzdTried to import onnxruntime, but it was not found. Please install it using `pip install onnxruntime`r+zJThe onnx inference framework is selected, but tflite models were provided!ZCPUExecutionProvider)Z sess_optionsÚ providersr'zJThe tflite inference framework is selected, but onnx models were provided!)Z model_pathZ num_threadsÚshapeÚindexcSsi|] }t|ƒt|ƒ“qSr!)Ústrr(r!r!r"Ú ¶sz"Model.__init__..FÚrbaICustom verifier models were provided, but some were not matched with a base model! Make sure that the keys provided in the `custom_verifier_models` dictionary argument exactly match that of the `.models` attribute of an instantiated openWakeWord Model object that has the same base models but doesn't have custom verifier models.é©Úmaxlen)ÚNoiseSuppressioné é€>rr!)@Ú openwakewordZget_pretrained_model_pathsÚlistZMODELSÚkeysÚlenÚ enumeraterrr,ÚappendÚsplitextÚbasenameÚ ValueErrorÚformatÚmodelsÚ model_inputsÚ model_outputsÚmodel_prediction_functionÚ class_mappingrrZtflite_runtime.interpreterÚ interpreterÚ ImportErrorÚloggingÚwarningÚallZ onnxruntimeÚzipZSessionOptionsZinter_op_num_threadsZintra_op_num_threadsZInferenceSessionr.r2Z get_outputsÚ functoolsrZ InterpreterZallocate_tensorsZget_input_detailsZget_output_detailsr3ÚgetZmodel_class_mappingsÚrangeÚ isinstanceÚdictÚpickleÚloadÚopenrrÚprediction_bufferZ speexdsp_nsr:ÚcreateÚspeex_nsrZVADÚvadrÚ preprocessor)Úselfr rrrrrrÚkwargsZpretrained_model_pathsZwakeword_model_namesZndxZmatching_modelr r&Zortr0Zmdl_pathZmdl_nameZsessionOptionsZ pred_functionZtflite_input_indexZtflite_output_indexr:r!rr"Ú__init__%s² /         €÷   ÿ  ÿ        ÿÿ    zModel.__init__cCsJd}|j ¡D]}||j| ¡vr|}q||j ¡vr"||kr"|}q|S)z>Gets the parent model associated with a given prediction labelÚ)rKr?Úvalues)r_ÚlabelÚ parent_modelÚmdlr!r!r"Úget_parent_model_from_label×s€z!Model.get_parent_model_from_labelcCs tttddƒ|_|j ¡dS)z•Reset the prediction and audio feature buffers. Useful for re-initializing the model, though may not be efficient when called too frequently.r7r8N)rrrrZr^Úreset)r_r!r!r"rhâsz Model.resetçr%ÚpatienceÚ thresholdÚ debounce_timeÚtimingc Cs¼t|tjƒstdt|ƒ›dƒ‚|ri}i|d<t ¡}|jr(| | |¡¡}n| |¡}|r9t ¡||dd<i} |j   ¡D]} |rIt ¡} |dkr‚g} t  |dddd¡D]} |   |j | |jj|j| |j|  | dƒ¡qZt | ¡jd d d }nV|dkr•|j | |j |j| ¡ƒ}nC|dkrØ|j| dkrºt|j| ƒd kr´|j| dggg}n$d ggg}n|j| dkrØtd d „|j|   ¡Dƒƒ}d g|dgg}|j| dkrê|d d d | | <n|j|  ¡D]\}}|d d t|ƒ| |<qñ|jikr<|   ¡D]/}| ||jkr:| |¡}|j |d¡r:|j| |j |j| ¡¡d d}|| |<q |   ¡D]}t|j|ƒdkrPd| |<q@|r_t ¡| |d| <q@|iksj|d krú|ikrstdƒ‚|ikr|d krtdƒ‚|   ¡D]t} | | ¡}| | dkrø||  ¡vr¼t |j| ¡|| d…}|||k ¡||krºd| | <q…|d krø||  ¡vrøtt ||d¡ƒ}t |j| ¡| d…}| | ||krø|||k ¡d krød| | <q…|   ¡D] } |j|  | | ¡qþ|j d krU|rt ¡}| !|¡|r+t ¡||dd<t"|j!jƒdd…}t|ƒd krAt |¡nd }|   ¡D] } ||j krSd| | <qG|r\| |fS| S)aPredict with all of the wakeword models on the input audio frames Args: x (ndarray): The input audio data to predict on with the models. Ideally should be multiples of 80 ms (1280 samples), with longer lengths reducing overall CPU usage but decreasing detection latency. Input audio with durations greater than or less than 80 ms is also supported, though this will add a detection delay of up to 80 ms as the appropriate number of samples are accumulated. patience (dict): How many consecutive frames (of 1280 samples or 80 ms) above the threshold that must be observed before the current frame will be returned as non-zero. Must be provided as an a dictionary where the keys are the model names and the values are the number of frames. Can reduce false-positive detections at the cost of a lower true-positive rate. By default, this behavior is disabled. threshold (dict): The threshold values to use when the `patience` or `debounce_time` behavior is enabled. Must be provided as an a dictionary where the keys are the model names and the values are the thresholds. debounce_time (float): The time (in seconds) to wait before returning another non-zero prediction after a non-zero prediction. Can preven multiple detections of the same wake-word. timing (bool): Whether to return timing information of the models. Can be useful to debug and assess how efficiently models are running on the current hardware. Returns: dict: A dictionary of scores between 0 and 1 for each model, where 0 indicates no wake-word/wake-phrase detected. If the `timing` argument is true, returns a tuple of dicts containing model predictions and timing information, respectively. zSThe input audio data (x) must by a Numpy array, instead received an object of type Ú.rGr^érr)Z start_ndxr)Zaxisr$cSsg|]}t|ƒ‘qSr!)Úintr(r!r!r"r#6r)z!Model.predict..FérizjError! When using the `patience` argument, threshold values must be provided via the `threshold` argument!zLError! The `patience` and `debounce_time` arguments cannot be used together!Nr<r]iùÿÿÿéüÿÿÿ)#rUÚnpÚndarrayrEÚtypeÚtimer\r^Ú_suppress_noise_with_speexrGr?ZarangeÚextendrJÚ get_featuresrHÚarrayÚmaxrIr@rZrKÚitemsrprrrgrSZ predict_probaÚsumÚceilrBrr]r>)r_r%rjrkrlrmZ timing_dictZ feature_startZn_prepared_samplesÚ predictionsrfZ model_startZgroup_predictionsr Z predictionZ n_classesZ int_labelÚclsreZverifier_predictionZscoresZn_framesZrecent_predictionsZ vad_startZ vad_framesZ vad_max_scorer!r!r"ÚpredictèsÌ  þÿÿÿ     ÿþþ€ €€   € €     €z Model.predictrroÚclipÚpaddingc Ksèt|tƒr*tj|dd}tj| | ¡¡tjd}Wdƒn1s$wYnt|tj ƒr2|}|rNt  t  d|¡  tj¡|t  d|¡  tj¡f¡}g}|}t d|jd||ƒD]} | |j|| | |…fi|¤Ž¡q]|S)aCPredict on an full audio clip, simulating streaming prediction. The input clip must bit a 16-bit, 16 khz, single-channel WAV file. Args: clip (Union[str, np.ndarray]): The path to a 16-bit PCM, 16 khz, single-channel WAV file, or an 1D array containing the same type of data padding (int): How many seconds of silence to pad the start/end of the clip with to make sure that short clips can be processed correctly (default: 1) chunk_size (int): The size (in samples) of each chunk of audio to pass to the model kwargs: Any keyword arguments to pass to the class `predict` method Returns: list: A list containing the frame-level prediction dictionaries for the audio clip r6©Úmode©ZdtypeNr<r)rUr4ÚwaverYrsÚ frombufferÚ readframesÚ getnframesÚint16rtZ concatenateZzerosZastyperTr2rBr) r_r‚rƒÚ chunk_sizer`ÚfÚdatarÚ step_sizer r!r!r"Ú predict_clip„s& þ€ ýÿ &zModel.predict_clipçà?ÚfeaturesÚfileÚ return_typecKs>tj|dd}tj| | ¡¡tjd}Wdƒn1swYttƒ}d}t d|j d||ƒD]U} |j || | |…fi|¤Ž} |   ¡D]?} | | |kr‰|  | ¡} |j |j| ¡} |dkrk||  | ¡|dkr‰|td| d ƒ| d …}t|ƒd kr‰||  |¡qJq5i}|  ¡D] } t || ¡|| <q‘|S) a- Gets predictions for the input audio data, and returns the audio features (embeddings) or audio data for all of the frames with a score above the `threshold` argument. Can be a useful way to collect false-positive predictions. Args: file (str): The path to a 16-bit 16khz WAV audio file to process threshold (float): The minimum score required for a frame of audio features to be returned. return_type (str): The type of data to return when a positive prediction is detected. Can be either 'features' or 'audio' to return audio embeddings or raw audio data, respectively. kwargs: Any keyword arguments to pass to the class `predict` method Returns: dict: A dictionary with filenames as keys and N x M arrays as values, where N is the number of examples and M is the number of audio features, depending on the model input shape. r6r„r†Nrorr’Zaudioi€»r<iú)r‡rYrsrˆr‰rŠr‹rr>rTr2rr?rgr^ryrHrBr{r@Zvstack)r_r“rkr”r`rrŽZ positive_datarr rZlblrfr’ÚcontextZpositive_data_combinedr!r!r"Ú_get_positive_prediction_frames¬s.þ    €÷ z%Model._get_positive_prediction_framesr;Ú frame_sizecCs^g}td|jd|ƒD]}||||…}| |j | ¡¡¡q d |¡}t |tj ¡}|S)aÄ Runs the input audio through the SpeexDSP noise suppression algorithm. Note that this function updates the state of the existing Speex noise suppression object, and isn't intended to be called externally. Args: x (ndarray): The 16-bit, 16khz audio to process. Must always be an integer multiple of `frame_size`. frame_size (int): The frame size to use for the Speex Noise suppressor. Must match the frame size specified during the initialization of the noise suppressor. Returns: ndarray: The input audio with noise suppression applied ró) rTr2rBr\ÚprocessÚtobytesÚjoinrsrˆr‹)r_r%r—Úcleanedr ÚchunkZcleaned_bytestringZ cleaned_arrayr!r!r"rwás z Model._suppress_noise_with_speexN)rro)r‘r’)r;)Ú__name__Ú __module__Ú __qualname__Ú__doc__rrr4rVÚboolÚfloatrargrhrsrtrrrprr–rwr!r!r!r"r sb øþýüûúù ø2 ÿÿÿ ÿ+üþý ü5r )Únumpyrsr=Zopenwakeword.utilsrrr‡rrNrRrWÚ collectionsrrrrvÚtypingrrr r r r!r!r!r"Ús