o i, @s ddlZddlZddlZddlmZddlmZddlm Z m Z ddl Z ddl Z ddl m Z ddlZddlmZddlmZmZmZmZddlZGdd d Z d!d eedeedededef ddZd"ddZd#ddZgejee j!"ddfdeedefddZ#dd Z$dS)$N)deque) ThreadPool)ProcessQueue)tqdm) open_memmap)UnionListCallableDequec @seZdZdZ      d8deded ed ed ed ef d dZddZddfdee j e fde fddZ ddZd9de j dedefddZd:ded efd d!Zd;d#d$Zd;d%d&Zd;d'd(Zd)d*Zd+d,Zd-d.Zdonnxcpumelspec_model_pathembedding_model_pathsrncpuinference_frameworkdevicec s|dkrzddl}Wn tytdw|dkr)tjttj ddd}|dkrWsz(AudioFeatures.__init__..csjdd|idS)NZinput_1r)embedding_modelrsqueezerr!r#r$r%]stflitezmTried to import the TFLite runtime, but it was not found.Please install it using `pip install tflite-runtime`zmelspectrogram.tflitezembedding_model.tflite.onnxzJThe tflite inference framework is selected, but onnx models were provided!)Z model_pathZ num_threadsrTstrictindexcs|jddkr!jjdd|jdgddj|jd_njdkr9jjdddgddjd_j|jjS)Nrr*rTr+)shaperresize_tensor_inputallocate_tensors"_tflite_current_melspec_input_size set_tensorinvoke get_tensorr)melspec_input_indexmelspec_output_indexr"r#r$tflite_melspec_predictzs     z6AudioFeatures.__init__..tflite_melspec_predictcs|jddkr#jjd|jddddgddj|jd_njdkr>jjdgdddj|jd_j|jjS)NrrL Tr+)rr8r9r) r.r&r/r0$_tflite_current_embedding_batch_sizer2r3r4r'r)embedding_input_indexembedding_output_indexr"r#r$tflite_embedding_predicts      z8AudioFeatures.__init__..tflite_embedding_predict )maxlenr8r9ix).Z onnxruntime ImportError ValueErrorospathjoinpathlibPath__file__parentresolveZSessionOptionsZinter_op_num_threadsZintra_op_num_threadsZInferenceSessionrZ get_providersonnx_execution_providermelspec_model_predictr&embedding_model_predictZtflite_runtime.interpreter interpreterZ Interpreterr/r0Zget_input_detailsZget_output_detailsr1r:rraw_data_buffernponesmelspectrogram_buffermelspectrogram_max_lenaccumulated_samplesemptyraw_data_remainder_get_embeddingsrandomrandintastypeint16feature_bufferfeature_buffer_max_len) r"rrrrrrZortZsessionOptionsr(r7r=r#)r;r<r5r6r"r$__init__&s~          zAudioFeatures.__init__cCsL|jtd|_d|_td|_|tj ddd tj |_ dS)zReset the internal buffersr@rrArBrCN)rSclearrTrUrVrXrYrZr[r\r]r^r_r`r!r#r#r$resets   $zAudioFeatures.resetcCs |ddS)Nr>r#rr#r#r$r%s zAudioFeatures.r melspec_transformcCst|trt|tjn|}|jtjkrtd|jdt|j dkr*|dn|}|jtj kr8|tj n|}| |}t |d}||}|S)a Function to compute the mel-spectrogram of the provided audio samples. Args: x (Union[np.ndarray, List]): The input audio data to compute the melspectrogram from melspec_transform (Callable): A function to transform the computed melspectrogram. Defaults to a transform that makes the ONNX melspectrogram model closer to the native Tensorflow implementation from Google (https://tfhub.dev/google/speech_embedding/1). Return: np.ndarray: The computed melspectrogram of the input audio data zIInput data must be 16-bit integers (i.e., 16-bit PCM audio).You provided z data.reNr) isinstancelistrTarrayr^r_dtyperFlenr.float32rPr')r"r rfZoutputsspecr#r#r$_get_melspectrograms    z!AudioFeatures._get_melspectrogramcCs$|jddkr |d}||}|S)z Computes the Google `speech_embedding` features from a melspectrogram input Args: melspec (np.ndarray): The input melspectrogram Returns: np.ndarray: The computed audio features/embeddings rrrg)r.rQ)r"melspec embeddingr#r#r$_get_embeddings_from_melspecs  z*AudioFeatures._get_embeddings_from_melspecr8 window_size step_sizec Ks|j|fi|}g}td|jddD]}||||}|jd|kr*||qtjt|ddtj} | | } | S)z@Function to compute the embeddings of the provide audio samples.rrs)Zaxis) roranger.appendrTZ expand_dimsrjr^rmrQ) r"r rtrukwargsrnwindowsiwindowbatchrqr#r#r$r[s  zAudioFeatures._get_embeddings audio_lengthcCs0tjddt||dtj}||jS)zjFunction that determines the size of the output embedding array for a given audio clip length (in seconds)rvri)rTr\uniformintr^r_r[r.)r"r~rr r#r#r$get_embedding_shapes$ z!AudioFeatures.get_embedding_shapec Csd}d|jvr t|d}tt|jddd}d}tj|jd||ftjd }tdt ||jd|D]D}||||} d |jvrK| | } n|ri| jd|kr[| jd|nd} t |j |j | | d } | ||||ddddf<q6|r||S) a& Compute the melspectrogram of the input audio samples in batches. Note that the optimal performance will depend in the interaction between the device, batch size, and ncpu (if a CPU device is used). The user is encouraged to experiment with different values of these parameters to identify which combination is best for their data, as often differences of 1-4x are seen. Args: x (ndarray): A numpy array of 16 khz input audio data in shape (N, samples). Assumes that all of the audio data is the same length (same number of samples). batch_size (int): The batch size to use when computing the melspectrogram ncpu (int): The number of CPUs to use when computing the melspectrogram. This argument has no effect if the underlying model is executing on a GPU. Returns: ndarray: A numpy array of shape (N, frames, melbins) containing the melspectrogram of all N input audio examples NCPUZ processesrr9rrkCUDA chunksize)rOrrrTceilr.rYrmrwmaxrorjmapr'close) r"r batch_sizerpooln_framesZmel_binsmelspecsr{r}resultrr#r#r$_get_melspectrogram_batchs&      $z'AudioFeatures._get_melspectrogram_batchcCs|jddkr tdd}d|jvrt|d}|jdddd}d}tj|jd ||ftjd }g}g} t|D]\} } d} td | jd dD]} | | | | }|jd | kr_| |qI| | t ||kst| d|jd krt | tj}d |jvr| |}n|r|jd |kr|jd |nd}t |j|j||d }ttd |jd || D]\}}||||||ddddf<qg}g} q:|r||S) a  Compute the embeddings of the input melspectrograms in batches. Note that the optimal performance will depend in the interaction between the device, batch size, and ncpu (if a CPU device is used). The user is encouraged to experiment with different values of these parameters to identify which combination is best for their data, as often differences of 1-4x are seen. Args: x (ndarray): A numpy array of melspectrograms of shape (N, frames, melbins). Assumes that all of the melspectrograms have the same shape. batch_size (int): The batch size to use when computing the embeddings ncpu (int): The number of CPUs to use when computing the embeddings. This argument has no effect if the underlying model is executing on a GPU. Returns: ndarray: A numpy array of shape (N, frames, embedding_dim) containing the embeddings of all N input melspectrograms rr8zMEmbedding model requires the input melspectrograms to have at least 76 framesNrrrs`rrrr)r.rFrOrrTrYrm enumeraterwrxrlrjr^rQrrrzipr)r"r rrrrZ embedding_dim embeddingsr}Zndcsndxrprtr{r|rrjZndx2r#r#r$_get_embeddings_batch$sF         $z#AudioFeatures._get_embeddings_batchcCs<|j|||d}|j|dddddddf||d}|S)a Compute the embeddings of the input audio clips in batches. Note that the optimal performance will depend in the interaction between the device, batch size, and ncpu (if a CPU device is used). The user is encouraged to experiment with different values of these parameters to identify which combination is best for their data, as often differences of 1-4x are seen. Args: x (ndarray): A numpy array of 16 khz input audio data in shape (N, samples). Assumes that all of the audio data is the same length (same number of samples). batch_size (int): The batch size to use when computing the embeddings ncpu (int): The number of CPUs to use when computing the melspectrogram. This argument has no effect if the underlying model is executing on a GPU. Returns: ndarray: A numpy array of shape (N, frames, embedding_dim) containing the embeddings of all N input audio clips rrN)rr)r"r rrrrr#r#r$ embed_clipsfs(zAudioFeatures.embed_clipscCsxt|jdkr tdt|j|t|j| ddf|_|jjd|j kr:|j|j dddf|_dSdS)aNote! There seem to be some slight numerical issues depending on the underlying audio data such that the streaming method is not exactly the same as when the melspectrogram of the entire clip is calculated. It's unclear if this difference is significant and will impact model performance. In particular padding with 0 or very small values seems to demonstrate the differences well. izHThe number of input frames must be at least 400 samples @ 16khz (25 ms)!iNr) rlrSrFrTvstackrVrorir.rW)r"Z n_samplesr#r#r$_streaming_melspectrograms" z'AudioFeatures._streaming_melspectrogramcCs*|jt|tjr|dS|dS)z9 Adds raw audio data to the input buffer N)rSextendrhrTndarraytolistr"r r#r#r$_buffer_raw_datas*zAudioFeatures._buffer_raw_datacCsd}|jjddkrt|j|f}td|_|j|jddkrh|j|jdd}|dkrN|d| }|||jt|7_|| d|_n)|dkrg|||j|jd7_td|_n|j|jd7_|||jdkr|jddkr||jt |jddddD];}d|}|dkr|nt|j }|j d|| tj ddddddf}|jddkrt |j||f|_q|j}d|_|jjd|jkr|j|j dddf|_|dkr|S|jS)Nrr*rrviir8)rZr.rTZ concatenaterYrXrrlrZarangerVr^rmrr`rQra)r"r Zprocessed_samples remainderZ x_even_chunksr{rr#r#r$_streaming_featuressD      .z!AudioFeatures._streaming_featuresrvn_feature_frames start_ndxcCsv|dkr&||dkr|t|nt|j}|j||ddfdtjS|jtd|dddfdtjS)Nrvrrg)rrlr`r^rTrm)r"rrZend_ndxr#r#r$ get_featuress  "*zAudioFeatures.get_featurescCs ||Srg)rrr#r#r$__call__s zAudioFeatures.__call__N)r r rrrr)r8rs)r)rr)rrv)__name__ __module__ __qualname____doc__strrrbrdrrTrr r rorrr[floatrrrrrrrrrr#r#r#r$r !sF $  1 B- r predict_cliprr( file_pathswakeword_modelsprediction_functionrrc s<tdt|t|}fddtdtdt|D}td|dD]}||dd|q+g} gt|D].} ddD} tjd||d| } | fd d } | t| | fd qD| D]}| qug}| D]} rt d  s| qd d|DS)a Bulk predict on the provided input files in parallel using multiprocessing using the specified model. Args: input_paths (List[str]): The list of input file to predict wakeword_models (List[str])): The paths to the wakeword model files prediction_function (str): The name of the method used to predict on the input audio files (default is the `predict_clip` method) ncpu (int): How many processes to create (up to max of available CPUs) inference_framework (str): The inference framework to use when for model prediction. Options are "tflite" or "onnx". The default is "tflite" as this results in better efficiency on common platforms (x86, ARM64), but in some deployment scenarios ONNX models may be preferable. kwargs (dict): Any other keyword arguments to pass to the model initialization or specified prediction function Returns: dict: A dictionary containing the predictions for each file, with the filepath as the key rcsg|] }||qSr#r#.0r{)r n_batchesr#r$ z bulk_predict..rrvcSs&i|]\}}|tjjjjvr||qSr#) openwakewordModelrb__code__ co_varnamesrkeyvaluer#r#r$ s z bulk_predict..)rrcsZg}|D]!}tdfddD}|||fi|iq|dS)Nrvcs"i|] \}}|jjvr||qSr#)rrrfuncr#r$rs z+bulk_predict..f..)getattritemsrxput)ZclipsresultsZclipfiltered_kwargs)rymdlsrqrr$fs zbulk_predict..f)targetargsg{Gz?cSs*i|]}t|dt|dqS)r)rikeysvaluesrr#r#r$rs*Nr#)rrlrwrxrrrrrstartrYtimesleeprget)rrrrrryrchunksr{ZpschunkrZowwrprr#)rryrrrrr$ bulk_predicts8 *    rrcCs`ddlm}t|d}||d}||d|df} t|dtj| d} d} t|} | jd} | |kr>t d|d | d |j | | d }|| | | |jdd d d d f<| |jd7} | t ||| d dD]>} | |krsn7|j | | |d}| |jd|kr|d|| }|| | | |jdd d d d f<| |jd7} | qk||d S)a> Computes audio features from a generator that produces Numpy arrays of shape (batch_size, samples) containing 16-bit PCM audio data. Args: generator (Generator): The generator that process the arrays of audio data n_total (int): The total number of rows (audio clips) that the generator will produce. Ideally this is precise, but it can be approximate as well as the output .npy file will be automatically trimmed to remove empty values. clip_duration (float): The duration (in samples) of the audio produced by the generator output_file (str): The output file (.npy) containing the audio features. Note that this file will be written to using memmap arrays, so it can be substantially larger than the available system memory. device (str): The device ("cpu" or "gpu") to use for computing features. ncpu (int): The number of cores to use when process the audio features (if computing on CPU) Returns: None r) trim_mmap)rrrzw+)moderkr.zThe value of 'n_total' (z) is less than the batch size (z1). Please increase 'n_total' to be >= batch size.)rNzComputing features)totaldescr) Zopenwakeword.datarr rrrTrmnextr.rFrflushr) generatorZn_totalZ clip_duration output_filerrrFZn_feature_colsZ output_shapefpZ row_counterZ audio_datarfeaturesr#r#r$compute_features_from_generators0   $$  rc Cs|dd}tj|ddW}|durt|dd|d}nt|jdd }t|dd|d}ttj ||d }|j d d D]}| || t |q@Wdn1sYwYWdn1shwY|dS) zcA simple function to download a file from a URL with a progress bar using only the requests library/rvT)streamNZiB)runitZ unit_scalerzcontent-lengthrwbi ) chunk_size)splitrequestsrrrheadersopenrGrHrI iter_contentwriteupdaterlr) urltarget_directory file_sizeZlocal_filenamer progress_barZ total_sizerrr#r#r$ download_file]s  rrr model_namesrc st|ts tdtj|st|tj D]&}tjtj ||d dds?t |d|t |d dd|qtj D]}tjtj ||d dds`t |d|qEddtj D}d dtj D}|gkr|D]6fd dt||D}|gkrtjtj ||d  ddst |d |t |d  dd|q{d S|D] }tjtj || ddst ||t | dd|qd S) a Download the specified models from the release assets in the openWakeWord GitHub repository. Uses the official urls in the MODELS dictionary in openwakeword/__init__.py. Args: model_names (List[str]): The names of the models to download (e.g., hey_jarvis_v0.1). Both ONNX and tflite models will be downloaded. If not provided (the default), the latest versions of all models will be downloaded. target_directory (str): The directory to save the models to. Defaults to the install location of openWakeWord (i.e., the `resources/models` directory). Returns: None z2The model_names argument must be a list of strings download_urlrrvrr)cSsg|]}|dqS)rr#rr#r#r$rsz#download_models..cSsg|] }|dddqS)rrrv)rrr#r#r$rscsg|] \}}|vr|qSr#r#)rr{rZ model_namer#r$rrrN)rhrirFrGrHexistsmakedirsrZFEATURE_MODELSrrIrrreplaceZ VAD_MODELSZMODELSr)rrZ feature_modelZ vad_modelZofficial_model_urlsZofficial_model_namesrZofficial_model_urlr#rr$download_modelsqs<   $$$  rcsfdd}|S)Ncsfdd}|S)NcsXi}|D]\}}|vrtd|d|d||||<q|i|S)NzDEPRECATION: keyword argument 'zB' is no longer valid and will be removed in future releases. Use 'z ' instead.)rloggingwarningr)rryZ new_kwargskv)r kwarg_mapr#r$wrappeds  z*re_arg..decorator..wrappedr#)rrrrr$ decoratorszre_arg..decoratorr#)rrr#rr$re_args  r)rrr()rrrg)%rGnumpyrTrJ collectionsrZmultiprocessing.poolrmultiprocessingrrrrrrZnumpy.lib.formatrtypingrr r r rr rrrrrrHrIrKrLrMrNrrr#r#r#r$sR    8 K ? 4