{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Transfer learning: training shallow classifiers on embedding models outputs\n", "\n", "If you want to adapt BirdNET, Perch, HawkEars, or another foundation model to a new set of classes or a new domain, you'll be doing what machine learning experts call transfer learning. This tutorial demonstrates tranfer learning for PyToch models, but you can do the same with TensorFlow models such as BirdNET and Perch - examples are in a separate notebook `training_birdnet_and_perch.ipynb` since you might need to set your python environment up differently (by installing tensorflow and tensorflow-hub packages).\n", "\n", "This notebook shows examples of how to train simple one-layer or multi-layer fully-connected neural networks (aka multi-layer perceptron networks, MLPs) on embedding (aka features) generated by a pre-trained deep learning model. This workflow is called transfer learning because the learned feature extraction of the embedding model is transfered to a new domain. Ghani et al. [1] demonstrated that gobal bird classification models can act as feature extractors that can be used to train shallow classifiers on novel tasks and domains, even when few training samples are available.\n", "\n", "Training a shallow classifier on embeddings, rather than training or fine-tuning an entire deep learning model, has three advantages: (1) classifiers can be developed with just a handful of training examples; (2) models fit very quickly, enabling an iterative human-in-the-loop workflow for active learning; (3) any model that generates embeddings can be used as the feature extractor; in particular, compiled models without open-source weights (e.g. BirdNET [2]) can be used as the embedding model.\n", "\n", "Users can develop flexible and customizable transfer-learning workflow by generating embeddings then using PyTorch or sklearn directly. This notebook demonstrates both (1) high-level functions and classes in OpenSoundscape that simplify the code needed to perform transfer learning; and (2) examples demonstrating the embedding and model fitting steps explicitly line-by-line.\n", "\n", "> Note: to use models from the model zoo, install the bioacoustics_model_zoo as a package in your python environment:\n", "\n", "`pip install bioacoustics-model-zoo==0.12.0`\n", "\n", "[1] Ghani, B., T. Denton, S. Kahl, H. Klinck, T. Denton, S. Kahl, and H. Klinck. 2023. Global birdsong embeddings enable superior transfer learning for bioacoustic classification. Scientific Reports 13:22876.\n", "\n", "[2] Kahl, Stefan, et al. \"BirdNET: A deep learning solution for avian diversity monitoring.\" Ecological Informatics 61 (2021): 101236.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Run this tutorial\n", "\n", "This tutorial is more than a reference! It's a Jupyter Notebook which you can run and modify on Google Colab or your own computer.\n", "\n", "|Link to tutorial|How to run tutorial|\n", "| :- | :- |\n", "| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kitzeslab/opensoundscape/blob/master/docs/tutorials/train_cnn.ipynb) | The link opens the tutorial in Google Colab. Uncomment the \"installation\" line in the first cell to install OpenSoundscape. |\n", "| [![Download via DownGit](https://img.shields.io/badge/GitHub-Download-teal?logo=github)](https://minhaskamal.github.io/DownGit/#/home?url=https://github.com/kitzeslab/opensoundscape/blob/master/docs/tutorials/train_cnn.ipynb) | The link downloads the tutorial file to your computer. Follow the [Jupyter installation instructions](https://opensoundscape.org/en/latest/installation/jupyter.html), then open the tutorial file in Jupyter. |" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# if this is a Google Colab notebook, install opensoundscape in the runtime environment\n", "if 'google.colab' in str(get_ipython()):\n", " %pip install \"opensoundscape==0.12.1\" \"jupyter-client<8,>=5.3.4\" \"ipykernel==6.17.1\" \"bioacoustics-model-zoo==0.12.0\"\n", " num_workers=0\n", "else:\n", " # choose cpu parallelization count\n", " num_workers=4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Import needed packages" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "#other utilities and packages\n", "import torch\n", "import pandas as pd\n", "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "import random \n", "from glob import glob\n", "import sklearn\n", "\n", "from tqdm.autonotebook import tqdm\n", "from sklearn.metrics import average_precision_score, roc_auc_score\n", "from pathlib import Path\n", "\n", "#set up plotting\n", "from matplotlib import pyplot as plt\n", "plt.rcParams['figure.figsize']=[15,5] #for large visuals\n", "%config InlineBackend.figure_format = 'retina'\n", "\n", "# opensoundscape transfer learning tools\n", "from opensoundscape.ml.shallow_classifier import MLPClassifier, quick_fit, fit_classifier_on_embeddings\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set random seeds\n", "\n", "Set manual seeds for Pytorch and Python. These essentially \"fix\" the results of any stochastic steps in model training, ensuring that training results are reproducible. You probably don't want to do this when you actually train your model, but it's useful for debugging." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "torch.manual_seed(0)\n", "random.seed(0)\n", "np.random.seed(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Download and prepare training data\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Download example files\n", "Download a set of aquatic soundscape recordings with annotations of _Rana sierrae_ vocalizations\n", "\n", "Option 1: run the cell below\n", "\n", "- if you get a 403 error, DataDryad suspects you are a bot. Use Option 2. \n", "\n", "Option 2:\n", "\n", "- Download and unzip the `rana_sierrae_2022.zip` folder containing audio and annotations from this [public Dryad dataset](https://datadryad.org/stash/dataset/doi:10.5061/dryad.9s4mw6mn3#readme)\n", "- Move the unzipped `rana_sierrae_2022` folder into the current folder" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# # Note: the \"!\" preceding each line below allows us to run bash commands in a Jupyter notebook\n", "# # If you are not running this code in a notebook, input these commands into your terminal instead\n", "# !wget -O rana_sierrae_2022.zip https://datadryad.org/stash/downloads/file_stream/2722802;\n", "# !unzip rana_sierrae_2022;" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Prepare audio data\n", "See the train_cnn.ipynb tutorial for step-by-step walkthrough of this process, or just run the cells below to prepare a trainig set." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/annotations.py:333: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n", " all_annotations_df = pd.concat(all_file_dfs).reset_index(drop=True)\n" ] } ], "source": [ "# Set the current directory to where the folder `rana_sierrae_2022` is located:\n", "dataset_path = Path(\"./rana_sierrae_2022/\")\n", "\n", "# let's generate clip labels of 5s duration (to match HawkEars) using the raven annotations\n", "# and some utility functions from opensoundscape\n", "from opensoundscape.annotations import BoxedAnnotations\n", "\n", "audio_and_raven_files = pd.read_csv(f\"{dataset_path}/audio_and_raven_files.csv\")\n", "# update the paths to where we have the audio and raven files stored\n", "audio_and_raven_files[\"audio\"] = audio_and_raven_files[\"audio\"].apply(\n", " lambda x: f\"{dataset_path}/{x}\"\n", ")\n", "audio_and_raven_files[\"raven\"] = audio_and_raven_files[\"raven\"].apply(\n", " lambda x: f\"{dataset_path}/{x}\"\n", ")\n", "\n", "annotations = BoxedAnnotations.from_raven_files(\n", " raven_files=audio_and_raven_files[\"raven\"],\n", " audio_files=audio_and_raven_files[\"audio\"],\n", " annotation_column=\"annotation\",\n", ")\n", "# generate labels for 5s clips, including any labels that overlap by at least 0.2 seconds\n", "labels = annotations.clip_labels(clip_duration=5, min_label_overlap=0.2)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Inspect labels\n", "\n", "Count number of each annotation type: \n", "\n", "Note that the 'X' label is for when the annotator was uncertain about the identity of a call. Labels A-E denote distinct call types." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "A 512\n", "E 128\n", "D 62\n", "B 24\n", "C 74\n", "X 118\n", "dtype: int64" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "labels.sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### split into training and validation data\n", "We'll just focus on class 'A', the call type with the most annotations. We'll randomly split the clips into training and validation data, acknowledging that this approach does not test the ability of the model to generalize. Since the samples in the training and validation sets could be adjascent 2-second audio clips, good performance could simply mean the model has memorized the training samples, and the validation set has very similar samples. " ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "labels_train, labels_val = sklearn.model_selection.train_test_split(labels[[\"A\"]])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train shallow classifiers on embedding model outputs\n", "\n", "We'll train our classifiers on a small annotated dataset with HawkEars, Perch, and BirdNET as feature extractors." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Downloading model from URL...\n", "File hgnet1.ckpt already exists; skipping download.\n", "Loading model from local checkpoint /Users/SML161/opensoundscape/docs/tutorials/hgnet1.ckpt...\n", "Downloading model from URL...\n", "File hgnet2.ckpt already exists; skipping download.\n", "Loading model from local checkpoint /Users/SML161/opensoundscape/docs/tutorials/hgnet2.ckpt...\n", "Downloading model from URL...\n", "File hgnet3.ckpt already exists; skipping download.\n", "Loading model from local checkpoint /Users/SML161/opensoundscape/docs/tutorials/hgnet3.ckpt...\n", "Downloading model from URL...\n", "File hgnet4.ckpt already exists; skipping download.\n", "Loading model from local checkpoint /Users/SML161/opensoundscape/docs/tutorials/hgnet4.ckpt...\n", "Downloading model from URL...\n", "File hgnet5.ckpt already exists; skipping download.\n", "Loading model from local checkpoint /Users/SML161/opensoundscape/docs/tutorials/hgnet5.ckpt...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/preprocess/preprocessors.py:512: DeprecationWarning: sample_shape argument is deprecated. Please use height, width, channels arguments instead. \n", " The current behavior is to override height, width, channels with sample_shape \n", " when sample_shape is not None.\n", " \n", " warnings.warn(\n", "/Users/SML161/opensoundscape/opensoundscape/ml/cnn.py:599: UserWarning: \n", " This architecture is not listed in opensoundscape.ml.cnn_architectures.ARCH_DICT.\n", " It will not be available for loading after saving the model with .save() (unless using pickle=True). \n", " To make it re-loadable, define a function that generates the architecture from arguments: (n_classes, n_channels) \n", " then use opensoundscape.ml.cnn_architectures.register_architecture() to register the generating function.\n", "\n", " The function can also set the returned object's .constructor_name to the registered string key in ARCH_DICT\n", " to avoid this warning and ensure it is reloaded correctly by opensoundscape.ml.load_model().\n", "\n", " See opensoundscape.ml.cnn_architectures module for examples of constructor functions\n", " \n", " warnings.warn(\n", "/Users/SML161/opensoundscape/opensoundscape/ml/cnn.py:623: UserWarning: Failed to detect expected # input channels of this architecture.Make sure your architecture expects the number of channels equal to `channels` argument 1). Pytorch architectures generally expect 3 channels by default.\n", " warnings.warn(\n" ] } ], "source": [ "import bioacoustics_model_zoo as bmz\n", "\n", "hawk = bmz.HawkEars()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a shallow classifier that we'll train with embeddings as inputs. The input size needs to match the size of the embeddings produced by our embedding model. HawkEars embeddings are vectors of length 2048. " ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "clf = MLPClassifier(\n", " input_size=2048, output_size=labels_train.shape[1], hidden_layer_sizes=()\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can run a single function that will embed the training and validation samples, then train the classifier.\n", "\n", "This will take a minute or two, since all of the samples need to be embedded with HawkEars." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from opensoundscape import CNN\n", "\n", "m = CNN(\"resnet18\", [0], 2)" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0;31mSignature:\u001b[0m\n", "\u001b[0mhawk\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0membed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0msamples\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0mtarget_layer\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0mprogress_bar\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0mreturn_preds\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0mavgpool\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0mreturn_dfs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0maudio_root\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mdataloader_kwargs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n", "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", "\u001b[0;31mDocstring:\u001b[0m\n", "Generate embeddings (intermediate layer outputs) for audio files/clips\n", "\n", "Note: to capture embeddings on multiple layers, use self.__call__ with intermediate_layers\n", "argument directly. This wrapper only allows one target_layer.\n", "\n", "Note: Output can be n-dimensional array (return_dfs=False) or pd.DataFrame with multi-index\n", "like .predict() (return_dfs=True). If avgpool=False, return_dfs is forced to False since we\n", "can't create a DataFrame with >2 dimensions.\n", "\n", "Args:\n", " samples: same as CNN.predict(): list of file paths, OR pd.DataFrame with index\n", " containing audio file paths, OR a pd.DataFrame with multi-index (file, start_time,\n", " end_time)\n", " target_layers: layers from self.model._modules to\n", " extract outputs from - if None, attempts to use self.model.embedding_layer as\n", " default\n", " progress_bar: bool, if True, shows a progress bar with tqdm [default: True]\n", " return_preds: bool, if True, returns two outputs (embeddings, logits)\n", " avgpool: bool, if True, applies global average pooling to intermediate outputs\n", " i.e. averages across all dimensions except first to get a 1D vector per sample\n", " return_dfs: bool, if True, returns embeddings as pd.DataFrame with multi-index like\n", " .predict(). if False, returns np.array of embeddings [default: True]. If\n", " avg_pool=False, overrides to return np.array since we can't have a df with >2\n", " dimensions\n", " audio_root: optionally pass a root directory (pathlib.Path or str)\n", " - `audio_root` is prepended to each file path\n", " - if None (default), samples must contain full paths to files\n", " dataloader_kwargs are passed to self.predict_dataloader()\n", "\n", "Returns: (embeddings, preds) if return_preds=True or embeddings if return_preds=False\n", " types are pd.DataFrame if return_dfs=True, or np.array if return_dfs=False\n", "\u001b[0;31mFile:\u001b[0m ~/opensoundscape/opensoundscape/ml/cnn.py\n", "\u001b[0;31mType:\u001b[0m method" ] } ], "source": [ "hawk.embed?\n", "#(labels_train)" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Embedding the training samples without augmentation\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "af8b703844804329b163700adc20f6eb", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/8 [00:00\n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Embedding the validation samples\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "05adabd8b54a4546bc10a8fe68e58bae", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3 [00:00\n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Fitting the classifier\n", "Epoch 100/1000, Loss: 0.550883948802948, Val Loss: 0.5516770482063293\n", "val AU ROC: 0.804\n", "val MAP: 0.768\n", "Epoch 200/1000, Loss: 0.5038763880729675, Val Loss: 0.5115477442741394\n", "val AU ROC: 0.810\n", "val MAP: 0.775\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:110: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_features = torch.tensor(train_features, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:111: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_labels = torch.tensor(train_labels, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:115: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " validation_features = torch.tensor(\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:118: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " validation_labels = torch.tensor(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 300/1000, Loss: 0.48154526948928833, Val Loss: 0.49445003271102905\n", "val AU ROC: 0.815\n", "val MAP: 0.779\n", "Epoch 400/1000, Loss: 0.4678399860858917, Val Loss: 0.48483508825302124\n", "val AU ROC: 0.818\n", "val MAP: 0.783\n", "Epoch 500/1000, Loss: 0.4578171670436859, Val Loss: 0.47837916016578674\n", "val AU ROC: 0.822\n", "val MAP: 0.787\n", "Epoch 600/1000, Loss: 0.449672132730484, Val Loss: 0.47367215156555176\n", "val AU ROC: 0.824\n", "val MAP: 0.790\n", "Epoch 700/1000, Loss: 0.44261693954467773, Val Loss: 0.4701281487941742\n", "val AU ROC: 0.825\n", "val MAP: 0.791\n", "Epoch 800/1000, Loss: 0.43625712394714355, Val Loss: 0.46743637323379517\n", "val AU ROC: 0.826\n", "val MAP: 0.793\n", "Epoch 900/1000, Loss: 0.43037843704223633, Val Loss: 0.4654019773006439\n", "val AU ROC: 0.826\n", "val MAP: 0.795\n", "Epoch 1000/1000, Loss: 0.42485761642456055, Val Loss: 0.4638913869857788\n", "val AU ROC: 0.827\n", "val MAP: 0.797\n", "Training complete\n" ] } ], "source": [ "emb_train, label_train, emb_val, label_val = fit_classifier_on_embeddings(\n", " embedding_model=hawk,\n", " classifier_model=clf,\n", " train_df=labels_train,\n", " validation_df=labels_val,\n", " steps=1000,\n", " embedding_batch_size=128,\n", " embedding_num_workers=num_workers,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "let's evaluate our shallow classifier on the test set:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "average precision score: 0.7966096322008143\n", "area under ROC: 0.8268707482993197\n" ] } ], "source": [ "# predict classes with the shallow classifier starting from the embeddings\n", "preds = clf(emb_val.to(torch.device(\"cpu\"))).detach().numpy()\n", "\n", "# evaluate with threshold agnostic metrics: MAP and ROC AUC\n", "print(\n", " f\"average precision score: {average_precision_score(label_val,preds,average=None)}\"\n", ")\n", "print(f\"area under ROC: {roc_auc_score(label_val,preds,average=None)}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "to visualize the performance, let's plot histograms of classifier logit scores for positive and negative samples\n", "\n", "it shows that precision is good for scores above >0 (few negatives get high scores), but recall is only moderate (many positive samples get low scores)" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/site-packages/matplotlib_inline/config.py:68: DeprecationWarning: InlineBackend._figure_format_changed is deprecated in traitlets 4.1: use @observe and @unobserve instead.\n", " def _figure_format_changed(self, name, old, new):\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAACXMAAANZCAYAAABH0T+uAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAB7CAAAewgFu0HU+AABpb0lEQVR4nOzdC5SVVf34/z0wDndQQVQYVFARb5nXJPOC11KE0LS0VEhDzcrMzEy/5vWr5gXLsrxg6FdDE80StSwTVMSQUlMxURDlpgJekevA/Nbe/875DzAzXGQ4e4bXa61nzXPOec4z+wwzrlrrvT67rLq6ujoAAAAAAAAAAABQUs1K++0BAAAAAAAAAACIxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABspLvYDGYMGCBeHFF19M55tsskkoL/djAwAAAAAAAACA9VlVVVWYNWtWOt95551Dy5YtP/U9VUmrIIZce+21V6mXAQAAAAAAAAAAZGjcuHFhzz33/NT3sc0iAAAAAAAAAABABkzmWgVxa8WaFd3mm29e0vUAAAAAAAAAAAClNXPmzOJufzX7ok9DzLUKysv//x9TDLkqKytLuh4AAAAAAAAAACDPvujTsM0iAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGyku9AAAAAAAAAACApm7p0qVh7ty54aOPPgqLFi0KS5YsKfWSYL3XvHnz0Lp167DhhhuGli1bhhyIuQAAAAAAAAAAGtDHH38cpk+fHqqrq0u9FKCGqqqqsHDhwvD++++HDh06hM033zyUlZWFUhJzAQAAAAAAAACsw5ArxiJxIhBQ+pir4MMPPwwVFRWhU6dOoZTEXAAAAAAAAAAADbS1Ys2Qq23btmHjjTdO27qVevoPENJ2px988EF499130+NZs2aF9u3bp6irVJqV7DsDAAAAAAAAADRhc+fOXSbkqqysDG3atBFyQSaaN28eOnbsmI6af7elJOYCAAAAAAAAAGgAH330UfE8TuQScUGe2rdvXzz/5JNPSroWMRcAAAAAAAAAQANYtGhR+hojrri1IpCnFi1aFGPLwt9tqYi5AAAAAAAAAAAawJIlS4pbuZnKBfkqKytLf6fR0qVLS7oWMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAACQgWHDhoWysrJ0TJky5VPd64ADDkj3iV9pPMRcAAAAAAAAAAAAGShvqBt/9NFH4eGHHw7PPvtsGD9+fJg+fXqYNWtWmD9/fthwww3DDjvsEA4//PBw8sknh44dO670fk8//XS48cYbw5NPPhneeeeddI9ddtklDBw4MBx33HEN9TEAAAAAAAAAANaJIX+dGJqysw7pWeolNHqxk7n99tvDlltu+aknd7GexVzjxo2rM7KKUdfo0aPTcfXVV4c777wzHHbYYXXe66KLLgqXXnppWLp0afG5GHQ9+uij6bjrrrvCiBEjQsuWLRvkswAAAAAAAAAAwLqIteKxNowaNWqt3IcmtM1it27dwoknnhh+/vOfh/vvvz+MHTs2jBkzJtxzzz3hmGOOCc2bNw+zZ88O/fr1Cy+88EKt97jpppvCxRdfnEKurbfeOgwdOjSFYg888EDo06dPuuahhx4K3/zmNxvyowAAAAAAAAAAADTOyVwxtHrrrbfqfP3YY49NQdaAAQPCokWLUrAVg6+a3nvvvXDuueem8y222CI888wzoVOnTsXX+/btm97/4IMPhuHDh4fBgweHAw44oKE+EgAAAAAAAAAAQOObzBWnbq3Ml7/85bDddtul8yeffHKF12+99dbw4YcfpvOrrrpqmZCr8D1uvPHG4veKWzYCAAAAAAAAALB+uuiii0JZWVk6og8++CD89Kc/DTvuuGNo27Zt2HjjjdOAojg0aGWmTJkSzjrrrPTedu3ahdatW4dtt902nHrqqeHFF19c6fv/8Ic/pDamsrIytGjRIt2jR48eYd999w3/8z//k3amW96wYcOK64/ff/nPdfvtt6fHb775ZvG6mkdNcSBSfG75wUhx97v4fKtWrcLHH3+80s8R2554/V577VXr60uWLEnrikOZunTpkj5rx44dwxe+8IVw3XXXhfnz59d7/3/+85/h5JNPDj179gxt2rQJLVu2TLsB7r777uGMM84If/rTn0J1dXVYXzTYZK5VFX9RowULFqzwWpzcFbVv3z4cddRRtb4//sIffPDB4S9/+Ut47LHH0i9Z4Z4AAAAAAAAAAKyf3njjjXDIIYeESZMmFZ/75JNPwqhRo9IRu5S77rorlJevmM/ccccdaYe4hQsXLvP866+/no6hQ4eGSy+9NJx33nm1xk3HHXdcuPfee5d5Pu5cN3fu3LSup556KjzyyCNh/PjxYV37+te/Hn7729+mVifuonfSSSfVeW1c38SJE4vvW17cta9fv37hhRdeWGE3vjFjxqTj17/+dXjooYdSrLW8IUOGhB/+8Idh6dKlyzw/bdq0dPzrX/9Kg55iDxRjvPVBg03mWhWvvvpqeP7559N5r169VvgFLhSIvXv3DhUVFXXeZ//9909f4x9QKX7JAQAAAAAAAADIy1e/+tUUTp122mnhb3/7W3j22WdThFWIin7/+9+Hc845Z4X3xfBo4MCBqUOJAVGc7BV3nBs7dmy49tpr085yMdj6yU9+kkKl5cXnCiFXnE4Vp23F98cw6a9//Wu6R4zMVmXXu4Jvf/vbaRpY//790+M4ASs+Xv5YFXEyWXx/FGO2+vzud79LX+Nav/a1ry3z2pw5c9LniyFXnMb1ne98J33u+HN+/PHHU+gWp5nF+O1LX/pScXe+gn//+9/FkKt79+7p5/LYY4+F5557LjzxxBPhlltuCccff3ya1rU+WeeTuebNmxemT58eHnzwwfCzn/0sVFVVpee///3vL3NdrPriL35todfyar7+yiuvpF+61RFLvvrMnDlzte4HAAAAAAAAAEBpxagoxkhxSlbBHnvsEY455pi01WGMkH7xi1+kLf522mmn9PrixYvTRK64rV8MuWKE9dnPfrb4/r333jscffTRaTBR7ElijBTvFwOvghiJRZ/73OdS1LT85K+4A90PfvCDNL1qVXXu3DkdG264YXq8wQYbFNe8upo1a5bCrLgF4t///vfwzjvvhE033XSF62Jkdc8996Tzgw46aIVrvve974WpU6eGLbfcMn3OGGTVFLd3LPysJ0+enDqhyy+/vPj6iBEj0veIsVYM5Za//7777htOOeWUFIHFKGx9sU4mc9XczzP+A8TC8eyzz06/DNGPf/zjVNLVFVjFrRTrE/fJLIi/JKsrvr++o649PwEAAAAAAAAAyFPfvn2XCbkK2rVrF26++eZ0HmOi3/zmN8XX/vCHP4QZM2ak8wsuuGCZkKsgxktXX311cahR3LKwprfffjt9/fznP1/rFo4FG2+8cSiVwpaJcdDS3XffXes1MdAq/CyW32JxypQpxdDrl7/85QohV8Guu+4azjjjjGI/VNvPKXZEtcVkBR06dEgB2vpinU/mqin+wsc/jj333HOF1+JelwUr2/Oy5ji1uLcoAKwVj19R6hWsf/qsuKc4NBn+m7Lu+W8KAAAAAMB6bdCgQXW+Fgf77LjjjuHll19OWzAWFM7jwKJvfvObdb4/TpyKkVKcGhXfU3O7xs033zy89tprade6uBVjzaldudhtt93STnj/+c9/0vSyM888s84tFlu1ahUGDBiwwlaUMQSLE7PiFor12W+//dJUrhiGvfXWW2GLLbYo/pyiCRMmhHHjxhm29F/rJFv78pe/XNybM/7whw8fnv6Rn3/++VRAjhw5coX3LFiwoHheUVFR7/3jvpsF8+fPX+31xWle9R1xzQAAAAAAAAAANB61DReqqRAPTZw4MSxatCidv/TSS+lrnDS1ySab1Pne2LLEqVM131Nw0kknpa+vv/562GabbVIUFluZmrvU5aAwbSt2MXGtNS1cuDDcf//96bxfv35pmllN48ePL04mi9PHCjv21XbECWnLT+OKYjMUt4uM32ufffYJRx55ZJqSFn+ecZvL9dU6ibnifp1xn854xD+UuO9m/Ae/44470p6Y/fv3X2GUWsuWLYvnhT+YusR/1IJYA66uuI1jfUehBAQAAAAAAAAAoHHo3Llzva8XtvaL4dD777+fzt97771Vem+02WabLfOeghhvxYlcMXKKk7viNozHH3986NatW4q7zj777NTLlFpcU8Fdd921wuStDz74oNYtFqN33313jb5njL8K4mSwGLlttNFGoaqqKg2DOv3008POO++cfv4nnHBCePLJJ8P6pqQbSsYfehw7F/cf/c53vrPML3fNom9lWyd+8sknq7wlIwAAAAAAAAAATV+cClWK90aXX355mnYVvx544IFpO8Jo0qRJ4brrrkshU5xCVUo9evQIvXv3XmZLxYLC444dO4YvfvGLK7w3brEYxS0kC7v1rcqx/LS0o48+OrzxxhvhpptuCkcddVRxGtrs2bPDnXfembZoHDhwYGqL1hcljbmiOJWrEGT9+c9/Lj4fJ2IVrGzMXNwKsSBWjAAAAAAAAAAArN/eeeedVXo9hltxOlS08cYbr9J7a24ZWHjP8rbccss0oeuxxx5LU67GjBkTzjzzzLRb3eLFi8O3v/3t8Nxzz4VSKkzdiltNFrZO/Oijj9JkrigOaYpbIS4vRl7Rxx9/HLbffvvijn0rO9q0abPCvTp06BAGDx4c7rvvvjTxa8KECeGKK64IXbp0Sa/ffvvt4YYbbgjri5LHXDX3F33zzTeL5z179gzNmzdP5//5z3/qvUfN1+MvCAAAAAAAAAAA67dnn312lV7fdtttQ0VFRTqPwVEUp0XNmjWrzvfGGKsQYhXeU58YRH3+858P119/fXHqVdzeccSIEet0Ytjyjj322LQdZFRYV4yqFixYUOcWi9Guu+6avi5cuLAYga0t22+/ffjxj38cnnnmmWL89fvf/z6sL0oec02fPr3WLRLjH8lee+2VzseOHRsWLVpU5z1Gjx6dvrZo0SLsscceDbpeAAAAAAAAAADyFyc61RdyvfTSS+n84IMPLj5fOI+h1W9/+9s63x8jrA8//HCF96+Kgw46qHgetxNcHXGqVyGiWltDmA499NB0fvfdd6ftDAtRV5wsts8++9T6viOPPLIYlsVArSF069YtDYNak59TY1bymOvee+8tnu+8887LvPblL3+5OL7t/vvvr/X9cQvGv/3tb8Vf9nbt2jXoegEAAAAAAAAAyN+f/vSnWic6zZ07N5x66qnpvFmzZsXzQqtS2N7v8ssvDy+++OIK7586dWr44Q9/mM5bt24dBg0atMzrd955Z6iqqqpzXY8++mjxvHv37qv1mTbffPP0NW5HGLc4XBsK07dmzpyZQq7HH388PT7++OPrnAS23XbbpS0YCxHYddddV+/3iJPOhg8fvsxzDzzwQNp+si5Tp04t7ta3uj+nxqzBYq5hw4YVR67VZciQIeHhhx8u/tD33XffZV4/5ZRT0r6YURyfNmfOnGVeX7JkSdo/NH6NzjnnnLX8KQAAAAAAAAAAaIzi7m4xSDrjjDNSoPTPf/4zTduKzxe2SIyvfeYzn1lmJ7mbb745RUxx+FCcTHXppZeGp59+OvzjH/9IrUt8/4wZM9L111xzTejUqdMy3/eEE04IlZWVqWmJYVfckS5+vz//+c/h7LPPDieeeGJxB7u6tjGsS9yqMYoTtE477bS0FeHrr79ePNZE//79i9sZfve73y12OCtb269//evQo0ePdB4/1/777x+GDh2a1hQ/bxzOdO2114ZDDjkkbLPNNmn7xpriRK+uXbumrR5/85vfpJ35nn/++fRvdfXVV6ef/fz589O18bOuL/6/TS8bwEUXXZT+oY4++ujwhS98IWy99dbplzBWgbFavOuuu8KYMWOW+UNo3rz5MvfYeOONw1VXXZX+Qd58883wuc99Lpx//vlpglf8o4j/qIUa8LjjjgsHHHBAQ30cAAAAAAAAAAAakTiVK+7yduONN6ZjebFpqW2i1BFHHJGirzixK3YuF154YTpqio1LjLxOP/30Wr/3O++8k2KneNQmDjeKE63iVoKr48ADDwx77713CqbiFK3ClogFcXvI1RVDrjiRLLY8hUlZu+yyS9hxxx3rfV/semL7E2OsJ598MjzxxBPpqEv79u1XeG7evHlpV7+aO/vVFCenXXzxxcXd/dYHDRZzRe+991645ZZb0lGXWCLedtttde4fGv8wYrgV/wAmTZoUvvnNb65wzeGHH57uAQAAAAAAAAAAhV3i4jSuOD3rD3/4QxoktMEGG6RQafDgwfVOnjrppJPSpKk4aChui/jWW2+laVhxC8YYVMUJVnEYUW1eeuml8NBDD4WnnnoqtS4x7IqRVLt27UKvXr3CYYcdliKwTTfddLU/U4yb4np+9rOfhQcffDDd/5NPPlmjiKum+LOIMVfNx6tis802SwFX/LxxG8U4heztt98OixcvDhtuuGHYdtttQ+/evUO/fv3Cfvvtt8x74/UjR44Mo0aNChMmTEjvmz17dmjZsmXYcsst0/VxAFTNyWnrg7LqT/uvWYdXX301/UPFAi+OcYu/mHGbxFatWoXOnTuHz372s6Fv376pzov7h65MHFf3q1/9KpV88V7xHzz+ccV9R+NUroY0bdq0YgkZ9+OMARoA64HHryj1CtY/fc4r9Qqg4fhvyrrnvykAAAAAQIm99tproaqqKpSXl6eohYYXd5KLk5yiBkpiaKJeW4O/14ZoihpsMtd2222Xjh/84Adr5X5xz8/Cvp8AAAAAAAAAAABNTbNSLwAAAAAAAAAAAAAxFwAAAAAAAAAAQBbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAQJNw0UUXherq6nRAYyTmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAA1nOjRo0KZWVl6YjnlIaYCwAAAAAAAAAAIAPlpV4AAAAAAAAAAAAhhMevCE1an/NKvYL1zpQpU0L37t3T+W9/+9swcODAUi+JlRBzAQAAAAAAAADAeu6AAw4I1dXVpV7Ges82iwAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAQJNw0UUXhbKysnRECxYsCFdffXXYbbfdQrt27dKx1157hV/+8pehqqqq3nvF98brDjrooLDZZpuFioqK0Llz53DwwQeHoUOHrvT90VNPPRWOPvro9P6WLVuGHj16hNNOOy28/vrrxa0N41rj19rMnDkz3HjjjeErX/lK2HbbbUObNm1CixYtQteuXUP//v3DPffcE5YuXVrre+N9u3fvXnw8aNCg4s+mcMSfV8GoUaOKz8fzgjfffDM0a9YsPX/++eev9DMPHz68eJ+HH3641mvi5z/rrLPCzjvvHDp06BBatWqVfjYDBw4M48ePX+m/yy9+8Yv0M9tkk03CBhtsEDbeeOOw3XbbhS996UvhuuuuC1OmTAmNVXmpFwAAAAAAAAAAAGvbO++8E774xS+G559/fpnnn3322XQ8+uij4YEHHkih0vJeeOGFFEvFkKmmWbNmhcceeywdN910U3jwwQfDpptuWuv3v+qqq8J5550Xqquri8+98cYb6X2/+93vwogRI+pd/5IlS0JlZWWtsdaMGTPCn/70p3TEsOz+++8Pbdu2DQ1hyy23DPvss08K02Kodfnll9d7/V133ZW+xtDq0EMPXeH1a665JvzkJz8JixcvXub5+LN54403wh133BEuuOCCcMkll9Qat8WYbsKECcs8//7776dj4sSJ4c9//nP6+cTv0xiJuQAAAAAAAAAAaHKOOuqoFP1873vfC0ceeWSa3vTqq6+GSy+9NLzyyispxLrlllvCqaeeusLUqP333z98+OGHoX379uGMM85I07y6desW5syZkwKqGGTFICwGX08++WSaDlXT73//+/DjH/84ncfve+6554Z99903PY7XX3nlleFrX/taCp7qUojADjzwwDRxKk6xitd//PHHYfLkyWntY8eODX/961/TGm+//fZl3v/iiy+mqOmwww5Ljy+77LK03pripLFV8fWvfz3FXDG2evrpp8PnP//5Wq+LP58YyUXHHntsKC9fNk2KU9J+9KMfpfPPfOYz4fTTT08TxzbccMP0b/PLX/4yfab4b9SpU6f0b1fTd7/73WLI9Y1vfCP9G3fp0iU0b948hV5xqtcf//jH0JiJuQAAAAAAAAAAaHIK07dqbmEYt1uMcdMOO+yQJnfFLQyXj7lOOumkFHLtuuuu6f0xKqopTpvq27dvOOKII8I//vGPMGzYsPCtb32r+PrChQuLEVJ8b4yTttlmm+LrvXv3Dl/+8pfT1zhJqi4xUIqBU833FsTYLG6b+NOf/jRNsPq///u/NM0qhlEFO+200zLTuuLWjPG5NXHMMcekzxSnacWpYnXFXPfee29x4lYMwGqKEVZhm8a47ngUtsOMdt999xS4xZ//nXfema494YQTwkYbbVTcXjGGdNHZZ59d6+StGO1dfPHF4b333guN1Ypz4gAAAAAAAAAAoJGLU5xqhlwFcVJWDKEK06tiuFUQp2bFyVNRnHS1fMhVELdv/MpXvpLOY8xVU9y6MYZi0UUXXVRrjNWzZ88UM9Unhk61vbemCy+8MK0xTvEqhE4NoWPHjukzF6aOVVVV1bvFYo8ePVKsVtO1116bQq899thjhZCroFmzZuGGG24ILVq0CHPnzl1mK8oYaBVCsf3226/e9cZ/48ZKzAUAAAAAAAAAQJOz/GSomuIUqChGUHHrwIJCELXddtulbQ3rUwiK4gSwmnHT3/72t2KYVN8a4jaBtQVNdVm6dGnaNjFO63rppZfSEbeLrKysTK+/8MILoSEVPsusWbPS1o7Le+utt8KYMWPS+fHHH7/C63Fby+joo4+u93NvuOGGxZ99nGpWMyirqKhI53ESWV1BWWMn5gIAAAAAAAAAoMnp1avXKk1u+vjjj4vn48ePT19jMBWDo/qO73znO+naOC2q5rZ+MbIqTKeKYVJ9a4jX1CfGZnHLwT59+qQtE+NWifFzxdipcDz//PPp2tmzZ4eG1K9fv9CuXbtlJnDVNHz48LTeaPmI7c0330wRWHTeeeet9Gc7/r//Dm+//XbxHnFa11e/+tV0Hid2xallP/rRj8LDDz8cPvjgg9BUiLkAAAAAAAAAAGhyWrduXedrcWpWwZIlS4rn77777hp9r3nz5hXP33///fR1k002Wen76rtmwYIF4YgjjggnnHBCGDVqVJg/f36991rZ659Wq1atwoABA4pbSdb8zDUDr912222FkG5t/FyjX/7yl+HII48sBmJXX311+hnFqV177rlnelxz28zGqLzUCwAAAAAAAAAAgBwUwq5ddtklTcRaVXFi1tp2+eWXh0ceeSSd77///uGMM85IodRmm22WwqpCkBa3e3zyySeLU7EaUpy4dccdd4RPPvkk/PGPfwzHHXdcev7ll18OL774YvGa5dUM5i688MJwzDHHrNL3a9OmzTKP27dvn7bCHDduXPj973+fIrc4mSzeP07zisc111yTYrPevXuHxkjMBQAAAAAAAAAAIaQJT9HcuXPDTjvttEb32GijjdLXwraC9anrmhhm3Xrrrel83333DX//+9+XmSZWU80tHhvaQQcdFDbddNPwzjvvpElchZirMJUrrvFrX/tanT/XaIMNNljjn23BXnvtlY7CNpkx6ho2bFi4//770xSwo48+OkyaNClFb42NbRYBAAAAAAAAACCEsOuuu6avkydPDm+//fYa3WPHHXcs3qOw5WJdEVa8pq7XCt8/TrGqK+SK0dmrr75a5/coKysLa1Pz5s2Lsdajjz4a5syZk8Kz4cOHp+f69OkTunTpssL7evToETp06JDOx4wZs1bX1K5du7T14n333Re+973vpedmzpwZnnrqqdAYibkAAAAAAAAAACCE0K9fv/Q1Bko///nP13h6VbR06dLwu9/9rs7r4jaOdW2NWFVVVTyPWxrWJU7vqnnt8lq2bFk8X7hwYVgbCtsoLl68OG11+PTTT4cpU6Ys81ptEdjhhx9ejMBeeeWV0BAO+u/PPpo9e3ZojMRcAAAAAAAAAAAQQjj00EOL2/ddffXVKVaqz4svvhgefPDBZZ4bMGBA6Ny5czq/6KKL0nZ/y3vttdfCxRdfXOd9N9lkk7Dhhhum8zj1qrYQ69lnnw3/8z//U+/64vaGFRUV6by2dayJPffcM2y77bbF7RULwVoMx+L2hnU577zzUtQVI7evfOUrYdq0aXVeu2TJknTvmtfEKWajR4+ud20xFCvo3r17aIzKS70AAAAAAAAAAADIRYyTYtAVtzr86le/miZoxa8xYIox0rvvvhuee+65FHE988wz4eyzz07b/BXEqOn6668Pxx9/fJoO9bnPfS6ce+65Yd99902vP/HEE+Gqq65KUVO8Zwy7lt8OMW6rGKdc/epXvwr//ve/wxe+8IXwgx/8IF3/4YcfhocffjjceOONoW3btmlbw4kTJ9b6WcrLy1N8Fbc2vO2229I2kp/97GfDBhtskF7feOON07G64tpiqBancr300kvpub59+4b27dvX+Z6dd945XHPNNeGss84KEyZMCDvttFMYPHhwOPDAA8Omm24aFixYkCZ8jR07NowYMSJtlRhjucrKyvT+t956K23juMMOO6Rgbo899ghdu3ZNr02dOjXcc889xfgufsb4c2+MxFwAAAAAAAAAAPBfW2+9dQqK4pSpGCrFaGv56Vs11RYwHXfccWmSVJycNWfOnPCjH/1omddbt24d7r333nDllVemmKvmdogFl19+eYqwnn/++TB+/PgUh9UUI6z77rsvXHjhhXXGXIWJWDE2i+tY/h4//elPU5S1pjFX3CYyxmWF51bm+9//fmjTpk36Gt8Xp5/FozYVFRW1/lxiCBaPuvTq1Svcf//9KwRyjYWYCwAAAAAAAAAgB33OK/UK+K+ePXumiCpOeorBVNzScNasWWn7v7h14XbbbZemZcUJUbvttlut9zj//PPDfvvtF6677ro0wSrGS5tttlk46KCDwg9/+MOw/fbbh5/85Cfp2g4dOqzw/vhcjLni++M6YvQVJ21169YtHHHEEeHMM88sTq2qT7z2scceCz//+c+Ln2Px4sWf6uezzTbbpOll48aNS4832mijcPjhh6/Se7/1rW+Ffv36hZtuuilti/jqq6+GDz74ILRo0SJN2ooTvA455JAU03Xq1Kn4vjjZbNSoUeEvf/lLmogWp3G98847aaJXDNt22WWXcNRRR4WBAwemezVWZdUxkaNecf/N+IcQxV+EVflDAKAJePyKUq9g/eP/oNCU+W/Kuue/KQAAAABAicX4pqqqKgU4cXs8qCkGVTHYmj9/frjgggvCpZdeWuolrddeW4O/14Zoipp96jsAAAAAAAAAAACr5YEHHkghV7T33nuXejlkQswFAAAAAAAAAABr2euvv17na1OmTAk/+MEP0vmmm24aDjvssHW4MnJWXuoFAAAAAAAAAABAU9OrV69w+OGHh759+4Ydd9wxtGnTJrz77rvh8ccfD7/5zW/CBx98kK675ppr0tZ+EPlNAAAAAAAAAACAtWzJkiXhwQcfTEdtmjVrFi677LLwjW98Y52vjXyJuQAAAAAAAAAAYC2LEdcjjzwSnn766fDOO++EOXPmhBYtWoSuXbuGAw44IJxxxhlhp512KvUyyYyYCwAAAAAAAAAA1rK4vWI8YHU0W62rAQAAAAAAAAAAaBBiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAgAbQvHnz9HXJkiWhurq61MsB6hD/PuPfadSsWWlzKjEXAAAAAAAAAEADqKioKIYi8+bNK/VygDosXLiwGFwW/m5LRcwFAAAAAAAAANAA2rdvXzx/7733TOeCTH300UfF8zZt2pR0LWIuAAAAAAAAAIAG0LZt21BWVpbO586dG6ZNmxY++eQTURdkYsmSJWHOnDnpqPl3W0rlJf3uAAAAAAAAAABNVLNmzULXrl3D9OnTU8AVg654xMCrefPmpV4erNeqq6tTzFXTJptsUvJtFsVcAAAAAAAAAAANpF27dssEXVH8WlVVVeqlATV06NAhdOzYMZSamAsAAAAAAAAAoIGDrp49e6apXB999FFYtGjRChOBgHUvTshr3bp12HDDDUPLli1DDsRcAAAAAAAAAADrYMvF9u3bpwOgLs3qfAUAAAAAAAAAAIB1RswFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkoLzUCwAAKHr8ilKvYP3S57xSrwAAAAAAAACowWQuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAmnrMNX78+HDJJZeEQw89NFRWVoYWLVqEtm3bhp49e4ZBgwaFp556aqX3GDZsWCgrK1ulI14LAAAAAAAAAADQGJU31I3322+/8OSTT67w/KJFi8Jrr72WjhhfnXjiieGWW24JFRUVDbUUAAAAAAAAAACA9TfmmjFjRvrapUuXcMwxx4R99903bLHFFmHJkiVh7Nix4dprrw3Tp08Pd9xxR1i8eHH43e9+t9J7/uUvf0n3q0uc/gUAAAAAAAAAANAYNVjM1atXr/C///u/4eijjw7Nmzdf5rW99947nHDCCWGfffYJEydODMOHDw+nnXZamuZVn7g941ZbbdVQSwYAAAAAAAAAACiZZg1145EjR4Zjjz12hZCroFOnTmk6V8GIESMaaikAAAAAAAAAAADrb8y1Kvr06VM8nzRpUimXAgAAAAAAAAAAsP7GXAsXLiye1zXBCwAAAAAAAAAAYH1QXspvPnr06OL59ttvv9LrBw0aFF599dUwe/bs0L59+7DNNtuEgw8+OJx++umha9eua7yOadOm1fv6zJkz1/jeAAAAAAAAAAAAWcdcS5cuDVdeeWXx8bHHHrvS94waNap4PmfOnHT84x//CNdee224/vrrw6mnnrpGa+nWrdsavQ8AAAAAAAAAAKDRx1xDhgwJ48aNS+dHHXVU2H333eu8tkePHuma3r17F8OryZMnh/vuuy+MGDEiLFiwIJx22mmhrKwsDB48eJ19BgAAAAAAAAAAgLWlrLq6ujqUYHvFuD1iVVVV6Ny5c3jxxRfT19p8+OGHaUvFGGrVZuTIkSn0Wrx4cWjdunWYNGlS2Gyzzdb6Not77bVXOp86dWqorKxcrfsD0Eg9fkWpVwANq895pV7B+sV/U9Y9v+MAAAAAAEADis1RYTDV2mqKmoV17OWXXw4DBgxIIVfLli3DvffeW2fIFXXo0KHOkCvq27dvuPDCC9P5vHnzwtChQ1d7TfEHWd+x+eabr/Y9AQAAAAAAAAAAso253njjjXDooYeG999/PzRv3jzcfffdYb/99vvU941bKxaCrzj1CwAAAAAAAAAAoLFZZzHXjBkz0taK8WsMr2677bbQv3//tXLvONmrY8eO6Xz69Olr5Z4AAAAAAAAAAABNLuaaPXt2OOSQQ8LkyZPT4xtuuCGceOKJa/V71LcVIwAAAAAAAAAAQFjfY64PP/wwHHbYYWHChAnp8ZVXXhnOOOOMtfo9Zs2alYKxqEuXLmv13gAAAAAAAAAAAI0+5po3b1444ogjwr/+9a/0+Pzzzw/nnnvuWv8+N998c6iurk7n+++//1q/PwAAAAAAAAAAQKONuRYtWhQGDBgQxowZkx6feeaZ4bLLLlute0yZMiU899xz9V4zcuTIcMkll6TzVq1ahUGDBn2KVQMAAAAAAAAAAJRGeUPd+LjjjguPPvpoOj/wwAPDySefHF566aU6r6+oqAg9e/ZcIebq06dP6N27dzjyyCPDLrvsEjp37pxemzx5chgxYkQ6ClO5rrnmmtC1a9eG+kgAAAAAAAAAAACNL+a6//77i+d///vfw2c+85l6r99yyy1TvFWbsWPHpqMurVu3DkOGDAmDBw/+FCsGAAAAAAAAAABogjHX2rD77ruHO++8M4Vc48ePDzNnzgyzZ88OVVVVYaONNgo77rhjOOigg8Ipp5xSnNgFAAAAAAAAAADQGDVYzFXY+vDTaNeuXfj617+eDgAAAAAAAAAAgKasWakXAAAAAAAAAAAAgJgLAAAAAAAAAAAgC2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyUF7qBQCwGh6/otQrAAAAAAAAAAAaiMlcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAA0NRjrvHjx4dLLrkkHHrooaGysjK0aNEitG3bNvTs2TMMGjQoPPXUU6t1v0ceeSQMGDCgeK/4NT6OzwMAAAAAAAAAADRm5Q114/322y88+eSTKzy/aNGi8Nprr6Vj2LBh4cQTTwy33HJLqKioqPNeS5cuDYMHDw5Dhw5d5vnp06en44EHHginnHJKuOmmm0KzZoaNAQAAAAAAAAAAjU+DlU8zZsxIX7t06RLOPPPMMGLEiDBu3LgwduzYcN1114WuXbum1++4444wcODAeu91/vnnF0OuXXfdNQwfPjzdK36Nj6Nbb701XHDBBQ31cQAAAAAAAAAAABpUWXV1dXVD3Lhv375p6tbRRx8dmjdvvsLrs2fPDvvss0+YOHFiejx69Og0zWt58fUdd9wxVFVVhT322CM88cQToVWrVsXX582bF/bff/+0pWN5eXl45ZVXwjbbbLNWP8u0adNCt27d0vnUqVPT9o4AJfH4FaVeAdCU9Dmv1CtYv/hv+LrndxwAAAAAAGhADdEUNdhkrpEjR4Zjjz221pAr6tSpU7j22muLj+Pkrtpcf/31KeSKbrjhhmVCrqh169bp+SheN2TIkLX4KQAAAAAAAAAAANaNBou5VkWfPn2K55MmTVrh9Tg07I9//GM679WrV9h7771rvU98frvttkvn8foGGjYGAAAAAAAAAADQNGOuhQsXFs9rm+D1xhtvhBkzZqTzuJVifQqvT58+PUyZMmWtrxUAAAAAAAAAAKDJxlyjR48unm+//fYrvD5hwoTieZzMVZ+ar7/yyitrbY0AAAAAAAAAAADrQnkokaVLl4Yrr7yy+PjYY49d4Zpp06YVzysrK+u9X7du3YrnU6dOXa211Pw+tZk5c+Zq3Q8AAAAAAAAAAKDRxFxDhgwJ48aNS+dHHXVU2H333Ve45uOPPy6et23btt77tWnTpng+d+7c1VpLzRAMAAAAAAAAAABgvYm54vaKP/7xj9N5586dw69//etar1uwYEHxvKKiot57tmjRong+f/78tbZWAIAm6/ErSr0CAAAAAAAAoJQx18svvxwGDBgQqqqqQsuWLcO9996bgq7axNcLFi1aVO99Fy5cWDxv1arVaq1pZdsyxm0W99prr9W6JwAAAAAAAAAAQLYx1xtvvBEOPfTQ8P7774fmzZuHu+++O+y33351Xt+uXbtV3jrxk08+WeUtGZdXWVm5WtcDAAAAAAAAAACsbc3COjJjxoxw8MEHp69lZWXhtttuC/3791/lyGratGmrPF2rW7dua2HFAAAAAAAAAAAATSzmmj17djjkkEPC5MmT0+MbbrghnHjiiSt93w477FA8/89//lPvtTVf33777T/VegEAAAAAAAAAAJpczPXhhx+Gww47LEyYMCE9vvLKK8MZZ5yxSu/t3r176NKlSzofPXp0vdc+8cQT6WvXrl3DVltt9anXDQAAAAAAAAAA0GRirnnz5oUjjjgi/Otf/0qPzz///HDuueeu8vvjdoyFrRjj5K1nnnmm1uvi84XJXPH6+D4AAAAAAAAAAIDGpMFirkWLFoUBAwaEMWPGpMdnnnlmuOyyy1b7Pt///vdD8+bN0/l3v/vdMH/+/GVej4/j81F5eXm6HgAAAAAAAAAAoLEpb6gbH3fcceHRRx9N5wceeGA4+eSTw0svvVTn9RUVFaFnz54rPB+fO+ecc9L2jOPHjw/77LNPmu619dZbh0mTJoWrrroqPPfcc+naeN22227bUB8JAAAAAAAAAACgwZRVV1dXN8iNV3Orwy233DJMmTKl1teWLl0avvWtb4XbbrutzvfHWOzmm28OzZqt/WFj06ZNC926dUvnU6dODZWVlWv9ewCsksevKPUKAKDx6HNeqVcAAAAAAAA0YdMaoClqsG0W16YYaA0dOjQ89NBDoX///qFLly5pklf8Gh8//PDD4dZbb22QkAsAAAAAAAAAAKBRb7PYEAO/Dj/88HQAAAAAAAAAAAA0NUZZAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAAAAAAAAABkQcwEAAAAAAAAAAGRAzAUAAAAAAAAAAJABMRcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZKC81AsAGrnHryj1CgAAaud/p6xbfc4r9QoAAAAAAKDRM5kLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACAph5zvfvuu2HkyJHhwgsvDF/60pdCp06dQllZWToGDhy4SvcYNmxY8T0rO+K1AAAAAAAAAAAAjVF5Q9580003bcjbAwAAAAAAAAAANBkNGnPVtMUWW4RevXqFRx99dI3v8Ze//CV06dKlztcrKyvX+N4AAAAAAAAAAABNNuaK2yvuueee6YhTuqZMmRK6d+++xvfr2bNn2GqrrdbqGgEAAAAAAAAAAJp8zHXxxRc35O0BAAAAAAAAAACajGalXgAAAAAAAAAAAABiLgAAAAAAAAAAgCw0qphr0KBBoUuXLqGioiJ06tQp7L333uGCCy4I06dPL/XSAAAAAAAAAAAAPpXy0IiMGjWqeD5nzpx0/OMf/wjXXnttuP7668Opp566RvedNm1ava/PnDlzje4LAAAAAAAAAADQpGKuHj16hKOOOir07t07dOvWLT03efLkcN9994URI0aEBQsWhNNOOy2UlZWFwYMHr/b9C/cEAAAAAAAAAAAolexjrgEDBoSTTjophVo17bnnnuGrX/1qGDlyZAq9Fi9eHM4666zQr1+/sNlmm5VsvQAAAAAAAAAAAGuiWchchw4dVgi5aurbt2+48MIL0/m8efPC0KFDV/t7TJ06td5j3Lhxn+ozAAAAAAAAAAAANPqYa1XErRULwdfo0aNX+/2VlZX1HptvvnkDrBoAAAAAAAAAAKCJxVydO3cOHTt2TOfTp08v9XIAAAAAAAAAAADWz5grqm8rRgAAAAAAAAAAgNw1iZhr1qxZYfbs2em8S5cupV4OAAAAAAAAAADA+hlz3XzzzaG6ujqd77///qVeDgAAAAAAAAAAQNOKuaZMmRKee+65eq8ZOXJkuOSSS9J5q1atwqBBg9bR6gAAAAAAAAAAANae8tCAnnrqqfD6668XHxe2Qozi88OGDVvm+oEDB64Qc/Xp0yf07t07HHnkkWGXXXYJnTt3Tq9Nnjw5jBgxIh2FqVzXXHNN6Nq1a0N+JAAAAAAAAAAAgMYXc916663h9ttvr/W1MWPGpKO+mKtg7Nix6ahL69atw5AhQ8LgwYM/5YoBAAAAAAAAAACaYMz1ae2+++7hzjvvTCHX+PHjw8yZM9N0r6qqqrDRRhuFHXfcMRx00EHhlFNOKU7sAgAAAAAAAAAAaIzKqgt7FFKnadOmhW7duqXzqVOnhsrKylIvCfLx+BWlXgEAADnoc16pVwAAAAAAAI2+KWq2FtYFAAAAAAAAAADApyTmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMlBe6gUAAAAAAAAArI+G/HViqZfAf511SM9SLwEAEpO5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAoKnHXO+++24YOXJkuPDCC8OXvvSl0KlTp1BWVpaOgQMHrvb9HnnkkTBgwIBQWVkZWrRokb7Gx/F5AAAAAAAAAACAxqy8IW++6aabrpX7LF26NAwePDgMHTp0meenT5+ejgceeCCccsop4aabbgrNmhk2BgAAAAAAAAAAND7rrHzaYostwqGHHrpG7z3//POLIdeuu+4ahg8fHsaNG5e+xsfRrbfeGi644IK1umYAAAAAAAAAAIAmMZkrbq+45557piNO6ZoyZUro3r37at1j4sSJ4Zprrknne+yxR3jiiSdCq1at0uN43379+oX9998/jB8/Plx99dXhm9/8Zthmm20a5PMAAAAAAAAAAAA0yslcF198cejbt++n2m7x+uuvD1VVVen8hhtuKIZcBa1bt07PR/G6IUOGfMpVAwAAAAAAAAAANOFtFtdEdXV1+OMf/5jOe/XqFfbee+9ar4vPb7fdduk8Xh/fBwAAAAAAAAAA0JhkHXO98cYbYcaMGek8bqVYn8Lr06dPT9s5AgAAAAAAAAAANCblIWMTJkwonsfJXPWp+forr7wSunfvvsrfZ9q0afW+PnPmzFW+FwAAAAAAAAAAQJOLuWpGVpWVlfVe261bt+L51KlTV+v71HwvAAAAAAAAAABAKWQdc3388cfF87Zt29Z7bZs2bYrnc+fObdB1kbnHryj1CgCARm7s5DmlXgJkp3ePjvVf4H+Hr1t9ziv1CgAAGo0hf51Y6iXwX2cd0rPUSwAAgOxlHXMtWLCgeF5RUVHvtS1atCiez58/f7W+z8omecVtFvfaa6/VuicAAAAAAAAAAECTiblatmxZPF+0aFG91y5cuLB43qpVq9X6PivbwhEAAAAAAAAAAKChNQsZa9eu3SpvnfjJJ5+s8paMAAAAAAAAAAAAuck65qo5MWvatGmrvFVit27dGnRdAAAAAAAAAAAA61XMtcMOOxTP//Of/9R7bc3Xt99++wZdFwAAAAAAAAAAwHoVc3Xv3j106dIlnY8ePbrea5944on0tWvXrmGrrbZaJ+sDAAAAAAAAAABYL2KusrKy0L9//+LkrWeeeabW6+Lzhclc8fr4PgAAAAAAAAAAgMYk65gr+v73vx+aN2+ezr/73e+G+fPnL/N6fByfj8rLy9P1AAAAAAAAAAAAjU15Q978qaeeCq+//nrx8ezZs4vn8flhw4Ytc/3AgQNXuEfPnj3DOeecE6688sowfvz4sM8++4Rzzz03bL311mHSpEnhqquuCs8991y6Nl637bbbNuRHAgAAAAAAAAAAaHwx16233hpuv/32Wl8bM2ZMOlYWc0WXX355ePfdd8Ntt92Wwq2vfe1rK1xz8sknh8suu2wtrRwAAAAAAAAAAGDdyn6bxahZs2Zh6NCh4aGHHgr9+/cPXbp0CRUVFelrfPzwww+ncCxeBwAAAAAAAAAA0Bg16GSuuI3i8lspfhqHH354OgAAAAAAAAAAAJoao6wAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMlJd6AQCsn8ZOnlPqJfBfvXt0LPUSAAAAAAAAADCZCwAAAAAAAAAAIA9iLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMlBe6gUAAAAAAI3fkL9OLPUS+K+zDulZ6iUAAAAAa8hkLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyEB5qRcAAAAArKbHryj1CtYvfc4r9QoAVsuQv04s9RIAABod/xsqH2cd0rPUSwAoKZO5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADIi5AAAAAAAAAAAAMiDmAgAAAAAAAAAAyICYCwAAAAAAAAAAIANiLgAAAAAAAAAAgAyIuQAAAAAAAAAAADIg5gIAAAAAAAAAAMiAmAsAAAAAAAAAACADYi4AAAAAAAAAAIAMiLkAAAAAAAAAAAAyIOYCAAAAAAAAAADIgJgLAAAAAAAAAAAgA2IuAAAAAAAAAACADJSXegEAAAAAAADAujPkrxNLvQQAAOpgMhcAAAAAAAAAAEAGxFwAAAAAAAAAAAAZEHMBAAAAAAAAAABkQMwFAAAAAAAAAACQATEXAAAAAAAAAABABsRcAAAAAAAAAAAAGRBzAQAAAAAAAAAAZEDMBQAAAAAAAAAAkAExFwAAAAAAAAAAQAbEXAAAAMD/a+/eY6wqzz0Av+AAcqsoiqKgqCMFb7VRqSKKqGgNXrAUrIaiWFvv2lOst8aWGk2NSNXyh0pFkdgW0VgM0lhtg6Molqo0akUtCugAR4WqoIxcCidrncOcWmaYEYdZ397zPMnO/vbsb/a8/POx9lq/9X4AAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAkoizNWqVatGPY499tiiSwUAAAAAAAAAACjfMBcAAAAAAAAAAEC5q4gSctFFF8XFF19c7/sdO3Zs1noAAAAAAAAAAABaZJirW7duceCBBxZdBgAAAAAAAAAAQJOzzSIAAAAAAAAAAEAChLkAAAAAAAAAAAASIMwFAAAAAAAAAACQgIooIQ899FBMmzYtFi1aFNttt13stttu0b9//zj33HNj0KBBW/251dXVW3x/2bJlW/3ZAAAAAAAAAAAAZRfmeu211z73esGCBfljypQpMXTo0Jg8eXLssMMOX/hze/bs2YRVAgAAAAAAAAAAlGmYq0OHDnHaaafF8ccfH3369IlOnTrFBx98EFVVVXHXXXfFihUrYvr06XH66afHk08+GW3atCm6ZAAoGXPeXlF0CQBAI/l/uxjPr39zs5/91+DehdQCAAAA5e62Jzf/Hk7zc+4DilMSYa4lS5ZEly5dNvv54MGD47LLLouTTz455s2bl4e77rzzzrj88su/0Oe/++67DW6z2K9fvy9cNwAAAAAAAAAAQFmFueoKcm2y6667xsMPP5x37Fq3bl1MmDDhC4e5evTo0QRVAgAAAAAAAAAAbL3WUQb22WefvEtXZsGCBbF06dKiSwIAAAAAAAAAAGh5Ya7M/vvv/7ltGQEAAAAAAAAAAEpJ2YS5WrVqVXQJAAAAAAAAAAAAW61swlyvvfZa7Xj33XcvtBYAAAAAAAAAAIAWGeZauHBhPPnkk/l43333jT322KPokgAAAAAAAAAAAMorzDVjxoxYv359ve+/9957MWzYsFi7dm3++uKLL27G6gAAAAAAAAAAAJpGRSTusssui3Xr1uWBrSOPPDJ69eoV7du3j+XLl8dTTz0Vd999dz7ODBgwIC655JKiSwYAAAAAAAAAACi/MFdm6dKlMWHChPxRnyzsdc8990S7du2atTYAAAAAAAAAAIAWEea6//77o6qqKubMmRNvv/123oVr5cqV0alTp+jZs2f0798/zjnnnLxrFwAAAAAAAAAAQKlKPsw1cODA/AEAAAAAAAAAAFDOWhddAAAAAAAAAAAAAMJcAAAAAAAAAAAASRDmAgAAAAAAAAAASIAwFwAAAAAAAAAAQAKEuQAAAAAAAAAAABIgzAUAAAAAAAAAAJAAYS4AAAAAAAAAAIAECHMBAAAAAAAAAAAkQJgLAAAAAAAAAAAgAcJcAAAAAAAAAAAACRDmAgAAAAAAAAAASIAwFwAAAAAAAAAAQAIqii4AAACA9M15e0XRJQDU6bYn3yy6hJJ3xDsTm+ZzmuRTyt/ze/6g6BIACuP/bQAAaJjOXAAAAAAAAAAAAAkQ5gIAAAAAAAAAAEiAMBcAAAAAAAAAAEAChLkAAAAAAAAAAAASIMwFAAAAAAAAAACQAGEuAAAAAAAAAACABAhzAQAAAAAAAAAAJECYCwAAAAAAAAAAIAHCXAAAAAAAAAAAAAkQ5gIAAAAAAAAAAEiAMBcAAAAAAAAAAEAChLkAAAAAAAAAAAASIMwFAAAAAAAAAACQAGEuAAAAAAAAAACABAhzAQAAAAAAAAAAJECYCwAAAAAAAAAAIAHCXAAAAAAAAAAAAAkQ5gIAAAAAAAAAAEiAMBcAAAAAAAAAAEAChLkAAAAAAAAAAAASIMwFAAAAAAAAAACQAGEuAAAAAAAAAACABAhzAQAAAAAAAAAAJKCi6AIAAAAA4HNm/aLRU494Z8U2LQWa2hHvTCy6hBbl+T1/UHQJAABQkm578s2iS+D//Nfg3kWXQDPTmQsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACSgougCAJrbnLdXFF0CAABQQo54Z+JmP5szqZBSAKAk/99k23l+zx8UXQIAANDEdOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABFUUXAAAAAAAA28IR70wsugQAvgTrOOXs+T1/UHQJACRKZy4AAAAAAAAAAIAECHMBAAAAAAAAAAAkQJgLAAAAAAAAAAAgAcJcAAAAAAAAAAAACRDmAgAAAAAAAAAASIAwFwAAAAAAAAAAQAKEuQAAAAAAAAAAABIgzAUAAAAAAAAAAJAAYS4AAAAAAAAAAIAECHMBAAAAAAAAAAAkQJgLAAAAAAAAAAAgAcJcAAAAAAAAAAAACRDmAgAAAAAAAAAASIAwFwAAAAAAAAAAQAJKLsy1ePHiGDNmTPTp0yc6duwYO+20Uxx++OExbty4WL16ddHlAQAAAAAAAAAAbJWKKCEzZsyIkSNHxsqVK2t/lgW4Xnjhhfxxzz33xMyZM6OysrLQOgEAAAAAAAAAAMq2M9e8efPizDPPzINcnTp1iptuuimee+65+POf/xzf//738zlvvvlmDBkyJFatWlV0uQAAAAAAAAAAAOXZmeuKK66ImpqaqKioiCeeeCKOPPLI2veOO+642G+//eKqq67KA13jx4+PsWPHFlovAAAAAAAAAABA2XXmmjt3bjzzzDP5+Hvf+97nglybjBkzJvr27ZuP77jjjli3bl2z1wkAAAAAAAAAAFDWYa7p06fXjkePHl3nnNatW8eoUaPy8UcffRSzZs1qtvoAAAAAAAAAAABaRJhr9uzZ+XPHjh3j0EMPrXfewIEDa8fPPvtss9QGAAAAAAAAAADQYsJc8+fPz58rKyujoqKi3nl9+vTZ7HcAAAAAAAAAAABKQf3JqER89tlnsXz58nzco0ePLc7dcccd8+5dn376abz77ruN/hvV1dVbfP/fP2vZsmWN/lwK8sHHRVdA4t7/56qiSwAAAAAA+NI+av/fRZcA25Tz+ZQzazjQWNXVHYougS349xzR+vXro0WEuVat+v+DtE6dOjU4f1OY65NPPmn03+jZs2ej5/br16/RcwEAAAAAALadiUUXAMBWs4YDjXND0QXQaB988EH06tUryn6bxawz1yZt27ZtcH67du3y55qamm1aFwAAAAAAAAAAQFNKvjPX9ttvXzteu3Ztg/PXrFmTP7dv377Rf6OhLRmzQNnrr78eu+66a+yyyy5RUVGxzVuwbeoANnfu3Ojevfs2/XsA5cyaCtB0rKkATceaCtB0rKkATceaCtB0rKnQMqxfvz7vyJU56KCDWkaYq3PnzrXjxmydmG2x2NgtGTfp0aNHg3MqKyujCNmC3pj6AGiYNRWg6VhTAZqONRWg6VhTAZqONRWg6VhTobz1aoKtFUtqm8WsM1fXrl3zcXV19Rbnfvjhh7Vhrp49ezZLfQAAAAAAAAAAAC0izJXZf//98+cFCxbk7cnqk22FuEnfvn2bpTYAAAAAAAAAAIAWE+YaMGBA/px13XrxxRfrnVdVVVU7Puqoo5qlNgAAAAAAAAAAgBYT5ho6dGjt+L777qtzzoYNG2LKlCn5uEuXLjFo0KBmqw8AAAAAAAAAAKBFhLn69esXRx99dD6eNGlSzJkzZ7M548ePj/nz5+fjK664Itq0adPsdQIAAAAAAAAAAGytiigRd9xxR751Yk1NTZx44olx3XXX5d23stdTp06NiRMn5vN69+4dY8aMKbpcAAAAAAAAAACA8gxzff3rX48HH3wwRo4cGStXrszDXP8pC3LNnDkzOnfuXEiNAAAAAAAAAAAAW6vVxo0bN0YJWbx4cd6lKwttVVdXR9u2baOysjKGDx8el156aXTo0KHoEgEAAAAAAAAAAMo/zAUAAAAAAAAAAFCOWhddAAAAAAAAAAAAAMJcAAAAAAAAAAAASRDmAgAAAAAAAAAASIAwFwAAAAAAAAAAQAKEuQAAAAAAAAAAABIgzAUAAAAAAAAAAJAAYS4AAAAAAAAAAIAECHMBAAAAAAAAAAAkQJirxMycOTPGjh0bQ4YMib59+8bOO+8cbdq0iR133DEOPfTQGDNmTLzxxhtFlwlQEhYtWhQTJkyIYcOGxX777RcdOnSI7bffPnr06BFDhw6NqVOnxvr164suE6AkfPLJJ/H000/HrbfeGiNGjIi99947WrVqlT969epVdHkASVm8eHH+/b1Pnz7RsWPH2GmnneLwww+PcePGxerVq4suDyB577//fjz22GPx05/+NE4++eT8HOmmY89zzz236PIASsoLL7wQN9xwQ5x44on5edF27dpFp06donfv3jF69OiYPXt20SUClISVK1fm15Wy7/sDBw6MysrK2GGHHaJt27bRrVu3OPbYY+OWW26JFStWFF0qUAJabdy4cWPRRdA4WaAgC241JJuTHXhfc801zVIXQCm6/vrr46abboqG/hvMLqo9/PDDseeeezZbbQClaNCgQfHUU0/V+d5ee+2VB2gBiJgxY0aMHDkyP8lbl+yiWXYjV3bSF4C6ZaGt+pxzzjkxefLkZq0HoFQdc8wx8cwzzzQ4b9SoUfHrX/86DyQAULc//elPMXjw4AbnZTciPPDAA3HSSSc1S11AaaoougC+mCy9m6V2v/GNb8Q+++wT3bt3zzvJLF26NL94du+998bHH38c1157bXTp0iUuvPDCoksGSNKyZcvyIFfWCeGMM86I448/Pu/OlXXmmj9/fvzqV7+Kv/71r/njhBNOiJdeeim/Iw2Auv17ODbrMHPYYYfFc889l3fsAuB/zZs3L84888yoqanJjy2z7+5ZGDZ7nd29m10ge/PNN/Nu3FmHhM6dOxddMkDyspuvsk6HTzzxRNGlAJSc7NpSZvfdd4/hw4fH0Ucfna+r//rXv2LOnDkxfvz4WLJkSUyZMiXWrVsXv/3tb4suGSBpPXv2zL/nZztqZePsWv6GDRuiuro6bxzwyCOPxPLly+O0006LuXPnxte+9rWiSwYSpTNXickOoLfbbrt631+4cGH+n8OHH34Yu+yySx5W2NJ8gJbq6quvjq5du8ZFF11U50WybL09++yzY9q0afnrn//85/n2DQDUbeLEifl6mnU03NRNJtteMdtKTGcugM93PqioqMi3pj3yyCM/9362zeJVV12Vj3/2s5/F2LFjC6oUIG3ZGpkdd2aPXXfdNT/WzLb5zujMBdB4p5xySt51a9iwYXVeS8oCB0cddVR+w0GmqqoqP6YF4Itfx89Mnz49bzCQyZ6zcBdAXYS5ylDWjevuu+/Ox6+++moccMABRZcEUJKyfcuzu9LWrl0bBx10ULz88stFlwRQUoS5AP5fdsdt1mU7c8EFF8Rdd9212Zzsbt0DDzww7xSbddt+//33o02bNgVUC1BahLkAtp3HHnssTj311Hx82WWX5TsaALD1sq6yb7zxRr7d4gcffFB0OUCiWhddAE3v3zvMfPbZZ4XWAlDKss5dBx98cD5+6623ii4HAIASlt19u8no0aPrnNO6deu8M0Lmo48+ilmzZjVbfQAAUJdsu7BNnCMFaLpr+a7jA1sizFVmampq4tFHH609Cdy7d++iSwIoaWvWrMmfbVkLAMCXMXv27Py5Y8eOceihh9Y7b+DAgbXjZ599tllqAwCAhs6PZpwjBfhyso5cf/vb32o7dAHUR5irDKxbty7eeeedmDp1avTv3z/+8Y9/5D8/77zzPtelC4AvJtvWJtviJtO3b9+iywEAoIRtOq6srKyMioqKeuf9+8ncTb8DAABFqaqqqh07Rwrwxa1evTq/fv/LX/4yv4Fr/fr1+c9/+MMfFl0akLD6zx6StEWLFsXee+9d7/snnXRSjB8/vllrAig348aNqz2oHjFiRNHlAABQorKtE5YvX56Pe/ToscW5O+64Y96969NPP4133323mSoEAIDNbdiwIW6++eba186RAjTO5MmTY/To0fW+f80118TZZ5/drDUBpUVnrjKz8847x4MPPhgzZ86Mr3zlK0WXA1Cy/vKXv8Ttt99ee8HtoosuKrokAABK1KpVq2rHnTp1anB+FubKfPLJJ9u0LgAA2JLbbrst5s6dm4+/9a1vbXG7cAAadsghh+Tr6i9+8Yto1apV0eUACdOZq0Ttscce8corr+TjrGvMkiVL4vHHH49JkybFhRdeGG+99VZce+21RZcJUJLee++9+Pa3v52vr9nB9P333x8dOnQouiwAAEq4M9cmbdu2bXB+u3bt8ueampptWhcAAGxpe8Wsc0ymW7duceeddxZdEkDJGDp0aBx22GG13+2za/fTpk2L3//+93HWWWflzQROOeWUossEEqYz1zaQXfj/so+s9eKWtGnTJg488MD8kSV4hwwZEhMmTIjnn38+//3rrrsuzjvvvGb7NwOU8pr6n10TsjW1uro6f521ET/uuOO24b8QoHzXVAD+1/bbb187Xrt2bYPz16xZkz+3b99+m9YFAAB1+fvf/x5nnHFGfrNrdiz70EMP5YEuABqnS5cutdfyDz/88PjOd74TjzzySEyZMiXefvvtOP30051nBbZImKvMHHzwwXHjjTfm4/vuuy+eeOKJoksCKKmOCdkB9Isvvpi/vvLKK+Oqq64quiwAAEpc586da8eN2Trx008/bfSWjAAA0JQWLlwYJ554Ynz44Yex3XbbxdSpU+OYY44puiyAsvDd7343hg8fHhs2bIhLL700/vnPfxZdEpAo2yxuA/Pnz//Sn9G9e/et/t0siHDxxRfn44cffjg/6AYoVc21pmZ3mY0YMSJmzZqVvz7//PNj3LhxX/pvA6Sk6ONUgJYq62bQtWvXWLFiRW0H2PpkF802hbl69uzZTBUCAEDE0qVL44QTTsifs+7c9957b37NCYCmk62r2ZaL2Xf/xx9/PM4+++yiSwISJMy1DfTp06fQv7/LLrvUjhcvXlxoLQClsKZmd0Bkd0PMmDEjf33mmWfG3Xffvc3/LkBLO04FaMn233//eOaZZ2LBggX5jQQVFXWfknn99ddrx3379m3GCgEAaMmWL18egwcPzrf/ykyYMCFGjRpVdFkAZce1fKAxbLNYhpYsWVI7tiUDQMMuuOCCvF145tRTT40HHnggWrf2XyQAAE1nwIAB+XN25+2mbb3rUlVVVTs+6qijmqU2AABato8//jhOOumkeO211/LXN998c1xyySVFlwVQllzLBxrDleoy9NBDD9WODzrooEJrAUjdj370o7jnnnvy8fHHH5+vofV1SQAAgK01dOjQ2vF9991Xb8fYKVOm5OMuXbrEoEGDmq0+AABaptWrV8eQIUPipZdeyl//5Cc/iauvvrrosgDKlmv5QGMIc5WQ6dOnx7Jly7Y45+mnn44bbrghH2dhhLPOOquZqgMoPWPHjo3bbrstH/fv3z8effTRaNeuXdFlAQBQhvr16xdHH310Pp40aVLMmTNnsznjx4+P+fPn5+Mrrrgi2rRp0+x1AgDQcqxduzbOOOOMePbZZ2uPQW+88caiywIoSZMnT47PPvtsi3Oya1J/+MMf8vHee+9de54A4D+12rhx48bNfkqSzj333Pjd736X3yGRdY854IAD8jt116xZE2+99VbMmDEjpk2blt/Jm8lCXddff33RZQMkacKECXH55Zfn4z322CMefPDB2GGHHbb4O1/96lddUAOox4IFC2L27Nmf+9mVV14ZK1asiK5du8att976ufe++c1vxm677dbMVQIUa968efnWiTU1NflWCtddd13efSt7nW37PXHixHxe796944UXXojOnTsXXTJAkrLjzuz4c5Ply5fHj3/843ycrbPnn3/+ZudVAdjcsGHD4pFHHsnHxx13XNx+++3RqlWreue3bds2P1YFYHO9evWKVatW5WvrgAEDYt99982/+2c/e+WVV+I3v/lNbXg2W09nzpwZJ5xwQtFlA4kS5ioh2UmH+++/v8F57du3z++cyLYOA6Buxx57bFRVVX2h31m4cGF+MA5A3XeejR49utHzZ82ala/FAC1NdiPWyJEjY+XKlXW+n10cy07oVlZWNnttAOV2nnQTp8AB6ral4FZd9tprr1i0aNE2qweglGXXjxYvXtzgvB49esS9994bgwcPbpa6gNJUUXQBNN4tt9wSAwcOzLdSfPXVV+O9996L999/P1q3bh077bRT3qkru3Ni1KhR0b1796LLBQAAAP7DqaeeGi+//HLccccdeWiruro6vyM3C28NHz48Lr300ujQoUPRZQIAAABfwB//+Mf8e37WfSvrIptdy892LcgasXTr1i0OOeSQOOWUU2LEiBG+9wMN0pkLAAAAAAAAAAAgAa2LLgAAAAAAAAAAAABhLgAAAAAAAAAAgCQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAkABhLgAAAAAAAAAAgAQIcwEAAAAAAAAAACRAmAsAAAAAAAAAACABwlwAAAAAAAAAAAAJEOYCAAAAAAAAAABIgDAXAAAAAAAAAABAAoS5AAAAAAAAAAAAEiDMBQAAAAAAAAAAEMX7Hyp4yC4ng23oAAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": { "image/png": { "height": 428, "width": 1209 } }, "output_type": "display_data" } ], "source": [ "plt.hist(preds[label_val == 1], bins=20, alpha=0.5, label=\"positives\")\n", "plt.hist(preds[label_val == 0], bins=20, alpha=0.5, label=\"negatives\")\n", "plt.legend()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, we can embed the training and validation sets first, then train as many different variants as we want.\n", "\n", "(note that the `fit_classifier_on_embeddings` returns the embeddings on the training and validation set, so if you've already run that function you don't need to re-generate the embeddings)\n", "\n", "Generally, embedding may take a while for large datasets, but training the shallow classifier will be very fast because the network is small and there is no preprocessing or data loading. \n", "\n", "For example, here we compare fitting classifiers with one or two hidden layers on the same data:" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "# uncomment the lines below to generate training and validation set embeddings, if you don't have them from the previous cells\n", "# emb_train = hawk.embed(labels_train, return_dfs=False, batch_size=128, num_workers=num_workers)\n", "# emb_val = hawk.embed(labels_val, return_dfs=False, batch_size=128, num_workers=num_workers)" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:110: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_features = torch.tensor(train_features, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:115: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " validation_features = torch.tensor(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 100/1000, Loss: 0.42801418900489807, Val Loss: 0.46339496970176697\n", "val AU ROC: 0.827\n", "val MAP: 0.796\n", "Epoch 200/1000, Loss: 0.3507393002510071, Val Loss: 0.4778691232204437\n", "val AU ROC: 0.808\n", "val MAP: 0.787\n", "Epoch 300/1000, Loss: 0.27483296394348145, Val Loss: 0.5363744497299194\n", "val AU ROC: 0.787\n", "val MAP: 0.774\n", "Epoch 400/1000, Loss: 0.2075444459915161, Val Loss: 0.6220154166221619\n", "val AU ROC: 0.769\n", "val MAP: 0.760\n", "Epoch 500/1000, Loss: 0.15723513066768646, Val Loss: 0.7255039811134338\n", "val AU ROC: 0.758\n", "val MAP: 0.749\n", "Epoch 600/1000, Loss: 0.12125258147716522, Val Loss: 0.8389410972595215\n", "val AU ROC: 0.753\n", "val MAP: 0.743\n", "Epoch 700/1000, Loss: 0.09492244571447372, Val Loss: 0.9559184312820435\n", "val AU ROC: 0.749\n", "val MAP: 0.738\n", "Epoch 800/1000, Loss: 0.07483015209436417, Val Loss: 1.073399305343628\n", "val AU ROC: 0.745\n", "val MAP: 0.733\n", "Epoch 900/1000, Loss: 0.05932421237230301, Val Loss: 1.193526268005371\n", "val AU ROC: 0.740\n", "val MAP: 0.726\n", "Epoch 1000/1000, Loss: 0.047229062765836716, Val Loss: 1.3133726119995117\n", "val AU ROC: 0.737\n", "val MAP: 0.720\n", "Training complete\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:110: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_features = torch.tensor(train_features, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:115: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " validation_features = torch.tensor(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 100/1000, Loss: 0.352482408285141, Val Loss: 0.48463407158851624\n", "val AU ROC: 0.805\n", "val MAP: 0.784\n", "Epoch 200/1000, Loss: 0.18686117231845856, Val Loss: 0.7288596630096436\n", "val AU ROC: 0.758\n", "val MAP: 0.745\n", "Epoch 300/1000, Loss: 0.09721919149160385, Val Loss: 1.0628374814987183\n", "val AU ROC: 0.739\n", "val MAP: 0.724\n", "Epoch 400/1000, Loss: 0.0494694784283638, Val Loss: 1.4031246900558472\n", "val AU ROC: 0.732\n", "val MAP: 0.712\n", "Epoch 500/1000, Loss: 0.02575032413005829, Val Loss: 1.736872673034668\n", "val AU ROC: 0.728\n", "val MAP: 0.705\n", "Epoch 600/1000, Loss: 0.013688751496374607, Val Loss: 2.0353024005889893\n", "val AU ROC: 0.727\n", "val MAP: 0.702\n", "Epoch 700/1000, Loss: 0.007962996140122414, Val Loss: 2.2820372581481934\n", "val AU ROC: 0.725\n", "val MAP: 0.700\n", "Epoch 800/1000, Loss: 0.0050779590383172035, Val Loss: 2.4839770793914795\n", "val AU ROC: 0.722\n", "val MAP: 0.696\n", "Epoch 900/1000, Loss: 0.003480414394289255, Val Loss: 2.651313066482544\n", "val AU ROC: 0.722\n", "val MAP: 0.695\n", "Epoch 1000/1000, Loss: 0.002518742810934782, Val Loss: 2.7919795513153076\n", "val AU ROC: 0.721\n", "val MAP: 0.693\n", "Training complete\n" ] } ], "source": [ "# define classifier with one hidden layer, and fit\n", "classifier_model_1 = MLPClassifier(2048, 1, hidden_layer_sizes=(100,))\n", "classifier_model_1.fit(\n", " emb_train, labels_train.values, emb_val, labels_val.values, steps=1000\n", ")\n", "\n", "# define classifier with two hidden layers, and fit\n", "classifier_model_2 = MLPClassifier(2048, 1, hidden_layer_sizes=(100, 100))\n", "classifier_model_2.fit(\n", " emb_train, labels_train.values, emb_val, labels_val.values, steps=1000\n", ")" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "classifier_model_1 area under ROC: 0.7372260015117157\n", "classifier_model_2 area under ROC: 0.7212396069538927\n" ] } ], "source": [ "# evaluate\n", "preds = classifier_model_1(emb_val)\n", "print(\n", " f\"classifier_model_1 area under ROC: {roc_auc_score(labels_val.values,preds.detach().numpy(),average=None)}\"\n", ")\n", "\n", "preds = classifier_model_2(emb_val)\n", "print(\n", " f\"classifier_model_2 area under ROC: {roc_auc_score(labels_val.values,preds.detach().numpy(),average=None)}\"\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### train on variants of the embeddings generated with audio-space augmentations\n", "\n", "The `fit_classifier_on_embeddings` function supports generating variants of training samples with augmentation via the parameter `n_augmentation_variants`. The default 0 does not perform augentation. Specifying a positive integer tells the function to generate each sample n times using stochastic augmentation. The specific augmentations performed are defined by the embedding model's `.preprocessor`. \n", "\n", "We can also generate the augmented samples directly using `opensoundscape.ml.shallow_classifier.augmented_embed`, similarly to how we generated embeddings above then trained various models on them. Note that preprocessing and sample loading is repeated for each iteration of augmented data creation, so augmented_embed will take `n_augmentation_variants` times longer than embedding without augmentation. The benefit is that augmenting the audio samples before embedding tends to improve model performance more than simply augmenting the embeddings themselves (e.g. by adding random noise). \n", "\n", "For the sake of speed, we demonstrate augmented embedding here on only a subset of the training data" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "9657c7e5c0f44cd89f128e51163b1204", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4 [00:00\n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n" ] } ], "source": [ "from opensoundscape.ml.shallow_classifier import augmented_embed\n", "\n", "train_emb_aug, train_label_aug = augmented_embed(\n", " hawk,\n", " labels_train.sample(512),\n", " batch_size=128,\n", " num_workers=num_workers,\n", " n_augmentation_variants=4,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "we embed the validation set as normal, without any augmentation" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "# uncomment and run if you don't already have emb_val from previous steps\n", "# emb_val = hawk.embed(labels_val, return_dfs=False, batch_size=128, num_workers=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "fitting the classifier on the augmented variants' embeddings looks the same as before:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:110: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_features = torch.tensor(train_features, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:111: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " train_labels = torch.tensor(train_labels, dtype=torch.float32, device=device)\n", "/Users/SML161/opensoundscape/opensoundscape/ml/shallow_classifier.py:115: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " validation_features = torch.tensor(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Epoch 100/1000, Loss: 0.6358025074005127, Val Loss: 0.5764970183372498\n", "val AU ROC: 0.810\n", "val MAP: 0.777\n", "Epoch 200/1000, Loss: 0.6124716997146606, Val Loss: 0.532778263092041\n", "val AU ROC: 0.815\n", "val MAP: 0.784\n", "Epoch 300/1000, Loss: 0.5981889367103577, Val Loss: 0.5130545496940613\n", "val AU ROC: 0.818\n", "val MAP: 0.789\n", "Epoch 400/1000, Loss: 0.5879126191139221, Val Loss: 0.5014684796333313\n", "val AU ROC: 0.819\n", "val MAP: 0.791\n", "Epoch 500/1000, Loss: 0.5796589851379395, Val Loss: 0.4930368959903717\n", "val AU ROC: 0.822\n", "val MAP: 0.792\n", "Epoch 600/1000, Loss: 0.5726227760314941, Val Loss: 0.4864462912082672\n", "val AU ROC: 0.824\n", "val MAP: 0.795\n", "Epoch 700/1000, Loss: 0.5664262175559998, Val Loss: 0.4813237488269806\n", "val AU ROC: 0.826\n", "val MAP: 0.796\n", "Epoch 800/1000, Loss: 0.560860276222229, Val Loss: 0.4774751663208008\n", "val AU ROC: 0.827\n", "val MAP: 0.797\n", "Epoch 900/1000, Loss: 0.5557926893234253, Val Loss: 0.4747283160686493\n", "val AU ROC: 0.828\n", "val MAP: 0.797\n", "Epoch 1000/1000, Loss: 0.5511319041252136, Val Loss: 0.47291892766952515\n", "val AU ROC: 0.828\n", "val MAP: 0.798\n", "Training complete\n" ] } ], "source": [ "classifier_model = MLPClassifier(2048, 1, hidden_layer_sizes=())\n", "quick_fit(\n", " classifier_model,\n", " train_emb_aug,\n", " train_label_aug,\n", " emb_val,\n", " labels_val.values,\n", " steps=1000,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "evaluate:" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/var/folders/d8/265wdp1n0bn_r85dh3pp95fh0000gq/T/ipykernel_59299/1566684614.py:1: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n", " preds = classifier_model(torch.tensor(emb_val).to(torch.device(\"cpu\")))\n" ] }, { "data": { "text/plain": [ "0.8280423280423281" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "preds = classifier_model(torch.tensor(emb_val).to(torch.device(\"cpu\")))\n", "roc_auc_score(labels_val.values, preds.detach().numpy(), average=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit SKLearn Classifiers on embeddings\n", "scikit-learn provides various classification algorithms as alternatives to the MLPClassifier implemented in OpenSoundscape via PyTorch. It's straightforward to fit any sklearn model on embeddings: " ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.7523809523809524" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.ensemble import RandomForestClassifier\n", "\n", "# initialize a random forest class from sklearn\n", "rf = RandomForestClassifier()\n", "\n", "# fit the model on training set embeddings\n", "rf.fit(emb_train, labels_train.values[:, 0])\n", "\n", "# evaluate on the validation set\n", "preds = rf.predict(emb_val)\n", "roc_auc_score(labels_val.values, preds, average=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "here's another example with K nearest neighbors classification:" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.7396825396825397" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.neighbors import KNeighborsClassifier\n", "\n", "# initialize classifier\n", "knc = KNeighborsClassifier()\n", "\n", "# fit on training set embeddings\n", "knc.fit(emb_train, labels_train.values[:, 0])\n", "\n", "# evaluate on validation set\n", "preds = knc.predict(emb_val)\n", "roc_auc_score(labels_val.values, preds, average=None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit a classifier that is a layer in an exisisting OpenSoundscape model\n", "\n", "If you have a fully connected layer at the end of an existing OpenSoundscape model, training that layer works similarly to training a separate MLPClassifier object. We can use the `quick_fit` function to train the layer on pre-generated embeddings (output of previous network layer) to avoid the slow-down associated with preprocessing samples for every training step. \n", "\n", "For example, let's load up a CNN trained in OpenSoundscape from the model zoo. This CNN was trained to detect the A and B call types of Rana Sierrae vocalizations. It has a resnet18 architecture that ends with a fully connected classifier layer. " ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f7191818709e417b8b1f1118c07f7fe8", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/8 [00:00\n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "48989c96337544c4a21d6e57300c7997", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3 [00:00\n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n", "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.9/tempfile.py:821: ResourceWarning: Implicitly cleaning up \n", " _warnings.warn(warn_message, ResourceWarning)\n" ] } ], "source": [ "import bioacoustics_model_zoo as bmz\n", "\n", "rana_sierrae_cnn = bmz.RanaSierraeCNN()\n", "train_emb = rana_sierrae_cnn.embed(\n", " labels_train, return_dfs=False, batch_size=128, num_workers=num_workers\n", ")\n", "val_emb = rana_sierrae_cnn.embed(\n", " labels_val, return_dfs=False, batch_size=128, num_workers=num_workers\n", ")" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 100/1000, Loss: 0.4183506369590759, Val Loss: 0.47312527894973755\n", "val AU ROC: 0.845\n", "val MAP: 0.775\n", "Epoch 200/1000, Loss: 0.3923477232456207, Val Loss: 0.4774077236652374\n", "val AU ROC: 0.846\n", "val MAP: 0.779\n", "Epoch 300/1000, Loss: 0.3722660541534424, Val Loss: 0.4802170395851135\n", "val AU ROC: 0.844\n", "val MAP: 0.777\n", "Epoch 400/1000, Loss: 0.3556406795978546, Val Loss: 0.4827151894569397\n", "val AU ROC: 0.842\n", "val MAP: 0.778\n", "Epoch 500/1000, Loss: 0.3414366841316223, Val Loss: 0.4857030510902405\n", "val AU ROC: 0.839\n", "val MAP: 0.777\n", "Epoch 600/1000, Loss: 0.3290245831012726, Val Loss: 0.4892197549343109\n", "val AU ROC: 0.838\n", "val MAP: 0.778\n", "Epoch 700/1000, Loss: 0.3179768919944763, Val Loss: 0.49323374032974243\n", "val AU ROC: 0.836\n", "val MAP: 0.779\n", "Epoch 800/1000, Loss: 0.3079927861690521, Val Loss: 0.4977739453315735\n", "val AU ROC: 0.834\n", "val MAP: 0.780\n", "Epoch 900/1000, Loss: 0.2988574802875519, Val Loss: 0.5028876066207886\n", "val AU ROC: 0.833\n", "val MAP: 0.780\n", "Epoch 1000/1000, Loss: 0.29041510820388794, Val Loss: 0.5086027979850769\n", "val AU ROC: 0.831\n", "val MAP: 0.779\n", "Training complete\n" ] } ], "source": [ "# modify the last layer of the CNN to have a single output for the class 'A'\n", "rana_sierrae_cnn.change_classes([\"A\"]) # replace fc layer with 1-output layer\n", "\n", "# fit the fc layer within the opso CNN by passing the layer to the `quick_fit` function\n", "quick_fit(\n", " rana_sierrae_cnn.network.fc,\n", " train_emb,\n", " labels_train.values,\n", " val_emb,\n", " labels_val.values,\n", " steps=1000,\n", ")" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "01931c9683d54b009ad7eb0b753fbfb8", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/3 [00:00