{ "cells": [ { "cell_type": "markdown", "id": "7416bd7c", "metadata": {}, "source": [ "# Agile Bioacoustic Modeling with SongSpace\n", "\n", "SongSpace provides a workflow for active or \"agile\" learning for bioacoustics data. Embed audio into a databse, query the database with vector search or classifieres, and select clips for active learning review or final verification for ecological analyses. \n", "\n", "Embeddings are saved in a HopLite database. The same folder storing the (sql) embedding database will also store classifiers and tables for labeled datasets. The full workspace can be saved and loaded with ss.save(path) and SongSpace.load(path). \n" ] }, { "cell_type": "markdown", "id": "7e4c6df3", "metadata": {}, "source": [ "## Run this tutorial\n", "\n", "If running in Colab, uncomment the installation line below." ] }, { "cell_type": "code", "execution_count": 1, "id": "ab51e7e7", "metadata": {}, "outputs": [], "source": [ "# if 'google.colab' in str(get_ipython()):\n", "# %pip install \"opensoundscape==0.12.1\" \"bioacoustics-model-zoo==0.12.0\"" ] }, { "cell_type": "code", "execution_count": 2, "id": "352a2a8b", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "\n", "import bioacoustics_model_zoo as bmz\n", "\n", "from opensoundscape.annotations import BoxedAnnotations\n", "from opensoundscape.vector_database import load_or_create_hoplite_usearch_db\n", "from opensoundscape.ml.song_space import SongSpace\n", "from opensoundscape.ml.shallow_classifier import select_from_hoplite\n", "from opensoundscape.visualization import annotate, inspect" ] }, { "cell_type": "markdown", "id": "1f281981", "metadata": {}, "source": [ "#### Download example files\n", "Download a set of aquatic soundscape recordings with annotations of _Rana sierrae_ vocalizations\n", "\n", "Option 1: run the cell below\n", "\n", "- if you get a 403 error, DataDryad suspects you are a bot. Use Option 2. \n", "\n", "Option 2:\n", "\n", "- Download and unzip the `rana_sierrae_2022.zip` folder containing audio and annotations from this [public Dryad dataset](https://datadryad.org/stash/dataset/doi:10.5061/dryad.9s4mw6mn3#readme)\n", "- Move the unzipped `rana_sierrae_2022` folder into the current folder" ] }, { "cell_type": "code", "execution_count": 3, "id": "1d43f6c3", "metadata": {}, "outputs": [], "source": [ "# # Note: the \"!\" preceding each line below allows us to run bash commands in a Jupyter notebook\n", "# # If you are not running this code in a notebook, input these commands into your terminal instead\n", "# !wget -O rana_sierrae_2022.zip https://datadryad.org/stash/downloads/file_stream/2722802;\n", "# !unzip rana_sierrae_2022;" ] }, { "cell_type": "markdown", "id": "f1b666b8", "metadata": {}, "source": [ "#### Prepare audio data\n", "See the train_cnn.ipynb tutorial for step-by-step walkthrough of this process, or just run the cells below to prepare a training set." ] }, { "cell_type": "code", "execution_count": 4, "id": "f3765908", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/annotations.py:347: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n", " all_annotations_df = pd.concat(all_file_dfs).reset_index(drop=True)\n" ] } ], "source": [ "# Set this variable to specify where the folder `rana_sierrae_2022` is located:\n", "dataset_path = Path(\"./rana_sierrae_2022/\")\n", "\n", "# let's generate clip labels of 5s duration (to match HawkEars) using the raven annotations\n", "# and some utility functions from opensoundscape\n", "from opensoundscape.annotations import BoxedAnnotations\n", "\n", "audio_and_raven_files = pd.read_csv(f\"{dataset_path}/audio_and_raven_files.csv\")\n", "# update the paths to where we have the audio and raven files stored\n", "audio_and_raven_files[\"audio\"] = audio_and_raven_files[\"audio\"].apply(\n", " lambda x: f\"{dataset_path}/{x}\"\n", ")\n", "audio_and_raven_files[\"raven\"] = audio_and_raven_files[\"raven\"].apply(\n", " lambda x: f\"{dataset_path}/{x}\"\n", ")\n", "\n", "annotations = BoxedAnnotations.from_raven_files(\n", " raven_files=audio_and_raven_files[\"raven\"],\n", " audio_files=audio_and_raven_files[\"audio\"],\n", " annotation_column=\"annotation\",\n", ")\n", "# generate labels for 5s clips, including any labels that overlap by at least 0.2 seconds\n", "labels = annotations.clip_labels(\n", " clip_duration=3, min_label_overlap=0.2, final_clip=None\n", ")" ] }, { "cell_type": "markdown", "id": "0b83317b", "metadata": {}, "source": [ "## Prepare labels" ] }, { "cell_type": "code", "execution_count": 5, "id": "2a831276", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/annotations.py:347: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n", " all_annotations_df = pd.concat(all_file_dfs).reset_index(drop=True)\n" ] } ], "source": [ "dataset_path = Path(\"./rana_sierrae_2022/\")\n", "audio_and_raven_files = pd.read_csv(dataset_path / \"audio_and_raven_files.csv\")\n", "audio_and_raven_files[\"audio\"] = audio_and_raven_files[\"audio\"].apply(\n", " lambda x: str(dataset_path / x)\n", ")\n", "audio_and_raven_files[\"raven\"] = audio_and_raven_files[\"raven\"].apply(\n", " lambda x: str(dataset_path / x)\n", ")\n", "\n", "annotations = BoxedAnnotations.from_raven_files(\n", " raven_files=audio_and_raven_files[\"raven\"],\n", " audio_files=audio_and_raven_files[\"audio\"],\n", " annotation_column=\"annotation\",\n", ")\n", "\n", "labels = annotations.clip_labels(clip_duration=3, min_label_overlap=0.2)" ] }, { "cell_type": "code", "execution_count": 6, "id": "0ff5ac69", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "seed_train: (4, 1)\n", "validation: (536, 1)\n", "pool: (2148, 1)\n" ] } ], "source": [ "target_source_class = \"C\"\n", "target_model_class = \"RanaSierrae_C\"\n", "\n", "# start with one recording of target class\n", "binary_labels = labels[[target_source_class]].rename(\n", " columns={target_source_class: target_model_class}\n", ")\n", "seed_train = binary_labels.loc[\n", " [\"rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220623_060000_0-10s.mp3\"]\n", "]\n", "other = binary_labels.drop(seed_train.index)\n", "validation, unlabeled = train_test_split(other, test_size=0.8, random_state=0)\n", "\n", "print(\"seed_train:\", seed_train.shape)\n", "print(\"validation:\", validation.shape)\n", "print(\"pool:\", unlabeled.shape)" ] }, { "cell_type": "markdown", "id": "1876817c", "metadata": {}, "source": [ "All audio clips from the single audio file we'll start with for positives:" ] }, { "cell_type": "code", "execution_count": 7, "id": "146752f6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "_ = inspect(seed_train, bandpass_range=(0, 2500))" ] }, { "cell_type": "markdown", "id": "390830fa", "metadata": {}, "source": [ "## Build database and SongSpace\n", "\n", "The default Machine Learning embedding model is Perch V2, a TensorFlow model provided via the Bioacoustics Model Zoo. If you wish to avoid installing TensorFlow, consider specifying `feature_extractor='perch2_onnx'` for an ONNX formatted version (currently CPU only), or selecting another model such as 'birdnet' (TFLite) or 'bs-convnext' (PyTorch). Alternatively, advanced users can provide a custom embedding model with a .embed() method matching the opensoundscape.CNN.embed() API.\n", "\n", "It is critical to maintain consistency of the machine learning model within a single SongSpace: you cannot change embedding models (or model versions) for different datasets or tasks within a single SongSpace. You'll need to make a new SongSpace and re-ingest all audio files if you change backbone embedding models. " ] }, { "cell_type": "code", "execution_count": 10, "id": "8fec817a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connected to existing database with 2,691 embeddings from 672 files.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.13/site-packages/bioacoustics_model_zoo/perch_v2.py:215: UserWarning: Disabling TensorFlow's XLA compilation (setting tf.config.optimizer.set_jit(False)) because otherwise TF models on Mac hang at runtime as of Tensorflow 2.21.0\n", " warnings.warn(\n" ] } ], "source": [ "ss = SongSpace(\"./Perch2SongSpace\")" ] }, { "cell_type": "code", "execution_count": 11, "id": "cb59c932", "metadata": {}, "outputs": [], "source": [ "import opensoundscape as opso\n", "\n", "opso.set_seed(0)" ] }, { "cell_type": "code", "execution_count": 12, "id": "c97de950", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "all samples already have embeddings in the database\n", "all samples already have embeddings in the database\n", "all samples already have embeddings in the database\n" ] }, { "data": { "text/plain": [ "['round1_train', 'validation', 'pool_unlabeled']" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Embed and register datasets in SongSpace.\n", "ss.ingest_audio(\n", " seed_train,\n", " dataset_name=\"round1_train\",\n", " batch_size=32,\n", ")\n", "ss.ingest_audio(\n", " validation,\n", " dataset_name=\"validation\",\n", " allow_training=False,\n", " batch_size=32,\n", ")\n", "ss.ingest_audio(\n", " unlabeled,\n", " dataset_name=\"pool_unlabeled\",\n", " batch_size=32,\n", ")\n", "\n", "ss.list_datasets()" ] }, { "cell_type": "code", "execution_count": 13, "id": "dcbbc04d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved SongSpace to ./Perch2SongSpace with 0 classifiers and 3 datasets.\n" ] } ], "source": [ "ss.save()" ] }, { "cell_type": "markdown", "id": "144d3e87", "metadata": {}, "source": [ "## Similarity search for similar samples" ] }, { "cell_type": "code", "execution_count": 14, "id": "f8f533a3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "embedding query samples\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/cnn.py:2955: UserWarning: The columns of input samples df differ from `model.classes`. Discarding sample df columns.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3e5fb70900324b65a99afd55e520a29a", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timeRanaSierrae_CAcceptReject
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.04.038275NoneNone
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.04.014554NoneNone
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.934664NoneNone
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.896363NoneNone
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.894268NoneNone
\n", "" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "\n", " RanaSierrae_C Accept Reject \n", "0 4.038275 None None \n", "1 4.014554 None None \n", "2 3.934664 None None \n", "3 3.896363 None None \n", "4 3.894268 None None " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool_scores = ss.predict_on_dataset(\n", " classifier_name=\"rana_round1\", dataset_name=\"pool_unlabeled\"\n", ")\n", "# drop samples already labeled\n", "labeled_idx = set(search_labels.index).union(set(seed_train.index))\n", "pool_scores = pool_scores[~pool_scores.index.isin(labeled_idx)]\n", "topk = pool_scores.nlargest(20, target_model_class).reset_index()\n", "\n", "# Review and annotate interactively.\n", "_ = annotate(\n", " topk, bandpass_range=(0, 2500), annotation_buttons=[\"Accept\", \"Reject\"], N=20\n", ")\n", "topk.head()" ] }, { "cell_type": "markdown", "id": "b1ee84a3", "metadata": {}, "source": [ "ingest labels" ] }, { "cell_type": "code", "execution_count": 23, "id": "6d0a3e9a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "all samples already have embeddings in the database\n", "Saved SongSpace to ./Perch2SongSpace with 1 classifiers and 5 datasets.\n" ] }, { "data": { "text/plain": [ "RanaSierrae_C\n", "0 8\n", "1 4\n", "Name: count, dtype: int64" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_pos = topk[topk[\"Accept\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_pos[target_model_class] = 1\n", "\n", "new_neg = topk[topk[\"Reject\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_neg[target_model_class] = 0\n", "\n", "round2_train = (\n", " pd.concat([new_pos, new_neg], ignore_index=True)\n", " .drop_duplicates()\n", " .set_index([\"file\", \"start_time\", \"end_time\"])[[target_model_class]]\n", ")\n", "\n", "ss.ingest_audio(\n", " round2_train,\n", " dataset_name=\"round2_train\",\n", " batch_size=32,\n", " num_workers=0,\n", ")\n", "ss.save()\n", "round2_train[target_model_class].value_counts()" ] }, { "cell_type": "markdown", "id": "a268fa2d", "metadata": {}, "source": [ "## build a new classifier" ] }, { "cell_type": "code", "execution_count": 24, "id": "81cb349e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "training classifier for 1 classes with 24 training samples and 536 validation samples\n", "Finding matching window IDs for samples in label_df...\n", "Finding matching window IDs for samples in label_df...\n", "Epoch 30/200, Loss: 0.342, Val Loss: 0.629\n", "\tval AU ROC: 0.818\n", "\tval MAP: 0.575\n", "Epoch 60/200, Loss: 0.198, Val Loss: 0.539\n", "\tval AU ROC: 0.812\n", "\tval MAP: 0.558\n", "Epoch 90/200, Loss: 0.131, Val Loss: 0.494\n", "\tval AU ROC: 0.805\n", "\tval MAP: 0.563\n", "Epoch 120/200, Loss: 0.094, Val Loss: 0.469\n", "\tval AU ROC: 0.801\n", "\tval MAP: 0.560\n", "Epoch 150/200, Loss: 0.071, Val Loss: 0.454\n", "\tval AU ROC: 0.796\n", "\tval MAP: 0.555\n", "Epoch 180/200, Loss: 0.056, Val Loss: 0.444\n", "\tval AU ROC: 0.793\n", "\tval MAP: 0.548\n", "Loaded best model with validation loss: 0.444 at step 180 of 200\n", "Training complete\n", "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "text/plain": [ "{'RanaSierrae_C': {'average_precision': 0.5429201413929045,\n", " 'roc_auc': 0.7913408137821604},\n", " 'macro_average_precision': np.float64(0.5429201413929045),\n", " 'macro_roc_auc': np.float64(0.7913408137821604)}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_round2 = ss.fit_classifier(\n", " classes=[target_model_class],\n", " train_datasets=[\"round1_train\", \"search_labels\", \"round2_train\"],\n", " validation_dataset=\"validation\",\n", " weak_negatives_proportion=1.0,\n", " weak_negatives_weight=0.001,\n", " steps=200,\n", " batch_size=128,\n", " validation_interval=30,\n", " logging_interval=30,\n", ")\n", "if \"rana_round2\" in ss.list_classifiers():\n", " ss.remove_classifier(\"rana_round2\")\n", "ss.add_classifier(\"rana_round2\", clf_round2)\n", "\n", "round2_metrics = ss.evaluate(\"rana_round2\", \"validation\")\n", "round2_metrics" ] }, { "cell_type": "markdown", "id": "957249aa", "metadata": {}, "source": [ "## Active learning round 2: review high-scoring candidates" ] }, { "cell_type": "code", "execution_count": 25, "id": "04c0d528", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e2522a410dde4003a2fd7579217e5c23", "version_major": 2, "version_minor": 0 }, "text/plain": [ "GridBox(children=(VBox(children=(HTML(value='\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timeRanaSierrae_CAcceptReject
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.05.725680NoneNone
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.05.206980NoneNone
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.04.048398NoneNone
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.03.675750NoneNone
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...9.012.03.413406NoneNone
\n", "" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 9.0 12.0 \n", "\n", " RanaSierrae_C Accept Reject \n", "0 5.725680 None None \n", "1 5.206980 None None \n", "2 4.048398 None None \n", "3 3.675750 None None \n", "4 3.413406 None None " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool_scores = ss.predict_on_dataset(\n", " classifier_name=\"rana_round2\", dataset_name=\"pool_unlabeled\"\n", ")\n", "# drop samples already labeled\n", "labeled_idx = (\n", " set(search_labels.index).union(set(seed_train.index)).union(set(round2_train.index))\n", ")\n", "pool_scores = pool_scores[~pool_scores.index.isin(labeled_idx)]\n", "topk = pool_scores.nlargest(20, target_model_class).reset_index()\n", "\n", "# Review and annotate interactively.\n", "_ = annotate(\n", " topk, bandpass_range=(0, 2500), annotation_buttons=[\"Accept\", \"Reject\"], N=20\n", ")\n", "topk.head()" ] }, { "cell_type": "markdown", "id": "f8bf98aa", "metadata": {}, "source": [ "ingest labels" ] }, { "cell_type": "code", "execution_count": 26, "id": "7e3bf094", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved SongSpace to ./Perch2SongSpace with 2 classifiers and 6 datasets.\n" ] }, { "data": { "text/plain": [ "Series([], Name: count, dtype: int64)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_pos = topk[topk[\"Accept\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_pos[target_model_class] = 1\n", "\n", "new_neg = topk[topk[\"Reject\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_neg[target_model_class] = 0\n", "\n", "round3_train = (\n", " pd.concat([new_pos, new_neg], ignore_index=True)\n", " .drop_duplicates()\n", " .set_index([\"file\", \"start_time\", \"end_time\"])[[target_model_class]]\n", ")\n", "\n", "ss.ingest_audio(\n", " round3_train,\n", " dataset_name=\"round3_train\",\n", " batch_size=32,\n", " num_workers=0,\n", ")\n", "ss.save()\n", "round3_train[target_model_class].value_counts()" ] }, { "cell_type": "markdown", "id": "14d38b36", "metadata": {}, "source": [ "build new classifier" ] }, { "cell_type": "code", "execution_count": 27, "id": "d18b4150", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "training classifier for 1 classes with 24 training samples and 536 validation samples\n", "Finding matching window IDs for samples in label_df...\n", "Finding matching window IDs for samples in label_df...\n", "Epoch 30/200, Loss: 0.349, Val Loss: 0.644\n", "\tval AU ROC: 0.818\n", "\tval MAP: 0.557\n", "Epoch 60/200, Loss: 0.203, Val Loss: 0.553\n", "\tval AU ROC: 0.811\n", "\tval MAP: 0.549\n", "Epoch 90/200, Loss: 0.134, Val Loss: 0.506\n", "\tval AU ROC: 0.804\n", "\tval MAP: 0.553\n", "Epoch 120/200, Loss: 0.096, Val Loss: 0.481\n", "\tval AU ROC: 0.799\n", "\tval MAP: 0.551\n", "Epoch 150/200, Loss: 0.073, Val Loss: 0.466\n", "\tval AU ROC: 0.793\n", "\tval MAP: 0.540\n", "Epoch 180/200, Loss: 0.058, Val Loss: 0.456\n", "\tval AU ROC: 0.790\n", "\tval MAP: 0.528\n", "Loaded best model with validation loss: 0.456 at step 180 of 200\n", "Training complete\n", "Saved SongSpace to ./Perch2SongSpace with 3 classifiers and 6 datasets.\n", "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "text/plain": [ "{'RanaSierrae_C': {'average_precision': 0.5286647807313382,\n", " 'roc_auc': 0.7881672900374023},\n", " 'macro_average_precision': np.float64(0.5286647807313382),\n", " 'macro_roc_auc': np.float64(0.7881672900374023)}" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_round3 = ss.fit_classifier(\n", " classes=[target_model_class],\n", " train_datasets=[\"round1_train\", \"search_labels\", \"round2_train\", \"round3_train\"],\n", " validation_dataset=\"validation\",\n", " weak_negatives_proportion=1.0,\n", " weak_negatives_weight=0.001,\n", " steps=200,\n", " batch_size=128,\n", " validation_interval=30,\n", " logging_interval=30,\n", ")\n", "if \"rana_round3\" in ss.list_classifiers():\n", " ss.remove_classifier(\"rana_round3\")\n", "ss.add_classifier(\"rana_round3\", clf_round3)\n", "\n", "ss.save()\n", "\n", "round3_metrics = ss.evaluate(\"rana_round3\", \"validation\")\n", "round3_metrics" ] }, { "cell_type": "markdown", "id": "81d9f131", "metadata": {}, "source": [ "we now have a solid classifier to use for downstream tasks.\n", "\n", "## Select clips for manual verification\n", "\n", "Use stratified or thresholded selection from the full embedded database.\n", "\n", "select_from_hoplite provides several options for filtering. We can loop over the variables of interest to select stratified clips. \n", "\n", "Filtering options include:\n", "- first and last date\n", "- earliest and latest time\n", "- minimum and maximum score\n", "- list of recordings (audio file paths)\n", "- list of deployments\n", "- list of projects\n", "\n", "We also specify which classes we want to extract clips for, how many, and under which strategy:\n", "- top_k: highest scoring k (eg, 5) clips matching the filters\n", "- random_k: randomly selected k clips matching the filters\n", "- all: all clips matching the filters\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3a5bf170", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timedatetimedeploymentprojectwindow_idclass
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-23 06:15:00mp3round1_train1264RanaSierrae_C
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-22 20:15:00mp3round1_train1220RanaSierrae_C
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.02022-06-22 18:15:00mp3round1_train599RanaSierrae_C
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-22 20:15:00mp3round1_train1219RanaSierrae_C
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-23 06:00:00mp3round1_train3RanaSierrae_C
\n", "
" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "\n", " datetime deployment project window_id class \n", "0 2022-06-23 06:15:00 mp3 round1_train 1264 RanaSierrae_C \n", "1 2022-06-22 20:15:00 mp3 round1_train 1220 RanaSierrae_C \n", "2 2022-06-22 18:15:00 mp3 round1_train 599 RanaSierrae_C \n", "3 2022-06-22 20:15:00 mp3 round1_train 1219 RanaSierrae_C \n", "4 2022-06-23 06:00:00 mp3 round1_train 3 RanaSierrae_C " ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# select the global 5 most confident 'RanaSierrae_C' clips from the pool according to the round 3 classifier\n", "clips = select_from_hoplite(\n", " db=ss.db,\n", " classifier=ss.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"top_k\",\n", " k=5,\n", ")\n", "inspect(clips, bandpass_range=(0, 2500))\n", "clips" ] }, { "cell_type": "markdown", "id": "028b40f4", "metadata": {}, "source": [ "We can re-load the SongSpace in another Python session, which will retain all the saved classifiers, labeled datasets, and embeddings. The clip dataframes created by these examples can be saved to CVS for annotation in Dipper or other review software." ] }, { "cell_type": "code", "execution_count": 26, "id": "bdea7ee5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connecting to existing db at Perch2SongSpace\n", "Connected database has 2,691 embeddings from 672 files.\n", "Classifiers: ['rana_round1', 'rana_round2', 'rana_round3']\n", "Datasets: ['round1_train', 'validation', 'pool_unlabeled', 'search_labels', 'round2_train', 'round3_train']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.13/site-packages/bioacoustics_model_zoo/perch_v2.py:208: UserWarning: Disabling TensorFlow's XLA compilation (setting tf.config.optimizer.set_jit(False)) because otherwise TF models on Mac hang at runtime as of Tensorflow 2.21.0\n", " warnings.warn(\n" ] } ], "source": [ "# reload the SongSpace, as we would in a new script/notebook\n", "from opensoundscape import SongSpace\n", "from opensoundscape.visualization import inspect\n", "\n", "ss_reloaded = SongSpace.open(\"./Perch2SongSpace\")\n", "print(f\"Classifiers: {ss_reloaded.list_classifiers()}\")\n", "print(f\"Datasets: {ss_reloaded.list_datasets()}\")" ] }, { "cell_type": "markdown", "id": "1ef92598", "metadata": {}, "source": [ "#### Example: Stratify by date range and deployment\n", "\n", "a typical stratification pattern for reviewing clips for an occupancy analysis" ] }, { "cell_type": "code", "execution_count": null, "id": "2bda328f", "metadata": {}, "outputs": [], "source": [ "date_ranges = [\n", " (\"2022-06-20\", \"2022-06-21\"),\n", " (\"2022-06-22\", \"2022-06-23\"),\n", " (\"2022-06-24\", \"2022-06-25\"),\n", " (\"2022-06-26\", \"2022-06-27\"),\n", "]\n", "clips = ss_reloaded.stratified_selection(\n", " ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " stratify_deployments=True,\n", " k=1,\n", " date_ranges=date_ranges,\n", ")\n", "\n", "\n", "# table ready for Dipper review with stratification by \"date_range\" and \"deployment\" in binary annotation mode\n", "# selected.to_csv('RanaSierrae_C_clips_for_review.csv')" ] }, { "cell_type": "markdown", "id": "eeb9f895", "metadata": {}, "source": [ "#### Other stratification and filteringpatterns\n", "\n", "Let's now select the highest scorking global k=2 clips for each of 4 date ranges. We'll enforce a score threshold of 0. " ] }, { "cell_type": "code", "execution_count": 27, "id": "f0023e11", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-20 to 2022-06-21\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-22 to 2022-06-23\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-24 to 2022-06-25\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-26 to 2022-06-27\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "selected = ss_reloaded.stratified_selection(\n", " classifier=ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"top_k\",\n", " k=2,\n", " min_score=0,\n", " date_ranges=date_ranges,\n", ")\n", "for date_range, clips in selected.groupby(\"date_range\"):\n", " print(f\"Date range: {date_range}\")\n", " inspect(selected, bandpass_range=(0, 2500))" ] }, { "cell_type": "markdown", "id": "0daf55e5", "metadata": {}, "source": [ "Example of clip selection for all clips above a threshold" ] }, { "cell_type": "code", "execution_count": 30, "id": "4371ec01", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "538" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search_results_df = ss_reloaded.select(\n", " ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"all\",\n", " min_score=0,\n", ")\n", "len(search_results_df)" ] }, { "cell_type": "markdown", "id": "7a089e82", "metadata": {}, "source": [ "select random clips in a score bin\n", "\n", "plus: use random state to create reproducible results \n", "\n", "plus: restrict the time range" ] }, { "cell_type": "code", "execution_count": 31, "id": "9e38be08", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timedatetimedeploymentprojectwindow_idscoreclass
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-26 01:00:00mp3round1_train15270.630615RanaSierrae_C
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...9.012.02022-06-21 07:00:00mp3round1_train2200-0.090810RanaSierrae_C
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-20 04:30:00mp3round1_train1343-0.179905RanaSierrae_C
\n", "
" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 9.0 12.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "\n", " datetime deployment project window_id score \\\n", "0 2022-06-26 01:00:00 mp3 round1_train 1527 0.630615 \n", "1 2022-06-21 07:00:00 mp3 round1_train 2200 -0.090810 \n", "2 2022-06-20 04:30:00 mp3 round1_train 1343 -0.179905 \n", "\n", " class \n", "0 RanaSierrae_C \n", "1 RanaSierrae_C \n", "2 RanaSierrae_C " ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import datetime\n", "\n", "search_results_df = ss_reloaded.select(\n", " classifier=\"rana_round3\",\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"random_k\",\n", " min_score=-2,\n", " max_score=1,\n", " random_state=0,\n", " # accepts either time strings or datetime.time objects\n", " time_range=(\"00:00:00\", datetime.time(8, 0, 0)),\n", ")\n", "search_results_df.head(3)" ] }, { "cell_type": "markdown", "id": "e3f60e13", "metadata": {}, "source": [ "Finally, we can also directly count the number of clips scoring in a bin, without retrieving clip information. \n", "\n", "This is memory-efficient on large datasets because we don't need to aggregate the clip information. " ] }, { "cell_type": "code", "execution_count": null, "id": "d6cb55ab", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
RanaSierrae_C
-2-1734
-101012
01383
12106
\n", "
" ], "text/plain": [ " RanaSierrae_C\n", "-2 -1 734\n", "-1 0 1012\n", " 0 1 383\n", " 1 2 106" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from opensoundscape.ml.shallow_classifier import count_dets_hoplite\n", "import pandas as pd\n", "\n", "counts = count_dets_hoplite(\n", " db=ss_reloaded.db,\n", " classifier=ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " score_bins=[(-2, -1), (-1, 0), (0, 1), (1, 2)],\n", ")\n", "pd.DataFrame(counts)" ] }, { "cell_type": "code", "execution_count": null, "id": "c85dd43f", "metadata": {}, "outputs": [], "source": [ "# Optional cleanup: uncomment and run to remove the SongSpace folder containing the embedding database, classifiers, and datasets\n", "# import shutil\n", "# shutil.rmtree('./songspace_agile_db/')" ] } ], "metadata": { "kernelspec": { "display_name": "opso_dev", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }