{ "cells": [ { "cell_type": "markdown", "id": "7416bd7c", "metadata": {}, "source": [ "# Agile Bioacoustic Modeling with SongSpace\n", "\n", "SongSpace provides a workflow for active or \"agile\" learning for bioacoustics data. Embed audio into a databse, query the database with vector search or classifieres, and select clips for active learning review or final verification for ecological analyses. \n", "\n", "Embeddings are saved in a HopLite database. The same folder storing the (sql) embedding database will also store classifiers and tables for labeled datasets. The full workspace can be saved and loaded with ss.save(path) and SongSpace.load(path). \n" ] }, { "cell_type": "markdown", "id": "7e4c6df3", "metadata": {}, "source": [ "## Run this tutorial\n", "\n", "If running in Colab, uncomment the installation line below." ] }, { "cell_type": "code", "execution_count": 1, "id": "ab51e7e7", "metadata": {}, "outputs": [], "source": [ "# if 'google.colab' in str(get_ipython()):\n", "# %pip install \"opensoundscape==0.12.1\" \"bioacoustics-model-zoo==0.12.0\"" ] }, { "cell_type": "code", "execution_count": 2, "id": "352a2a8b", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "import numpy as np\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "\n", "import bioacoustics_model_zoo as bmz\n", "\n", "from opensoundscape.annotations import BoxedAnnotations\n", "from opensoundscape.vector_database import load_or_create_hoplite_usearch_db\n", "from opensoundscape.ml.song_space import SongSpace\n", "from opensoundscape.ml.shallow_classifier import select_from_hoplite\n", "from opensoundscape.visualization import annotate, inspect" ] }, { "cell_type": "markdown", "id": "0b83317b", "metadata": {}, "source": [ "## Prepare labels\n", "\n", "This uses the same _Rana sierrae_ example files as the agile Hoplite tutorial." ] }, { "cell_type": "code", "execution_count": 3, "id": "2a831276", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/annotations.py:347: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.\n", " all_annotations_df = pd.concat(all_file_dfs).reset_index(drop=True)\n" ] } ], "source": [ "dataset_path = Path(\"./rana_sierrae_2022/\")\n", "audio_and_raven_files = pd.read_csv(dataset_path / \"audio_and_raven_files.csv\")\n", "audio_and_raven_files[\"audio\"] = audio_and_raven_files[\"audio\"].apply(\n", " lambda x: str(dataset_path / x)\n", ")\n", "audio_and_raven_files[\"raven\"] = audio_and_raven_files[\"raven\"].apply(\n", " lambda x: str(dataset_path / x)\n", ")\n", "\n", "annotations = BoxedAnnotations.from_raven_files(\n", " raven_files=audio_and_raven_files[\"raven\"],\n", " audio_files=audio_and_raven_files[\"audio\"],\n", " annotation_column=\"annotation\",\n", ")\n", "\n", "labels = annotations.clip_labels(clip_duration=3, min_label_overlap=0.2)" ] }, { "cell_type": "code", "execution_count": 4, "id": "0ff5ac69", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "seed_train: (4, 1)\n", "validation: (536, 1)\n", "pool: (2148, 1)\n" ] } ], "source": [ "target_source_class = \"C\"\n", "target_model_class = \"RanaSierrae_C\"\n", "\n", "# start with one recording of target class\n", "binary_labels = labels[[target_source_class]].rename(\n", " columns={target_source_class: target_model_class}\n", ")\n", "seed_train = binary_labels.loc[\n", " [\"rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220623_060000_0-10s.mp3\"]\n", "]\n", "other = binary_labels.drop(seed_train.index)\n", "validation, unlabeled = train_test_split(other, test_size=0.8, random_state=0)\n", "\n", "print(\"seed_train:\", seed_train.shape)\n", "print(\"validation:\", validation.shape)\n", "print(\"pool:\", unlabeled.shape)" ] }, { "cell_type": "markdown", "id": "1876817c", "metadata": {}, "source": [ "All audio clips from the single audio file we'll start with for positives:" ] }, { "cell_type": "code", "execution_count": 5, "id": "146752f6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "_ = inspect(seed_train, bandpass_range=(0, 2500))" ] }, { "cell_type": "code", "execution_count": 6, "id": "be08ae58", "metadata": {}, "outputs": [], "source": [ "import opensoundscape as opso" ] }, { "cell_type": "markdown", "id": "390830fa", "metadata": {}, "source": [ "## Build database and SongSpace" ] }, { "cell_type": "code", "execution_count": 7, "id": "8fec817a", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.13/site-packages/tensorflow_hub/__init__.py:61: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.\n", " from pkg_resources import parse_version\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Connecting to existing db at Perch2SongSpace\n", "Connected database has 2,691 embeddings from 672 files.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.13/site-packages/bioacoustics_model_zoo/perch_v2.py:208: UserWarning: Disabling TensorFlow's XLA compilation (setting tf.config.optimizer.set_jit(False)) because otherwise TF models on Mac hang at runtime as of Tensorflow 2.21.0\n", " warnings.warn(\n" ] } ], "source": [ "ss = SongSpace(\"./Perch2SongSpace\", feature_extractor=\"perch2\")" ] }, { "cell_type": "code", "execution_count": 8, "id": "cb59c932", "metadata": {}, "outputs": [], "source": [ "import opensoundscape as opso\n", "\n", "opso.set_seed(0)" ] }, { "cell_type": "code", "execution_count": 9, "id": "c97de950", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "all samples already have embeddings in the database\n", "all samples already have embeddings in the database\n", "all samples already have embeddings in the database\n" ] }, { "data": { "text/plain": [ "['round1_train', 'validation', 'pool_unlabeled']" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Embed and register datasets in SongSpace.\n", "ss.ingest_audio(\n", " seed_train,\n", " dataset_name=\"round1_train\",\n", " batch_size=32,\n", ")\n", "ss.ingest_audio(\n", " validation,\n", " dataset_name=\"validation\",\n", " allow_training=False,\n", " batch_size=32,\n", ")\n", "ss.ingest_audio(\n", " unlabeled,\n", " dataset_name=\"pool_unlabeled\",\n", " batch_size=32,\n", ")\n", "\n", "ss.list_datasets()" ] }, { "cell_type": "code", "execution_count": 10, "id": "dcbbc04d", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved SongSpace to ./Perch2SongSpace with 0 classifiers and 3 datasets.\n" ] } ], "source": [ "ss.save()" ] }, { "cell_type": "markdown", "id": "144d3e87", "metadata": {}, "source": [ "## Similarity search for similar samples" ] }, { "cell_type": "code", "execution_count": 11, "id": "f8f533a3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "embedding query samples\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/opensoundscape/opensoundscape/ml/cnn.py:2954: UserWarning: The columns of input samples df differ from `model.classes`. Discarding sample df columns.\n", " warnings.warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "dac1d4cfbd97475bb22109007f023d8a", "version_major": 2, "version_minor": 0 }, "text/plain": [ " 0%| | 0/4 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timeRanaSierrae_CAcceptReject
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.04.038275NoneNone
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.04.014554NoneNone
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.934664NoneNone
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.896363NoneNone
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.03.894268NoneNone
\n", "" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "\n", " RanaSierrae_C Accept Reject \n", "0 4.038275 None None \n", "1 4.014554 None None \n", "2 3.934664 None None \n", "3 3.896363 None None \n", "4 3.894268 None None " ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool_scores = ss.predict_on_dataset(\n", " classifier_name=\"rana_round1\", dataset_name=\"pool_unlabeled\"\n", ")\n", "# drop samples already labeled\n", "labeled_idx = set(search_labels.index).union(set(seed_train.index))\n", "pool_scores = pool_scores[~pool_scores.index.isin(labeled_idx)]\n", "topk = pool_scores.nlargest(20, target_model_class).reset_index()\n", "\n", "# Review and annotate interactively.\n", "_ = annotate(\n", " topk, bandpass_range=(0, 2500), annotation_buttons=[\"Accept\", \"Reject\"], N=20\n", ")\n", "topk.head()" ] }, { "cell_type": "markdown", "id": "b1ee84a3", "metadata": {}, "source": [ "ingest labels" ] }, { "cell_type": "code", "execution_count": 23, "id": "6d0a3e9a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "all samples already have embeddings in the database\n", "Saved SongSpace to ./Perch2SongSpace with 1 classifiers and 5 datasets.\n" ] }, { "data": { "text/plain": [ "RanaSierrae_C\n", "0 8\n", "1 4\n", "Name: count, dtype: int64" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_pos = topk[topk[\"Accept\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_pos[target_model_class] = 1\n", "\n", "new_neg = topk[topk[\"Reject\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_neg[target_model_class] = 0\n", "\n", "round2_train = (\n", " pd.concat([new_pos, new_neg], ignore_index=True)\n", " .drop_duplicates()\n", " .set_index([\"file\", \"start_time\", \"end_time\"])[[target_model_class]]\n", ")\n", "\n", "ss.ingest_audio(\n", " round2_train,\n", " dataset_name=\"round2_train\",\n", " batch_size=32,\n", " num_workers=0,\n", ")\n", "ss.save()\n", "round2_train[target_model_class].value_counts()" ] }, { "cell_type": "markdown", "id": "a268fa2d", "metadata": {}, "source": [ "## build a new classifier" ] }, { "cell_type": "code", "execution_count": 24, "id": "81cb349e", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "training classifier for 1 classes with 24 training samples and 536 validation samples\n", "Finding matching window IDs for samples in label_df...\n", "Finding matching window IDs for samples in label_df...\n", "Epoch 30/200, Loss: 0.342, Val Loss: 0.629\n", "\tval AU ROC: 0.818\n", "\tval MAP: 0.575\n", "Epoch 60/200, Loss: 0.198, Val Loss: 0.539\n", "\tval AU ROC: 0.812\n", "\tval MAP: 0.558\n", "Epoch 90/200, Loss: 0.131, Val Loss: 0.494\n", "\tval AU ROC: 0.805\n", "\tval MAP: 0.563\n", "Epoch 120/200, Loss: 0.094, Val Loss: 0.469\n", "\tval AU ROC: 0.801\n", "\tval MAP: 0.560\n", "Epoch 150/200, Loss: 0.071, Val Loss: 0.454\n", "\tval AU ROC: 0.796\n", "\tval MAP: 0.555\n", "Epoch 180/200, Loss: 0.056, Val Loss: 0.444\n", "\tval AU ROC: 0.793\n", "\tval MAP: 0.548\n", "Loaded best model with validation loss: 0.444 at step 180 of 200\n", "Training complete\n", "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "text/plain": [ "{'RanaSierrae_C': {'average_precision': 0.5429201413929045,\n", " 'roc_auc': 0.7913408137821604},\n", " 'macro_average_precision': np.float64(0.5429201413929045),\n", " 'macro_roc_auc': np.float64(0.7913408137821604)}" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_round2 = ss.fit_classifier(\n", " classes=[target_model_class],\n", " train_datasets=[\"round1_train\", \"search_labels\", \"round2_train\"],\n", " validation_dataset=\"validation\",\n", " weak_negatives_proportion=1.0,\n", " weak_negatives_weight=0.001,\n", " steps=200,\n", " batch_size=128,\n", " validation_interval=30,\n", " logging_interval=30,\n", ")\n", "if \"rana_round2\" in ss.list_classifiers():\n", " ss.remove_classifier(\"rana_round2\")\n", "ss.add_classifier(\"rana_round2\", clf_round2)\n", "\n", "round2_metrics = ss.evaluate(\"rana_round2\", \"validation\")\n", "round2_metrics" ] }, { "cell_type": "markdown", "id": "957249aa", "metadata": {}, "source": [ "## Active learning round 2: review high-scoring candidates" ] }, { "cell_type": "code", "execution_count": 25, "id": "04c0d528", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "e2522a410dde4003a2fd7579217e5c23", "version_major": 2, "version_minor": 0 }, "text/plain": [ "GridBox(children=(VBox(children=(HTML(value='\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timeRanaSierrae_CAcceptReject
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.05.725680NoneNone
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.05.206980NoneNone
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.04.048398NoneNone
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.03.675750NoneNone
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...9.012.03.413406NoneNone
\n", "" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 9.0 12.0 \n", "\n", " RanaSierrae_C Accept Reject \n", "0 5.725680 None None \n", "1 5.206980 None None \n", "2 4.048398 None None \n", "3 3.675750 None None \n", "4 3.413406 None None " ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pool_scores = ss.predict_on_dataset(\n", " classifier_name=\"rana_round2\", dataset_name=\"pool_unlabeled\"\n", ")\n", "# drop samples already labeled\n", "labeled_idx = (\n", " set(search_labels.index).union(set(seed_train.index)).union(set(round2_train.index))\n", ")\n", "pool_scores = pool_scores[~pool_scores.index.isin(labeled_idx)]\n", "topk = pool_scores.nlargest(20, target_model_class).reset_index()\n", "\n", "# Review and annotate interactively.\n", "_ = annotate(\n", " topk, bandpass_range=(0, 2500), annotation_buttons=[\"Accept\", \"Reject\"], N=20\n", ")\n", "topk.head()" ] }, { "cell_type": "markdown", "id": "f8bf98aa", "metadata": {}, "source": [ "ingest labels" ] }, { "cell_type": "code", "execution_count": 26, "id": "7e3bf094", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved SongSpace to ./Perch2SongSpace with 2 classifiers and 6 datasets.\n" ] }, { "data": { "text/plain": [ "Series([], Name: count, dtype: int64)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "new_pos = topk[topk[\"Accept\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_pos[target_model_class] = 1\n", "\n", "new_neg = topk[topk[\"Reject\"] == True][[\"file\", \"start_time\", \"end_time\"]].copy()\n", "new_neg[target_model_class] = 0\n", "\n", "round3_train = (\n", " pd.concat([new_pos, new_neg], ignore_index=True)\n", " .drop_duplicates()\n", " .set_index([\"file\", \"start_time\", \"end_time\"])[[target_model_class]]\n", ")\n", "\n", "ss.ingest_audio(\n", " round3_train,\n", " dataset_name=\"round3_train\",\n", " batch_size=32,\n", " num_workers=0,\n", ")\n", "ss.save()\n", "round3_train[target_model_class].value_counts()" ] }, { "cell_type": "markdown", "id": "14d38b36", "metadata": {}, "source": [ "build new classifier" ] }, { "cell_type": "code", "execution_count": 27, "id": "d18b4150", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "training classifier for 1 classes with 24 training samples and 536 validation samples\n", "Finding matching window IDs for samples in label_df...\n", "Finding matching window IDs for samples in label_df...\n", "Epoch 30/200, Loss: 0.349, Val Loss: 0.644\n", "\tval AU ROC: 0.818\n", "\tval MAP: 0.557\n", "Epoch 60/200, Loss: 0.203, Val Loss: 0.553\n", "\tval AU ROC: 0.811\n", "\tval MAP: 0.549\n", "Epoch 90/200, Loss: 0.134, Val Loss: 0.506\n", "\tval AU ROC: 0.804\n", "\tval MAP: 0.553\n", "Epoch 120/200, Loss: 0.096, Val Loss: 0.481\n", "\tval AU ROC: 0.799\n", "\tval MAP: 0.551\n", "Epoch 150/200, Loss: 0.073, Val Loss: 0.466\n", "\tval AU ROC: 0.793\n", "\tval MAP: 0.540\n", "Epoch 180/200, Loss: 0.058, Val Loss: 0.456\n", "\tval AU ROC: 0.790\n", "\tval MAP: 0.528\n", "Loaded best model with validation loss: 0.456 at step 180 of 200\n", "Training complete\n", "Saved SongSpace to ./Perch2SongSpace with 3 classifiers and 6 datasets.\n", "Finding matching window IDs for samples in label_df...\n" ] }, { "data": { "text/plain": [ "{'RanaSierrae_C': {'average_precision': 0.5286647807313382,\n", " 'roc_auc': 0.7881672900374023},\n", " 'macro_average_precision': np.float64(0.5286647807313382),\n", " 'macro_roc_auc': np.float64(0.7881672900374023)}" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "clf_round3 = ss.fit_classifier(\n", " classes=[target_model_class],\n", " train_datasets=[\"round1_train\", \"search_labels\", \"round2_train\", \"round3_train\"],\n", " validation_dataset=\"validation\",\n", " weak_negatives_proportion=1.0,\n", " weak_negatives_weight=0.001,\n", " steps=200,\n", " batch_size=128,\n", " validation_interval=30,\n", " logging_interval=30,\n", ")\n", "if \"rana_round3\" in ss.list_classifiers():\n", " ss.remove_classifier(\"rana_round3\")\n", "ss.add_classifier(\"rana_round3\", clf_round3)\n", "\n", "ss.save()\n", "\n", "round3_metrics = ss.evaluate(\"rana_round3\", \"validation\")\n", "round3_metrics" ] }, { "cell_type": "markdown", "id": "81d9f131", "metadata": {}, "source": [ "we now have a solid classifier to use for downstream tasks.\n", "\n", "## Select clips for manual verification\n", "\n", "Use stratified or thresholded selection from the full embedded database.\n", "\n", "select_from_hoplite provides several options for filtering. We can loop over the variables of interest to select stratified clips. \n", "\n", "Filtering options include:\n", "- first and last date\n", "- earliest and latest time\n", "- minimum and maximum score\n", "- list of recordings (audio file paths)\n", "- list of deployments\n", "- list of projects\n", "\n", "We also specify which classes we want to extract clips for, how many, and under which strategy:\n", "- top_k: highest scoring k (eg, 5) clips matching the filters\n", "- random_k: randomly selected k clips matching the filters\n", "- all: all clips matching the filters\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3a5bf170", "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timedatetimedeploymentprojectwindow_idclass
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-23 06:15:00mp3round1_train1264RanaSierrae_C
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-22 20:15:00mp3round1_train1220RanaSierrae_C
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...0.03.02022-06-22 18:15:00mp3round1_train599RanaSierrae_C
3rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-22 20:15:00mp3round1_train1219RanaSierrae_C
4rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-23 06:00:00mp3round1_train3RanaSierrae_C
\n", "
" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 0.0 3.0 \n", "3 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "4 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "\n", " datetime deployment project window_id class \n", "0 2022-06-23 06:15:00 mp3 round1_train 1264 RanaSierrae_C \n", "1 2022-06-22 20:15:00 mp3 round1_train 1220 RanaSierrae_C \n", "2 2022-06-22 18:15:00 mp3 round1_train 599 RanaSierrae_C \n", "3 2022-06-22 20:15:00 mp3 round1_train 1219 RanaSierrae_C \n", "4 2022-06-23 06:00:00 mp3 round1_train 3 RanaSierrae_C " ] }, "execution_count": 46, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# select the global 5 most confident 'RanaSierrae_C' clips from the pool according to the round 3 classifier\n", "clips = select_from_hoplite(\n", " db=ss.db,\n", " classifier=ss.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"top_k\",\n", " k=5,\n", ")\n", "inspect(clips, bandpass_range=(0, 2500))\n", "clips" ] }, { "cell_type": "markdown", "id": "028b40f4", "metadata": {}, "source": [ "We can re-load the SongSpace in another Python session, which will retain all the saved classifiers, labeled datasets, and embeddings. The clip dataframes created by these examples can be saved to CVS for annotation in Dipper or other review software." ] }, { "cell_type": "code", "execution_count": 26, "id": "bdea7ee5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Connecting to existing db at Perch2SongSpace\n", "Connected database has 2,691 embeddings from 672 files.\n", "Classifiers: ['rana_round1', 'rana_round2', 'rana_round3']\n", "Datasets: ['round1_train', 'validation', 'pool_unlabeled', 'search_labels', 'round2_train', 'round3_train']\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/SML161/miniconda3/envs/opso_dev/lib/python3.13/site-packages/bioacoustics_model_zoo/perch_v2.py:208: UserWarning: Disabling TensorFlow's XLA compilation (setting tf.config.optimizer.set_jit(False)) because otherwise TF models on Mac hang at runtime as of Tensorflow 2.21.0\n", " warnings.warn(\n" ] } ], "source": [ "# reload the SongSpace, as we would in a new script/notebook\n", "from opensoundscape import SongSpace\n", "from opensoundscape.visualization import inspect\n", "\n", "ss_reloaded = SongSpace.open(\"./Perch2SongSpace\")\n", "print(f\"Classifiers: {ss_reloaded.list_classifiers()}\")\n", "print(f\"Datasets: {ss_reloaded.list_datasets()}\")" ] }, { "cell_type": "markdown", "id": "1ef92598", "metadata": {}, "source": [ "#### Example: Stratify by date range and deployment\n", "\n", "a typical stratification pattern for reviewing clips for an occupancy analysis" ] }, { "cell_type": "code", "execution_count": null, "id": "2bda328f", "metadata": {}, "outputs": [], "source": [ "date_ranges = [\n", " (\"2022-06-20\", \"2022-06-21\"),\n", " (\"2022-06-22\", \"2022-06-23\"),\n", " (\"2022-06-24\", \"2022-06-25\"),\n", " (\"2022-06-26\", \"2022-06-27\"),\n", "]\n", "clips = ss_reloaded.stratified_selection(\n", " ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " stratify_deployments=True,\n", " k=1,\n", " date_ranges=date_ranges,\n", ")\n", "\n", "\n", "# table ready for Dipper review with stratification by \"date_range\" and \"deployment\" in binary annotation mode\n", "# selected.to_csv('RanaSierrae_C_clips_for_review.csv')" ] }, { "cell_type": "markdown", "id": "eeb9f895", "metadata": {}, "source": [ "#### Other stratification and filteringpatterns\n", "\n", "Let's now select the highest scorking global k=2 clips for each of 4 date ranges. We'll enforce a score threshold of 0. " ] }, { "cell_type": "code", "execution_count": 27, "id": "f0023e11", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-20 to 2022-06-21\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-22 to 2022-06-23\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-24 to 2022-06-25\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Date range: 2022-06-26 to 2022-06-27\n" ] }, { "data": { "text/html": [ "\n", " \n", "\n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " \n", " \n", "
\n", " \n", "
\n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "selected = ss_reloaded.stratified_selection(\n", " classifier=ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"top_k\",\n", " k=2,\n", " min_score=0,\n", " date_ranges=date_ranges,\n", ")\n", "for date_range, clips in selected.groupby(\"date_range\"):\n", " print(f\"Date range: {date_range}\")\n", " inspect(selected, bandpass_range=(0, 2500))" ] }, { "cell_type": "markdown", "id": "0daf55e5", "metadata": {}, "source": [ "Example of clip selection for all clips above a threshold" ] }, { "cell_type": "code", "execution_count": 30, "id": "4371ec01", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "538" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "search_results_df = ss_reloaded.select(\n", " ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"all\",\n", " min_score=0,\n", ")\n", "len(search_results_df)" ] }, { "cell_type": "markdown", "id": "7a089e82", "metadata": {}, "source": [ "select random clips in a score bin\n", "\n", "plus: use random state to create reproducible results \n", "\n", "plus: restrict the time range" ] }, { "cell_type": "code", "execution_count": 31, "id": "9e38be08", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
filestart_timeend_timedatetimedeploymentprojectwindow_idscoreclass
0rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...6.09.02022-06-26 01:00:00mp3round1_train15270.630615RanaSierrae_C
1rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...9.012.02022-06-21 07:00:00mp3round1_train2200-0.090810RanaSierrae_C
2rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220...3.06.02022-06-20 04:30:00mp3round1_train1343-0.179905RanaSierrae_C
\n", "
" ], "text/plain": [ " file start_time end_time \\\n", "0 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 6.0 9.0 \n", "1 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 9.0 12.0 \n", "2 rana_sierrae_2022/mp3/sine2022a_MSD-0558_20220... 3.0 6.0 \n", "\n", " datetime deployment project window_id score \\\n", "0 2022-06-26 01:00:00 mp3 round1_train 1527 0.630615 \n", "1 2022-06-21 07:00:00 mp3 round1_train 2200 -0.090810 \n", "2 2022-06-20 04:30:00 mp3 round1_train 1343 -0.179905 \n", "\n", " class \n", "0 RanaSierrae_C \n", "1 RanaSierrae_C \n", "2 RanaSierrae_C " ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import datetime\n", "\n", "search_results_df = ss_reloaded.select(\n", " classifier=\"rana_round3\",\n", " classes=[\"RanaSierrae_C\"],\n", " strategy=\"random_k\",\n", " min_score=-2,\n", " max_score=1,\n", " random_state=0,\n", " # accepts either time strings or datetime.time objects\n", " time_range=(\"00:00:00\", datetime.time(8, 0, 0)),\n", ")\n", "search_results_df.head(3)" ] }, { "cell_type": "markdown", "id": "e3f60e13", "metadata": {}, "source": [ "Finally, we can also directly count the number of clips scoring in a bin, without retrieving clip information. \n", "\n", "This is memory-efficient on large datasets because we don't need to aggregate the clip information. " ] }, { "cell_type": "code", "execution_count": null, "id": "d6cb55ab", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
RanaSierrae_C
-2-1734
-101012
01383
12106
\n", "
" ], "text/plain": [ " RanaSierrae_C\n", "-2 -1 734\n", "-1 0 1012\n", " 0 1 383\n", " 1 2 106" ] }, "execution_count": 33, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from opensoundscape.ml.shallow_classifier import count_dets_hoplite\n", "import pandas as pd\n", "\n", "counts = count_dets_hoplite(\n", " db=ss_reloaded.db,\n", " classifier=ss_reloaded.classifiers[\"rana_round3\"],\n", " classes=[\"RanaSierrae_C\"],\n", " score_bins=[(-2, -1), (-1, 0), (0, 1), (1, 2)],\n", ")\n", "pd.DataFrame(counts)" ] }, { "cell_type": "code", "execution_count": null, "id": "c85dd43f", "metadata": {}, "outputs": [], "source": [ "# Optional cleanup: uncomment and run to remove the SongSpace folder containing the embedding database, classifiers, and datasets\n", "# import shutil\n", "# shutil.rmtree('./songspace_agile_db/')" ] } ], "metadata": { "kernelspec": { "display_name": "opso_dev", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }