{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,12,20]],"date-time":"2023-12-20T15:25:45Z","timestamp":1703085945879},"reference-count":38,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T00:00:00Z","timestamp":1674777600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T00:00:00Z","timestamp":1674777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"name":"National Academies Keck Futures Initiative","award":["NAKFI CB4"]},{"DOI":"10.13039\/100000185","name":"Defense Advanced Research Projects Agency","doi-asserted-by":"publisher","award":["HR0011-16-2-0019"]},{"name":"Christopher Family Foundation"},{"DOI":"10.13039\/100000057","name":"National Institute of General Medical Sciences","doi-asserted-by":"publisher","award":["R01GM117467"]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Barcode-based tracking of individuals is revolutionizing animal behavior studies, but further progress hinges on whether in addition to determining an individual\u2019s location, specific behaviors can be identified and monitored. We achieve this goal using information from the barcodes to identify tightly bounded image regions that potentially show the behavior of interest. These image regions are then analyzed with convolutional neural networks to verify that the behavior occurred. When applied to a challenging test case, detecting social liquid transfer (trophallaxis) in the honey bee hive, this approach yielded a 67% higher sensitivity and an 11% lower error rate than the best detector for honey bee trophallaxis so far. We were furthermore able to automatically detect whether a bee donates or receives liquid, which previously required manual observations. By applying our trophallaxis detector to recordings from three honey bee colonies and performing simulations, we discovered that liquid exchanges among bees generate two distinct social networks with different transmission capabilities. Finally, we demonstrate that our approach generalizes to detecting other specific behaviors. We envision that its broad application will enable automatic, high-resolution behavioral studies that address a broad range of previously intractable questions in evolutionary biology, ethology, neuroscience, and molecular biology.<\/jats:p>","DOI":"10.1038\/s41598-022-26825-4","type":"journal-article","created":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T13:03:50Z","timestamp":1674824630000},"update-policy":"http:\/\/dx.doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Automated monitoring of honey bees with barcodes and artificial intelligence reveals two distinct social networks from a single affiliative behavior"],"prefix":"10.1038","volume":"13","author":[{"given":"Tim","family":"Gernat","sequence":"first","affiliation":[]},{"given":"Tobias","family":"Jagla","sequence":"additional","affiliation":[]},{"given":"Beryl M.","family":"Jones","sequence":"additional","affiliation":[]},{"given":"Martin","family":"Middendorf","sequence":"additional","affiliation":[]},{"given":"Gene E.","family":"Robinson","sequence":"additional","affiliation":[]}],"member":"297","published-online":{"date-parts":[[2023,1,27]]},"reference":[{"key":"26825_CR1","doi-asserted-by":"publisher","first-page":"1","DOI":"10.3389\/fevo.2015.00103","volume":"3","author":"F Wario","year":"2015","unstructured":"Wario, F., Wild, B., Couvillon, M. J., Rojas, R. & Landgraf, T. Automatic methods for long-term tracking and the detection and decoding of communication dances in honeybees. Front. Ecol. Evol. 3, 1\u201314 (2015).","journal-title":"Front. Ecol. Evol."},{"key":"26825_CR2","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1371\/journal.pone.0136487","volume":"10","author":"JD Crall","year":"2015","unstructured":"Crall, J. D., Gravish, N., Mountcastle, A. M. & Combes, S. A. BEEtag: A low-cost, image-based tracking system for the study of animal behavior and locomotion. PLoS ONE 10, 1\u201313 (2015).","journal-title":"PLoS ONE"},{"key":"26825_CR3","doi-asserted-by":"publisher","first-page":"1536","DOI":"10.1111\/2041-210X.13005","volume":"9","author":"G Alarc\u00f3n-Nieto","year":"2018","unstructured":"Alarc\u00f3n-Nieto, G. et al. An automated barcode tracking system for behavioural studies in birds. Methods Ecol. Evol. 9, 1536\u20131547 (2018).","journal-title":"Methods Ecol. Evol."},{"key":"26825_CR4","doi-asserted-by":"publisher","first-page":"1090","DOI":"10.1126\/science.1234316","volume":"340","author":"DP Mersch","year":"2013","unstructured":"Mersch, D. P., Crespi, A. & Keller, L. Tracking individuals shows spatial fidelity is a key regulator of ant social organization. Science 340, 1090\u20131093 (2013).","journal-title":"Science"},{"key":"26825_CR5","doi-asserted-by":"publisher","first-page":"683","DOI":"10.1126\/science.aat1598","volume":"362","author":"JD Crall","year":"2018","unstructured":"Crall, J. D. et al. Neonicotinoid exposure disrupts bumblebee nest behavior, social networks, and thermoregulation. Science 362, 683\u2013686 (2018).","journal-title":"Science"},{"key":"26825_CR6","doi-asserted-by":"publisher","first-page":"941","DOI":"10.1126\/science.aat4793","volume":"362","author":"N Stroeymeyt","year":"2018","unstructured":"Stroeymeyt, N. et al. Social network plasticity decreases disease transmission in a eusocial insect. Science 362, 941\u2013945 (2018).","journal-title":"Science"},{"key":"26825_CR7","doi-asserted-by":"publisher","first-page":"1201","DOI":"10.1038\/s41467-018-03561-w","volume":"9","author":"JD Crall","year":"2018","unstructured":"Crall, J. D. et al. Spatial fidelity of workers predicts collective response to disturbance in a social insect. Nat. Commun. 9, 1201 (2018).","journal-title":"Nat. Commun."},{"key":"26825_CR8","doi-asserted-by":"publisher","first-page":"1433","DOI":"10.1073\/pnas.1713568115","volume":"115","author":"T Gernat","year":"2018","unstructured":"Gernat, T. et al. Automated monitoring of behavior reveals bursty interaction patterns and rapid spreading dynamics in honeybee social networks. Proc. Natl. Acad. Sci. 115, 1433\u20131438 (2018).","journal-title":"Proc. Natl. Acad. Sci."},{"key":"26825_CR9","doi-asserted-by":"publisher","first-page":"17663","DOI":"10.1038\/s41598-017-17863-4","volume":"7","author":"C Blut","year":"2017","unstructured":"Blut, C. et al. Automated computer-based detection of encounter behaviours in groups of honeybees. Sci. Rep. 7, 17663 (2017).","journal-title":"Sci. Rep."},{"key":"26825_CR10","doi-asserted-by":"publisher","first-page":"541","DOI":"10.1016\/j.tree.2013.06.002","volume":"28","author":"J Krause","year":"2013","unstructured":"Krause, J. et al. Reality mining of animal social systems. Trends Ecol. Evol. 28, 541\u2013551 (2013).","journal-title":"Trends Ecol. Evol."},{"key":"26825_CR11","doi-asserted-by":"publisher","first-page":"436","DOI":"10.1038\/nature14539","volume":"521","author":"Y LeCun","year":"2015","unstructured":"LeCun, Y., Bengio, Y. & Hinton, G. E. Deep learning. Nature 521, 436\u2013444 (2015).","journal-title":"Nature"},{"key":"26825_CR12","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1038\/s41592-018-0295-5","volume":"16","author":"F Romero-Ferrero","year":"2019","unstructured":"Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C., Heras, F. J. H. & de Polavieja, G. G. Idtracker.Ai: Tracking all individuals in small or large collectives of unmarked animals. Nat. Methods 16, 179\u2013182 (2019).","journal-title":"Nat. Methods"},{"key":"26825_CR13","doi-asserted-by":"publisher","unstructured":"Bozek, K., Hebert, L., Mikheyev, A. S. & Stephens, G. J. Towards dense object tracking in a 2D honeybee hive. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 4185\u20134193 (2018) doi: https:\/\/doi.org\/10.1109\/CVPR.2018.00440.","DOI":"10.1109\/CVPR.2018.00440"},{"key":"26825_CR14","doi-asserted-by":"publisher","first-page":"1","DOI":"10.7554\/eLife.58145","volume":"9","author":"A Gal","year":"2020","unstructured":"Gal, A., Saragosti, J. & Kronauer, D. J. C. anTraX, a software package for high-throughput video tracking of color-tagged insects. Elife 9, 1\u201332 (2020).","journal-title":"Elife"},{"key":"26825_CR15","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1038\/s41593-018-0209-y","volume":"21","author":"A Mathis","year":"2018","unstructured":"Mathis, A. et al. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21, 1281\u20131289 (2018).","journal-title":"Nat. Neurosci."},{"key":"26825_CR16","doi-asserted-by":"publisher","first-page":"117","DOI":"10.1038\/s41592-018-0234-5","volume":"16","author":"TD Pereira","year":"2019","unstructured":"Pereira, T. D. et al. Fast animal pose estimation using deep neural networks. Nat. Methods 16, 117\u2013125 (2019).","journal-title":"Nat. Methods"},{"key":"26825_CR17","doi-asserted-by":"crossref","unstructured":"Pereira, T. D. et al. SLEAP: Multi-animal pose tracking. Preprint at bioRxiv 276246 (2020).","DOI":"10.1101\/2020.08.31.276246"},{"key":"26825_CR18","doi-asserted-by":"publisher","first-page":"1","DOI":"10.7554\/eLife.47994","volume":"8","author":"JM Graving","year":"2019","unstructured":"Graving, J. M. et al. Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning. Elife 8, 1\u201342 (2019).","journal-title":"Elife"},{"key":"26825_CR19","doi-asserted-by":"crossref","unstructured":"Bohnslav, J. P. et al. DeepEthogram: A machine learning pipeline for supervised behavior classification from raw pixels. Preprint at bioRxiv 312504 (2020).","DOI":"10.1101\/2020.09.24.312504"},{"key":"26825_CR20","doi-asserted-by":"publisher","first-page":"4724","DOI":"10.1109\/CVPR.2017.502","volume-title":"2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"J Carreira","year":"2017","unstructured":"Carreira, J. & Zisserman, A. Quo vadis, action recognition? A New model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (eds Carreira, J. & Zisserman, A.) 4724\u20134733 (IEEE, 2017). https:\/\/doi.org\/10.1109\/CVPR.2017.502."},{"key":"26825_CR21","doi-asserted-by":"publisher","first-page":"E5716","DOI":"10.1073\/pnas.1719367115","volume":"115","author":"MS Norouzzadeh","year":"2018","unstructured":"Norouzzadeh, M. S. et al. Automatically identifying, counting and describing wild animals in camera-trap images with deep learning. Proc. Natl. Acad. Sci. U. S. A. 115, E5716\u2013E5725 (2018).","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"26825_CR22","unstructured":"Zou, Z., Shi, Z., Guo, Y. & Ye, J. Object detection in 20 Years: A survey 1\u201339 (2019)."},{"key":"26825_CR23","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-021-21212-5","volume":"12","author":"B Wild","year":"2021","unstructured":"Wild, B. et al. Social networks predict the life and death of honey bees. Nat. Commun. 12, 1\u201312 (2021).","journal-title":"Nat. Commun."},{"key":"26825_CR24","doi-asserted-by":"publisher","first-page":"1441","DOI":"10.1111\/2041-210X.13646","volume":"12","author":"L Baltiansky","year":"2021","unstructured":"Baltiansky, L., Sarafian-Tamam, E., Greenwald, E. & Feinerman, O. Dual-fluorescence imaging and automated trophallaxis detection for studying multi-nutrient regulation in superorganisms. Methods Ecol. Evol. 12, 1441\u20131457 (2021).","journal-title":"Methods Ecol. Evol."},{"key":"26825_CR25","volume-title":"The Biology of the Honey Bee","author":"ML Winston","year":"1991","unstructured":"Winston, M. L. The Biology of the Honey Bee (Harvard University Press, 1991)."},{"key":"26825_CR26","doi-asserted-by":"publisher","first-page":"e20375","DOI":"10.7554\/eLife.20375","volume":"5","author":"AC LeBoeuf","year":"2016","unstructured":"LeBoeuf, A. C. et al. Oral transfer of chemical cues, growth proteins and hormones in social insects. Elife 5, e20375 (2016).","journal-title":"Elife"},{"key":"26825_CR27","doi-asserted-by":"publisher","first-page":"e62850","DOI":"10.7554\/eLife.62850","volume":"9","author":"BM Jones","year":"2020","unstructured":"Jones, B. M. et al. Individual differences in honey bee behavior enabled by plasticity in brain gene regulatory networks. Elife 9, e62850 (2020).","journal-title":"Elife"},{"key":"26825_CR28","doi-asserted-by":"publisher","first-page":"55","DOI":"10.1007\/s00114-006-0157-3","volume":"94","author":"WM Farina","year":"2007","unstructured":"Farina, W. M., Gr\u00fcter, C., Acosta, L. & Mc Cabe, S. Honeybees learn floral odors while receiving nectar from foragers within the hive. Naturwissenschaften 94, 55\u201360 (2007).","journal-title":"Naturwissenschaften"},{"key":"26825_CR29","doi-asserted-by":"publisher","first-page":"343","DOI":"10.1007\/BF00295707","volume":"19","author":"TD Seeley","year":"1986","unstructured":"Seeley, T. D. Social foraging by honeybees: How colonies allocate foragers among patches of flowers. Behav. Ecol. Sociobiol. 19, 343\u2013354 (1986).","journal-title":"Behav. Ecol. Sociobiol."},{"key":"26825_CR30","doi-asserted-by":"publisher","first-page":"10406","DOI":"10.1073\/pnas.2002268117","volume":"117","author":"AC Geffre","year":"2020","unstructured":"Geffre, A. C. et al. Honey bee virus causes context-dependent changes in host social behavior. Proc. Natl. Acad. Sci. 117, 10406\u201310413 (2020).","journal-title":"Proc. Natl. Acad. Sci."},{"key":"26825_CR31","doi-asserted-by":"publisher","first-page":"124","DOI":"10.2307\/3495099","volume":"70","author":"KD Waddington","year":"1987","unstructured":"Waddington, K. D. & Herbst, L. H. Body size and the functional length of the proboscis of honey bees. Fla. Entomol. 70, 124\u2013128 (1987).","journal-title":"Fla. Entomol."},{"key":"26825_CR32","doi-asserted-by":"publisher","first-page":"249","DOI":"10.1016\/j.neunet.2018.07.011","volume":"106","author":"M Buda","year":"2018","unstructured":"Buda, M., Maki, A. & Mazurowski, M. A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249\u2013259 (2018).","journal-title":"Neural Netw."},{"key":"26825_CR33","unstructured":"Kingma, D. P. & Ba, J. L. Adam: A method for stochastic optimization. 3rd Int. Conf. Learn. Represent. ICLR 2015\u2014Conf. Track Proc. 1\u201315 (2015)."},{"key":"26825_CR34","doi-asserted-by":"publisher","first-page":"360","DOI":"10.1007\/PL00001791","volume":"48","author":"WM Farina","year":"2001","unstructured":"Farina, W. M. & Wainselboim, A. J. Thermographic recordings show that honeybees may receive nectar from foragers even during short trophallactic contacts. Insectes Soc. 48, 360\u2013362 (2001).","journal-title":"Insectes Soc."},{"key":"26825_CR35","doi-asserted-by":"publisher","first-page":"201","DOI":"10.1016\/0003-3472(60)90028-2","volume":"8","author":"MD Allen","year":"1960","unstructured":"Allen, M. D. The honeybee queen and her attendants. Anim. Behav. 8, 201\u2013208 (1960).","journal-title":"Anim. Behav."},{"key":"26825_CR36","first-page":"818","volume-title":"Computer Vision\u2014ECCV 2014. Lecture Notes in Computer Science","author":"MD Zeiler","year":"2014","unstructured":"Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision\u2014ECCV 2014. Lecture Notes in Computer Science Vol. 8689 (eds Fleet, D. et al.) 818\u2013833 (Springer, 2014)."},{"key":"26825_CR37","volume-title":"Epidemiology and Infection","author":"RM Anderson","year":"1992","unstructured":"Anderson, R. M. & May, R. M. Infectious diseases of humans: Dynamics and control. In Epidemiology and Infection (eds Anderson, R. M. & May, R. M.) (Oxford University Press, 1992)."},{"key":"26825_CR38","doi-asserted-by":"publisher","first-page":"709","DOI":"10.1038\/s41598-017-18450-3","volume":"8","author":"M Li","year":"2018","unstructured":"Li, M., Rao, V. D., Gernat, T. & Dankowicz, H. Lifetime-preserving reference models for characterizing spreading dynamics on temporal networks. Sci. Rep. 8, 709 (2018).","journal-title":"Sci. Rep."}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26825-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26825-4","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26825-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,27]],"date-time":"2023-01-27T13:08:16Z","timestamp":1674824896000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26825-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,27]]},"references-count":38,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["26825"],"URL":"http:\/\/dx.doi.org\/10.1038\/s41598-022-26825-4","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":["Multidisciplinary"],"published":{"date-parts":[[2023,1,27]]},"assertion":[{"value":"27 September 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 December 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 January 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"1541"}}