Krystle Hewitt

Light in a bottle: U of T researchers use AI to capture photons in motion

Christopher.Sorensen — Tue, 19 Nov 2024 15:07:26 +0000

Light in a bottle: U of T researchers use AI to capture photons in motion

Christopher.Sorensen Tue, 11/19/2024 - 10:07

Cutline

A scene rendered using videos from an ultra-high-speed camera shows a pulse of light travelling through a pop bottle, scattering off liquid, hitting the ground, focusing on the cap and reflecting back (supplied image)

Krystle Hewitt

Topic

Breaking Research

Artificial Intelligence

Computer Science

Faculty of Arts & Science

Graduate Students

Research & Innovation

Subheadline

A novel AI algorithm simulates what an ultra-fast scene – such as a pulse of light speeding through a pop bottle – would look like from any vantage point

Close your eyes and picture the iconic “bullet time” scene from The Matrix – the one where Neo, played by Keanu Reeves, dodges bullets in slow motion. Now imagine being able to witness the same effect, but instead of speeding bullets, you’re watching something that moves one million times faster: light itself.

Computer scientists from the ��Ƶ have built an advanced camera setup that can visualize light in motion from any perspective, opening avenues for further inquiry into new types of 3D sensing techniques.

The researchers developed a sophisticated AI algorithm that can simulate what an ultra-fast scene – a pulse of light speeding through a pop bottle or bouncing off a mirror – would look like from any vantage point.

David Lindell (supplied image)

David Lindell, an assistant professor in the department of computer science in the Faculty of Arts & Science, says the feat requires the ability to generate videos where the camera appears to “fly” alongside the very photons of light as they travel.

“Our technology can capture and visualize the actual propagation of light with the same dramatic, slowed-down detail,” says Lindell. “We get a glimpse of the world at speed-of-light timescales that are normally invisible.”

The researchers believe the approach, which was recently presented at the 2024 European Conference on Computer Vision, can unlock new capabilities in several important research areas, including: advanced sensing capabilities such as non-line-of-sight imaging, a method that allows viewers to “see” around corners or behind obstacles using multiple bounces of light; imaging through scattering media, such as fog, smoke, biological tissues or turbid water; and 3D reconstruction, where understanding the behaviour of light that scatters multiple times is critical.

In addition to Lindell, the research team included U of T computer science PhD student Anagh Malik, fourth-year engineering science undergraduate Noah Juravsky, Professor Kyros Kutulakos and Stanford University Associate Professor Gordon Wetzstein and PhD student Ryan Po.

The researchers’ key innovation lies in the AI algorithm they developed to visualize ultrafast videos from any viewpoint – a challenge known in computer vision as “novel view synthesis.”

Traditionally, novel view synthesis methods are designed for images or videos captured with regular cameras. However, the researchers extended this concept to handle data captured by an ultra-fast camera operating at speeds comparable to light, which posed unique challenges – including the need for their algorithm to account for the speed of light and model how it propagates through a scene.

Through their work, researchers observed a moving-camera visualization of light in motion, including refracting through water, bouncing off a mirror or scattering off a surface. They also demonstrated how to visualize phenomena that only occur at a significant portion of the speed of light, as predicted by Albert Einstein. For example, they visualize the “searchlight effect” that makes objects brighter when moving toward an observer, and “length contraction,” where fast-moving objects look shorter in the direction they are travelling. The researchers were also able to create a way to see how objects would appear to contract in length when moving at such high speeds.

Anagh Malik (supplied image)

While current algorithms for processing ultra-fast videos typically focus on analyzing a single video from a single viewpoint, the researchers say their work is the first to extend this analysis to multi-view light-in-flight videos, allowing for the study of how light propagates from multiple perspectives.

“Our multi-view light-in-flight videos serve as a powerful educational tool, offering a unique way to teach the physics of light transport,” says Malik. “By visually capturing how light behaves in real-time – whether refracting through a material or reflecting off a surface – we can get a more intuitive understanding of the motion of light through a scene.

“Additionally, our technology could inspire creative applications in the arts, such as filmmaking or interactive installations, where the beauty of light transport can be used to create new types of visual effects or immersive experiences.”

The research also holds significant potential for improving LIDAR (Light Detection and Ranging) sensor technology used in autonomous vehicles. Typically, these sensors process data to immediately create 3D images right away. But the researchers’ work suggests the potential to store the raw data, including detailed light patterns, to help create systems that perform better than conventional LIDAR to see more details, look through obstacles and understand materials better.

While the researchers’ project focused on visualizing how light moves through a scene from any direction, they note that carries “hidden information” about the shape and appearance of everything it touches. As the researchers look to their next steps, they want to unlock this information by developing a method that uses multi-view light-in-flight videos to reconstruct the 3D geometry and appearance of the entire scene.

“This means we could potentially create incredibly detailed, three-dimensional models of objects and environments – just by watching how light travels through them,” Lindell says.

U of T initiative encourages computer science students to incorporate ethics into their work

Christopher.Sorensen — Fri, 26 Apr 2024 19:22:07 +0000

U of T initiative encourages computer science students to incorporate ethics into their work

Christopher.Sorensen Fri, 04/26/2024 - 15:22

Cutline

(photo by urbazon/Getty Images)

Krystle Hewitt

Topic

Our Community

Schwartz Reisman Institute for Technology and Society

Artificial Intelligence

Computer Science

Faculty of Arts & Science

Subheadline

Total enrolment in courses featuring Embedded Ethics Education Initiative modules exceeded 8,000 students this year

Computer science students at the ��Ƶ are learning how to incorporate ethical considerations into the design and development of new technologies such as artificial intelligence with the help of a unique undergraduate initiative.

The Embedded Ethics Education Initiative (E3I) aims to provide students with the ability to critically assess the societal impacts of the technologies they will be designing and developing throughout their careers. That includes grappling with issues such as AI safety, data privacy and misinformation.

Program co-creator Sheila McIlraith, a professor in the department of computer science in the Faculty of Arts & Science and an associate director at the Schwartz Reisman Institute for Technology and Society (SRI), says E3I aims to help students “recognize the broader ramifications of the technology they’re developing on diverse stakeholders, and to avoid or mitigate any negative impact.”

First launched in 2020 as a two-year pilot program, the initiative is a collaborative venture between the department of computer science and SRI in association with the department of philosophy. It integrates ethics modules into select undergraduate computer science courses – and has reached thousands of U of T students in this academic year alone.

Malaikah Hussain is one of the many U of T students who has benefited from the initiative. As a first-year student enrolled in CSC111: Foundations of Computer Science II, she participated in an E3I module that explored how a data structure she learned about in class laid the foundation of a contact tracing system and raised ethical issues concerning data collection.

“The modules underlined how the software design choices we make extend beyond computing efficiency concerns to grave ethical concerns such as privacy,” says Hussain, who is now a third-year computer science specialist.

Hussain adds that the modules propelled her interest in ethics and computing, leading her to pursue upper year courses on the topic. During a subsequent internship, she organized an event about the ethics surrounding e-waste disposal and the company’s technology life cycle.

“The E3I modules have been crucial in shaping my approach to my studies and work, emphasizing the importance of ethics in every aspect of computing,” she says.

The program, which initially reached 400 students, has seen significant growth over the last four years. This academic year alone, total enrolment in computer science courses with E3I programming has exceeded 8,000 students. Another 1,500 students participated in E3I programming in courses outside computer science.

Clockwise from top left: Steven Coyne, Diane Horton, David Liu and Sheila McIlraith (supplied images)

In recognition of the program’s impact on the undergraduate student learning experience, McIlraith and her colleagues – Diane Horton and David Liu, a professor and associate professor, teaching stream, respectively, in the department of computer science, and Steven Coyne, an assistant professor who is jointly appointed to the departments of computer science and philosophy – were recently recognized with the 2024 Northrop Frye Award (Team), one of the prestigious U of T Alumni Association Awards of Excellence.

Horton, who leads the initiative’s assessment efforts, points to the team’s recently published paper showing that after participating in modules in only one or two courses, students are inspired to learn more about ethics and are benefiting in the workplace.

“We have evidence that they are better able to identify ethical issues arising in their work, and that the modules help them navigate those issues,” she says.

Horton adds that the findings build on earlier assessment work showing that after experiencing modules in only one course, students became more interested in ethics and tech, and more confident in their ability to deal with ethical issues they might encounter.

The team says the initiative’s interdisciplinary nature is key to delivering both a curriculum and experience with an authentic voice, giving instructors and students the vocabulary and depth of knowledge to engage on issues such as privacy, well-being and harm.

“As a philosopher and ethicist, I love teaching in a computer science department,” says Coyne. “My colleagues teach me about interesting ethical problems that they’ve found in their class material, and I get to reciprocate by finding distinctions and ideas that illuminate those problems. And we learn a lot from each other – intellectually and pedagogically – when we design a module for that class together.”

E3I is founded upon three key principles: teach students how – not what – to think; encourage ethics-informed design choices as a design principle; and make discussions safe, not personal.

“Engaging with students and making them feel safe, not proselytizing, inviting the students to participate is especially important,” says Liu.

The modules support this type of learning environment by using stakeholders with fictional character profiles that include names, pictures and a backstory.

“Fictional stakeholders help add a layer of distance so students can think through the issues without having to say, ‘This is what I think,’” Horton says. “Stakeholders also increase their awareness of the different kinds of people who might be impacted.”

McIlraith adds that having students advocate for an opinion that is not necessarily their own encourages empathy, while Liu notes that many have a “real hunger” to learn about the ethical considerations of their work.

“An increasing number of students are thinking, ‘I want to be trained as a computer scientist and I want to use my skills after graduation,’ but also ‘I want to do something that I think will make a positive impact on the world,’” he says.

Together, the E3I team works with course instructors to develop educational modules that tightly pair ethical concepts with course-specific technical material. In an applied software design course, for example, students learn about accessible software and disability theory; in a theoretical algorithms course, they learn about algorithmic fairness and distributive justice; and in a game design course, they learn about addiction and consent.

Steve Engels, a computer science professor, teaching stream, says integrating an ethics module about addiction into his fourth-year capstone course on video game design felt like a natural extension of his lecture topic on ludology – in particular, the psychological techniques used to make games compelling – instead of something that felt artificially inserted into the course.

“Project-based courses can sometimes compel students to focus primarily on the final product of the course, but this module provided an opportunity to pause and reflect on what they were doing and why,” Engels says. “It forced them to confront their role in the important and current issue of gaming addiction, so they would be more aware of the ethical implications of their future work and thus be better equipped to handle it.”

By next year, each undergraduate computer science student will encounter E3I modules in at least one or two courses every year throughout their program. The team is also exploring the adoption of the E3I model in other STEM disciplines, from ecology to statistics. Beyond U of T, the team plans to share their expertise with other Canadian universities that are interested in developing a similar program.

“This initiative is having a huge impact,” McIlraith says. “You see it in the number of students we’re reaching and in our assessment results. But it’s more than that – we’re instigating a culture change.”

U of T researchers develop video camera that captures 'huge range of timescales'

Christopher.Sorensen — Mon, 13 Nov 2023 15:12:10 +0000

U of T researchers develop video camera that captures 'huge range of timescales'

Christopher.Sorensen Mon, 11/13/2023 - 10:12

Cutline

Researchers Sotiris Nousias and Mian Wei work on an experimental setup that uses a specialized camera and an imaging technique that timestamps individual particles of light to replay video across large timescales (photo by Matt Hintsa)

Topic

Faculty of Applied Science & Engineering

Faculty of Arts & Science

Graduate Students

Research & Innovation

Subheadline

“Our work introduces a unique camera capable of capturing videos that can be replayed at speeds ranging from the standard 30 frames per second to hundreds of billions of frames per second”

Computational imaging researchers at the ��Ƶ have built a camera that can capture everything from light bouncing off a mirror to a ball bouncing on a basketball court – all in a single take.

Dubbed by one researcher as a “microscope for time,” the imaging technique could lead to improvements in everything from medical imaging to the LIDAR (Light Detection and Ranging) technologies used in mobile phones and self-driving cars.

“Our work introduces a unique camera capable of capturing videos that can be replayed at speeds ranging from the standard 30 frames per second to hundreds of billions of frames per second,” says Sotiris Nousias, a post-doctoral researcher who is working with Kyros Kutulakos, a professor of computer science in the Faculty of Arts & Science.

“With this technology, you no longer need to predetermine the speed at which you want to capture the world.”

The research by members of the Toronto Computational Imaging Group – including computer science PhD student Mian Wei, electrical and computer engineering alumnus Rahul Gulve and David Lindell, an assistant professor of computer science – was recently presented at the 2023 International Conference on Computer Vision, where it received one of two best paper awards.

“Our camera is fast enough to even let us see light moving through a scene,” Wei says. “This type of slow and fast imaging where we can capture video across such a huge range of timescales has never been done before.”

Wei compares the approach to combining the various video modes on a smartphone: slow motion, normal video and time lapse.

“In our case, our camera has just one recording mode that records all timescales simultaneously and then, afterwards, we can decide [what we want to view],” he says. “We can see every single timescale because if something’s moving too fast, we can zoom in to that timescale, if something’s moving too slow, we can zoom out and see that, too.”

Postdoctoral researcher Sotiris Nousias, PhD student Mian Wei, Assistant Professor David Lindell and
Professor Kyros Kutulakos (photos supplied)

While conventional high-speed cameras can record video up to around one million frames per second without a dedicated light source – fast enough to capture videos of a speeding bullet – they are too slow to capture the movement of light.

The researchers say capturing an image much faster than a speeding bullet without a synchronized light source such as strobe light or a laser creates a challenge because very little light is collected during such a short exposure period – and a significant amount of light is needed to form an image.

To overcome these issues, the research team used a special type of ultra-sensitive sensor called a free-running single-photon avalanche diode (SPAD). The sensor operates by time-stamping the arrival of individual photons (particles of light) with precision down to trillionths of a second. To recover a video, they use a computational algorithm that analyzes when the photons arrive and estimates how much light is incident on the sensor for any given instant in time, regardless of whether that light came from room lights, sunlight or from lasers operating nearby.

Reconstructing and playing back a video is a matter of retrieving the light levels corresponding to each video frame.

The researchers “passive ultra-wideband imaging” approach uses a set of timestamps to detect the arrival of individual photons.

The researchers refer to the novel approach as “passive ultra-wideband imaging” that enables post-capture refocusing in time – from transient to everyday timescales.

“You don’t need to know what happens in the scene, or what light sources are there. You can record information and you can refocus on whatever phenomena or whatever timescale you want,” Nousias explains.

Using an experimental setup that employed multiple external light sources and a spinning fan, the team demonstrated their method’s ability to allow for post-capture timescale selection. In their demonstration, they used photon timestamp data captured by a free-running SPAD camera to play back video of a rapidly spinning fan at both 1,000 frames per second and 250 billion frames per second.

The technology could have myriad applications.

“In biomedical imaging, you might want to be able to image across a huge range of timescales at which biological phenomena occur. For example, protein folding and binding happen across timescales from nanoseconds to milliseconds,” says Lindell. “In other applications, like mechanical inspection, maybe you’d like to image an engine or a turbine for many minutes or hours and then after collecting the data, zoom in to a timescale where an unexpected anomaly or failure occurs.”

In the case of self-driving cars, each vehicle may use an active imaging system like LIDAR to emit light pulses that can create potential interference with other systems on the road. However, the researchers say their technology could “turn this problem on its head” by capturing and using ambient photons. For example, they say it might be possible to create universal light sources that any car, robot or smartphone can use without requiring the explicit synchronization that is needed by today’s LIDAR systems.

Astronomy is another area that could see potential imaging advancements – including when it comes to studying phenomena such as fast radio bursts.

“Currently, there is a strong focus on pinpointing the optical counterparts of these fast radio bursts more precisely in their host galaxies. This is where the techniques developed by this group, particularly their innovative use of SPAD cameras, can be valuable,” says Suresh Sivanandam, interim director of the Dunlap Institute for Astronomy & Astrophysics and associate professor at the David A. Dunlap department of astronomy and astrophysics.

The researchers say that while the sensors that have the capability to timestamp photons already exist – it’s an emerging technology that’s been deployed on iPhones in their LIDAR and their proximity sensor – no one has used the photon timestamps in this way to enable this type of ultra-wideband, single-photon imaging.

“What we provide is a microscope for time,” Kutulakos says. “So, with the camera you record everything that happened and then you can go in and observe the world at imperceptibly fast timescales.

“Such capability can open up a new understanding of nature and the world around us."

Researchers find similarities in the way both children and societies alter words' meanings

Christopher.Sorensen — Tue, 22 Aug 2023 13:22:31 +0000

Researchers find similarities in the way both children and societies alter words' meanings

Christopher.Sorensen Tue, 08/22/2023 - 09:22

Cutline

(photo by Steve Debenport/Getty Images)

Krystle Hewitt

Topic

Breaking Research

Computer Science

Faculty of Arts & Science

Research & Innovation

Subheadline

"Our hypothesis is that these processes are fundamentally the same"

An international team of researchers is using computer science to explore the knowledge foundation of word meaning in both child language development and the evolution of word meanings across languages.

Through a computational framework they developed, the researchers show how patterns of children’s language innovation can be used to predict patterns of language evolution, and vice versa.

The interdisciplinary work by ��Ƶ computer science researcher Yang Xu and computational linguistics and cognitive science researchers from Universitat Pompeu Fabra and ICREA in Spain, was recently published in the journal Science.

Associate Professor Yang Xu (photo by Matt Hintsa)

In the paper, the team investigates what is known as word meaning extension, which is the creative use of known words to express novel meanings.

The research aimed to look at how this type of human lexical creativity observed in both children and adult language users can be understood in a unified framework, says Xu, a senior author on the paper and an associate professor in the department of computer science in the Faculty of Arts & Science and the cognitive science program at University College.

“A common strategy of human lexical creativity is to use words we know to express something new, so that we can save the effort of creating new words. Our paper offers a unified view of the various processes of word meaning extension observed at different timescales, across populations and within individuals.”

Word meaning extension is often observed in the historical change or evolution of language, Xu adds. For example, the word “mouse” in English originally meant a type of rodent, but now also refers to a computer device.

On the other hand, word meaning extension is also observed in children as early as two years of age. For example, children sometimes use the word “ball” to refer to “balloon,” presumably because they haven’t yet acquired the right word to describe the latter object, so they overextend a known word to express what they want to say.

“In this study, we ask whether processes of word meaning extension at two very different timescales, in language evolution, which takes place over hundreds or thousands of years, and in child’s language development, which typically occurs in the order of months or years, have something in common with each other,” Xu says. “Our hypothesis is that these processes are fundamentally the same.

“We find, indeed, there’s a unified foundation underlying these processes. There is a shared repertoire of knowledge types that underlies word meaning extension in both language evolution and language development.”

To test their hypothesis and figure out what is in common between products of language learning and language evolution, the team built a computational model that takes pairs of meanings or concepts as input, such as “ball” versus “balloon,” “door” versus “key” or “fire” versus “flame,” and makes a prediction about how likely these concepts can be co-expressed under the same word.

In building their model, the researchers constructed a knowledge base that helps identify similarity relations between concepts as they “believe it is the key that makes people relate meanings in word meaning extension,” Xu says.

The knowledge base consists of four primary knowledge types grounded in human experience: visual perception, associative knowledge, taxonomic knowledge and affective knowledge. Pairs of concepts score high if they are measured to be similar in one or some of these knowledge types.

“The pair of concepts like ‘ball’ and ‘balloon’ would score high due to their visual similarity, whereas ‘key’ and ‘door’ would score high because they are thematically related or often co-occur together in daily scenarios,” Xu explains. “On the contrary, for a pair of concepts such as ‘water’ and ‘pencil,’ they would have little similarity measured in any of the four knowledge types, so that pair would receive a low score. As a result, the model would predict they can’t, or they are unlikely to, extend to each other.”

Figure A: Researchers demonstrate examples of child overextension that are also found in language evolution.
Figure B: The team developed a computational framework for investigating the possibility of a common foundation in lexical creativity. (images by Yang Xu)

Xu notes the team found all four knowledge types contributed to word meaning extension and a model that incorporates these types tends to better predict data than alternative models that rely on fewer or individual knowledge types.

“This finding tells us that word meaning extension relies on multifaceted and grounded knowledge based on people’s perceptual, affective and common-sense knowledge,” he says.

Built exclusively from children’s word meaning extension data, the model can successfully predict word meaning extension patterns from language evolution and can also make predictions in the reverse direction on children’s overextension when trained on language evolution data.

“This cross-predictive analysis suggests that there are shared knowledge types between children’s word meaning extension and the products of language evolution, despite the fact that they occur at very different timescales. These processes both rely on a common core knowledge foundation – together these findings help us understand word meaning extension in a unified way,” Xu says.

Xu stresses that existing research on child overextension has been typically discussed in the context of developmental psychology whereas word meaning extension in history is typically discussed in historical and computational linguistics, so this project aims to build a tighter connection between the two fields of research.

The researchers hope that additional computational modelling will shed light on other potential lines of inquiry, including the basic mechanisms at play in the historical evolution of word meanings and the emergence of word meanings in child development, as well as the origins of different semantic knowledge types and how they are acquired.

Researchers develop interactive ‘Stargazer’ camera robot that can help film tutorial videos

siddiq22 — Fri, 19 May 2023 19:02:56 +0000

Researchers develop interactive ‘Stargazer’ camera robot that can help film tutorial videos

siddiq22 Fri, 05/19/2023 - 15:02

Cutline

Research led by U of T computer science PhD candidate Jiannan Li explores how an interactive camera robot can assist instructors and others in making how-to videos (photo by Matt Hintsa)

Topic

Our Community

Computer Science

Faculty of Arts & Science

Faculty of Information

machine learning

Research & Innovation

A group of computer scientists from the ��Ƶ wants to make it easier to film how-to videos.

The team of researchers have developed Stargazer, an interactive camera robot that helps university instructors and other content creators create engaging tutorial videos demonstrating physical skills.

For those without access to a cameraperson, Stargazer can capture dynamic instructional videos and address the constraints of working with static cameras.

“The robot is there to help humans, but not to replace humans,” explains lead researcher Jiannan Li, a PhD candidate in U of T's department of computer science in the Faculty of Arts & Science.

“The instructors are here to teach. The robot’s role is to help with filming – the heavy-lifting work.”

The Stargazer work is outlined in a published paper presented this year at the Association for Computing Machinery Conference on Human Factors in Computing Systems, a leading international conference in human-computer interaction.

Li’s co-authors include fellow members of U of T's Dynamic Graphics Project (dgp) lab: postdoctoral researcher Mauricio Sousa, PhD students Karthik Mahadevan and Bryan Wang, Professor Ravin Balakrishnan and Associate Professor Tovi Grossman; as well as Associate Professor Anthony Tang (cross-appointed with the Faculty of Information); recent U of T Faculty of Information graduates Paula Akemi Aoyaui and Nicole Yu; and third-year computer engineering student Angela Yang.

A study participant uses the interactive camera robot Stargazer to record a how-to video on skateboard maintenance (supplied photo)

Stargazer uses a single camera on a robot arm, with seven independent motors that can move along with the video subject by autonomously tracking regions of interest. The system’s camera behaviours can be adjusted based on subtle cues from instructors, such as body movements, gestures and speech that are detected by the prototype’s sensors.

The instructor’s voice is recorded with a wireless microphone and sent to Microsoft Azure Speech-to-Text, a speech-recognition software. The transcribed text, along with a custom prompt, is then sent to the GPT-3 program, a large language model which labels the instructor’s intention for the camera – such as a standard versus high angle and normal versus tighter framing.

These camera control commands are cues naturally used by instructors to guide the attention of their audience and are not disruptive to instruction delivery, the researchers say.

For example, the instructor can have Stargazer adjust its view to look at each of the tools they will be using during a tutorial by pointing to each one, prompting the camera to pan around. The instructor can also say to viewers, "If you look at how I put ‘A’ into ‘B’ from the top,” Stargazer will respond by framing the action with a high angle to give the audience a better view.

In designing the interaction vocabulary, the team wanted to identify signals that are subtle and avoid the need for the instructor to communicate separately to the robot while speaking to their students or audience.

“The goal is to have the robot understand in real time what kind of shot the instructor wants," Li says. "The important part of this goal is that we want these vocabularies to be non-disruptive. It should feel like they fit into the tutorial."

Stargazer’s abilities were put to the test in a study involving six instructors, each teaching a distinct skill to create dynamic tutorial videos.

Using the robot, they were able to produce videos demonstrating physical tasks on a diverse range of subjects, from skateboard maintenance to interactive sculpture-making and setting up virtual-reality headsets, while relying on the robot for subject tracking, camera framing and camera angle combinations.

The participants were each given a practice session and completed their tutorials within two takes. The researchers reported all of the participants were able to create videos without needing any additional controls than what was provided by the robotic camera and were satisfied with the quality of the videos produced.

While Stargazer’s range of camera positions is sufficient for tabletop activities, the team is interested in exploring the potential of camera drones and robots on wheels to help with filming tasks in larger environments from a wider variety of angles.

They also found some study participants attempted to trigger object shots by giving or showing objects to the camera, which were not among the cues that Stargazer currently recognizes. Future research could investigate methods to detect diverse and subtle intents by combining simultaneous signals from an instructor’s gaze, posture and speech, which Li says is a long-term goal the team is making progress on.

While the team presents Stargazer as an option for those who do not have access to professional film crews, the researchers admit the robotic camera prototype relies on an expensive robot arm and a suite of external sensors. Li notes, however, that the Stargazer concept is not necessarily limited by costly technology.

“I think there’s a real market for robotic filming equipment, even at the consumer level. Stargazer is expanding that realm, but looking farther ahead with a bit more autonomy and a little bit more interaction. So realistically, it could be available to consumers,” he says.

Li says the team is excited by the possibilities Stargazer presents for greater human-robot collaboration.

“For robots to work together with humans, the key is for robots to understand humans better. Here, we are looking at these vocabularies, these typically human communication behaviours,” he explains.

“We hope to inspire others to look at understanding how humans communicate ... and how robots can pick that up and have the proper reaction, like assistive behaviours."

Add new author/reporter

Krystle Hewitt