Topic Discussion: Use of Sound for Sensing

Introduction

The age of sensors is upon us. Dr. Janusz Bryzek, former VP Development of MEMS and Sensing Solutions at Fairchild Semiconductor, has shared a vision of 1 trillion sensors in use by 2022.[1] Gartner says that by 2020 there will be 20 billion connected things in use worldwide—most of them, we believe, will have more than one sensor embedded.[2] Sensors are important because they collect numerous streams of data that can be analysed and interpreted to increase productivity, strengthen security, predict threats, and improve lives. In this quarterly report, Fluxus examines the use of sound for sensing, a well-exploited method that is having a renaissance due to advancements in data processing and increasing concern for privacy. We begin by discussing various basic concepts of sound and sound-based sensors and continue with applications of sound for sensing.

Basic Concepts

Sounds are vibrations that travel through a medium (air, water, steel, etc.) in the form of a sound wave. A sound wave is generated and emitted when an object vibrates. In the case of a vibrating object in air, the vibrating object moves the surrounding air molecules back and forth at the same rate at which the object is vibrating. These moving air molecules in turn move adjacent air molecules back and forth, which then move their neighbouring air molecules back and forth. The end result is that a sound wave travels outward from the vibrating object. Physically speaking, the sound wave is made up of fluctuations in air pressure that result from air molecules moving back and forth. These fluctuations in air pressure can be thought of as vibrations and can be detected with a variety of different sensors.

When speaking about sound, the most relevant descriptors are the frequency, amplitude, and phase of the sound wave. The frequency of a sound wave is related to how fast the object generating the sound is vibrating. Objects vibrating very quickly generate higher frequency sound waves, and, when heard by the human ear, sounds with a higher frequency have a higher pitch. The amplitude of a sound wave is a measure of how strong the sound wave is. High amplitude sound waves will be louder when heard by the human ear. Lastly, the phase of a sound wave has to do with the difference in timing of two or more sound waves. Simply put, two sound waves that are in phase with each other will combine their amplitudes to make a wave that sounds twice as loud. Two sound waves that are completely out of phase will cancel each other out and dampen the sound. Differences in phase give rise to sound interference are can be cleverly used in devices like noise-cancelling headphones. These measures of frequency, amplitude, and phase characterize the properties of a sound wave at any given time. By observing how these properties change over time, one can gain useful insights about the environment through which a sound wave is traveling.

Sounds can have a wide range of frequencies, so it is instructive to categorize sounds into several general categories. At the high end of the scale, sounds with frequency greater than 20 kHz are characterized as ultrasound. Ultrasounds cannot be detected by the human ear and have many technological applications. Acoustic sounds are those that are audible to the average human. These sounds have frequencies between 20 Hz and 20 kHz. There are technical applications for acoustic waves, but their amplitude must be limited because the human ear can be damaged by loud sounds with these frequencies. Sounds with frequencies below 20 Hz are called infrasounds. Infrasounds can be used for applications like earthquake detection and detection of nuclear explosions. Acoustic sounds, ultrasounds, and infrasounds are all sound waves that have the same general form. The only key difference is that they have differing frequencies. Due to the differing frequencies these different types of sound waves interact with the physical world in slightly different ways, and, as a result, each type of sound wave can be used for differing specific applications.

To measure sound waves and output electrical signals for analysis, sound sensors or microphones are used. There are several different types of microphones that use differing mechanisms, but they all aim to convert sound waves, and the vibrations they cause within the medium they are traveling, into electrical signals. Dynamic microphones are commonly used for measuring acoustic sounds and are essentially loudspeakers operated in reverse. When a sound wave enters a dynamic microphone, the wave causes the diaphragm in the device to vibrate at a specific frequency. The movement of the diaphragm causes an induction coil within the device to move in response, which generates an electric current. Piezoelectric sensors are often used for more sophisticated applications and for detecting sound waves of a wide range of frequencies. This type of sensor utilizes piezoelectric materials, which directly generate electrical signals when they vibrate. Thus, when the material within the sensor is caused to vibrate when a sound wave travels through it, an electrical signal is generated, and the sound is detected. There are several other methods of detecting sound as well. However, each of these methods utilize the principle that sound waves generate vibrations and movements in the medium through which they travel. By detecting these vibrations, one can convert a sound wave into an electrical signal that can be measured and analysed.

Applications

Intrusion Detection and Passive Monitoring

We start from the most obvious application, intrusion detection. For households, a Swede company called Minut has an IoT product that uses a combination of sensors (including microphones) to act as an alarm for intrusion. The device has a microphone array that measures a wide range of sounds from very low to very high frequencies and a Passive Infra-Red (PIR) sensor to detect motion when you are not at home (suggesting an intruder) and detect whether an alarm has been set off (essentially adding connectivity to any traditional alarm systems). Minut plans to release glass-window-break detection and no-tampering (the device will know if it is being moved) features soon, expanding its functionality. The company seems to be doing well, it has recently closed a USD 2.5 million funding and Crunchbase points out that companion app downloads and web traffic are increasing at double-digit rates. We take special note that this company, which uses microphones and PIR sensors, comes from Europe which is culturally seen as being highly concerned of privacy.

In securing infrastructure over a very large area (such as roads, railways, and pipes), a new technology called Distributed Acoustic Sensing (DAS) promises a very cost-effective and reliable system to detect intrusion and tampering. The technology allows detection of sound waves using a long fibre optic cable that vibrates in response to the sound waves travelling through the ground. It can be used to look for disturbances and anomalies along the cable with 10 metre resolution.[3] Essentially, sound waves traveling above or through the ground cause the long fibre optic cable to vibrate at specific frequencies. These vibrations can be detected with extremely high accuracy using the DAS system. The system can detect footsteps, digging, and even low-flying drones.[4] There is also significant potential for detection of other things, greatly expanding the use cases, as the system “learns” and associates various sounds with activities or events. OptaSense, the leading provider of DAS systems, has joined forces with Deutsche Bahn, a leading rail provider in Europe, to validate OptaSense’s solution in detecting various train related events: Hot axle box, rock and tree fall, animal detection, train integrity tracking, wheel flats, etc.[5] Optasense is a subsidiary of QinetiQ Group, a company specialising in defence technology. Their solution is currently deployed across multiple industries in over 40 countries.[6]

DAS has also found potential use below ground. A group of scientists at Berkeley Lab published a paper in Nature describing their experiment in using DAS techniques to detect changes in underground water content, permafrost thaw, and seismic activities (i.e. earthquakes). This finding is significant because it increases the resolution of the data in a cost-effective manner. Continuous, high-resolution monitoring is important in providing early warning for ground movements that may cause severe and life-threatening damage to structures and people above ground.[7] Some notable sinkhole events include two sinkholes in Brussels in 2017, a sinkhole in Florence in 2016, and one in Miami in 2017.[8],[9],[10] All of the events are related to underground water changes. Although we are not aware of any sinkhole swallowing a large office building, the construction of Lotte World Tower was suspected to be related to the appearance of several sinkholes in the neighbourhood around the construction site in 2014.[11]

Besides environment monitoring, sound is also being used to monitor machineries. Just like an experienced mechanic who would listen to a car’s engine to discern signs of troubles, machineries are currently being fitted with IoT devices that continuously listen to the sounds coming out of the machineries to look for early signs of failure or predict when machines will fail. These devices are the backbone of a new maintenance concept called Predictive Maintenance (PdM). In one of its document, NASA lists a few machine types that are suited for sound-based predictive maintenance sensors (by continuous detection of vibration or ultrasonic noise from the machines): pumps, electric motors, diesel generators, condensers, valves, heat-exchangers, electrical systems, and transformers.[12]

Augury is one of the companies in the PdM space that listens to and analyses sounds coming out of machines. It recently raised a USD 17 million Series B round, bringing its total funding amount to USD 26 million.[13] Augury has announced partnerships with various brand name companies such as Grundfos and Johnson Controls to test and sell its solutions.[14],[15] A similar company to Augury, called Dynamic Components GmbH analyses data from sensors (one of them being sound data) to gauge the conditions of elevators and escalators. Dynamic Components was established in 2016 and received a USD 25 thousand non-equity assistance from DB Accelerator in the same year.[16]

Beyond PdM, sound can also be used to monitor pipes for leakages. This is important because water leakage from pipes will most likely lead to further, more severe damage, especially in buildings. One company called AquaSeca has an IoT sensor that can be wrapped around an existing pipe and “listen” to the pipes for leaks. A collection of these sensors placed strategically on pipes around a building can provide continuous monitoring for leakage, ensuring that the maintenance team will be notified the minute a leak happens. Its sensors are compatible with a variety of water pipes such as copper, stainless steel, PVC, PEX, and flex metal. AquaSeca was established in 2015 and based in San Jose, California, US.

Startups like Augury and AquaSeca are taking advantage of recent advances in machine learning to analyse the frequencies of the sounds generated by machines and pipes. Machine learning algorithms can be trained to find and detect sound signatures that are out of the ordinary or that indicate a known failure mechanism. Up until recently, this type of analysis was not possible without a large amount of computing power. Now, companies can monitor the data streamed from sound sensors in real-time and detect anomalies on the fly using battery powered sensors with low power usage.

Object Detection and Active Monitoring

Use of sound for object detection is not a new thing, especially underwater where sound travels four times as fast as in air and where visibility is poor due to lack of light. Submarines use SONAR (SOund Navigation And Ranging) technology to detect other submarines and submerged objects. There are two types of SONAR, active and passive SONAR. Passive SONAR is like a microphone passively listening to underwater sounds while active SONAR emits sound waves and listens for reflected sound waves. SONAR can extract information such as distance, bearing and a rough size estimate of objects. Although very useful, traditional SONAR technology only gives one or two-dimensional representation of objects. We present below two companies with 3D sound-based sensing technology and applications.

The company CodaOctopus has a product called Echoscope which it claims to be the world’s highest definition real-time 3D imaging sonar. Established in 1994, CodaOctopus has several interesting case studies outlining the use of Echoscope, mostly in submarine construction and engineering projects. One case study is on how Echoscope is used in a project to join two submarine power cables by providing the project with georeferenced 3D image of the subsurface. This precious image allows contractors to properly plan the connection of the two cables. The Echoscope sensors were also used during the connection progress, providing visuals on the cables under the sea.[17]

Another company, called Toposens uses ultrasound to provide 3D imaging capabilities above the ground. It claims that its sensor can detect, count, and track objects and people in the near field (0.01 – 4 metres).[18] Established in 2015, the company is targeting three markets: automotive, smart building, and robotics. In automotive, the company believes that its sensor will very useful for autonomous driving and in-car gesture control. In real estate, the sensor will compete or complement traditional sensors such as WiFi triangulation, BLE Beacons and standard cameras for people counting and tracking (providing building owners with valuable analytics for better decision-making). In robotics, Toposens hopes to use its sensors as the “eyes” of the robots. Our diligence indicates that 3D ultrasound sensor technology “would be a huge, useful leap for autonomous driving”. Its application in smart building is also very interesting to follow, since anecdotally, a lot of existing sensors (WiFi, camera) evoke privacy concerns.

Other Applications

Another interesting application of sound-based sensors is in pinpointing gunfire. A company based in the Netherlands called Microflown AVISA has a sensor suite that relies on sound waves to detect and locate the origin of explosive projectiles. Microflown AVISA’s sensors can detect shots fired from small firearms, RPGs (rocket propelled grenades), up to artillery fire.[19] A similar company called ShotSpotter uses an array of sensors deployed strategically in cities to detect and pinpoint gunfire incidents. Its solution has been deployed in 75 cities in the United States and other countries and allows law enforcement to rapidly deploy its resources to tackle any gunfire incidents. [20]

As we have outlined in the report, many unique solutions can be developed with advanced sound sensing technology and data processing. We firmly believe that sound should not be overlooked as a valuable data stream. Using sound detection and analysis for predictive maintenance applications, anomaly detection, and people counting/tracking, should be of great interest for real estate players going forward.

[1]https://www-bsac.eecs.berkeley.edu/scripts/show_pdf_publication.php?pdfID=1365520205

[2] https://www.gartner.com/newsroom/id/3165317

[3] https://www.railengineer.uk/2014/04/24/acoustic-sensing-monitoring/

[4] https://www.economist.com/science-and-technology/2014/01/04/the-ear-underground

[5] http://www.optasense.com/optasense-announces-contract-with-europes-biggest-rail-provider-deutsche-bahn-netz-ag/

[6] http://www.optasense.com/about-us/global-company/

[7] https://www.nature.com/articles/s41598-017-11986-4

[8] http://www.flanderstoday.eu/current-affairs/second-sinkhole-brussels-sees-evacuation-and-damage-rail-line

[9] https://www.nytimes.com/2016/05/26/world/europe/sinkhole-florence-italy.html

[10] https://edition.cnn.com/2017/07/14/us/sinkhole-florida/index.html

[11] http://edition.cnn.com/2014/08/05/world/asia/seoul-skyscraper-sinkholes/index.html

[12] https://www.hq.nasa.gov/office/codej/codejx/Assets/Docs/RCMGuideMar2000.pdf

[13] Crunchbase

[14] https://www.businesswire.com/news/home/20160927005164/en/Augury-Unveils-OEM-Partnership-Program-Industrial-IoT

[15] http://www.johnsoncontrols.com/media-center/news/press-releases/2016/06/22/advanced-vibration-evaluation

[16] Crunchbase

[17] http://www.codaoctopus.com/touch-down-monitoring-cable-jointing-using-codaoctopus-real-time-3d-sonar-echoscope

[18] https://toposens.com/ts-alpha/

[19] http://microflown-avisa.com/audible-threats/small-arms-fire-saf/

[20] http://www.businessinsider.com/the-nypd-has-a-new-high-tech-system-for-pinpointing-gunfire-2015-3/?IR=T