Audio is often overlooked in the security and video surveillance industry. There are some intercom installations where audio plays a key role, but it’s not typically thought about when it comes to security and event management. Audio takes a back seat in many security systems because audio captured from a surveillance camera can have a different impact on the privacy of those being monitored.

Audio surveillance is therefore subject to strict laws that vary from state to state. Many states require a clearly posted sign indicating audio recording is taking place in an area before a person enters. Analytic information derived from audio can be a useful tool and when implemented correctly, removes any concerns over privacy or legal compliance.

Audio analytics on the edge overcomes legal challenges as it never passes audio outside of the camera

Focused responses to events

Audio analytics processed in the camera, has been a niche and specialised area for many installers and end users. This could be due to state laws governing audio recording, however, audio analytics on the edge overcomes legal challenges as it never passes audio outside of the camera

Processing audio analytics in-camera provides excellent privacy since audio data is analysed internally with a set of algorithms that only compare and assess the audio content. Processing audio analytics on the edge also reduces latency compared with any system that needs to send the raw audio to an on-premises or cloud server for analysis.

Audio analytics can quickly pinpoint zones that security staff should focus on, which can dramatically shorten response times to incidents. Audio-derived data also provides a secondary layer of verification that an event is taking place which can help prioritise responses from police and emergency personnel.

Having a SoC allows a manufacturer to reserve space for specialised features, and for audio analytics, a database of reference sounds is needed for comparison

Microphones and algorithms

Many IP-based cameras have small microphones embedded in the housing while some have a jack for connecting external microphones to the camera. Microphones on indoor cameras work well since the housing allows for a small hole to permit sound waves to reach the microphone. Outdoor cameras that are IP66 certified against water and dust ingress will typically have less sensitivity since the microphone is not exposed. In cases like these, an outdoor microphone, strategically placed, can significantly improve outdoor analytic accuracy.

There are several companies that make excellent directional microphones for outdoor use, some of which can also combat wind noise. Any high-quality external microphone should easily outperform a camera’s internal microphone in terms of analytic accuracy, so it is worth considering in areas where audio information gathering is deemed most important.

In-built audio-video analytics

Surveillance cameras with a dedicated SoC (System on Chip) have become available in recent years with in-built video and audio analytics that can detect and classify audio events and send alerts to staff and emergency for sounds such as gunshots, screams, glass breaks and explosions. Having a SoC allows a manufacturer to reserve space for specialised features. For audio analytics, a database of reference sounds is needed for comparison.

The camera extracts the characteristics of the audio source collected using the camera's internal or externally connected microphone and calculates its likelihood based on the pre-defined database. If a match is found for a known sound, e.g., gunshot, explosion, glass break, or scream, an event is triggered, and the message is passed to the VMS.

Each camera must be custom configured for its particular environment to detect audio levels which exceed a user-defined level
If a match is found for a known sound, e.g., gunshot, explosion, glass break, or scream, an event is triggered, and the message is passed to the VMS

Configuring a camera for audio analytics

Audio detection
The first job of a well-configured camera or camera/mic pair is to detect sounds of interest while rejecting ancillary sounds and noise below a preset threshold. Each camera must be custom configured for its particular environment to detect audio levels which exceed a user-defined level. Since audio levels are typically greater in abnormal situations, any audio levels exceeding the baseline set levels are detected as being a potential security event. Operators can be notified of any abnormal situations via event signals allowing the operator to take suitable measures. Finding a baseline of background noise and setting an appropriate threshold level is the first step.

Installers should be able to enable or disable the noise reduction function and view the results to validate the optimum configuration during setup

Noise reduction
A simple threshold level may not be adequate enough to reduce false alarms depending on the environment where a camera or microphone is installed. Noise reduction is a feature on cameras that can reduce background noise greater than 55dB-65dB for increased detection accuracy. Installers should be able to enable or disable the noise reduction function and view the results to validate the optimum configuration during setup. With noise reduction enabled, the system analyses the attenuated audio source. As such, the audio source classification performance may be hindered or generate errors, so it is important to use noise reduction technology sparingly.

Audio source classification
It’s important to supply the analytic algorithm with a good audio level and a high signal-to-noise ratio to reduce the chance of generating false alarms under normal circumstances. Installers should experiment with ideal placement for both video as well as audio. While a ceiling corner might seem an ideal location for a camera, it might also cause background audio noise to be artificially amplified. Many cameras provide a graph which visualises audio source levels to allow for the intuitive checking of noise cancellation and detection levels.

Analytics take privacy concerns out of the equation and allow installers and end users to use camera audio responsibly

Messages and events
It’s important to choose a VMS that has correctly integrated the camera’s API (application programming interface) in order to receive comprehensive audio analytic events that include the classification ID (explosion, glass break, gunshot, scream). A standard VMS that only supports generic alarms, may not be able to resolve all of the information. More advanced VMS solutions can identify different messages from the camera.

Well configured audio analytics can deliver critical information about a security event, accelerating response times and providing timely details beyond video-only surveillance. Analytics take privacy concerns out of the equation and allow installers and end users to use camera audio responsibly. Hanwha Techwin's audio source classification technology, available in its X Series cameras, features three customisable settings for category, noise cancellation and detection level for optimum performance in a variety of installation environments.

Download PDF version

Author profile

In case you missed it

Virtual worlds disrupt building security & facility management
Virtual worlds disrupt building security & facility management

From satellite imagery to street views to indoor mapping, technology has disrupted our past world. This has left us dependent upon new ways to visualise large spaces. This new world has brought many benefits and risks. But what does that mean for the security professional or facility manager today and what technologies can be used to secure buildings and improve facility operations? A brief history of 3D technology Starting May 5, 2007 (inception 2001), Google rolled out Google Street View to augment Google Maps and Google Earth; documenting some of the most remote places on earth using a mix of sensors (Lidar/GSP/Radar/Imagery). The mission to map the world moved indoors May 2011 with Google Business Photos mapping indoor spaces with low cost 360° cameras under the Trusted Photographer program. In the earlier days, 3D scanning required a high level of specialisation, expensive hardware and unavailable computing power With the growth of 3D laser scanning from 2007 onwards, the professional world embraced scanning as effective method to create digitised building information modelling (BIM), growing fast since 2007. BIM from scanning brought tremendous control, time and cost savings through the design and construction process, where As-Built documentation offered an incredible way to manage large existing facilities while reducing costly site visits. In the earlier days, 3D scanning required a high level of specialisation, expensive hardware, unavailable computing power and knowledge of architectural software. Innovation during the past 8 year, have driven ease of use and lower pricing to encourage market adoption. Major investments in UAVs in 2014 and the commercial emergence of 360° photography began a new wave of adoption. While 3D scanners still range from $20K – $100K USD, UAVs can be purchased for under $1K USD and 360° cameras for as low as $100. UAVs and 360° cameras also offer a way to document large spaces in a fraction of the time of terrestrial laser scanners with very little technical knowledge.  Access to building plans, satellite imagery, Google Street View, indoor virtual tours and aerial drone reconnaissance prove effective tools to bad actors The result over the past 10+ years of technology advancement has been a faster, lower cost, more accessible way to create virtual spaces. However, the technology advances carry a major risk of misuse by bad actors at the same time. What was once reserved to military personal is now available publicly. Access to building plans, satellite imagery, Google Street View, indoor virtual tours and aerial drone reconnaissance prove effective tools to bad actors. Al Qaeda terror threats using Google Maps, 2007 UK troops hit by terrorists in Basra, 2008 Mumbai India attacks, 2016 Pakistan Pathankot airbase attacks, ISIS attacks in Syria using UAVs, well-planned US school shootings and high casualty attacks show evidence that bad actors frequently leverage these mapping technologies to plan their attacks. The weaponization of UAVs is of particular concern to the Department of Homeland Security: "We continue to face one of the most challenging threat environments since 9/11, as foreign terrorist organisations exploit the internet to inspire, enable or direct individuals already here in the homeland to commit terrorist acts."   Example comparison of reality capture on the left of BIM on the right. A $250 USD 360° camera was used for the capture in VisualPlan.net software What does this mean for the security or facility manager today? An often overlooked, but critical vulnerability to security and facility managers is relying on inaccurate drawing. Most facilities managers today work with outdated 2D plan diagrams or old blueprints which are difficult to update and share.Critical vulnerability to security and facility managers is relying on inaccurate drawing Renovations, design changes and office layout changes leave facility managers with the wrong information, and even worse is that the wrong information is shared with outside consultants who plan major projects around outdated or wrong plans. This leads to costly mistakes and increased timelines on facility projects.  Example benefits of BIM There could be evidence of a suspect water value leak which using BIM could be located and then identified in the model without physical inspection; listing a part number, model, size and manufacture. Identification of vulnerabilities can dramatically help during a building emergency. First Responders rely on facilities managers to keep them updated on building plans and they must have immediate access to important building information in the event of a critical incident. Exits and entrances, suppression equipment, access control, ventilation systems, gas and explosives, hazmat, water systems, survival equipment and many other details must be at their fingertips. In an emergency situation this can be a matter of life or death. Example benefit of reality capture First Responders rely on facilities managers to keep them updated on building plans A simple 360° walk-through can help first responders with incident preparedness if shared by the facility manager. Police, fire and EMS can visually walk the building, locating all critical features they will need knowledge of in an emergency without ever visiting the building. You don’t require construction accuracy for this type of visual sharing. This is a solution and service we offer as a company today. Reality capture is rapidly becoming the benchmark for facility documentation and the basis from which a security plan can be built. Given the appropriate software, plans can be easily updated and shared.  They can be used for design and implementation of equipment, training of personnel and virtual audits of systems or security assessments by outside professionals. Our brains process visual information thousands of times faster than text. Not only that, we are much more likely to remember it once we do see it. Reality capture can help reduce the need for physical inspections, walk-throughs and vendor site-visits but more importantly, it provides a way to visually communicate far more effectively and accurately than before. But be careful with this information. You must prevent critical information falling into the hands of bad actors. You must watch out for bad actors attempting to use reality capture as a threat, especially photo/video/drones or digital information and plans that are posted publicly. Have a security protocol to prevent and confront individuals taking photos or video on property or flying suspect drones near your facility and report to the authorities. Require authorisation before capturing building information and understand what the information will be used for and by who.There are a number of technologies to combat nefarious use of UAVs today Nefarious use of UAVs There are a number of technologies to combat nefarious use of UAVs today, such as radio frequency blockers and jammers, drone guns to down UAVs, detection or monitoring systems. Other biometrics technologies like facial recognition are being employed to counter the risk from UAVs by targeting the potential operators. UAVs are being used to spy and monitor for corporate espionage and stealing intellectual property. They are also used for monitoring security patrols for the purpose of burglary. UAVs have been used for transport and delivery of dangerous goods, delivering weapons and contraband and have the ability to be weaponised to carry a payload.Investigating reality capture to help with accurate planning and visualisation of facilities is well worth the time The Federal Aviation Administration has prevented UAV flights over large event stadiums, prisons and coast guard bases based on the risks they could potentially pose, but waivers do exist. Be aware that it is illegal today to use most of these technologies and downing a UAV, if you are not Department of Justice or Homeland Security, could carry hefty penalties. Facility managers must have a way to survey and monitor their buildings for threats and report suspicious UAV behaviours immediately to authorities. At the same time, it’s critical to identify various potential risks to your wider team to ensure awareness and reporting is handled effectively. Having a procedure on how identify and report is important. Investigating reality capture to help with accurate planning and visualisation of facilities is well worth the time. It can help better secure your facilities while increasing efficiencies of building operations. Reality capture can also help collaboration with first responders and outside professionals without ever having to step a foot in the door. But secure your data and have a plan for bad actors who will try to use the same technologies for nefarious goals.

Intellectual honesty: the growth of Cobalt Robotics and robots in security
Intellectual honesty: the growth of Cobalt Robotics and robots in security

The best route to greater adoption of robotics in the field of physical security is intellectual honesty, says Travis Deyle, CEO and co-founder of Cobalt Robotics. “Robots are not a panacea, so we must be clear and honest about capabilities and use cases,” he says. “If you are dishonest, people will lose faith. We must have clear expectations about what’s feasible today and possible tomorrow.” The robotics tide is turning in the security market, which is notoriously slow to embrace new technologies. “The tone has changed at recent security events,” says Deyle. “Previously, robots were thought of as a science experiment. But now, there are big-name users wanting to discuss proof of concept. It has evolved from being a novelty to now it’s time to give it a serious look. They want us to help them sell the concept up the chain of command. It’s helpful to have conversations with other parts of the company because it has an impact on the culture of the company.” The robotics tide is turning in the security market, which is notoriously slow to embrace new technologies Cobalt’s robots are purpose-built for a specific use case: providing after-hours support and security for corporate locations. Indoor environments, confined and controlled, present fewer navigation challenges for robots, which can quickly become familiar with the surroundings and navigate easily through an office space. Indoor robots can provide benefits beyond security, too, such as facility management, promoting employee health and safety, and emergency response. Cobalt's human-centred design Cobalt’s robots also interact well with people. They are friendly and approachable and make employees feel safe and secure. The human-centered design promotes that interaction, and a real person (located remotely) can enter into any interaction instantly as needed. “We combine machines with people,” says Deyle. “We allow the machine to do what it does best, such as dull and boring activities, and add the flexibility and cultural relevancy of having a person there.” Cobalt’s robots also interact well with people, they are friendly and approachable and make employees feel safe and secure When a robot is deployed, it performs a brief mapping phase (about an hour), in which it moves around and builds up a “map” of its space and develops its patrol route. Over time, it lingers more in areas where it encounters more incidents. There are 60 sensors on the robot, including day/night cameras, high-resolution thermal cameras, a card reader that integrates with the corporate access control system, a microphone, and environmental sensors for temperature and humidity.  The robot builds models of what’s normal in its environment in terms of people, sound, motion, open doors and windows, and even leaks and spills. And then it detects anomalies and sends relevant notifications to Cobalt specialists, who respond and manage any events in real time. The machine provides unwavering attention, perfect recall, and accountability. Cobalt robots have been designed to help bridge the problems faced with utilising guards and cameras  Accommodating various anomalies  The Cobalt robot is designed to blend into a high-end office environment, with flexible fabric and a corporate design aesthetic. It is stable beyond 45-degrees, so it’s hard to topple over. The 5-foot-2-inch robot can see over desks and cubicles. It is designed to bridge the gap between guards, who are expensive and underutilised during uneventful night shifts, and cameras, which are unable to respond to nuanced situations. Cobalt Robotics already has customers in defense, finance and manufacturing, and a handful of Fortune 500 companies are looking at the service Autonomous navigation uses artificial intelligence (AI) and machine learning to avoid static and dynamic obstacles. Over time, the robot accommodates various anomalies such as loud machinery noise, and “semantic mapping” adds intelligence to its map. When the robot figures out that a picture on the wall is not a real person, for example, it stores that information for future reference.  The technologies enabling robotics in the indoor environment are mature – there have been variations of security robots in operation for decades. What has changed is the costs of the technologies, which are now inexpensive enough to make a robot affordable to businesses. Cobalt Robotics offers an all-inclusive service providing hardware, software, service and maintenance as well as the remote human interface. All together, the service is a third to half the cost of a man-guard, and it bills monthly, says Deyle. Cobalt Robotics offers an all-inclusive service providing hardware, software, service and maintenance as well as the remote human interface Cobalt Robotics already has customers in defense, finance and manufacturing, and a handful of Fortune 500 companies are looking at the service. They are currently operational in the San Francisco Bay area and Chicago and will be in six other geographies in the next three months (in response to customer needs). Uses include offices, museums, warehouses, technology centres, and innovation centres.  A former Google employee, Deyle’s experience in robotics goes back to his Ph.D. studies at Georgia Tech, where he worked on developing a robot to deliver healthcare to homebound patients. Deyle and Cobalt Robotics co-founder Erik Schluntz departed Google in 2016 to form Cobalt Robotics. In just 12 months, Cobalt went from the initial idea to paid robot deployments.

Which segments are under-served in the physical security industry?
Which segments are under-served in the physical security industry?

Physical security technologies operate successfully in many different markets, but in which markets do they fall short? Physical security is a difficult challenge that can sometime defy the best efforts of manufacturers, integrators and end users. This is especially the case in some of the more problematic markets and applications where even the best technology has to offer may not be good enough, or could it be that the best technology has not been adequately applied? We asked this week’s Expert Panel Roundtable to reflect on instances when the industry may fall short: Which segments of the physical security industry are most under-served and why?