Data storage devices across the security industry are routinely required to handle an enormous amount and many layers of raw data. As Safe City projects in varying sizes become more prevalent, the number of surveillance nodes has reached the hundreds of thousands. And due to the widespread use of high-definition monitoring, the amount of data involved in security surveillance has increased dramatically in a short time. Efficient collection, analysis, and application of data and the intelligent use of it are becoming ever more critical in this industry. Thus, improving video intelligence appears to be an inevitable, industry-wide goal.

Security users hope that their investment in new products will bring even more benefits beyond simply tracing and tracking persons of interest and evidence collection after a security event. Some examples of added benefits include using the latest technologies to replace the large amount of man-power previously required for searching surveillance footage, detecting anomalous data, and finding ever more efficient ways to allow surveillance to shift from post-incident tracing to alerts during incidents—or even pre-incident alerts. In order to satisfy these demands, new technologies are required. Intelligent video surveillance has been available for many years. However, the outcomes of its application have not been ideal. The emergence of deep learning has enabled these demands to become reality.

The insufficiency of traditional intelligent algorithms

Traditional intelligent video surveillance has especially strict requirements for a scene’s background. The accuracy of intelligent recognition and analysis in comparable scenarios remains inconsistent. This is primarily due to the fact that traditional intelligent video analysis algorithms still have many flaws. In an intelligent recognition and analysis process, such as human facial recognition, two key steps are required: First, features are extracted, and second, “classification learning” is performed.

The features in traditional intelligent algorithms are designed by humans and have always been heavily subjective

The degree of accuracy in this first step directly determines the accuracy of the algorithm. In fact, most of the system’s calculation and testing workload is consumed in this part. The features in traditional intelligent algorithms are designed by humans and have always been heavily subjective. More abstract features—those that humans have difficulty comprehending or describing—are inevitably missed. With shifting angles and lighting, and especially when the sample size is enormous, many features can be too difficult to detect. Therefore, while traditional intelligent algorithms perform well in very specific environments, subtle changes (image quality, environment, etc.) yield significant challenges to accuracy.

Target detection and attribute recognition

The second step—classification learning—mainly involves target detection and attribute recognition. As the number of available categories for classification rises, so does the difficulty level. Hence, traditional intelligent analysis technologies are highly accurate in vehicle analysis but not in human and object analysis. For example, in vehicle detection, a distinction is made between a vehicle and a non-vehicle, so the classification is simple and the level of difficulty is low. To recognise vehicle attributes requires recognition of different vehicle designs, logos, and so on. However, there are relatively few of these, making the classification results generally accurate. On the other hand, if recognition is to be performed on human faces, each person is a classification of its own, and the corresponding categories will be extremely numerous—naturally leading to a very high level of difficulty.

The accuracy of intelligent recognition and analysis in comparable scenarios remains inconsistent
Enhanced accuracy is the result of multi-layer learning and extensive data collection

Traditional intelligent algorithms generally use shallow learning models to handle situations with large amounts of data in complex classifications. The analysis results are far from ideal. Furthermore, these results directly restrict the breadth and depth of intelligent applications and further development. Hence the need for increasing the “depth” of intelligence in big data for the security industry is arising.

The advantages of Deep Learning and its algorithms

Traditional intelligent algorithms are designed by humans. Whether or not they are designed well depends greatly on experience and even luck, and this process requires a lot of time. So, is it even possible to get machines to automatically learn some of the features? Yes! This is actually the objective of Artificial Intelligence (AI).

The inspiration for deep learning comes from a human brain’s neural networks. Our brains can be seen as a very complex deep learning model. Brain neural networks are comprised of billions of interconnected neurons; deep learning simulates this structure. These multi-layer networks can collect information and perform corresponding actions. They also possess the ability for object abstraction and recreation. Deep learning is intrinsically different from other algorithms. The way it solves the insufficiencies of traditional algorithms is encompassed in the following aspects.

Algorithmic model for deep learning

The algorithmic model for deep learning has a much deeper structure than the two 3-layered structures of traditional algorithms. Sometimes, the number of layers can reach over a hundred, enabling it to process large amounts of data in complex classifications. Deep learning is very similar to the human learning process, and has a layer-by-layer feature-abstraction process. Each layer will have different “weighting,” and this weighting reflects on what was learned about the images’ “components.” The higher the layer level, the more specific the components. Simulating the human brain, an original signal in deep learning passes through layers of processing; next, it takes a partial understanding (shallow) to an overall abstraction (deep) where we can perceive the object.

Deep learning does not require manual intervention but relies on a computer to extract features by itself. This way it is able to extract as many features from the target as possible, including abstract features that are difficult or impossible to describe. The more features there are, the more accurate the recognition and classification will be. Some of the most direct benefits that deep learning algorithms can bring include achieving comparable or even better-than-human pattern recognition accuracy, strong anti-interference capabilities, and the ability to classify and recognise thousands of features.

Hikvision has operated in the security industry for many years with its own research and development capabilities

Key factors of Deep Learning

In total, there are three main reasons why deep learning only became popular in recent years and not earlier: the scale of data involved, computing power, and network architecture.

Improvements in data-driven algorithm performance have accelerated deep learning in various intelligent applications in a short amount of time. Specifically, with the increase in data scale, algorithmic performance improved as well. Accordingly, user experience has improved and more users are involved, further facilitating a larger scale of data.

Video surveillance data makes up 60% of big data, and the amount is rising at 20% annually. The speed and scale of this achievement is due to the popularisation of high definition video surveillance—HD 1080p is becoming more common, and 4K and higher resolutions are gradually being applied in many important applications.

Hikvision has operated in the security industry for many years with its own research and development capabilities, employing large amounts of real video and image data as training samples. With a large amount of good quality data, and over a hundred team members to label the video images, sample data with millions of categories have been accumulated. With this large amount of quality training data, human, vehicle, and object pattern recognition models will become more and more accurate for video surveillance use.

The deep learning model requires a large amount of samples, making a large amount of calculations inevitable
Enhanced accuracy is the result of multi-layer learning and extensive data collection

Higher computational power

Furthermore, high performance hardware platforms enable higher computational power. The deep learning model requires a large amount of samples, making a large amount of calculations inevitable. In the past, hardware devices were incapable of processing complex deep learning models with over a hundred layers. In 2011, Google’s DeepMind used 1,000 devices with 16,000 CPUs to simulate a neural network with approximately 1 billion neurons. Today, only a few GPUs are required to achieve the same sort of computational power with even faster iteration. The rapid development of GPUs, supercomputers, cloud computing, and other high performance hardware platforms has allowed deep learning to become possible.

Finally, the network architecture plays its own role in advancing deep learning. Through constant optimisation of deep learning algorithms, better target-object recognition can be achieved. For more complex applications such as facial recognition or in scenarios with different lighting, angles, postures, expressions, accessories, resolutions, etc., network architecture will impact the accuracy of recognition, i.e., the more layers in deep learning algorithms, the better the performance.

In 2016, Hikvision achieved the number one position in the Scene Classification category at the ImageNet Large Scale Visual Recognition Challenge 2016. The team from Hikvision Research Institute used inception-style networks and not-so-deep residual networks that perform better in considerably less training time, according to Hikvision’s experiments for training and testing.

Furthermore, Hikvision’s Optical Character Recognition (OCR) Technology, based on Deep Learning and led by the company’s Research Institute, also won the first price in the ICDAR 2016 Robust Reading Competition. The Hikvision team substantially surpassed both strong domestic and foreign competitors in three word-recognition challenges, including born-digital images, focused scene text, and incidental scene text, demonstrating that the word recognition technology by Hikvision reached the world’s top level.

In the past two years, deep learning technology has excelled in speech recognition, computer
vision, voice translation,
and much more

Application of Deep Learning products

In the past two years, deep learning technology has excelled in speech recognition, computer vision, voice translation, and much more. It has even surpassed human capabilities in the areas of facial verification and image classification; hence, it has been highly regarded in the field of video surveillance for the security industry.

In the application of intelligent video in target detection, tracking, and recognition, the rise of deep learning has had a profound influence. When applying those three functions, deep learning potentially touches upon every aspect of the security video surveillance industry: facial detection, vehicle detection, non-motor vehicle detection, facial recognition, vehicle brand recognition, pedestrian detection, human body feature detection, abnormal facial detection, crowd behaviour analysis, multiple target tracking, and so on.

These types of intelligent functions require a series of front-end surveillance cameras, back-end servers and other products which support deep learning algorithms. In small scale applications, front-end cameras can directly operate structured human and vehicle feature extraction, and tens of thousands of human facial images can be stored within the front-end devices to implement direct facial comparison, so as to reduce costs of communicating with a server. In large scale applications, front-end cameras can work with back-end servers. Specifically, the structured video task is handled by front-end devices, reducing the workload for back-end devices; matching and searching efficiency of back-end servers improve as well.

Hikvision new products with Deep Learning

This year, Hikvision will soon introduce a series of products with deep learning technology, such as the DeepInview Series cameras which can accurately detect, recognise, and analyse human, vehicle, and object features and behaviour, and can be widely used in indoor and outdoor scenarios. Another of products worth mentioning is Hikvision’s DeepInmind Series of NVRs which incorporate advanced deep learning algorithms and imitate human thoughts and memory. The DeepInmind products feature an innovative NVR+GPU mode, retaining the advantages of traditional NVRs and additional structured video analysis functions, which together greatly improve the value of video.

Deep learning is the next level of AI development. It is beyond machine learning where supervised classification of features and patterns are set into algorithms. Deep learning incorporates unsupervised or “self-learning” principles. Hikvision is developing this concept in its own analytics algorithms. Enhanced accuracy is the result of multi-layer learning and extensive data collection. Application of this algorithm into face recognition, vehicle recognition, human recognition, and other platforms will significantly advance the performance of analytics.

Download PDF version

In case you missed it

Where is it inappropriate to install video cameras?
Where is it inappropriate to install video cameras?

Video cameras are everywhere, and hundreds more are installed every day. Our society appears to be reaching a point of perpetual surveillance. It certainly feels as if we are always being watched even though it is not yet the case. But as cameras are becoming more common than ever, we are also entering a new era of privacy concerns and sensitivities, as evidenced by GDPR and other such initiatives. We presented this quandary to this week’s Expert Panel Roundtable: Surveillance cameras can go anywhere, right? Where is it “not OK?”

Development of integrated thermal imaging technology into evolving market verticals
Development of integrated thermal imaging technology into evolving market verticals

Global and domestic threats have highlighted the need for tighter security across all verticals. One of the technologies that has redefined situational awareness and intrusion detection is thermal imaging. Once a technology exclusively manufactured for the military operations, thermal cameras today are deployed across hundreds of security applications and continue to see strong demand in existing and emerging commercial markets. With thermal technology, security personnel can see in complete darkness as well as in light fog, smoke and rain Technology overview and early adoption What distinguishes thermal cameras from optical sensors is their ability to produce images based on infrared energy, or heat, rather than light. By measuring the heat signatures of all objects and capturing minute differences between them, thermal cameras produce clear, sharp video despite unfavorable environmental conditions. With thermal technology, security personnel can see in complete darkness as well as in light fog, smoke and rain. Originally a military developed, commercially qualified technology, the first thermal cameras for military and aircraft use appeared in the 1950s. By the 1960s, the technology had been declassified and the first thermal camera for commercial use was introduced. However, it wasn’t until the late 1990s - when FLIR Systems introduced a camera with an uncooled thermal detector - when the technology began to see substantial adoption beyond government defense deployments. Installations at critical infrastructure sites In the 2000s, industrial companies were some of the first adopters of thermal, using the technology for predictive maintenance to monitor overheating and machine malfunctions. In the years following the September 11 terrorist attacks in 2001, there was an increase in thermal camera installations across critical infrastructure sites. Stricter security requirements drove the deployment of thermal cameras for perimeter protection, especially in the nuclear power sector. Thermal cameras produce clear video in daylight, low light or no light scenarios and their sharp images result in higher performing analytics In 2010, the U.S. Nuclear Regulatory Committee released its 73.55 policy, which states nuclear facilities must “provide continuous surveillance, observation and monitoring” as a means to enhance threat detection and deterrence efforts onsite. Because thermal cameras produce clear video in daylight, low light or no light scenarios and because their sharp images result in higher performing analytics, thermal cameras quickly became the preferred option for nuclear facilities. Likewise, following the 2013 sniper attack on PG&E Corporation’s Metcalf transmission substation, the Federal Energy Regulation Commission introduced the Critical Infrastructure Protection Standard 014 (CIP-014). The policy requires utilities to identify threats to mission critical assets and implement a security system to mitigate those risks. This statute also led to more thermal installations in the utility sector as thermal cameras’ long-range capabilities are ideal for detection of approaching targets beyond the fence line. The demand from both industrial and critical infrastructure entities, as well as other factors, helped drive volume production and price reduction for thermal, making the technology more accessible to the commercial security marketplace. Commercial applications In recent years, the increasing affordability of thermal cameras along with the introduction of new thermal offerings has opened the door to new commercial applications for the technology. In the past, thermal cameras were designed for applications with enormous perimeters, where the camera needed to detect a human from 700 meters away. Locations like car dealerships, marinas and construction supply facilities can be protected by precise target detection, thermal analytic cameras providing an early warning to security personnel Today, there are thermal cameras specifically designed for short- to mid-range applications. Developed for small to medium enterprises, these thermal cameras ensure property size and security funds are no longer barriers to adoption. Lumber yards, recreation fields and sports arenas are some of the commercial applications now able to implement thermal cameras for 24-hour monitoring and intrusion detection. Affordable thermal cameras with onboard analytics have become attractive options for commercial businesses Innovation and advancements Innovation and advancements in the core technology have also spurred growth in thermal camera deployment, providing faster image processing, higher resolution, greater video analytic capabilities and better camera performance. In particular, affordable thermal cameras with onboard analytics have become attractive options for commercial businesses that need outdoor, wide area protection. Car dealerships, marinas and construction supply locations all store valuable merchandise and materials outside. Without protection, these assets are vulnerable to vandalism and theft. However, by providing precise target detection, thermal analytic cameras provide an early warning to security personnel so that they can intervene before a crime is committed. By helping to deter just one incident, the thermal solution delivers a clear ROI. New market opportunities Not only are there more thermal cameras in use today than ever before, but there are also more thermal sensors being integrated with other multi-sensor systems, driving the adoption of thermal in new markets. For large perimeter surveillance applications, thermal is repeatedly being integrated with radar and drones to expand situational awareness beyond the point of fixed cameras. Users get immediate, accurate alerts of approaching targets and evidentiary class video for target assessment In the commercial market, thermal imagers are combined with optical sensors, analytics and LED illuminators into one solution that integrates with central monitoring station platforms. By bringing these technologies together, users get immediate, accurate alerts of approaching targets and evidentiary class video for target assessment. The result is a lower number of false positives, reducing the total cost of ownership for the solution. These multi-sensor solutions also feature two-way audio capabilities, which enable remote security officers to act as “virtual guards” and speak to intruders in real-time to dissuade them from illegal activity. The introduction of solutions that integrate all these state-of-the-art technologies under one unit reduces the amount of capital and infrastructure needed for deployment. Consequently, more small businesses and alarm monitoring companies can implement advanced perimeter security technologies like thermal sensors, some for the very first time. Thermal cameras have gone from military defense devices to widespread commercial security cameras Multi-sensor thermal solutions Multi-sensor solutions featuring thermal are quickly gaining traction and opening the door to new business opportunities for the security channel. One of the primary reasons for the strong market interest in these systems is they enable integrators to increase their recurring monthly revenue (RMR). With intense price competition and eroding margins on CCTV equipment, integrators have to rely on RMR to grow their businesses. Offering remote video monitoring services and virtual guarding technologies is one of the best ways to do so.  Additionally, there is a clear demand for it. Central stations are continually looking for new technologies to offer their customers and businesses are interested in economical alternatives to physical guards. In conclusion, thermal cameras have gone from military defense devices to widespread commercial security cameras that are a substantial segment of the outdoor security protection market. From nuclear power plants to construction locations, thermal technology is being implemented to secure sites around the globe.

Highlighting the importance of security integrations and alliances
Highlighting the importance of security integrations and alliances

Most technology companies have one goal in mind: to provide customers with high-quality, affordable products that can efficiently help streamline operations. Whether it's surveillance cameras, video management software, access control technology or any other type of security device, today's leading organisations invest in expertise in these product segments and strive to produce the highest quality solutions. To effectively fulfill this task, technology providers are always searching for emerging components to make their products and services even stronger. Oftentimes, a key aspect necessary to build a comprehensively robust solution involves finding like-minded partners that share a common goal and are willing to work together to create an integration that increases insight and intelligence.The interoperability between systems, devices and different types of applications should be intuitive and fast Key factors for security integrations A basic factor in a partnership is openness. For an integration to perform seamlessly for the end user, the platform through which the technologies converge must follow standard protocols, easily operate with other platforms, allow freedom and customisation, and provide adaptability. The interoperability between systems, devices and different types of applications should be intuitive and fast, enabling more time to be spent on analysing critical data and responding to security events. The puzzle of a complete security solution contains many pieces, and it's often necessary to fuse together aspects from various providers to create a best-in-breed technology offering. When organisations collaborate, the end result is a simplified solution with an increased level of value. As threats become more severe and complex, customers demand solutions that combine different security and business elements into a single interface that can address a wide variety of risks. A unified security system requires a strong collaboration between technology providers and integrated solutions Interconnected security devices Users used to only look at specific security devices - such as cameras or door alarms - as each having a strong, autonomous purpose, but now, every device plays an important interconnected role. And the progression of the Internet of Things (IoT) has made this transition even easier, as maintaining a consistent and uniform communication and interconnectivity between devices has now become the norm. The IoT has also made it so that partnerships must not only exist between manufacturers, but also within the customer's organisational structure. Although exceptionally beneficial, the IoT brings with it an increased amount of cyber vulnerabilities. As security systems are networked to increase flexibility, the door is opened to a number of potential threats that could compromise the entire enterprise. This risk highlights the need for an internal partnership and knowledge sharing between a company's physical security professionals and its IT team. IT experts should be pulled into security decisions and positioned as collaborative partners to assist with software updates, data safety protocols and solving complex network challenges, ultimately leading to a more cyber secure solution.Partnerships are beneficial to both the companies involved and their customers Knowledge sharing and learning Aside from cybersecurity, the latest prominent security attacks and events have focused primarily on soft targets, such as schools, concerts or shopping malls. This has caused many technology providers to venture into different vertical markets, and strong partnerships streamline this crossover. Innovators can extend their geographic reach and purpose through integrations with other like-minded manufacturers or integrators to add new levels of functionalities. Of course, a partnership cannot operate properly and to the best of its ability without a core component: learning. In today's evolving business and risk environment, knowledge is critical. A shared knowledge base can open up new opportunities and lead to the strengthening of security across many levels. A truly powerful, unified security system requires a strong collaboration between technology providers and integrated solutions. Partnerships are beneficial to both the companies involved and their customers, and the results created through these alliances can reach far beyond a user's expectations, offering enhanced flexibility and extensive safety options.