Gaps in video from dropped video frames are a common problem. A web search on the phrase “dropped video frames” turns up links oriented towards gamers, YouTube videographers and even the world of video surveillance, discussing why frame drops happen and providing advice for coping with them.

On Windows computers, video frame drops are a fact of life. Much of the advice can help reduce the number of frame drops, though never completely eliminate them. If you are an amateur filmmaker, and you drop key video frames, you can always do another “take.”

Not so in the 24/7 world of security surveillance. You never know which few seconds of video might be critical in determining what really happened, who was really there and who could be held liable.General use IT products like a Windows computer are architected to be event-driven and time-shared

Causes of video frame drops in surveillance systems

To begin to address this problem, it’s important to have a better understanding of the major causes of video frame drops in a video surveillance system.

Many video surveillance systems stream the output from the video cameras over a network to an off-the-shelf computer such as a Windows machine, which acts as the video server and runs your Video Management Software (VMS). The video server stores all the video files to racks of disk drives (disk drives being an affordable choice for the large amount of storage needed in a multi-camera system). Let’s focus on these major sources of video frame drops:

  • Video servers dropping incoming video frames when various system tasks compete for resources
  • Disk heads in your disk drives that can’t write the incoming video files fast enough to keep up with the continuous stream of data

    Many integrators who use off-the-shelf Windows computers will ensure the computer is dedicated to being the video server

Video processing overload

General IT products like a Windows computer are architected to be event-driven and time-shared. They strive for fairness and are designed to share the processing capacity so that no single task is starved. Instead, all tasks get a smaller proportion of processing capacity.

This means new events, like a keystroke or automatic start of a common “background” task or system service, can interrupt and delay ongoing tasks like the handling of your incoming video streams.

In the networked world, that is usually a good choice, since the network protocols will recover by resending the data and the latency that adds is not a problem. This does not work with video. Your cameras are continuing to send new video, so even if you could resend, this would simply bog down the computer still further and cause even more frames to drop.

If the CPU, memory or disk handling required by both the new event and your ongoing video processing exceeds the capacity of the computer, its default way of dealing with the overload is to drop data coming in over its network connections.

By using disk systems that are optimised for video storage, you can improve your odds
Disk drives are optimised for reading and writing data quickly, generally in a variety of block sizes depending on what is being stored

Dropped video frame testing

Recently, we started monitoring and testing for dropped video frames on off-the-shelf Windows video servers. In our tests, just clicking on a Windows disk management command resulted in almost 4,000 frames being dropped, an 8-second gap in the video record – and as we all know, a lot can happen in 8 seconds.

Even we were surprised by how often and how consistently frames were dropped, coinciding with various demands on the servers from things like background tasks. Almost always, these drops happened as a cluster, multiple seconds of contiguous video simply gone, a much more serious issue than dropping a few random frames here or there.The best solution is to use a machine and operating system that are optimised to be part of a video surveillance system

Many integrators who use off-the-shelf Windows computers will ensure the computer is dedicated to being the video server, and they will try to tweak the Windows operating system to reduce the number of background tasks, like removing any third-party antivirus programs or performance programs. This reduces the number of video frame drops but does not come close to eliminating all of them.

Video surveillance system solution

The best solution is to use a machine and operating system that are optimised to be part of a video surveillance system, designed to funnel resources as needed to handling the incoming video streams versus attempting to share resources evenly across all tasks or dropping incoming video data when the system is overloaded. By giving priority to the incoming video data and being sized correctly for the number and type of cameras, these systems are much more robust and less prone to dropping frames.

Optimising storage for surveillance

The previous sections looked at how video frames get dropped before your VMS even sees them. However, frames can go missing on the other side of the system when the storage system cannot keep up with the incoming video data.

Most systems use some form of disk storage; because to store all the video from multiple cameras running 24/7 can require a massive amount of storage – frequently into the terabytes – and disk storage is still affordable on this scale.

Disk drives are optimised for reading and writing data quickly, generally in a variety of block sizes depending on what is being stored. If you think about a database, various data records are written once, read many times as people look up data and modified (written again). Storing the video output from multiple cameras breaks this model in several ways:

  • The block sizes for video data are larger than typical data files
  • The blocks are written once and very seldom read, so it is very write-intensive
  • New files from multiple cameras are continuously coming in to be written
    When you need high performance and reliability from a system that faces heavy demands, off-the-shelf may simply not be good enough

Avoid missing key video

All these factors keep the disk head extremely busy and lead to longer seek times. Essentially, the disk head cannot write fast enough and cannot keep up with the incoming stream of video data to be stored. The result is that some video frames get dropped.

By using disk systems that are optimised for video storage, you can improve your odds. Solutions that improve head seek time, consolidate multiple video files into one file to reduce the number of writes needed and find other ways for the disk head to work more efficiently can greatly reduce the chances that frames might be dropped.

The off-the-shelf components that are frequently used for video surveillance systems are general purpose machines that do a decent job of supporting a small system with a minimal number of cameras. However, even in a small system with only 3 to 4 cameras, our testing has shown that video frame drop is a given, due partly to Windows’ bias to share resources among tasks and default to dropping incoming data/video streams when busy, and partly to off-the-shelf disk drives that cannot keep up with the write-intensive nature of storing video feeds.

These issues are due to the size and nature of video files, but are not unique to surveillance systems. In the film industry, they too use custom machines as their video servers to handle incoming streams from their digital cameras and store those streams to custom storage hardware. When you need high performance and reliability from a system that faces heavy demands, off-the-shelf – though less expensive initially – may simply not be good enough. Only you can determine how much tolerance you have to the risk of missing key video sequences, and whether your potential liability and the risk of lawsuits is increased beyond an acceptable measure by a surveillance system that captures less than 100% of the action.

Download PDF version

Author profile

Sean Chang co-founder, CEO & President, Rasilient Systems, Inc.

In case you missed it

Smart home access control growth and the future of door security
Smart home access control growth and the future of door security

There’s growing noise around smart homes and smarter security. You’ve probably heard it. But there is a place where access control and more have been smart for decades: the workplace. Home automation and IoT are still playing catch-up with the commercial sector. A new insights report from ASSA ABLOY and IFSEC Global — “The Smart Door Locks Report 2018” — measures just how fast consumer smart technology is running. According to a survey conducted for the report, 61% of households now claim to own at least one smart home device or system. Energy monitors, home CCTV cameras, intruder alarms and smart door locks are the most popular, according to the report. All these functions, of course, have been available to businesses for years.61% of households now claim to own at least one smart home device or system Educating the smart home consumer Paradoxically, report data also questions how much consumers really know about their smarter home. A surprising 42% of those surveyed, for example, were unaware they could control a smart door lock from their phone. In fact, many leading smart door lock models offer this feature, delivered by Wi-Fi or Bluetooth and an app. Despite a wealth of features offered by the latest smart door locks — remote and location-based locking/unlocking; voice activation; timed access; emailed entry alerts; and integration with smart camera and lighting systems — most people still have limited knowledge of their capabilities.  Smart technology is increasingly becoming the new norm in terms of home security  Only 14% of survey respondents described themselves as “very familiar” with what a smart lock can do. Even though most of them probably use smart access control solutions at their workplace. Secure homes through smart technology Monitoring and security are not the only drivers for smart home adoption. We humans also love convenience, and modern living presents us with problems that smart home technology can solve. Ironically, given the report’s findings, it takes a smartphone to really unlock the convenient possibilities of smarter living. The device that’s “always to hand” is central to the newest generation of smart door locks.A smart door lock is a convenient way for a landlord or agency to offer round-the-clock check-in and check-out If homeowners wish to remotely manage property access for friends and family, many smart door locks oblige. You let in guests remotely, send them a virtual digital key, or provide a temporary or single-use PIN to unlock the door. It is just as easy to revoke a digital key, if you don’t want its owner to come around anymore. This is a significant improvement over sharing physical keys — or hiding one under the doormat. We cannot be totally sure where a metal key ends up and have no way to track or cancel it once it’s “out in the wild”. Commercial access control offers such functionality as standard, of course.  In addition, smart door locks offer more than just stand-alone operation and clever functions. In a domestic setting, magic happens when locks work in harmony with a home automation system, connected by protocols like Z-Wave, ZigBee or Wi-Fi. "Smart" security on the move  The smartphone is becoming a remote control for managing a connected life beyond just home (and even workplace) security. According to Accenture, the parcel delivery services market will grow by $343 billion by 2020. Just like home security, convenience is a major driver of change. Homeowners can send guests a virtual digital key to their phones, or provide a temporary or single-use PIN to unlock the door A recent PostNord pilot in Sweden aimed to remove the inconvenience of waiting home for a postal delivery. Selected customers of some major Scandinavian e-retailers could choose to have parcels delivered inside their front door, if it was equipped with a Yale smart door lock.  Home delivery is among potential smart services covered in “The Smart Door Locks Report 2018 ”. When asked whether the ability to receive parcels securely in a porch or lobby would make them more likely to invest in a smart door lock, 79% said it would.It is easy to revoke a digital key, if you don’t want its owner to come around anymore Holiday rentals and smart home tech ASSA ABLOY research published in 2017 forecasts continued growth in the European holiday rentals sector (at 5.8% CAGR). Smart door locks are also making an impact here, at both ends of the market: for service providers — agents and homeowners — and for travellers. A smart door lock is a convenient way for a landlord or agency to offer round-the-clock check-in and check-out, without creating extra work or staff costs. Both Intersoft, in Croatia, and Hoomvip in Spain have built holiday rentals management systems around an app and the ENTR® smart door lock. Agents issue, revoke, track and manage virtual keys for all their guests, saving everyone time and hassle. Travellers use their phones and an app to unlock their apartment. For these visitors the smartphone is already an essential travel accessory. It is a boarding pass, a credit card, a travel guide, and a postcard home... why not a door key, too? And if this key is backed by a trusted home security brand — and a company with vast experience in the mature market for commercial “smart” security — better still.

Bosch startup SAST addresses need for evolved solutions in security industry
Bosch startup SAST addresses need for evolved solutions in security industry

Security and Safety Things GmbH (SAST) is a new company that has announced its vision for an Internet of Things (IoT) platform for the next generation of security cameras. The Bosch startup plans to build a global ecosystem for the development of innovative security camera applications. Based on the Android Open Source Project (AOSP), SAST provides libraries, an API framework, and codecs for developers to work with. The SAST App Store will allow developers to build and market new applications, similar to today’s app stores for smartphone applications. We presented some questions to Nikolas Mangold-Takao, VP Product Management and Marketing, about the new venture, and here are his responses: Q: Why a new company now? What technology innovations have made this a good time to launch this company? The time is right to bring market needs and technological innovations together on one platform"Mangold-Takao: From a technical perspective we see two main drivers: increasing computing power at the edge and increasing internet connectivity, which will enable devices to directly communicate with each other and bring new technologies such as artificial intelligence also to the security and safety industry. At the same time, we see that this industry and its users are hungry for more innovative solutions – addressing new security needs while at the same leveraging the possibility to improve business operations for specific verticals, e.g. retail and transportation. The time is right to bring market needs and technological innovations together on one platform for this industry. Q: Why does SAST need to be a separate entity from Bosch? Mangold-Takao: SAST is setup as a wholly owned subsidiary of the Bosch Group. We wanted to make sure that SAST is able to underline its role as an industry standard platform across multiple players. SAST is open to get additional investors and is being setup as a startup in its own offices in Munich to foster the environment where speed and innovation can more easily take place. Having said that, several entities of the Bosch Group are very interesting partners for SAST. The SAST App Store will allow developers to build and market new applications, similar to today’s app stores for smartphone applications Q: Please explain your "value proposition" to the industry. Mangold-Takao: We will bring new innovations and possibilities to the security and safety industry by providing an open, secure and standardised Operating System for video security cameras, to also address pressing issues such as cyber security and data privacy concerns. Devices that run then with the SAST operating system will work with an application marketplace provided and operated by SAST. Integrators and users can then use these apps from this marketplace to deploy additional functionality on these devices. With our platform we will be able to build up a community of app developers, including the ones not yet developing for this industry who have expertise in computer vision and artificial intelligence. Q: It seems what you are doing has parallels with the Apple and Android "app" stores. How is your approach the same (and how is it different) than those approaches? We are setting up SAST as a user-centric company and involve selected users very early on in the process"Mangold-Takao: The approach is similar in the way that we plan to generate revenue by operating the application marketplace and thus participate in the app revenue. The difference is that there is much more needed than apps and cameras to create a complete working solution addressing a user problem in this industry – we need to make sure that our own platform as well as the new applications being created will work as a part of an end-to-end solution. Q: "Critical mass" and wide industry participation seem to be requirements for your success. How will you achieve those goals? Will you involve integrators, consultants, or other parties in addition to manufacturers (to drive awareness)? How? Mangold-Takao: SAST is in close exchange with device manufacturers, integrators and consultants, as well as application developers and large end-users at the moment to ensure that we are building the right platform and ecosystem for this industry. We are setting up SAST as a user-centric company and involve selected users very early on in the process. We will run dedicated programs and hackathons to attract app developers, already active and new to our industry. We will also run selected pilots with end-users throughout 2019 to ensure we have all partners involved early on. SAST sees the industry is hungry for more innovative solutions – with the retail vertical market a target for these solutions Q: What timeline do you foresee in terms of implementing these initiatives? Mangold-Takao: While we start with first app development programs and plan our first pilots already for this year, we are planning our commercial launch for end of 2019. Q: How does your new company relate to the new Open Security & Safety Alliance (OSSA)? Mangold-Takao: The Open Security and Safety Alliance has been working very closely with SAST over the past year, defining some important concepts and elements required. One of the most important elements is an open and standardised Operating System, specific to this industry, which will then bring forward new innovative technologies and solutions. SAST is actively working on this Operating System, based on Android Open Source Project (ASOP), but is evolved and hardened with industry-specific features. Q: What's the biggest thing you want the security industry to understand about SAST? What is your "message" to the industry? Mangold-Takao: Our message is simple: let’s build better security and safety systems – together! But for real, innovating an industry is a joint effort, we can only bring new innovation to this industry with partners who share our vision and are excited about new technology. At the same time, we strongly believe that our platform allows every partner to bring forward what they do best but also invite new partners to our industry.

What is the value of remotely monitoring a system's health and operation?
What is the value of remotely monitoring a system's health and operation?

When is it too late to learn that a video camera isn’t working properly? As any security professional will tell you, it’s too late when you find that the system has failed to capture critical video. And yet, for many years, system administrators “didn’t know what they didn’t know.” And when they found out, it was too late, and the system failed to perform as intended. Fortunately, in today’s technology-driven networked environment, monitoring a system’s health is much easier, and a variety of systems can be deployed to ensure the integrity of a system’s operation. We asked this week’s Expert Panel Roundtable: How can remote monitoring of a security system’s health and operation impact integrators and end users?