Video and Rich Media: Overcoming Performance and Scale Hurdles 

Video and rich media content is growing exponentially. Video now constitutes 50 percent of all data, and a rich media explosion is driving performance and capacity gains across countless markets.

This content enhances the capabilities of businesses across the globe. So, what’s the issue? The lack of infrastructure and management tools for video and rich media creates major performance and scale challenges for data centers serving both technical and non-technical environments.

Emerging Workloads

Data infrastructure and services for managing databases, virtual environments and traditional corporate file data are grounded in mature technologies that are not necessarily well-suited to support video. Some of them, like compression and deduplication, do not work at all. Others, like replication, are far too expensive. Uploading to the cloud is problematic because of large file sizes, and search and management tools for video are in their infancy.

At the same time, the use cases for video and rich media are expanding to new realms beyond media and entertainment, such as video surveillance, consumer images, voice and video, medical imagery, IoT and social media.

These use cases require advanced technology, but they are not technical computing markets. They need a usable, practical and adaptable approach to making video storage work for them. No matter what business they are in – intelligence, entertainment, healthcare or autonomous vehicle research – they all need the same capabilities: high-speed video ingestion, sophisticated high-volume analytics, catalogues instead of basic file systems, efficient distribution and economical storage tiering, including high-capacity media.

Usable, Practical, Adaptable

Businesses should not need to hire computer scientists to leverage rich media and video content.  Images, photographs, video, lidar, radar and IoT data require extremely low latency and high-speed storage, and it must be easy to use. For example:

• High-end auto makers might not manufacture Google AV test cars, but they are adding high-speed computing features to their cars. They don't want the heavyweight complexity of a supercomputing system - they need high-performance computing, massive storage at an economical price, as well as data analytics.
• New facilities such as state-of-the-art stadiums are turning to hyperconverged video and data to support their security operations.
• Video surveillance depends on video management software, analytics and license plate recognition.
• Sports video producers need to capture, edit, analyze and actively archive footage.
• IoT/sensor data processing generates massive raw data, and business operations account for a large share of data resources.
• Life sciences ingest massive volumes of raw data for processing and study. This data must be indexed and preserved to validate studies and for regulatory compliance, and kept available for future studies.

What to Expect

Video/rich media users expect the same sophisticated level of data services that they enjoy with text-based data: ingest, analyze, search and store. To achieve this level of rich data management, the market needs technology that manages these assets throughout their lifecycle.

The key to managing this content throughout its “workflow” or lifecycle is a compute/storage ecosystem that provides high-performance data movement, low cost/high capacity storage and tools for analysis and pattern recognition. For example, video surveillance continuously generates massive file sizes that users must protect and store. These data services need to efficiently search and identify matching patterns, such as license plates or faces. They may use pattern analysis to search for arguments between people, or a person holding a weapon, or an unattended briefcase under an airport chair. Video surveillance professionals need tools that keep their data immediately available and highly searchable.

How to Get There

Foundational technology, such as programmatic APIs, high speed flash, software data services and intelligent storage tiering, are making video/rich media lifecycle management a practical reality.

Tiering is key. From high performance ingestion to SSD flash tiers, to high speed disk and tape; intelligent, policy-driven tiering efficiently manages massive media files for performance and retention.

Promising technologies, such as high-speed NVMe, quickly ingests massive volumes of sensor data and video. From there, policy-driven storage tiering manages video for performance and capacity, keeping storage costs from spiraling out of control. Available today, these technologies have been used historically within the media and entertainment space and are now driving new, on-premises scalability architectures into other verticals and the enterprise.

For example, sports broadcasters recognize the value of using historical footage for fan engagement. Producers need immediate access to their footage to edit older video into new broadcasts. Cloud download speeds are far too slow for accelerated broadcast schedules, and disk-based mass storage is too expensive. In many cases broadcasters turn to tape, which is a cost-effective component in a cloud-like, video/rich media management infrastructure.

Users also need to search their assets, be it for intelligence on terrorist activity or stats from a baseball game. These queries require artificial intelligence that enriches the metadata upon ingestion, so users can intelligently search their assets for the information they require.

What’s Coming

Technology development is continuing to accelerate. Forward-thinking video lifecycle vendors are pouring R&D resources into AI development, on-premise storage scalability, higher performance and search and analytics. They are also moving towards bundling these technologies into managed services for their customers.

Another development to look for is cloud-managed, on-premise storage.  This approach combines the speed and cost savings of on-premise systems with the ease of use and elasticity of cloud offerings. In media and entertainment applications this approach could bridge the gap until full post-production can efficiently be done 100 percent in the cloud.

Customers expect their vendors to help them store, analyze, tier and protect their rich media in order to yield insights from it. It’s a challenge but also a rich opportunity for storage vendors who can step up and answer the call.

Jamie Lerner is president and CEO at Quantum Corp.