Storage and Back up

What is the next step in this process? When our character has a professional video recorder it all comes down to taking a couple of cassettes, copying the product and nothing else. This simple conclusion requires a significant investment, which many filmmakers cannot do. In fact, more and more small producers are setting up a project studio without having their own recorder. And in some cases the solution to be viable and remain competitive is given by the new storage options that have appeared on the market.

Suppose this character has a CD-R drive on his computer. If we talk about a 30-second commercial to which a 2:1 compression has been applied, the arithmetic tells us that it is possible to "print" a consolidated version of the commercial on a single disc ... that can be easily taken to a facility that has both a digital video system and the famous recorder that is needed to generate a master of the commercial. Total cost of the solution? Less than $5 per disk, with an investment of less than $500 for a good CD-R drive. And what the copy-to-tape service costs, which is still charged to the costs of the project.

In short, this character can go on the market and compete in favorable conditions without spending fifteen or twenty thousand dollars on a video recorder that would spend a good part of the time without working.

Like all solutions that come from "lateral thinking", these types of outputs usually have costs on other fronts. It is evident that our character will need to invest a little more time every time he needs to copy material to videotape. And that will become a relatively high response time that could prove inconvenient to meet the needs of some customers. It is clear that not all projects can benefit from a solution of this type: If we are talking about a 30-minute documentary, it may not be practical to generate eighteen or twenty discs to transport it.

However, it is a viable solution that comes from a single technical resource: The availability of CD-R discs that allow registering more than 600Mb in each dish. A few years ago it was quite difficult to store information packets of four or five Gb. Normally this required the use of linear storage systems, which were always characterized by their slowness and low reliability. How many times were your projects hurt by fortuitous failures in your backup systems? These kinds of problems were part of the everyday life of graphic designers until about three or four years ago. Obviously a CD-R or CD-RW system is not a direct replacement for a DAT streamer that can accommodate 4Gb in each cartridge... but it certainly makes things easier.

We are talking about dramatic changes in work systems. Users of digital video stations can benefit from this technology, thus breaking some of the restrictions that traditionally applied to non-linear editors.

What is the most tense point of the relationship with a customer? The moment of change of projects. The producer is forced to continue storing the customer's material while the customer decides what are the last changes he is going to ask for... and in the meantime the following client has to wait because the limited storage of the editing equipment is fully or partially occupied. And despite everything you have to handle this type of situation with a smile ...

When possible these types of problems are solved with brute force: More disks are purchased, which are managed as independent "banks", and the cost scheme is adjusted to accommodate the rental of the disks during the term of each project. So each client has their space, everyone keeps their projects "online" and everyone is still happy... minus the team owner, who has to increase the investment without getting a significant return.

Fortunately the costs of hard drives continue to fall, We talk about the special disks that are needed to handle audio and video ... they are especially expensive drives because they ensure very high performance for long periods, and are known as AV disks. The most commonly used AV disks are SCSI drives that are mostly used in ordinary data processing applications. Most network servers use AV disks because they offer significant performance improvements. Some people say that much of the web resides on AV disks... and perhaps this is true.

As always the increase in demand contributes to the reduction of costs by the increase in the scale of manufacturing. In fact, the most requested models of 9 and 18Gb hard drives are less expensive today than they were two years ago. Interestingly the technology seems to have frozen at this point. In recent years there have been no spectacular developments in SCSI technology, apart from the buried failure of most 23Gb drives that in many cases could not match the performance of their lower capacity relatives.

Very well... storage technology continues to advance by leaps and bounds... and most users still use the exact same hard drive models that were originally released two or three years ago. Some of these models have names that are already recognized in the industry, Barracudas, Fireballs and other bugs like that are part of our work tools. And temperamental SCSI technology remains the support of the industry.

How can this happen as compression rats decrease with consequent increased bandwidth requirements? –Or putting the question in Spanish, why do we continue to use the same discs even if the demand of the teams increases? – . The answer is very simple: The same discs are used, but in a different way.

The magic word is RAID: Redundant Array of Independent Disks. Which is simply a sophisticated way of describing the use of multiple disks in parallel to increase the capacity of digital video systems. Do I have a disc that gives me 5Mb per second? If I install two in parallel I can reach 10Mb/s.... and possibly already reach to make effects in real time. And if I put a third party maybe I can take my compression rat below 2:1.

The use of RAID technology has spread dramatically thanks to the fact that the most frequently used operating systems support it natively. And the industry very quickly acquired the know-how of this technology thanks to the convergence between digital video systems and ordinary personal computers. In fact, the 36Gb RAID system on SCSI disks that serves a network of 35 users is the same one that can eventually be used to edit a music video.

However, the importance of a conscious work of installing storage systems should not be minimized. A small delay in the delivery of information can lead to serious failures in a digital video system. In fact, the combination of the right elements to achieve predictable and satisfactory results is still very important.

This is why nonlinear editor manufacturers continue to sell fully integrated storage systems under their own brand: They know what they do, and they charge for it. Don't forget that a misconfigured storage system can become a nightmare.

SCSI technology remains the engine of the industry, at least on certain levels. Why change if with SCSI disks we can serve the PAL and NTSC market? In addition, we are at a time when we can obtain results similar to those accustomed with much lower investments in storage.

There are several factors in the industry that have led to the emergence of new solutions. The first is the emergence of systems based on the DV compression scheme, which operates with a fixed data rat of 25MBits/Second. Generating storage systems that achieve this performance is not difficult. In fact, some ordinary hard drives, the same ones that operate in home computers, can achieve this performance.

Let us clarify this point. We are talking about hard drives with IDE type interface. I'm storing this text on a 6.4Gb IDE disk that cost me about $150 less than a year ago. This particular disc could handle digital video with 6:1 compression without any effort. And if I had invested something more in a state-of-the-art UDMA disk with the appropriate controller and a good installation job I would perfectly reach a rat of 3.8Mbytes/s, equivalent to that of many SCSI disks and perfectly enough to handle DV material.

Let's take a short technical break. Until some time ago one of the fundamental differences between AV hard drives and ordinary disks was the ability to deliver uninterrupted digital information streams. This meant a very precise and difficult to execute mechanical and electronic design. Thanks to the ingenuity of the computer industry, ordinary hard drives now have the ability to deliver continuous streams thanks to the use of caching technology. Caching or intermediate storage is the use of a limited space of RAM that allows to "cover" the bumps in the performance of the disk, delivering information without any pause even if the disk mechanism stops reading the platters for a few fractions of a second.

Suppose we want to install a DV-based editing system that allows us to make some effects in real time. This implies that the storage system must have the ability to handle two simultaneous streams of digital video, at least for part of the time. What is the obvious solution? Install a RAID system with a pair of $250 IDE disks. This attitude leads to one of the most powerful trends in the storage industry in recent years: The use of ordinary hard drives for applications with limited requirements.

Let's leave a question before we continue. Doesn't it make sense to think about using IDE collections to back up my sophisticated SCSI array? It can be a fairly quick way to make backup copies, which in some cases can take less than three times the duration of the material while a DLT drive takes eight or nine times the actual time. Considering the costs, it sounds pretty good... although it is still a solution with many restrictions.

Another factor that simplifies storage problems at some levels of the market is the emergence of MPEG compression systems with editable streams. The temporal compression of MPEG systems is much more efficient than the spatial compression of conventional MJPEG systems, and this can significantly reduce the demand for storage systems.

Suppose we have an editing system that handles an MPEG stream of about 1.5 Mbytes/second, which can offer acceptable quality for many applications. It is perfectly possible that a single IDE disk can hold the two flows necessary to be able to make effects in real time. Obviously this would need a very clever design to avoid jumps in the image, but that can be solved by using RAM with some liberality.

At this moment, home-type editing systems based on this technology are beginning to appear on the market, offering a storage capacity of 45 minutes with real-time effects for a really ridiculous cost. Obviously, the convergence between MPEG compression and conventional desktop video is beginning to generate very powerful products, which are mostly oriented to the market for the production of material for DVD titles.

So what is the space for the most recently appeared products? We are talking about disks with FibreChannel connections, for example. For now, these devices have a niche in DTV/HDTV technology, or in the segment of systems that handle uncompressed material. In this segment of the market, the line that separates the issues of storage and connectivity is becoming increasingly diffuse, and this makes it evident that even on this front an interconnected future awaits us.

Another technology that arouses great expectations is DVD-RAM, which in its initial versions supports up to 4.7Gb on each disc. In addition to its possibilities as a backup system, it is perfectly possible that in the near future it will directly support the continuous playback of material for applications that require limited bandwidth. Wouldn't it be interesting to use an MPEG compression-based editing system that operates with interchangeable discs of almost 5GB?

Some time ago we filmmakers tended to worry too much about the issue of storage. Work on nonlinear editing stations included periodic reviews of the remaining space, accompanied by eternal commentary about how enjoyable it would be to have a room full of hard drives for just one.

Although the market offers us more and more space, we have become accustomed to operating with limited storage and many times this means significant improvements in the quality of the products. Despite the initial resistance we have ended up accepting the process of logging the material, which allows us to judiciously manage the available space. Latino producers have finally understood the power of off-line publishing and increasingly tend to adopt the work systems for which these devices were designed.

And from the hand of this process comes the adoption of a healthy attitude towards the issue of storage: The important thing is that it works...