Monthly Archives: February 2013

Documentary Data wrangling demystified, part III: Organizing for the Edit in FCPX

In part I of this post about data wrangling a feature-length documentary, we covered protecting your footage. In part II, we took a look at documentary film storage requirements. Today I’ll explain how we organized our footage in preparation for editing in Final Cut Pro X.

Here’s an overview of the workflow we used during production to get footage off our cards, stored safely, and organized for retrieval. Note that this is a dual-system workflow – we always recorded audio separately from video, requiring us to sync clips later in post in a workflow that I’ve previously covered.

1. Copy contents of video SD card into a new folder located within the project’s “Footage” folder (using naming convention below).
2. Copy contents of audio SD card as above into project’s “Audio” folder.

We stored the media on our Drobo Pro, a raid-like system that protects data in case of a drive failure.

Folder Structure for Organizing Media. The foundation of our organizational work begins at the Finder level, with a simple but powerful folder naming convention:

“YYYY-MM-DD two to four keywords”

Here’s a partial snapshot of our film’s audio folder:

You’ll notice that some dates have more than one folder. We created a new folder for each shoot, which helped immeasurably to quickly find footage later in the project.

The nice thing about using this date format as a prefix to tagging is that your folders will always sort in order. If you put the tags first, that wouldn’t be the case. And adding two to four keyword tags that succinctly describe the shoot allowed us instantly identify the contents.

3. Stills, if any, are copied into “Stills” folder, using same naming convention.
4. Back up data onto another hard drive which is stored at another location.

Item 4 is the hardest bit to keep up. In practice we tended to copy and move the files offsite at milestones in the project – after a particularly big shoot, about midway, after wrapping, etc.

In hindsight, our biggest workflow mistake was neglecting to view dailies. Most often, we would simply skim through footage after a shoot to ensure it was looking OK. But we never made time to properly review dailies. We figured we’d do that when it came time to edit. And that was a mistake.

One of our lavs developed a short halfway through production, and we never discovered the problem until post. Everything recorded in the second half of production with this mic sounded tinny. It required heroic EQ work to make it usable. Luckily we were paranoid and almost always recorded with a lav AND a boom mic.

Further, I wish we had begun to edit a few sequences together soon after shooting, instead of months later. That would have helped us discover whether we were covering scenes well enough to cut them together easily. Instead, we had to learn all of this the hard way months later when we finally got down to serious editing. Too often we discovered that we didn’t shoot enough b-roll (people without their lips moving, or just their hands, or ANYTHING that we could cut away to) to build the dialog easily.

Prepping for editing in Final Cut Pro X The day we wrapped production was the day Apple shipped Final Cut Pro X. We dove right in and never looked back. But it was difficult at first to understand Apple’s new “Event” and “Project” terminology.

I initially thought that it made the most sense to create one monolithic Event, and import the entire film into it. Wrong! Final Cut loads all open Events and Projects into memory, so everything slows to a crawl if too many clips are open concurrently. It’s best to have as few Events and Projects as possible open at a time.

So what we ended up doing, which worked very well, was to create an Event structure that mirrored our folder structure almost exactly. We created a new Event for every shoot, titled YYYY-MM-DD plus two or three descriptive keywords. So after import, like so:

Importing and organizing into FCPX is not trivial when you’re talking more than 4 terabytes of footage. It took us about a month and a half of work to get all of our footage moved over from raw storage and organized the way we wanted them in FCPX Events. Seriously, it was a LOT of work. It wasn’t as simple as hitting the “import” button.

To Proxy or not to Proxy? Proxy media is 1/4 resolution ProRes files, which FCPX can create for you on import. With FCPX, it’s a dead-simple process to create and use proxy media. When importing, just check the “proxy” box.

It takes quite awhile to generate proxy media, though, so with serious amounts of footage, be prepared for some lengthy transcode times. With Final Cut Pro X, you can be doing other things at the same time, albeit a bit sluggishly.

I would argue that it’s almost essential to use proxy media on a feature-length project, because you’ll likely need to have many Events open at a time. And proxy will give you the ability to open many more than you would with full resolution media. Luckily, it’s easy to switch back and forth between proxy and full resolution at any time. Press cmd-, to call up preferences, select Playback, and choose “Proxy.” We didn’t have to switch back to Original Media again until we were ready to start creating assemblies.

After making proxies, our next step, and the most tedious and time-consuming part of our workflow, was audio syncing. I’ve outlined our audio syncing process and it worked very well, but it added a lot of time to the process. I really wish Final Cut Pro X had a way of batch syncing audio automatically. You can use PluralEyes, but I’m not crazy about the mess it leaves behind. It requires a lot of cleanup if you like to maintain, as I do, a clean separation between media and projects. So I prefer DualEyes.

A note about this workflow. If we had been on FCPX from the beginning of our film, this process would have been much smoother, because we would have been doing this import > transcode > audio sync process daily, instead of facing it all at once at the end. I definitely recommend making this a part of your dailies ritual it time allows.

The most significant improvement for Final Cut Pro X to add, from my feature-length perspective, would be to make it possible to load an entire film all at once. I simply couldn’t do it with my hardware (3.4 GHz Intel Core i7 2011 iMac, 16GB ram, AMD Radeon 2 gig graphics card). Attempting it just displays the spinning beach ball. I’ve since tried it on my new 2012 MacBook Pro, on which I can just get the entire film to load. Even then, it’s too sluggish to make it practical. But it makes me suspect that Apple’s playing the long game with FCPX – by the time next year’s Macs are released, the hardware may well have caught up with the software on this, allowing FCPX to work acceptably well feature film situations.

As a result of this, we spent a stupid amount of time over the life of the project just moving Event and Project folders back and forth between Final Cut Events and Final Cut Events – Hidden folders. Each time, we had to restart FCPX. Tick, tick, tick…

It wasn’t until the very end, when I was audio mixing the film, that I discovered Event Manager X, a $5 utility that makes this process relatively effortless by allowing you to create lists of Projects associated with Events. It let’s you move them and restart FCPX with a single click. If you’re cutting your feature doc on FCPX, this app is indispensable.

Getting organized inside of Final Cut Pro X With everything imported, audio synced, and proxified, a final step remains: adding metadata. Final Cut Pro X provides a two incredibly powerful ways of organizing footage: Smart Folders and Keywords.

Smart Folders

Our first step is to create smart folders that split our media into types: audio only, video only and sync video. Smart folders are just standing searches, or filters, that allow you to display everything that matches your query. When you create a smart folder, you can specify what you’re looking for. In the case of our Audio only example, it looks like this:

For our Events, we generally wanted to be working on sync video when we got down to editing, because that was the best sound. But sometimes we had video for which there was no audio recorded separately. It’s easy to set up a Smart Folder that will tell you at a glance which files are which, by adding additional qualifiers to the search:

You can take this pretty far, but we found that for most Events, we could find everything we needed with just three or four smart folders.

Event structure for sharing Projects between editors As the project took shape over time, we shared Project files back and forth between two editors, my partner Lisa and I. We discovered that it’s very important to “lock” Events and add the inevitable new media that become necessary during post (such as sound effects, archival footage, archival stills, etc) to a new Event, rather than adding them to existing Events. That’s because if they’re added to existing Events, it’s too hard to keep track of who has the most current Event. Then, when you share a Project, you don’t know if there will be missing media when the other editor opens it.

Our solution was to create an Event called “BN Additional Footage” and another called “BN Music and SFX.” We added all the new files to the appropriate folder, and then whenever we shared a Project, we included just these two Events to ensure no missing media. This made Project sharing much more manageable.

Keywords

Keywords are the second most powerful way to organize footage in FCPX. I reach for Keywords whenever I can’t figure out a way to do it with a Smart Folder. For example, this shoot was rather long, and consisted of five different “scenes”, as follows:

Tagging each part of the shoot with a descriptive keyword gave me a way to instantly find only those clips. I tend to use Keywords to break apart Events into chunks like this. My partner, Lisa, tends to use them to identify her favorites, and clips that would make good b-roll. You can use Keywords however you like.

Favorites The last step of getting organized might actually be considered the first step of editing: making favorites. My process for this involved standing at a desk, with my finger hovering over the “i” and “o” keys, as I played back the footage. Whenever I saw or heard something that looked promising, I’d make a selection and press the “f” key to mark it as a favorite.

To make it possible to find your favorite later, it’s essential to add a text descriptor. I add mine by typing it into the “notes” field, which I reposition to display just to the right of the “name” field in the Event Browser (click and drag on the field to move it).

If the favorite is extra special, I use a simple notation I learned from Werner Herzog at Rogue Film School. Herzog has a notebook open on his lap when reviewing footage. When he sees a favorite, he writes down the timecode, with some brief descriptive notes. If thinks it’s pretty good, he adds a * with it. If he likes it a lot, he puts two, **. Finally, for footage that he “would have lived in vain if it doesn’t make it into the film,” he puts ***. I do the same, substituting the notebook with FCPX’s notes field.

So there you have it: what worked for us. Yet there are many different ways to accomplish the same thing in Final Cut Pro X. What’s your favorite organizational technique?

Documentary data wrangling demystified, Part II: Choosing storage

In this multi-part post, I share how we managed digital media on a feature-length documentary project. In Part I – Securing Your Footage, I covered how to prevent data loss. Let’s continue today with a look at primary storage for a tapeless workflow. Next time, I’ll cover organizing your media for editing in Final Cut Pro X.

Primary storage. On a documentary, it’s typical to shoot for a long time before production wraps. In our film Beyond Naked, two years elapsed from first shot till locked picture (nearly a year in production, and more than a year in post). The amount of resulting footage can easily surpass that of a narrative film. So if you’re making a doc, you’re likely to have some serious data demands.

On my first short film a few years back, I solved storage problems by simply filling up individual external hard drives. Then I’d back them up by mirroring them to other drives. The problem with this approach is that it’s a pain to perform this backup every time you add new media, and so like anything that’s a pain, you end up putting it off. Before you know it, it’s been weeks since you backed up. So when a drive fails, you’re really going to be screwed. And as the old data wrangler saying goes, “it’s not whether a drive will fail, but when.”

So forget primary storage on individual hard drives. As a first line of resistance against data loss, you MUST be able to replace a defective drive without losing anything. That means selecting a RAID. Or something like it. (For a thorough discussion of RAID types as they apply to video editing, check out this post by Larry Jordan.)

Throughout the production of Beyond Naked, we used a Drobo Pro for our primary storage. It uses a proprietary RAID-like system (called “Beyond RAID”) that has some advantages over traditional RAIDs. By default, it’s configured so that if one drive fails, you can replace it without any data loss. You can even configure them so that two drives can fail simultaneously. The tradeoff is that you get less storage space.

Drobos are also hot swappable, so recovering from a drive failure is as simple as pulling the failed drive, and shoving in a new one. Everything will magically pick up right where you left it (although it takes a few hours of read/write time before the recovery is complete). During production, our Drobo experienced a massive failure that prevented it from booting up. It was heart-stopping, but all we had to do was remove all of the hard drives, and send the unit back to Drobo. Even though it was a few months out of warranty, Drobo sent us a replacement Drobo Pro, no questions asked. When it arrived, we very carefully plugged the drives back in, held our breath, and turned the power on. The lights blinked on, red at first, then green. Good to go.

Another benefit of Drobos is that they can handle any type and size of SATA drive you can fit into them, including SATA 3. So as drives get bigger, faster and more affordable, you can replace smaller ones with bigger ones (if you have an older Drobo, a firmware upgrade is required to recognize drives larger than 2TB). We started with 4TB of storage on our Drobo when we purchased it in 2009, and today it’s grown to 16TB. We’ve still got plenty of room to expand.

But there is a major down side to using Drobo Pro: it’s slow. Really slow. Despite being advertised as connecting at up to 100MB/sec via gigabit ethernet, I have found this connection to be totally unstable. Using it invariably causes my 2011 iMac (as well as my previous Macbook Pro) to freeze and require a force-reboot. Repeated support tickets to Drobo have yet to resolve the issue.

So for Drobo Pro, Firewire 800 is the fastest connection I can count on. Here’s what that means on my editing suite:

Unfortunately, that’s nowhere near fast enough to edit HD video. As a safe place for simply storing media, though, it works fine.

If I were purchasing a primary storage today, I would stay far away from any system that didn’t support a Thunderbolt connection. If you want a recommendation, I would heartily recommend the Pegasus Promise R6, after the great experience we’ve had with our Pegasus R4, which I’ll elaborate on shortly).

Storage for Editing. By coincidence, the day we wrapped principle photography on our film was the same day that Apple released Final Cut Pro X. As a frustrated Final Cut 7 user, I made a snap decision to switch. I don’t regret that decision for a minute, despite all the venom that oozed from the professional editing community. FCPX has given me (an artist, not an engineer) god-like powers to skim through mountains of data, and has put wings on my editing. Still, there were challenges.

It turns out that in order to live up to its billing, FCPX demands fast everything (a newer computer, a Thunderbolt connection, and zippy drives). We couldn’t afford to purchase a Thunderbolt RAID that would hold all 4.5 terabytes of original media for the film, but we did scrape together enough cash from our remaining Kickstarter funds to purchase a 4TB RAID.

Calculating storage space. This raised a question: If we used FCPX to create proxy media for us, would the entire film fit on a 4TB RAID with enough free space left over for editing?

You guessed it, there’s an app for that. The best one is called Katadata. AJA Datacalc also works, but with a more rudimentary UI. The beauty of Katadata is that you can select your camera type, and options narrow immediately to those supported by your camera, making the menus much easier to navigate. The app also automatically adds multiple shoot info, and offers the option to email the results of your calculation. Katadata costs $4.99 in the App Store.

We calculated that after converting to proxy, we could fit the entire project into about 2.5 terabytes of proxy footage. So with the last of our Kickstarter funds, we purchased a Pegasus Promise R4 (the only Thunderbolt drive manufacturer actually shipping drives at the time) and set about the task of importing and organizing our footage for editing within FCPX. I’ll cover how we handled that task in part III of this post.

The Pegasus R4 turned out to be amazing for us. Just take a look at the performance we’re getting with ours today, with the project finished editing, and the drive way more than 3/4 full:

An important aside: avoid letting your drives grow beyond 3/4 full for best editing performance. Hard drives require some headroom in order to perform at their peak. Things can really slow down as a drive approaches getting full, so just be keenly aware of that and never let your editing storage drives get too full.

Another Pegasus R4 plus is that it’s relatively small, about the size of a toaster. At least, as long as this drive was parked on my desk, it seemed small. But after a month of daily packing it up and transporting it to my co-editor’s place for work, it began to seem rather large. The cardboard box that it shipped in began to fall apart, and I searched in vain for a Pelican case that would neatly contain it for secure travel. I looked everywhere, and discovered they don’t make one.

As an aside, if you’re thinking “I’ll just use my internal hard drive for editing storage,” that probably won’t work. On all but the newest computers with SSD drives or Apple’s new Fusion drives, an internal drive’s bus connection is slower than it will be via an external connection like ESATA or Thunderbolt. You’re better off using external storage for another reason, too: inevitably at some stage of the film, you will need to take your media with you (to a colorist like John Davidson or an audio mixing facility, for example). However, if you have a 2012 iMac, you might want to forget everything I just said and use your screaming fast Fusion drive to edit on, as long as you’re backing up regularly.

Fast, portable storage. As we got deeper and deeper into editing, Lisa and I needed to share files more and more often. We opted for Lacie 2big 4TB Thunderbolt drives. The most economical place to purchase them, we found, is MacMall, which routinely sells refurbished ones for under $300. We got two of them. We were initially reluctant to buy refurbished drives, but our budget constraints forced our hand, and they have performed flawlessly. You can fit one into a Pelican 1400, with room for the Thunderbolt cable stowed in the top lid under the foam, with the power cable off to the side below.

Because we wanted maximum performance, we left them at Raid level 0, their default. Which yields this:

At RAID 0, if one of the drives fails, you lose everything on the drive. But as long as you remember this is editing storage – not primary storage – it’s less scary. If something fails, you don’t lose footage – just the work you’ve done in editing it. So of course, regular backups are absolutely critical. It would be far more secure to set these drives to RAID level 1, but this results in significantly slower editing response times in FCPX. And I’ve found that if I have to wait for my hard drives when I’m editing, I get distracted, and pulled out of “the zone.” So we committed RAID 0 and to making regular backups.

Storing in more than one location. Having all the original media stored on the Drobo meant our footage was safe if one hard drive failed. But what if the house caught on fire? It’s a common-sense good idea to have your film media in two places just in case. We couldn’t afford to buy a second Drobo. So we simply filled up a bunch of older 1 and 2TB Lacie drives from our previous project. We copied all our original media onto these, and put them in a box in Lisa’s closet. Offsite backup – check.

Is this archival? No, but… for our purpose, it doesn’t need to be. Studies show that disconnected hard drives in storage lose about 1 percent of their data-holding capacity per year. But experts say that you can safely store data for 2-5 years without danger. So as a good, short-term solution, it works. Just don’t put them in a drawer and forget about them, or you may be sorry in 10 or 15 years.

Ultra-portable storage. One final note about storage. When I was in the UK over the holidays last year, I took the film (proxy only) with me on a single Lacie 2Big Lacie, intending to do first pass sound mix and color correction on my MacBook Pro. I realized after I arrived that I had forgotten about 1/4 of the files! Altogether they totaled about 600 gigs. Way too much to transfer over the sketchy internet connection at the place I was staying.

So Lisa, back home in Seattle, simply loaded the missing files onto a USB 3 2TB WD My Passport for Mac portable external hard drive. It is so tiny it fit into a DHL envelope. The cost of shipping from Seattle to England ($100) was almost as much as the cost of the drive ($150).

WD claims the drive is “ulta fast,” but even on my MacBook Pro 2012, with USB 3, it clocks modestly compared to Thunderbolt:

We will be tasking this drive with doing on-set file backups on many of our shoots in the coming year.

OK, so we’ve got storage covered. Next question: How does one organize media for feature-length documentary film editing? I’ll address that in my next post.