Documentary Data wrangling demystified, part III: Organizing for the Edit in FCPX

In part I of this post about data wrangling a feature-length documentary, we covered protecting your footage. In part II, we took a look at documentary film storage requirements. Today I’ll explain how we organized our footage in preparation for editing in Final Cut Pro X.

Here’s an overview of the workflow we used during production to get footage off our cards, stored safely, and organized for retrieval. Note that this is a dual-system workflow – we always recorded audio separately from video, requiring us to sync clips later in post in a workflow that I’ve previously covered.

1. Copy contents of video SD card into a new folder located within the project’s “Footage” folder (using naming convention below).
2. Copy contents of audio SD card as above into project’s “Audio” folder.

We stored the media on our Drobo Pro, a raid-like system that protects data in case of a drive failure.

Folder Structure for Organizing Media. The foundation of our organizational work begins at the Finder level, with a simple but powerful folder naming convention:

“YYYY-MM-DD two to four keywords”

Here’s a partial snapshot of our film’s audio folder:

You’ll notice that some dates have more than one folder. We created a new folder for each shoot, which helped immeasurably to quickly find footage later in the project.

The nice thing about using this date format as a prefix to tagging is that your folders will always sort in order. If you put the tags first, that wouldn’t be the case. And adding two to four keyword tags that succinctly describe the shoot allowed us instantly identify the contents.

3. Stills, if any, are copied into “Stills” folder, using same naming convention.
4. Back up data onto another hard drive which is stored at another location.

Item 4 is the hardest bit to keep up. In practice we tended to copy and move the files offsite at milestones in the project – after a particularly big shoot, about midway, after wrapping, etc.

In hindsight, our biggest workflow mistake was neglecting to view dailies. Most often, we would simply skim through footage after a shoot to ensure it was looking OK. But we never made time to properly review dailies. We figured we’d do that when it came time to edit. And that was a mistake.

One of our lavs developed a short halfway through production, and we never discovered the problem until post. Everything recorded in the second half of production with this mic sounded tinny. It required heroic EQ work to make it usable. Luckily we were paranoid and almost always recorded with a lav AND a boom mic.

Further, I wish we had begun to edit a few sequences together soon after shooting, instead of months later. That would have helped us discover whether we were covering scenes well enough to cut them together easily. Instead, we had to learn all of this the hard way months later when we finally got down to serious editing. Too often we discovered that we didn’t shoot enough b-roll (people without their lips moving, or just their hands, or ANYTHING that we could cut away to) to build the dialog easily.

Prepping for editing in Final Cut Pro X The day we wrapped production was the day Apple shipped Final Cut Pro X. We dove right in and never looked back. But it was difficult at first to understand Apple’s new “Event” and “Project” terminology.

I initially thought that it made the most sense to create one monolithic Event, and import the entire film into it. Wrong! Final Cut loads all open Events and Projects into memory, so everything slows to a crawl if too many clips are open concurrently. It’s best to have as few Events and Projects as possible open at a time.

So what we ended up doing, which worked very well, was to create an Event structure that mirrored our folder structure almost exactly. We created a new Event for every shoot, titled YYYY-MM-DD plus two or three descriptive keywords. So after import, like so:

Importing and organizing into FCPX is not trivial when you’re talking more than 4 terabytes of footage. It took us about a month and a half of work to get all of our footage moved over from raw storage and organized the way we wanted them in FCPX Events. Seriously, it was a LOT of work. It wasn’t as simple as hitting the “import” button.

To Proxy or not to Proxy? Proxy media is 1/4 resolution ProRes files, which FCPX can create for you on import. With FCPX, it’s a dead-simple process to create and use proxy media. When importing, just check the “proxy” box.

It takes quite awhile to generate proxy media, though, so with serious amounts of footage, be prepared for some lengthy transcode times. With Final Cut Pro X, you can be doing other things at the same time, albeit a bit sluggishly.

I would argue that it’s almost essential to use proxy media on a feature-length project, because you’ll likely need to have many Events open at a time. And proxy will give you the ability to open many more than you would with full resolution media. Luckily, it’s easy to switch back and forth between proxy and full resolution at any time. Press cmd-, to call up preferences, select Playback, and choose “Proxy.” We didn’t have to switch back to Original Media again until we were ready to start creating assemblies.

After making proxies, our next step, and the most tedious and time-consuming part of our workflow, was audio syncing. I’ve outlined our audio syncing process and it worked very well, but it added a lot of time to the process. I really wish Final Cut Pro X had a way of batch syncing audio automatically. You can use PluralEyes, but I’m not crazy about the mess it leaves behind. It requires a lot of cleanup if you like to maintain, as I do, a clean separation between media and projects. So I prefer DualEyes.

A note about this workflow. If we had been on FCPX from the beginning of our film, this process would have been much smoother, because we would have been doing this import > transcode > audio sync process daily, instead of facing it all at once at the end. I definitely recommend making this a part of your dailies ritual it time allows.

The most significant improvement for Final Cut Pro X to add, from my feature-length perspective, would be to make it possible to load an entire film all at once. I simply couldn’t do it with my hardware (3.4 GHz Intel Core i7 2011 iMac, 16GB ram, AMD Radeon 2 gig graphics card). Attempting it just displays the spinning beach ball. I’ve since tried it on my new 2012 MacBook Pro, on which I can just get the entire film to load. Even then, it’s too sluggish to make it practical. But it makes me suspect that Apple’s playing the long game with FCPX – by the time next year’s Macs are released, the hardware may well have caught up with the software on this, allowing FCPX to work acceptably well feature film situations.

As a result of this, we spent a stupid amount of time over the life of the project just moving Event and Project folders back and forth between Final Cut Events and Final Cut Events – Hidden folders. Each time, we had to restart FCPX. Tick, tick, tick…

It wasn’t until the very end, when I was audio mixing the film, that I discovered Event Manager X, a $5 utility that makes this process relatively effortless by allowing you to create lists of Projects associated with Events. It let’s you move them and restart FCPX with a single click. If you’re cutting your feature doc on FCPX, this app is indispensable.

Getting organized inside of Final Cut Pro X With everything imported, audio synced, and proxified, a final step remains: adding metadata. Final Cut Pro X provides a two incredibly powerful ways of organizing footage: Smart Folders and Keywords.

Smart Folders

Our first step is to create smart folders that split our media into types: audio only, video only and sync video. Smart folders are just standing searches, or filters, that allow you to display everything that matches your query. When you create a smart folder, you can specify what you’re looking for. In the case of our Audio only example, it looks like this:

For our Events, we generally wanted to be working on sync video when we got down to editing, because that was the best sound. But sometimes we had video for which there was no audio recorded separately. It’s easy to set up a Smart Folder that will tell you at a glance which files are which, by adding additional qualifiers to the search:

You can take this pretty far, but we found that for most Events, we could find everything we needed with just three or four smart folders.

Event structure for sharing Projects between editors As the project took shape over time, we shared Project files back and forth between two editors, my partner Lisa and I. We discovered that it’s very important to “lock” Events and add the inevitable new media that become necessary during post (such as sound effects, archival footage, archival stills, etc) to a new Event, rather than adding them to existing Events. That’s because if they’re added to existing Events, it’s too hard to keep track of who has the most current Event. Then, when you share a Project, you don’t know if there will be missing media when the other editor opens it.

Our solution was to create an Event called “BN Additional Footage” and another called “BN Music and SFX.” We added all the new files to the appropriate folder, and then whenever we shared a Project, we included just these two Events to ensure no missing media. This made Project sharing much more manageable.

Keywords

Keywords are the second most powerful way to organize footage in FCPX. I reach for Keywords whenever I can’t figure out a way to do it with a Smart Folder. For example, this shoot was rather long, and consisted of five different “scenes”, as follows:

Tagging each part of the shoot with a descriptive keyword gave me a way to instantly find only those clips. I tend to use Keywords to break apart Events into chunks like this. My partner, Lisa, tends to use them to identify her favorites, and clips that would make good b-roll. You can use Keywords however you like.

Favorites The last step of getting organized might actually be considered the first step of editing: making favorites. My process for this involved standing at a desk, with my finger hovering over the “i” and “o” keys, as I played back the footage. Whenever I saw or heard something that looked promising, I’d make a selection and press the “f” key to mark it as a favorite.

To make it possible to find your favorite later, it’s essential to add a text descriptor. I add mine by typing it into the “notes” field, which I reposition to display just to the right of the “name” field in the Event Browser (click and drag on the field to move it).

If the favorite is extra special, I use a simple notation I learned from Werner Herzog at Rogue Film School. Herzog has a notebook open on his lap when reviewing footage. When he sees a favorite, he writes down the timecode, with some brief descriptive notes. If thinks it’s pretty good, he adds a * with it. If he likes it a lot, he puts two, **. Finally, for footage that he “would have lived in vain if it doesn’t make it into the film,” he puts ***. I do the same, substituting the notebook with FCPX’s notes field.

So there you have it: what worked for us. Yet there are many different ways to accomplish the same thing in Final Cut Pro X. What’s your favorite organizational technique?

17 thoughts on “Documentary Data wrangling demystified, part III: Organizing for the Edit in FCPX

  1. John W

    Hey Dan, Just want to say that you’ve done a great job with this series. I read each installment and found lots of great tips. Thanks for taking the time out to document and share your experiences. I hope you’ll consider adding another installment when it comes time to finishing the film and finalizing the online.

    Reply
    1. Dan McComb

      Thanks John. I’ll be happy to do that. We’re looking at quite a few interesting options for distributing the film online – including working with a local company IndieFlix, VHX and now Vimeo is on the verge of allowing Pro members to distribute right on Vimeo, which looks promising. So it will be fascinating to see where this all takes us, and you can be sure I’ll be blogging every significant step we take. Are you working on a feature-length project?

      Reply
  2. Rhett Murphy

    Great post. I think a great, and easy, feature for a FCPX update would be the ability to “lock” an event. And I’m curious about your reasons for opting to use DualEyes over the new PluralEyes — I’ll have to read your audio post, I suspect. I’ve been syncing with PluralEyes and outputting new synced media files with only the high quality audio tracks; maybe I’m missing something in DualEyes that is a better (cleaner) workflow. Thanks for sharing this experience and congrats on the work.

    Reply
    1. Dan McComb

      Hi Rhett,
      My biggest complaint about PluralEyes is that it creates extraneous project files to accomplish it’s task. I like to have clear separation between footage (Events) and timelines (Projects). You CAN keep this clean separation using PluralEyes – but you have to remember to clean up the Projects it creates while it’s getting its job done. It’s actually not a bad workflow, as long as you do that. I just prefer DualEyes because it keeps everything separate from the beginning.

      Reply
  3. Shane Tilston

    Dan –

    I came across your site recently, and have been devouring your articles. I’m impressed both by your own work, as well as the depth of the sharing you’re doing! And I’m also very grateful! I’m embarking on my first feature in a few months in FCPX (after many shorts over the years, but single system on FCP 6 and 7) so all of your FCPX posts are very useful, and I’m sure I’ll continue to re-read them as I move forward with my project.

    One question I did have was about how you organized your projects (which I don’t think that you covered – unless I missed it!). Did you work with just one project file for the whole film? If so, when you disabled certain events, did you just live with the “media missing” red notation? Or – did you use multiple projects for different scenes, and then combine them? If you did that, how did you combine them? I haven’t played around with that yet, but I’ve heard that’s tricky.

    You mentioned that you were going to cover your outputting, so I’m happy to wait for that article, but just thought I’d ask!

    Thanks again for all that you’re doing!

    Shane

    Reply
    1. Dan McComb

      Hi Shane,
      Congrats on choosing FCPX for your feature-length project. With a little care about how you structure your project, you’ll be able to avoid the pitfalls and reap the benefits. Glad you’ve found the site helpful. Feel free to ping me with any questions along the way.

      Regarding project, you will most definitely want to create separate projects, many of them. We created a separate project file for each sequence in the film. The primary reason for doing this is that it allows you to load only those Events that you need for the given sequence, instead of having to load all the Events in the film, which would likely slow your computer to a crawl. At best FCPX behaves erratically when you load too many Events. The secondary reason for doing this is it just allows you to work faster: if you only need three Events open to cut a sequence, then you can find the footage you’re looking for much more quickly than if you have 30 or 50 Events in your Library.

      As you progress further into the edit, you will have more and more sequences finished. Our film had 31 sequences. When we got to the point where we were ready to put them all together into an assembly, we shared each sequence as a master file, created a new Event called “Screener” and imported each file into that. Then we created a new project called “Assembly 1” and dropped all the screener files, in order, into that. Then, we exported it, and viola, we had our first assembly. We probably repeated this process a dozen times or so, before we got to the final assembly.

      The only thing awkward about this process is that some sequences will have music playing between them, or other transitional sound or picture elements. In that case, it may make sense to combine two sequences into one. In two cases on our film this would have resulted in us having too many Events open, so we waited to lay in the connecting elements for those two sequences until the final assembly.

      I forgot about promising that post about final output, and now that we’ve done with all that, it’s high time I wrote it. Thanks for the reminder. I’ll post it as soon as I can.

      Tell me more about your film – what’s it about?

      Reply
  4. David Blumenfeld

    Hey Dan.
    You site is wealth of info. thanks. I shoot a lot of NGO work with c100 and am now starting to edit my first feature doc on fcpx…just starting to log the footage…I;m wondering if all your advice still applies with updates since then…Like to still create different Events for each shoot or each shoot day…thanks.

    Reply
    1. Dan McComb Post author

      Hi David,
      All that advice is definitely still relevant, because the thing the most sets FCPX apart from other NLEs is the strong metadata foundation, which lets you organize your footage better than anything else on the market. But FCPX has evolved a lot since I wrote this, and since I finished that film, I’ve been focused on much shorter projects, so I’m organizing my footage quite differently. But I read a fantastic article a couple weeks go that goes into great depth about how to organize your edit in FCPX for a feature doc, written by a filmmaker who is currently in the middle of a huge edit. Check it out, I’m sure you’ll find this useful: https://library.creativecow.net/angelo_michael/apple_fcpx_documentree/1

      Reply
  5. David Blumenfeld

    Dan,

    Would you still recommend creating new events for each “shoot” or Shoot day? In 10.2.2, you can close libraries, but is it possible to only open certain “events?” Trying to figure out best workflow so I don’t need to go back and redo things…thanks.

    Reply
    1. Dan McComb Post author

      Yes, that’s how I would do it. You can close events inside of libraries, opening only those that you are working on. I would use the Library container to hold your entire film, and events for each shoot day, or, for particularly huge shoot days, you might break it up further. On our big day we broke events up by shooter team, for example (we had 4 teams operating independently of each other and each was treated with its own event).

      Reply
  6. David Esp

    I just completed a 2.5 hour mostly-reality job using 10.2.3 on a 16GB+NVIDIA-GPU retinaMacbook with everything on USB3-attached 3TB WD Passport. Rather than your (31) “Master Files + Assembly” approach, I Compound-Clipted each sequence, the assembly (of these CClips) thereby automatically keeping up to date. Most sequences were cut from multicam clips, even if only one cam but plural audio (more flexible than Synch clips). And lots of audio level keyframing. Worked ok initially but around mid-way became sluggish and delicate (crash-worthy). Have you any similar experiences or relevant advice?

    Reply
    1. Dan McComb Post author

      Hi David,
      I’m guessing the reason you’re getting crashes is because you have too many Events open (that is, you are asking FCPX to hold too much footage in it’s brain at once). The compound clip method you describe sounds like a great approach, but it only works if you have all the events open simultaneously. FCPX is super fast and amazing in part because of the way it holds footage in fast-access caches, but when you have too much footage open at once, it gets cranky. I’m guessing you are pushing the limits of your system.

      Reply
  7. Lane

    I’ll immediately take hhold of your rss feed as I
    can’t in finding your e-mail subscription link or e-newsletter service.
    Do you have any? Please allow me recognise so that I may
    subscribe. Thanks.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.