introtitle copy.jpg
 
 

INTRO TO BASIC PIPELINE OPTIMIZATION THINKING

Pipeline optimization thinking is something I didn’t fully take advantage of in college. I used to accept a lot of tedious tasks as the only way to do them, but the more I learned about speeding up tasks with code, the more I saw what could be optimized. Time is the most valuable resource in production, so pipeline optimization thinking involves finding ways to save as much time as possible. This includes reducing time performing repetitive tasks so that you can focus on creative tasks that need more time and attention.


IDENTIFYING TIME WASTERS

I really started recognizing time wasters once I learned the power of coding and the command line. Time wasters on a team 3D animated project could include:

  • Manually setting Maya render settings for 20 similar shots, one at a time.

  • Manually creating file directory structures for 20 compositing shots, naming each folder and giving them each a nuke template.

  • Manually seeing what assets or shots are due for the next two weeks on Google Calendar or Trello, and typing it all in an email or posting on Slack.

  • Manually updating Slack or Trello once a Maya batch render has been kicked off for a shot.


For an example of how to speed up tasks, we’re going to look at “Manually creating file directory structures for 20 compositing shots, naming each folder and giving them each a nuke template.“ A tree view of the directory and file structure we want is below. We want to make a general nuke template and place them in folders for shots s010-s200. The nuke template would then take on the name of its shot.

shots
|____s010
| |____comp_files
| | |____s010_v001.nk
| |____media
| |____output_media
|____s020
| |____comp_files
| | |____s020_v001.nk
| |____media
| |____output_media
|____s030
| |____comp_files
| | |____s030_v001.nk
| |____media
| |____output_media
|____s040
| |____comp_files
| | |____s040_v001.nk
| |____media
| |____output_media
|____s050
| |____comp_files
| | |____s050_v001.nk
| |____media
| |____output_media
|____s060
| |____comp_files
| | |____s060_v001.nk
| |____media
| |____output_media
|____s070
| |____comp_files
| | |____s070_v001.nk
| |____media
| |____output_media
|____s080
| |____comp_files
| | |____s080_v001.nk
| |____media
| |____output_media
|____s090
| |____comp_files
| | |____s090_v001.nk
| |____media
| |____output_media
|____s100
| |____comp_files
| | |____s100_v001.nk
| |____media
| |____output_media
|____s110
| |____comp_files
| | |____s110_v001.nk
| |____media
| |____output_media
|____s120
| |____comp_files
| | |____s120_v001.nk
| |____media
| |____output_media
|____s130
| |____comp_files
| | |____s130_v001.nk
| |____media
| |____output_media
|____s140
| |____comp_files
| | |____s140_v001.nk
| |____media
| |____output_media
|____s150
| |____comp_files
| | |____s150_v001.nk
| |____media
| |____output_media
|____s160
| |____comp_files
| | |____s160_v001.nk
| |____media
| |____output_media
|____s170
| |____comp_files
| | |____s170_v001.nk
| |____media
| |____output_media
|____s180
| |____comp_files
| | |____s180_v001.nk
| |____media
| |____output_media
|____s190
| |____comp_files
| | |____s190_v001.nk
| |____media
| |____output_media
|____s200
| |____comp_files
| | |____s200_v001.nk
| |____media
| |____output_media

COUNTING THE STEPS

Step counting was an invaluable lesson I learned when I started out at ILM. It simply lays out all of the steps needed to complete a task so you can find inefficiencies. Let’s count the steps needed for creating the above directory structures for 20 compositing shots and their nuke templates.

  1. create a nuke template

  2. open Finder/Linux Terminal/Windows File Viewer

  3. Go to folder you want to build the comp folder structure in

  4. Create parent directory called “shots”

  5. create a sub directory for the first shot

  6. change directory for first shot to “s010“

  7. create sub directories of “comp_files, media, output“

  8. copy and paste nuke template into “comp_files”

  9. change name of nuke template to “s010_v001.nk”

  10. copy and paste “s010” to create “s020”

  11. change name of “s020”’s nuke file from “s010_v001.nk” to “s020_v001.nk”

  12. repeat steps 10 and 11 until you’ve made folders and nuke files for shot “s200”

Wow, that’s a lot of steps! And imagine if there were 40 or even 1000 shots!



REDUCING STEPS

  1. create nuke template

  2. open command line shell

  3. go to folder you want to build the comp folder structure in

  4. use a python tool that inputs the nuke template and your shot range of s010-s200, and the tool does all of the work

And that’s it! Step 4 is a python script that takes in your nuke template and your list of shots and does all of the copy/pasting, and folder/template naming for you. And even if there were 1000 shots, there are easy ways in Python to auto-create the shot names and place them into a file for you. If you are new to command line and Python, and step 4 seemed alien to you, don’t worry! Tutorials are provided at the end of this post that should make this easier to understand.



KNOWING WHEN TO SPEED THINGS UP

There are times when even reducing task steps from 5 to 3 can make all the difference if it’s for a task that you have to do repeatedly. Something equally important is not every pipeline optimization solution requires code. It might require using more robust task management software like Trello or Asana or downloading plugins for Maya or Nuke.

Other flags for speeding things up that have non-coding solutions are Maya load times and Maya render times. If your scene is dragging, research Maya optimizations like caching. And if you think your renders are taking a long time, know that each frame should be around the 1-3 hour mark. If it isn’t, look into both scene and render optimizations. One step is rendering with fewer samples and using Arnold’s denioser to do the rest of the work.
There are times, however, when pipeline optimization is not needed. If you are not working at scale, and your project is just a few shots long, it may not be worth making a list of your 3 shots to feed into a Python tool for it to make directories and do nuke template file naming for you. The time saved would be minimal or that could ironically waste time.



GET TO KNOW THE OPTIMIZATION TOOLS

Below are resources that I’ve personally used thoroughly to learn command line and Python fundamentals that I now use every day:

FREE Linux command line COURSE

Linux is the main OS at every VFX and animation studio, so this should be a standard course to learn and retake every Summer to brush up on.

(Focus on the first 3 lessons and you can use the Mac application “Terminal“)

https://linuxjourney.com/

FREE coding and python CLASSES

(Lynda is free through college email accounts or with public library cards)

https://www.lynda.com/Programming-Foundations-tutorials/Welcome/83603/90426-4.html

Codecademy Python Class

Lengthy but great concept by concept tutorials that helped me greatly. (Best gone through at a nice ABQ cafe on a Summer day)

https://docs.python.org/2.7/tutorial/introduction.html

https://docs.python.org/2.7/tutorial/controlflow.html

https://docs.python.org/2.7/tutorial/datastructures.html

https://docs.python.org/2.7/tutorial/modules.html

TUTORIAL ON COMBINING LINUX COMMANDS WITH PYTHON

Python Argparse Tutorial



Learning the Linux command line and Python is just the beginning. Learning the power they have to save time in production is when the magic starts. The next articles in this series will start putting these concepts to use.