A basic asset creation pipeline - batteries included.

• Keywords: Film, games, content creation, pipeline

• Objective:
  1. Provide an example of how one might go about building a pipeline around Pyblish.
  2. Demonstrate where publishing fits within a typical production pipeline
  3. Inspire further expansion upon basic ideas

• Mission: Demonstrate what is made possible with publishing.

• Motivation: I’m doing this for the same reason I created Pyblish. Because I see publishing as the single most important aspect of any production pipeline. It is on top of the advantages that it provides that the surrounding pipeline is made possible - e.g. browser and loader, builder and manager.

• Requirements: Reliably output correct data with minimal impact on artist productivity.

• Technology: Starter is built upon Pyblish, Python and bindings for Qt, and depends upon a Windows, Linux or MacOS operating system with Autodesk Maya.

• Audience: Technical directors interested in pipeline working in small- to mid-sized companies with a thirst for better ways of working.

• Discaimer: In the interest of simplicity, Starter is a very limited pipeline. If you have experience with an existing pipeline, or have thought about making your own, odds are this will look nothing like it. Take what you can from this project, and feel free to contribute your own ideas to make it simpler, or fork and expand upon it.

Prerequisities

Before you start looking into pyblish-starter, it is recommended that you first familiarise yourself with Pyblish.

• learn.pyblish.com

Target Audience

To make the most out of pyblish-starter, some knowledge and experience is assumed.

“Pyblish is elegantly designed and closely resembles our in-house production publishing tools here at Imageworks. The setup and documentation is straightforward and easy to understand, getting you setup with a production-quality asset publisher in no time.”

Dhruv Govil - Pipeline TD @ Sony Pictures Imageworks

Introduction

Welcome to pyblish-starter, a basic asset creation pipeline - batteries included.

Table of contents

• Install
• Usage
• Description
• Batteries
  • Creator
  • Loader
• Looking Ahead
• API
  • Terminology
  • Filesystem API
    • Information Hierarchy
    • Workspace, Stage and Share
    • ls()
    • asset.json
    • version.json
    • representation.json
  • Starter API
  • Host API
• Contract
  • starter.model
  • starter.rig
  • starter.animation
• Homework
• Contributing
• Help

Install

Starter takes the form of a Python package with embedded plug-ins.

$ pip install pyblish-starter --target .
$ set PYTHONPATH=%cd%    # Windows
$ export PYTHONPATH=$cd  # Linux
$ maya

Usage

Starter is initialised by calling install() with an interface for your host.

>>> from pyblish_starter import install, maya
>>> install(maya)

From here, you model, rig and animate as per the contract below.

Description

Build your own ASSET CREATION PIPELINE, starting with the basics.

Overview

Asset creation covers aspects related to producing content used in the production of film. A film is typically partitioned into sequences and shots, where each shot consists of one or more assets.

This project includes plug-ins and tools for 3 common types of ASSETS in a typical production pipeline.

• Modeling
• Rigging
• Animation

These illustrate how to (1) devise a contract - known as a family - for each kind of ASSET and (2) design their interface towards each other.

Batteries included

In addition to contracts, the project illustrates boundaries between publishing and setup. The definition of “setup” being anything that isn’t strictly output, such as creating geometry or assigning attributes.

It includes a series of graphical user interfaces to aid the user in conforming to these contracts. These interfaces represent placeholders for your pipeline to fill in; you are invited to either continue from where they finish, or build your own from scratch.

Limitations

Each of the concepts in this pipeline are compatible with pipelines of any complexity. In the interest of simplicity however, some concepts were excluded.

• Maya Only - The overall pipeline, tools and graphical user interfaces are all platform agnostic, but at this point in time only Maya is provided.
• Deformation Pipeline Only - That means only modeling, rigging and animation is considered. Additional pipeline steps are easily added, and are encouraged to by you the user, but are out of scope for the master project and documentation.
• No Asset Composition - Lack of asset composition means no asset can contain another asset, and that includes shots. A shot typically contains one or more assets and depend on the relationship between these assets to be configurable and maintained throughout the lifetime of a project. Starter includes the necessary foundation of this relationship, via containers, but does not implement the necessary tools to manage it.
• Static Root Directory - By default, only Maya is provided support for, and within Maya the root directory in which assets are published (and staged) defaults to the Maya project directory.

Batteries

In addition to providing a co-operative set of plug-ins, Starter also implements a minimal toochain for ASSET creation in a typical film pipeline.

Creator

Associate content with a family.

The family is what determins how the content is handled throughout your pipeline and tells Pyblish what it should look like when valid.

API

The creator respects families registered with Starter.

from pyblish_starter import api

api.register_family(
    name="my.family",
    help="My custom family"
)

For each family, a common set of data is automatically associated with the resulting instance.

{
    "id": "pyblish.starter.instance",
    "family": {chosen family}
    "name": {chosen name}
}

Additional common data can be added.

from pyblish_starter import api

api.register_data(
    key="myKey",
    value="My value",
    help="A special key"
)

Finally, data may be associated with a family.

from pyblish_starter import api

api.register_family(
    name="my.family",
    data=[
        {"key": "name", "value": "marcus", "help": "Your name"},
        {"key": "age", "value": 30, "help": "Your age"},
])

Loader

Visualise results from api.ls().

from pyblish_starter import api

for asset in api.ls():
    print(asset["name"])

API

The results from api.ls() depends on the currently registered root.

from pyblish_starter import api
api.register_root("/projects/gravity")

The chosen ASSET is passed to the load() function of the currently registered host.

from pyblish_starter import api, maya
api.register_host(maya)

A host is automatically registered on pyblish_starter.install().

Looking Ahead

Without publishing, in any shape or form, the following essential tools are but a dream.

API

Starter exposes a series of interrelated APIs to the end-user.

Terminology

Starter reserves the following words for private and public use. Public members are exposed to the user, private ones are internal to the implementation.

	Term	Public	Description	Example
	PROJECT	X	Root of information	Gravity, Dr. Strange
	ASSET	X	Unit of data	Ryan, Bicycle, Flower pot
	VERSION	X	An ASSET iteration	v1, v034
	REPRESENTATION		A data format	Maya file, pointcache, thumbnail
	FORMAT		A file extension	.ma, .abc, .ico, .png
	WORKSPACE	X	Private data	Scenefile for v034 of Ryan
	STAGE		Transient data	Outgoing VERSION from scenefile
	SHARED	X	Public data	v034 of Ryan
	PRODUCER		Creator of data	You
	CONSUMER		User of data	Me

Filesystem API

Data is organised into files and folders.

Some files and folders have special meaning in Starter.

Information Hierarchy

The mental and physical model for files and folders look like this.

Workspace, Stage and Share

During the course of the creation of any ASSET, data moves between 2 of 3 states.

• Starter does not take into consideration the workspace and is therefore workspace-agnostic.
• The staging area is both implicit and transparent to the PRODUCER and CONSUMER, except for debugging purposes. This is where automatic/scheduled garbage collection may run to optimise for space constraints.
• The shared space is where ASSETS ultimately reside once published.

Workspace and Shared separation

A naive approach to content creation might be to refer to ASSETS straight from another artists workspace. Starter separates between data considered work-in-progress and data exposed to others.

Workspace data is highly mutable and typically private to an individual artist.

• Mutable implies transient data that is likely to change at any given moment.
• Private implies personal, highly irregular and likely invalid data.

Shared data on the other hand is immutable, correct and impersonal.

• Immutable implies that the data may be dependent upon by other data.
• Correct implies passing validation of the associated family.
• Impersonal implies following strict organisational conventions.

Each ASSET reside within the top-level root directory as follows.

Hierarchy	Example

Each ASSET contain 0 or more versions which in turn contain 0 or more representations.

Hierarchy	Example

Every extraction is made into the staging directory, regardless of whether it integrates successfully or not.

Hierarchy	Example

Each directory will contain everything that did extract successfully, along with its metadata, for manual inspection and debugging.

ls()

Communication with the filesystem is made through ls().

ls() returns available assets - relative the currently registered root directory - in the form of JSON-compatible dictionaries. Each dictionary is strictly formatted according to three distinct “schemas” (see below).

See below for a full list of members.

Example

from pyblish_starter import api

for asset in api.ls():
    for version in asset["versions"]:
        for representation in version["representations"]:
            pass

Schema

Available schemas are organised hierarchically, with the former containing the latter.

• asset.json
  • version.json
    • representation.json

asset.json

A unit of data

version.json

An ASSET iteration

representation.json

A data FORMAT.

Starter API

pyblish-starter provides a stateful API.

State is set and modified by calling any of the exposed registration functions, prefixed register_*, or automatically on calling pyblish_starter.install().

Public members of pyblish_starter

Public members of pyblish_starter.api

Host API

A host must implement the following members.

Information hierarchy

Loaded data is stored in a container. A container hosts a loaded asset along with metadata used to associate assets that use other assets, such as a Wheel asset used in a Car asset.

Id

Internally, Pyblish instances and containers are distinguished from native content via an “id”. For example, in Maya, the id is a user-defined attribute.

Contract

Starter defines these families.

starter.model

A generic representation of geometry.

Target Audience

• Texturing
• Rigging
• Final render

Requirements

• All DAG nodes must be parented to a single top-level transform
• Normals must be unlocked

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• geometry_SEL (geometry): Meshes suitable for rigging
• aux_SEL (any, optional): Auxilliary meshes for e.g. fast preview, collision geometry

starter.rig

The starter.rig contains the necessary implementation and interface for animators to animate.

Target Audience

• Animation

Requirements

• All DAG nodes must be parented to a single top-level transform
• Must contain an objectSet for controls and cachable geometry

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• in_SEL (geometry, optional): Geometry consumed by this rig
• out_SEL (geometry): Geometry produced by this rig
• controls_SEL (transforms): All animatable controls
• resources_SEL (any, optional): Nodes that reference an external file

starter.animation

Point positions and normals represented as one Alembic file.

Target Audience

• Lighting
• FX
• Cloth
• Hair

Requirements

• None

Data

• name (str, optional): Pretty printed name in graphical user interfaces

Sets

• None

Legend

	Title	Description
	Target Audience	Who is the end result of this family intended for?
	Requirements	What is expected of this ASSET before it passes the tests?
	Data	End-user configurable options
	Sets	Collection of specific items for publishing or use further down the pipeline.

Homework

With an understanding of this asset creation pipeline, here are some suggestions for what to explore next.

Difficulty	Task	Description
	Loader tooltip	Pick any of the available data from an asset, and add it to the tooltip of the item.
	Shots	Think of and document how the kind of workflow exemplified here - (1) creating, (2) publishing and (3) loading assets - can be applied to a shot- and task-based pipeline.
	A Creator Help Page	The creator interface is very limited. Try including a help-page and picture alongside each family illustrating what each are for.
	Loader thumbnails	The loader interface is even more limited, with room for expansion. Consider how you might go about extracting a thumbnail alongside each asset, and add it to the interface.

Contributing

pyblish-starter, as Pyblish itself, is an open source effort and contributions are welcome.

For example, you could fork Starter, expand upon the graphical user interfaces and either make it your own or submit a pull-request to have it merge with the official project.

For more information on this, contact me and let’s have a conversation!