Dashboard_avatar
Jan 17, 2011
Post id: 306821 Report Item

I’m working with a game engine that uses proprietary DirectX shaders in Maya 2008, which are then exported into the game engine.  I want to bake a lighting pass in the environment into the vertices using Batch Bake (Mental Ray), but unfortunately Mental Ray won’t render shadows on meshes using the proprietary shaders.


One workaround is I can assign a Lambert shader to all the meshes in the scene, and bake the lighting.  But then I’d have to individually reassign the individual game shaders back onto the meshes afterwards.  Another option I considered is duplicating all the meshes as instances, assigning a lambert to the instanced duplicates, baking the lighting, and then deleting the instances.  But that’s a pain, especially as the scenes grow larger and more complex.


So, two questions:


1) If anyone’s had experience writing custom Direct X shader plugins in Maya, is there some kind of easy “enable shadows” solution that I can pass onto my programmer that will allow mental ray to render shadows?


or if not....


2) Is there a workflow-efficient way to assign a lambert to everything while baking the lights, save the resulting vertex colors, and then revert the assigned shaders back to the original proprietary shaders?


or.... is there some other obvious solution I’m not thinking of?


Any help here much appreciated!  Working on deadline as always.....


Matt

Avatar2
Jan 17, 2011
Post id: 306823 Report Item



Dont know if you can do all you work in maya 2009 because since this version there is a feature called "Render Passes". Its commonly used for creating separate images out of different color passes, even on a per objcect base via contribution maps. You should have a look at it. With it you could create an additional render layer, assign all the objects you want the lightmaps from and create a material override with your shadow shader for this special layer (the override material only applies to your objects while rendering this special layer, so no reassign of original shaders needed).

A quick setup would look like:

- select all the objects you want to have lightmaps from, then switch to render layer and choose Layer->Create Layer from Selected

- go into Hypershade window and right click on your shadow shader and set "Assign Material Override for layerX" (and make sure that your new render layer is selected, otherwise this option wont appear)

- everytime you now start baking lightmaps only make sure that in your render globals the current render layer is set to your override layer, done.


PS: if stuck with maya 2008 why do you bother with reassigning the original shader? Why not saving the scene, assigning the shadow shader, rendering out the lightmaps and reset the scene to saved state?


PPS: aah i see, vertex baking. So i am not even sure if the upper approach would work, i have never used vertex baking so far. So i would suggest that you use a little script that writes each objects material name into a custom string attribute before assigning the bake shader, then reassign the original shder with a second script after baking.


proc writeMatNamesToObject()
{
    string $selList[] = `ls -long -allPaths -dagObjects -shapes -sl`;
    for($obj in $selList)
    {
        string $SG;
        string $depNodes[] = `listConnections -destination 1 -plugs 0 $obj`;
        $depNodes = stringArrayRemoveDuplicates($depNodes);
            for($node in $depNodes)
            {
               if(`nodeType $node` == "shadingEngine") $SG = $node;
        };
        if(! `attributeExists "dxSG" $obj`)
        {
            addAttr -ln "dxSG" -dt "string" $obj;
            setAttr -e-keyable true ($obj + ".dxSG");
        };
        setAttr -type "string" ($obj + ".dxSG") $SG;
    };
};




proc assignOrigMat()
{
    string $selList[] = `ls -long -allPaths -dagObjects -shapes -sl`;
    for($obj in $selList)
    {
        print($obj + "\n");
        if( `attributeExists "dxSG" $obj`)
        {
            string $SG = `getAttr ($obj + ".dxSG")`;
            print("material: " + $SG + "\n");
            sets -e -forceElement $SG $obj;
        };
    };
};


Dashboard_avatar
Jan 18, 2011
Post id: 306825 Report Item

well the above may work. But i would say you both miss the ovious solution. See in the bake dialog (accessed trough the box next to the menuitem) theres a section of what to bake. And it includes a option for shader override. Use that!