ISET-Depth
From SCIEN
We are combining code from the RenderToolbox, SimToolbox, Radiance, ISET, and Maya to build 3D models with full spectral radiance data. We process these models through ISET to develop a more general rendering with depth of field and appropriate blurring.
The programs specific to this project are in the pdcprojects repository in the scene3d directory.
Andy Lin, Joyce Farrell and Brian Wandell
Contents |
Overview
ISET can simulate the responses to 3D scene data; these are spectral radiance data that include information about the distance from the lens to each point in the scene.This page describes how to create synthetic 3D ISET scenes.
The method relies heavily on Dave Brainard's Rendertoolbox (RTB). RTB is built on top of two other packages: Maya and Radiance.
Maya is an object modeling package that is freely available to universities (see below). This is a means for creating 3D objects and placing them in a scene.
Rendertoolbox reads the Maya objects and allows the user to specify spectral reflectance functions for the objects, the illuminant, and to specify bi-directional reflectance distribution functions (BRDF) for the objects (we haven't tried the last part).
Rendertoolbox combines the Maya data and these spectral functions in a wavelength-by-wavelength calculation and passes them to the Radiance package. That package creates the rendered image in each waveband to produce a spectral output.
ISET combines this output into an ISET scene structure that contains information about the scene spectral radiance, illuminant and depth map. These 3D scenes can be analyzed by the ISET optics, sensor and other modules to characterize depth-of-field and defocus, as well as other properties.
For another overview see http://white.stanford.edu/teach/index.php/Andy_Lin
For the user manual and technical report, please see http://www.stanford.edu/~ydna/Scene3DManual.pdf
Computer Requirements
We assume you are on a Linux system. We don't have installation instructions for other stuff.
We need 32-bit Matlab (there is no 64 bit version of XXX).
We have installed on Red Hat's Fedora OS version XXXX
We have also installed in Ubuntu 10.10 (Maverick).
Not sure about graphics cards requirements - probably none.
Download and Installation
We list the Ubuntu scene3D installation instructions separately from the Red Hat scene3D installation.
We are just starting to work on the Windows scene3D installation.
RenderToolBox
RenderToolBox is a collection of rendering tools developed by Professor David Brainard from University of Pennsylvania. RenderToolBox allows for the rendering of 3D multi-spectral scenes, using various rendering engines. RenderToolBox was modified slightly to allow for the output of depth-maps. Please refer to the RenderToolBox website for more details: http://rendertoolbox.org/
Initializing RenderToolBox
- Start Matlab
- Your path should include ISET, RenderToolbox, SimToolbox, psychtoolbox-3. Also, it is best to remove the SVN directories from your path. An example of a fairly complex startup program from wandell is:
A program to remove .svn directories from your path is here
Now, you should be ready to follow the example below.
Simple RenderToolBox Example
Change into the scene3D directory.
This is a simple example that loads a scene of a sphere on a table, in a room, along with its depth-map into an ISET scene. Run this example as a simple verification if the installation was successful or not. This part is designed to run on a Linux box that has all of the tools installed.
In Matlab, cd into the RenderToolbox directory and then into the sub-directory examples/table_sphere/
chdir(fullfile('/usr','share','RenderToolbox','examples','simple_square'))
Note that this directory is the one that will contain your scene. Each scene will have its own root directory. In this example, the example scene is called table_sphere, so it has it's own table_sphere directory. All the necessary files to render this scene will be in this directory, which we will refer to as "scene root".
Run the following series of commands from inside the Matlab prompt
unix('./clean.sh')
Render_BatchRender
On ochre this took about 10 sec.
To verify the output, change into sub-directory within the table_sphere directory, monitor_image_data, and open a .jpg file for verification.
im = imread('monitor_image_data/rad_MonitorImage.jpg'); image(im)
The multi-spectral data will be in the image_data directory. ISET will then use this data to construct multi-spectral scenes.
d = load('image_data/picMat_rad.mat');
vcNewGraphWin; imagesc(d.picMat{10}); colormap(gray)
Setting up Maya objects for Rendertoolbox
This section describes how to take Maya objects and render them as a set of multispectral files that will be converted into an ISET scene.
Either create a scene of Maya objects, or download Maya objects from an example site and place them in a Maya scene.
(At this point, we have a few scenes (table_sphere) that contain a room and wall and so forth. We have only learned how to place new objects inside this scene. We don't know yet how to arrange the geometry of a full scene from scratch.)
- Open the scene using Maya (We use Maya for Windows).
- If you have downloaded an object to place in a scene, you need to use Maya to scale the object and place it at the correct general location of the scene/camera. Some Maya objects are actually composed of multiple objects - you must scale, shift, and export all of them.
- In Windows->Settings/Preferences->Plug-in Manager, make sure objExport.mll is loaded (you can select auto-load if you will be using it frequently).
- Select object (must select all objects if your object consists of many objects)
- File...Export Selection... Name the exported file the same as the Maya objectName field. Here is more information on how to export from Maya.
- IMPORTANT: The exported objects need to be edited before RTB handles them properly. In Linux, using, say, gedit or xemacs, open and edit the .obj file. (It is a text file). Replace the strings 'usemtl XXX:YYY' with 'usemtl objectName'. Thus, if your object is 'Bench' and there is a line such as 'usemtl Bench1:default1' replace it with 'usemtl Bench'.
Using Rendertoolbox to create a multispectral image from Maya objects
The RTB builds multispectral images from data and parameter files stored in a scene directory. A relatively easy to way to begin creating an image of a scene is to modify an existing example. You might begin by copying the files in the "table_sphere" directory in "$RenderToolbox/examples/" to a new directory where you will do your work.
- Replace the objects in the scene_objects directory with the new Maya objects. In an example we did, we copied the table_sphere directory and into banana_bench. We then placed a set of maya objects called Ball_Thing1, banana1, and some others into the scene_objects directory. We removed the old objects.
- Edit the parameter files.
- Edit the objectProperties.txt in the scene root directory by adding the name of each object and setting the fields. There are a number of fields that must be filled in to this tab-delimited text file. To get started you can copy and paste previous entry if don't know what to put here.
- Edit the conditions.txt file in the scene root directory. Mainly you can set up the scene spatial resolution (300 is OK for testing), set up the wavelength sampling (400 100 3 produces samples at 400, 500, 600 which is good for testing). It is nice to have a good scene name, as well.
- Run Render_BatchRenderClean, a RenderToolbox script that cleans the directory and then invokes radiance and rpict. If all goes well, the multispectral data will be created and a simple RGB image of the result will be placed in the monitor_image_data directory.
Modifying RenderToolbox parameter files
- Add the name of new objects to objectProperties.txt (which is in the scene root folder). The objectProperties.txt contains the texture/reflectance information for each object. Refer to the rendertoolbox wiki for details.
Similarly,
- To add/edit lighting, edit lightProperties.txt
- To add/edit rendering parameters, edit conditions.txt
- To change view points, edit ./view_files/view.vf (or the current view point file specified by conditions.txt)
Refer to the rendertoolbox wiki for details.
When the directory is setup properly, within Matlab you can
- Change into the scene directory
- Run Render_BatchRender
- Or, if you are running multiple times, use Render_BatchRenderClean.
Importing RTB multispectral data to an ISET scene structure
Scene3D is a component of ISET which interfaces with RenderToolBox, and allows for the RenderToolBox output to become an ISET scene, with a depth map. Scene3D also allows for depth-of-field simulation of multispectral scenes with provided depth-map.
An example
To load the RenderToolBox multispectral output and a depth map into an ISET scene structure, first change directory into a scene directory, then run:
- scene = s3dRT2Scene([pwd '\image_data\picMac_orange_rad.mat'], 400:10:700);
The depthmap is stored in scene.depthMap.
The following command is under development:
Finally, to render the scene with optics blurring, run:
- s3dBlurScene
This command loads the scene from table_sphere into an ISET scene, then apply default blurring properties to the optical image. The command uses defaults, but you can run the code with different optics parameters set. We are still in the process of debugging and validating the blurring code.
PBRT Multispectral Modification
The next phase of the Scene3D project will consist of modifying PBRT to allow for multispectral inputs and outputs. The reason why PBRT is chosen is because PBRT has some nice native support for texture mapping, material modeling, is modern, and is easy to modify. There is already some native PBRT support for spectral imaging. However, the biggest concern is that it does not allow for spectral outputs.
Also, the multispectral inputs are somewhat limited. For example, PBRT does not natively allow for multispectral image maps for textures. This next phase of the project will consist of modifying PBRT for multispectral input and output.
See PBRTOutput for details on how PBRT was modified to allow for multispectral output.
See PBRTConversionMatrix for details on how PBRT was modified to allow for a conversion matrix multiplication, after processing.
See PBRTFileFormat for the input file format for conversion matrices.
TODO
- add flexible input range - this is a challenge... there are many vectors that are initialized with the currently provided constant. All of them would need to be modified to allow for dynamic allocation... plus, is it any faster? 31 channels vs. 310? need to try this out... see if it makes a difference!
- add flexible output (specify output by a matrix perhaps) - this is done. may need to polish
- add Matlab interface so ISET can read the files! - this is done, but need to improve this interface.
- get the depth-map - look at the pbrt website and look at Extended channel film plugin for reference.[1]
- Obtain multispectral textures/light maps/materials/measured subsurface scattering
- obtain bump-maps for material realism
- cleanup the DOF rendering algorithm that takes into account partial occlusion -> package this, perhaps into ISET
- allow for multispectral textures
- what is the best file format to use for this?
- allow for multispectral light maps (environment map)
- make some nice example scenes (buildings, outdoors?)
Some notes
- pbrt uses bison, a gnu parser?
- parsing is taken care of in the lex and ycc files, core/pbrtlex.ll and core/pbrtparse.yy
- Important to figure out how to input NEW PARAMETERS
- paramset.cpp/ paramsset.h has some answers
- pbrtparse.cpp/pbrtparse.yy has the bulk of the parsing procedures
- api.c/h ...
- InitParamSet in pbrtparse.yy is a good example of using paramset functions! don't need to worry about nItems because it detects that for you. We might need to make a new PARAM_TYPE enum for the transformation matrix! Need to make a new keyword for it too. Initially can just read from the .pbrt file, eventually want to have it read from a text file... although this will make it harder to implement...
- Api.cpp -> Film *MakeFilm .... perhaps this is where we put the new transformation matrix!!!!!
- perhaps use ParamSet::AddSampledSpectrumFiles as inspiration on the input interface
- add an entry to pbrtparse.yy InitParamSet for transformation matrix
- proposed data flow.... pbrtparse.yy/cpp -> initialize param set (Create new entry for matrix file?)... -> api.cpp MakeFilm .... -> Image.cpp, CreateImageFilm .... extract matrix filename from param set and perform parsing... may need to use paramset class to perform actual parsing (ParamSet::AddSampledSpectrumFiles inspiration)
- .pbrt files require you declare the type before specifying the parameter type (use string for the filename of the matrix)
3DS Format to PBRT
- meshes export ok
- however, materials are a mess - need to go into 3ds and adjust all the materials to PBRT types -> is there an easier way to go from 3ds to pbrt?!!
- need to specify a camera position, or else will give an error
Blender to PBRT
- must use Blender version 2.48!!
- put .py file in the directory that all the scripts are in (use the search function to find this)
- go to Render - then you will find PBRT export selection in there
- by default, there will be an infinite light source -this lighting is very flat
- do lights specified in blender work in the converter?! not sure about this
- C:\Users\Andy\AppData\Roaming\Blender Foundation\Blender\.blender\scripts\bpydata is the location of the exported pbrt files
- lights don't work! - lights DO work - the infinite light just drowns them out!! - remove infinite light perhaps, or make much dimmer
- lights in PBRT are using the global coordinate system!! (not eye coordinates/clip coordinates)
- must use .tga or .exr texture maps! but these WILL work!
Random Notes
- some pbrt scenes use bump-mapping - basically it uses a height-map, and uses the height-map to compute normal vectors, to approximate a bumpy surface. In the end, this is only an approximation, and I wonder if we can modify pbrt to utilize more advanced techniques than bump mapping.
- pbrt can measure subsurface scattering - see chapter 16 - some of this stuff is quite advanced
- in teapot subsurface scattering example,
- sigma_prime: Reduced scattering coefficient of the volume, measured in mmˆ-1
- sigma a: Absorption coefficient of the volume, measured in mmˆ-1.
- what do these parameters mean?
- “kdsubsurface” material is easier to specify the "look"
- Kd: diffuse reflection term
- Kr: specular reflection term
- meanfreepath: average distance light travels before being scattered
- working on extracting depth map - this has been finished!
- in film.cpp... we need to modify AddSample()
- however, can't find the original call to Addsample()... this is very curious
- may need to change the Ray data structure to allow for storage of depth (in geometry.h)
- add a Ray & currentRay input parameters to AddSample() in image.cpp, needed to change image.h and film.h as well
- added Z data member to Pixel struct in image.h
- 3/4 - finished getting depth map and importing into ISET, however, there are PROBLEMS with compatibility with ISET + drawing depth maps etc. - should ask Prof Wandell how exactly this works.
- working on blender to pbrt converter
- find files in this directory: "C:\Users\Andy\AppData\Roaming\Blender Foundation\Blender\.blender\scripts\bpydata"
- environment maps are in lat/longitude coordinates. in other words, they are in spherical, with theta and phi mapped to u and v.
TODO and Miscellaneous comments
rview (Radiance program that allows for interactive 3D viewing) is currently broken and should be debugged.
scene3DInitialize.m needs to be added to SVN
very early scene3D files should be packaged and sent
some preview visualization technique for scenes would be great
Main Page 
Software 
Measurement 
