Introduction:

Check here for Part 1 or Part 2

If you have been reading along with us, here we present part 3, some new ideas. I will discuss an idea we had for making a raindrop function for calculating reflection and refraction. Also we will get to the actual solution that worked and I will explain how the pieces go together. If you’re new to this set of articles, check out part-1 to read our assumptions that we are making. Then continue on to part-2 to read about the first ideas and the corresponding results we received with those methods.

Next Ideas:

Next we considered, well since interpolating the positions on the environment map didn’t work, we would instead interpolate the outgoing vectors from the raindrop. So rather than the intersection points with the environment map, we would interpolate the 3-dimensional refraction and reflection vectors for each “pixel” in our raindrop-screens. Unfortunately this also did not work and we ended up with the same results as before…peculiar. I am certain there is a mathematical explanation, after all, the outgoing vector is only the intersection point normalized so in some sense we expected the results we received a little bit, but it was easy enough to change the code and try it.

After that we came up with this concept of creating a raindrop function. For each pixel in some raindrop screen S_i we will create a function that returns the 3-dimensional position data of where that pixel’s refraction would intersect the environment map.

So our function returns a Vector3 and is of the form (Vector3 RaindropPosition, int PixelPositionU, int PixelPositionV). To create this function we kind of used a machine learning approach. We ray-traced a bunch of raindrops in a bunch of different positions and then tried to figure out a function that would give us the result we wanted. We started with a raindrop simply straight in front of the camera.

At the time of this research I was not very familiar with machine learning and what that field had to offer me so we first just plotted how where refraction rays were intersecting the environment map. Next we used Matlab to try and recreate the results of that function based on the set of inputs we had. As the raindrop moved to the left or right of the camera, the refracted rays started to vary, and the further away from the center the greater the difference between the ray-traced and function-based results. So, that meant we need to make another function to adjust for what we considered noise. This also meant that the image above was the closest we would ever get.

Looking back on this work now, years later, and having a far better understanding of machine learning I wonder what kind of results this function-based method could produce. Essentially I would ray trace a billion raindrops and save all the positions they were at, and their corresponding intersection points with the environment map. Then try to learn the common pieces that would create the parameters for my function, a neural network should be able to do a potentially good job at this.

However after acquiring the results shown above, we abandoned this idea as well and we moved onto our third and final idea, which is much more elegant and I think pretty clever.

Concept Three:

So one day, Marcos and I were working on miscellaneous paper work for god knows what, as is mandatory for all Universities world wide and after some time we went home. The next day he came in and more or less said, we’re thinking about the problem all wrong. See, up until now we were thinking of a solution that would work both for CPU and GPU, for really no other reason than that was what we had been considering prior. Starting with the interpolation idea, etc. The key was to change our way of thinking, and start thinking parallel.

So with parallel in mind, we came up with the idea of ray tracing a single raindrop at a specified set of coordinates, starting with unit distance from the camera.  Next we would create a mask for our raindrop. The mask is the same thing as shown in figure 5 from part two, shown here.

That let us store all the reflection and refraction values for each pixel in our mask. Next we performed 32-ray super-sampling along the edge and stored the alpha value in the fourth channel of the “color” vector for the refraction ray. The reflection’s fourth channel stored the Fresnel value for each mask-pixel.

The Mask is Half the Story:

Well just ray-tracing some values and storing them doesn’t exactly give us a screen full of raindrops. The next part is done on the GPU. So the ray-tracing step was performed at pre-processing, next we performed the magic during processing and on the graphics card.

For each raindrop we took it’s position and the camera’s position and acquired a vector. Next we compared that vector to our original vector between our mask and our camera. Naturally, there is likely some difference there, and that difference can be expressed by a rotation.

The rotation is shown as theta in the image above. So in order to get the values for Li and Ri in the image above, we take the values Lp and Rp and multiple those vectors by a rotation matrix which is derived from the theta.

Pre-processing Hierarchical Maps:

Well so far we would get pretty good results but that doesn’t describe raindrops entirely. To accurately calculate reflection and refraction, it is necessary to know the distance from the camera to the raindrop. This is because that distance affects the incident angle to the surface normal of the raindrop. To accommodate this we set a series of drops at specific distances to perfectly calculate their masks. We calculate the distance of those drops in a specific way in order to keep the size of the mask in accordance with the square root of 2, which maps well to the mip-mapping procedure. It would be possible to just create the largest mask and let the GPU perform mip-mapping of its own on the mask, but the results would not be quite as accurate. By creating our own mask for each level of the mip-map, we get a better level of accuracy to the true values of the reflection and refraction vectors. Here an image will hopefully help articulate what is described here.

LDR vs. HDR:

The next issue we addressed was the use of High Dynamic Range environment maps versus Low Dynamic Range environment maps. In previous works it was argued that reflection was not important, which at the time was true. Back in the early 2000’s HDR was still pretty new and uncommon. However by 2008 HDR had kind of become the norm and is present in a number of video games. Once you use HDR environment maps, reflection is no longer ignorable, but actually influences the results heavily. Even in the image above, figure 4, has a large bright white reflection present, a result of the HDR image we are using. Here another image compares the differences.

As shown above the HDR image has much more visible, and reflection shouldn’t be ignored.

Shader Tree:

Here I provide an image that describes the progression of the shader tree. Marcos did a great job creating this image, and it was presented in our poster at the I3D conference 2010 in Washington D.C. One of the great advantages of doing computer graphics, is you typically end up with excellent posters, because they can generally be filled with a lot of cool pictures.

In the image above, the hierarchical maps are pre-processed, with specific parameters to generate values to be used as a mip-map. The camera, and raindrop positions are used to make the rotation matrix. The rest is done in real time. Check our paper, Photorealistic real-time rendering of spherical raindrops with hierarchical reflective and refractive maps, for all the details.

Video Demo and Explanation:

Extra Points:

When writing a research paper, one often doesn’t have enough space to describe the progression of ideas and thoughts because there a page limits. However for my thesis, I included these things, which I find are the most interesting aspects of the work.  This work does a great job capturing how ideas progressed over time and developed. Starting first with our initial idea on how to solve the problem, followed by the next two iterations, and then the solution.

Though the progression of the ideas is captured in my Thesis at Hiroshima University, it is likely no one has or will ever read that work. Thankfully the internet exists and I can share those ideas, thoughts and experiences here, which will hopefully garner additional readers, after all it couldn’t be less. I hope you have enjoyed reading.

Conclusion:

Well that explains some of the interesting work that I did at Hiroshima University while doing my Master’s under the supervision of Dr. Kazufmi Kaneda. It was a great pleasure to work with the folks at Hiroshima, and I am proud of the work we were able to complete. I look forward to a future where I may get to work with them again or maybe even some of you. Thanks for reading and take care.