0
\$\begingroup\$

I'm trying to build a game that uses the Pix2Pix post-processing effect for Unity, which is an image-to-image translation with deep neural networks. It works well but I want to fade this effect with the non-post-processing render. The post-processing effect allows only an off-on feature.

pix2pix effect postprocessing

I'm trying to get an in-between that I could access via script in order to change the render

Here with the post processing : pix2pix effect

Here without : without effect

Is there a way to get an in-between?

EDIT:

I tried to follow Gabriele's idea of using color.lerp method to lerp the input and output (named as sourceRT and resultRT), It gives me a gray output, here is what I tried (in Postprocessing.cs) :

    // RenderTexture Source to Color[]
    Texture2D texSources = new Texture2D(256, 256, TextureFormat.RGB24, false);

    texSources.ReadPixels(new Rect(0, 0, 256, 256), 0, 0);

    Color[] sourcePixels = texSources.GetPixels(0, 0, 256, 256);


    // RenderTexture Result to Color[]

    Texture2D texResult = new Texture2D(256, 256, TextureFormat.RGB24, false);

    texResult.ReadPixels(new Rect(0, 0, 256, 256), 0, 0);

    Color[] resultPixels = texSources.GetPixels(0, 0, 256, 256);


    // Blend Pixel comporte un lerping de chaque pixel selon un float
    Color[] blendPixel = new Color[resultPixels.Length];

    Texture2D blendTexture = new Texture2D(256, 256, TextureFormat.RGB24, false);

    // Lerping via a loop in the array
    for (int i = 0; i < resultPixels.Length; i++)
    {
        Color c = Color.Lerp(sourcePixels[i], resultPixels[i], ratio);
        blendPixel[i] = c;
    }
    // tranformation of blendPixel to Texture2d
    blendTexture.SetPixels(blendPixel);


    // END LERP

    // blendpixel
    cmd.Blit(blendTexture, context.destination);
\$\endgroup\$

1 Answer 1

2
+50
\$\begingroup\$

Even though i am not familiar with this library the first solution that comes to mind is to do a lerp from the input to the output by creating a new Texture2D and interpolate each pixel between the first image and the second one. If Pix2Pix does not provide any way to extract the output as an image you can renderer the output of the camera to a texture.

\$\endgroup\$
5
  • \$\begingroup\$ Thanks for the answer, the project use as an BlitFullscreenTriangle() method to create the post processing effect, I have no idea how this method work; but it seems to generate a render texture — is there any documentation about lerping two render texture ? \$\endgroup\$
    – Felox
    Commented Sep 2, 2021 at 13:33
  • \$\begingroup\$ with the render texture generated by BlitFullscreenTriangle() method you can create a texture2d object that provides a set of tools to access each individual pixel, you can refer to the docs for texture 2d on how to create one (docs.unity3d.com/ScriptReference/Texture2D.html) there is also plenty of resources on how to make a texture2D from a RenderTexture \$\endgroup\$ Commented Sep 2, 2021 at 14:49
  • 1
    \$\begingroup\$ You will then use a new Texture 2D for each frame of the transition you want to create and manually set each pixel with the value returned by the Color.Lerp method (docs.unity3d.com/ScriptReference/Color.Lerp.html) with arguments the pixel of the original image and the one with post-procesing applied \$\endgroup\$ Commented Sep 2, 2021 at 14:52
  • \$\begingroup\$ I tried to follow your idea of lerping the colors to get the inbetween aspect. It gives me a gray output. I put the code above. \$\endgroup\$
    – Felox
    Commented Sep 3, 2021 at 14:10
  • \$\begingroup\$ After calling SetPixels you also have to call Apply on blendTexture \$\endgroup\$ Commented Sep 3, 2021 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .