Image Processing

Let’s say that you’re creating a hot new photo editing application whose job is to soften images, giving them a traditional “portrait” feel. A typical digital workflow for achieving this is to take an existing image, blur it, then blend the original with the blurred version to produce the softened output.

For our app, users will be able to specify the image to soften, and modify a set of parameters that control how much blurring will occur and how much of the blurred version is blended with the original. They’ll also have a “scale” parameter that controls the size of the output image. We want the app to be as responsive as possible, so it should only perform computations if they’re needed.

This is a perfect use-case for Graphcat! We’ll use it to create a set of tasks that represent our workflow and its parameters, and modify those parameters based on user input. Graphcat will make sure that every task is executed when - and only when - needed. The output from the final task in our network will be the softened image.

Note

For a more substantial image processing workflow based on Graphcat, see Imagecat (https://imagecat.readthedocs.io).

To get started, let’s import graphcat and create an empty computational graph:

[1]:
import graphcat
graph = graphcat.StaticGraph()

The first step in our workflow will be to load an image from disk. We’re going to use Pillow to do the heavy lifting, so you’ll need to install it with

$ pip install pillow

if you don’t already have it. With that out of the way, the first thing we need is a parameter with the filename of the image to be loaded. In Graphcat, everything that affects your computation - including parameters - should be represented as a task:

[2]:
graph.set_task("filename", graphcat.constant("astronaut.jpg"))

… note that graphcat.constant() is used to create a task function that returns the given value. Next, we need to create the task that will actually load the image:

[3]:
import PIL.Image

def load(graph, name, inputs):
    path = inputs.get("path")
    return PIL.Image.open(path)

graph.set_task("load", load)

The load function expects an input named path which will supply the filename to be loaded, and returns a Pillow image as output. Our “filename” task produces a filename, so we connect it to the “load” task’s path input:

[4]:
graph.set_links("filename", ("load", "path"))

Finally, let’s stop and take stock of what we’ve done so far, with a diagram of the current computational graph:

[5]:
import graphcat.notebook
graphcat.notebook.display(graph)
../_images/user-guide_image-processing_10_0.svg

For our next step, we’ll resize the incoming image:

[6]:
def resize(graph, name, inputs):
    image = inputs.get("image")
    scale = inputs.get("scale")
    return image.resize((int(image.width * scale), int(image.height * scale)))

graph.set_task("resize", resize)

Notice that the resize function expects an image for the image input, plus a scale factor for the scale input, and produces a modified image as output. The pattern of “parameter” tasks that feed into a larger task performing data manipulation is a common one. In fact, it’s so common that Graphcat provides a special helper - graphcat.static.StaticGraph.set_parameter() - to simplify the process. Let’s use it to setup the scale:

[7]:
graph.set_parameter(target="resize", input="scale", source="scale_parameter", value=0.2)

And of course, we need to connect the load function to resize:

[8]:
graph.set_links("load", ("resize", "image"))

graphcat.notebook.display(graph)
../_images/user-guide_image-processing_16_0.svg

Before going any further, let’s execute the current graph to see what the loaded image looks like:

[9]:
graph.output("resize")
[9]:
../_images/user-guide_image-processing_18_0.png

When we use graphcat.static.StaticGraph.output() to retrieve the output of a task, it implicitly executes any unfinished tasks that might have an impact on the result. If we look at the graph diagram again, we see that all of the tasks have been executed (are rendered with a black background):

[10]:
graphcat.notebook.display(graph)
../_images/user-guide_image-processing_20_0.svg

Creating the blurred version of the input image works much like the resize operation - the blur task function takes an image and a blur radius as inputs, and produces a modified image as output:

[11]:
import PIL.ImageFilter

def blur(graph, name, inputs):
    image = inputs.get("image")
    radius = inputs.get("radius")
    return image.filter(PIL.ImageFilter.GaussianBlur(radius))

graph.set_task("blur", blur)
graph.set_parameter("blur", "radius", "radius_parameter", 5)

graph.set_links("resize", ("blur", "image"))

graphcat.notebook.display(graph)
../_images/user-guide_image-processing_22_0.svg

Notice that the tasks we just added have white backgrounds in the diagram, indicating that they haven’t been executed yet.

Now, we’re ready to combine the blurred and unblurred versions of the image. Notably, our “blend” task will take three inputs: one for each version of the image, plus one for the “alpha” parameter that will control how much each image contributes to the final result:

[12]:
import PIL.ImageChops

def blend(graph, name, inputs):
    image1 = inputs.get("image1")
    image2 = inputs.get("image2")
    alpha = inputs.get("alpha")
    return PIL.ImageChops.blend(image1, image2, alpha)

graph.set_task("blend", blend)
graph.set_parameter("blend", "alpha", "alpha_parameter", 0.65)
graph.add_links("resize", ("blend", "image1"))
graph.set_links("blur", ("blend", "image2"))

graphcat.notebook.display(graph)
../_images/user-guide_image-processing_24_0.svg

That’s it! Now we’re ready to execute the graph and see the softened result:

[13]:
graph.output("blend")
[13]:
../_images/user-guide_image-processing_26_0.png

Don’t ask how, but I can confirm that the image now looks like it was taken in a department store, circa 1975.

Of course, executing the graph once doesn’t really demonstrate Graphcat’s true abilities. The real benefit of a computational graph only becomes clear when its parameters are changing, with the graph only executing the tasks that need to be recomputed.

To demonstrate this, we will use Jupyter notebook widgets - https://ipywidgets.readthedocs.io - to provide a simple, interactive user interface. In particular, we’ll use interactive sliders to drive the “scale”, “radius”, and “alpha” parameters in the computational graph. We won’t discuss how the widgets work in any detail, focusing instead on just the places where they are integrated with Graphcat. To begin, we will need to define some callback functions that will be called when the value of a widget changes:

[14]:
def set_graph_value(name):
    def implementation(change):
        graph.set_task(name, graphcat.constant(change["new"]))
    return implementation

… when the function is called, it will assign an updated graphcat.constant() function to the parameter task, with the widget’s new value.

Next, we’ll create the widgets, and connect them to their tasks:

[15]:
import ipywidgets as widgets

scale_widget = widgets.FloatSlider(description="scale:", min=0.01, max=1, value=0.2, step=0.01, continuous_update=False)
scale_widget.observe(set_graph_value("scale_parameter"), names="value")

radius_widget = widgets.FloatSlider(description="radius:", min=0, max=10, value=5, step=1, continuous_update=False)
radius_widget.observe(set_graph_value("radius_parameter"), names="value")

alpha_widget = widgets.FloatSlider(description="alpha:", min=0, max=1, value=0.7, step=0.01, continuous_update=False)
alpha_widget.observe(set_graph_value("alpha_parameter"), names="value")

We’ll also need an output widget where our results will be displayed:

[16]:
output_widget = widgets.Output()
output_widget.layout.height="1000px"

So we can see exactly which tasks are executed when a slider is moved, we will create our own custom logging function and connect it to the graph:

[17]:
def log_execution(graph, name, inputs):
    with output_widget:
        print(f"Executing {name}")

graph.on_execute.connect(log_execution);

This function will be called every time a task is executed.

We also need a function that will be called whenever the graph changes. This function will be responsible for clearing the previous output, displaying an up-to-date graph diagram, and displaying the new graph output:

[18]:
import IPython.display

def graph_changed(graph):
    with output_widget:
        IPython.display.clear_output(wait=True)
        graphcat.notebook.display(graph)
        IPython.display.display(graph.output("blend"))

graph.on_changed.connect(graph_changed);

Note that the “on_changed” event is emitted by the graph whenever tasks or connections are modified - in our case, every time we move a slider and change the corresponding parameter task. Finally, we’re ready to display our live user interface.

Note

If you’re reading this page online as part of the Graphcat documentation, the interface won’t be visible - to see it in operation, you need to run this notebook for yourself - if you don’t already have the full Graphcat sources from Github, you can download the individual notebook from https://github.com/shead-custom-design/graphcat/blob/main/docs/image-processing-case-study.ipynb

After executing the following, try dragging the sliders and watch the results change. Take note of the following:

  • The graph diagram shows which tasks have been affected by parameter changes (white backgrounds).
  • Below the graph diagram, our custom log output shows exactly which tasks are executed to produce the updated image.
  • Drag each slider, and notice how the diagram and log outputs change:
    • Only the tasks that are affected by the slider are executed.
[19]:
IPython.display.display(scale_widget)
IPython.display.display(radius_widget)
IPython.display.display(alpha_widget)
IPython.display.display(output_widget)

graph_changed(graph)