Graphviz just released a patch to a critical security issue I reported to them.
The following is the advisory:

Background

Graphviz is an open-source multi-platform graph visualization software. It takes a description of graphs in a simple text format (DOT language), and makes diagrams out of it in several useful formats (including SVG).

Description

A vulnerability exists in Graphviz's parsing engine which makes it possible to overflow a globally allocated array and corrupt memory by doing so. 
parser.y (Graphviz 2.20.2):
34:  static Agraph_t *Gstack[32];
35:  static int GSP; 
45:   static void push_subg(Agraph_t *g)
46:  {
47:    G = Gstack[GSP++] = g;
48:  }
As it can be seen, no bounds check is performed by the push_subg procedure, allowing one to overflow Gstack by pushing more than 32 (Agraph_t *) elements.

Impact/Severity

A malicious user can achieve an arbitrary code execution by creating a specially crafted DOT file and convince the victim to render it using Graphviz.

Below is a screenshot of a successful exploitation. When the  malicious DOT file is rendered, a shellcode which calls the MessageBoxA API is executed:

image

Affected versions

Graphviz 2.20.2 is affected by this vulnerability. Older versions are probably affected as well.

Workaround

Version 2.20.3 has been released in order to address this issue. A bounds check has been added in order to avoid an overflow.

parser.y (Graphviz 2.20.3):
34: #define GSTACK_SIZE 64
35: static Agraph_t *Gstack[GSTACK_SIZE];
36: static int GSP;
45:
46: static void push_subg(Agraph_t *g) 
47: {
48:   if (GSP >= GSTACK_SIZE) {
49:     agerr (AGERR, "Gstack overflow in graph parser\n"); exit(1);
50:   } 
51:   G = Gstack[GSP++] = g;
52: }

Acknowledgements

I would like to thank the Graphviz team (Stephen C. North, John Ellson, Emden R. Gansner and others) for their quick responses and fix (it took them only a day since my disclosure to release a patch!).