Cogsketch is a program that understands sketches (the type the people draw with pen and paper). You can download it for free, from the Qualitative Reasoning Group at Northwestern University.
Cogsketch is an impressive achievement, one reason being that people draw things in many different ways, and make mistakes (which Cogsketch can often advise them in how to correct)
Suppose a teacher uses CogSketch to draw a solution to an exercise, such as drawing the layers of the Earth. For example, he might draw three circles to start with and label the innermost circle “Inner Core” and the outermost circle “Crust.” He would also provide labels, along with labels for the other parts and distractors (here, “Lava” and “Rock)
Cogsketch then automatically constructs a variety of relationships between the visual entities.
For education, having students explicitly label their sketches with their intended concepts is important since an expert seeing an unlabeled student sketch like Fig. 1 might think it is correct because he or she is interpreting the circles differently from the student. In drawing the layers of the Earth, for example, the sketch of a student who swapped the mantle and crust would look the same as the sketch of a student who got it right: Both sketches consist of a set of concentric circles. Conceptual labels allow for correcting this kind of error.
Humans see objects in a qualitative way. For instance, rather than say the ceiling is 20 feet above the floor, we just say it is high above the floor. When Cogsketch interprets sketches, it relies on qualitative judgements including topology. (Two objects are said to be topologically equivalent if one can be elastically deformed into the other. For instance, a clay doughnut could be reshaped into the shape of a coffee cup, without tearing the clay apart in any way.)
Cogsketch also recognizes hierarchies of items that make up a picture (for instance, after recognizing edges, it can recognize the object the edges are part of).
To give an idea of what is involved just in interpreting edges (the lowest hierarchy), here is a table of edge properties (I abbreviated it):
|Edge Attributes||Edge Relations|
• CurvedArc length:
(several, not given here)
Adjacent corner relations:
Cogsketch uses a program called the Structure Mapping Engine to find important differences between the solution and student sketch and to offer advice based on those differences. SME takes as input the solution sketch and the student sketch and creates a mapping between the two. Each mapping consists of
(1) correspondences, which indicate how things in the solution correspond to things in the student sketch;
(2) candidate inferences, which indicate important differences between the solution and the student sketch.
The correspondences and candidate inferences are used to determine what advice should be given to the student.
Fig. 5. Greenhouses. These three sketches all are valid solutions to the Greenhouse effect worksheet. CogSketch views them as equivalent because they all satisfy the properties the author specified as important.
Look how different the pictures are, and yet Cogsketch sees that they are the same, on a deeper level, paying attention to as what the teacher thought was important when the teacher drew a prototype sketch.
In future, the team at Northwestern University want to better handle images with entities such as force fields and air masses. An example would be the changes in flow orientation as air moves around an airplane wing.
They also want to put in 3D representation and reasoning capabilities in CogSketch, to better interpret sketches about surfaces and complex shapes, which often occur in engineering, biology, and geoscience. Since sketches are 2D, much can currently be done by crafting advice based on 2D visual properties, but potentially more 3D representations could enable students to choose a wider variety of viewpoints when sketching.
For more info, including papers on analogical reasoning in general, take a look at: http://www.qrg.northwestern.edu/papers/papers.html