Reading 4 : K-Sketch

Citaiton:
K-sketch: A kinetic sketch pad for novice animators
RC Davis, B Colwell, JA Landay
Proceedings of the SIGCHI Conference on Human Factors in Computing System
Publication Link

Summary:
This paper discusses a scratchpad, K-Sketch that has been developed keeping in mind the primary needs of animators, while maintaining a simplistic interface.  The usage scenarios for this application were compiled after a series of interviews with both professional animators and amateurs.  The paper then discusses how the final set of possible animations was arrived at by modelling the supported animations against the number of steps and percentage of scenarios that were covered in that many steps, as an optimization problem. It then details the functionalities available in K-Sketch, and its trial run with users coming from a variety of backgrounds. Finally, the implications, results and comparisons of K-Sketch with conventional animation tools are discussed.

Discussion:
The main takeaways from this paper are:

• Surveying the needs of target users is always the first step of building any user interface.
• There is a trade-off between allowing a large variety of functions and minimizing then number of steps.
• Smaller learning curve is often preferred to perfection when it comes to learning something new or building informal mock ups.
• Some of the core functionalities include coordinated motions, overwriting and adding motion, ghosts, copy pasting and recording. 

Reading 3 : iCanDraw?

Citation:
iCanDraw: using sketch recognition and corrective feedback to assist a user in drawing human faces
D Dixon, M Prasad, T Hammond
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Publication Link

Summary:
This paper describes the second version of iCanDraw, which is essentially a software to help users learn to draw faces with step-wise guidance and live feedback. It discusses the the amends made to the previous version, which failed to yield a good experience or proper learning. At a top level, the main aim of this research is to help Left-Brained people (poor artists who draw based on a symbolic representation) adopt a gradual transition to the R-Mode (i.e Right-brained people who can perceive facial features and represent the drawing more realistically). The paper first discusses the user interface and options available to the user, and then proceeds to discuss the kind of corrective feedback mechanisms, evaluation metrics as well as the techniques which actually perform the comparison of a drawing to the actual image, by extracting a face template from both.


Discussion:
The first iteration of iCanDraw highlighted the conceptual mistakes in the program design, such as over-reliance on 'prep-sketching' and 'overtracing' to detect the user's intentions. This did not apply well for beginner drawers. The visual feedback was also not intuitive enough.

The following are some of the aspects of the user interface :

• Drawing area
• Drawing instructions
• Reference Image - It is manipulated at each step to assist in the R-Mode shift
• Corrective Feedback

Next, we discuss the implementation details outlined in the paper.

• Pre-processing reference imagery: Face recognition library is used to extract 40 points, and few are manually added to obtain 53 points.
• Setting the example template : Example template is centered and rescaled until head is drawn
• Processing of a stroke: Done using PaleoSketch
• New strokes are analyzed according to their positions as per the facial features of the features in that step using K-nearest neighbours algorithms (where k = 3) and classified accordingly. All strokes drawn outside an example template are treated as ignore spaces.

• Determining Correctness of Image: This is done using a sliding window that compares dissimilarity with the template. The average of all these distances is used to isolate windows that have a larger than acceptable standard deviation.

The paper also discusses a few principles that drawing through sketch recognition needs to follow:

1. Master template must be accurate
2. Application should complement the R-mode shift
3. Feedback should not be intrusive towards the creative process
4. Feedback should direct user towards final outcome
5. Erased strokes should be temporarily be visible
6. User should be able to override the applications suggestion, 
7. Sketch reco algo should be adaptive
8. Support with the drawing area
9. Support artistic techniques such as shading

Reading 2 : Sketch Recognition Algorithms for Comparing Complex and Unpredictable Shapes

Citation:
Sketch recognition algorithms for comparing complex and unpredictable shapes

M Field, S Valentine, J Linsey, T Hammond

IJCAI, 2436-2

Summary:

This paper describes the methods by which unusually shaped bodies are identified and compared. The identification is basically done by constantly trying to combine strokes to form larger shapes. The rest of the paper goes briefly over truss recognition and comparision techniques that were discussed in the Reading 1. This support for unspecified type of diagrams makes the software very flexible and adapt to any new type of diagrams that an instructor may want to assign questions on.

Discussion:

One of the main ideas used in this paper, is that the computation is done online. This basically means that every time a new stroke is made, only combinations including the new stroke need to be analyzed, as combinations excluding it are assumed to have already been performed.

The main concepts discussed here are on identifying and comparing bodies of complex/undefined shapes:

1. Body Identification

This is done by checking for a closed shape first, whereby an arbitrary end point is selected and we traverse along closest end points, until we reach the last segments second point.  This point should be close to the point first selected. The notion of close enough is defined as 9% of the total path length of the strokes.

2. Body Comparision

This step uses a modified version of the template comparison algorithms by Kara and Stockwich(2005), but uses stroke points instead of rasterizing an image. The steps followed here are :

(i)Resample the shape to 64 evenly spaced points (to prevent effect of stroke speed)

(ii)The shapes are scaled uniform to bring them to common bounding box for comparison

(iiI) Now we calculate three parameters - Hausdroff Distance, Modified Hausdroff Distance, and Tanimoto Coefficient. (note that nab is different from nba)
(iv) We use a combination of the three metrics for comparison, and set 0.65 as an acceptance threshold to determine if students sketch matches that of instructor.

Reading 1 : Mechanix and Free-Body Diagrams

Citation:

Mechanix: a sketch-based tutoring and grading system for free-body diagrams

S Valentine, F Vides, G Lucchese, D Turner, H Kim… - AI Magazine, 2012

Summary:

Scalar mechanics, beam stress analysis, etc. are some of the important techniques that every civil and mechanical engineer needs to be well versed in. This paper discussed Mechanix, an software that tries to teach these skills to students, allowing them to hand-draw figures, analyzes these strokes and provide intuitive annotations and feedback to help students draw a diagram correctly and understand the underlying concepts. The paper first discusses past work in this area, all of which have only been partial solutions, or do not give students the liberty to hand draw diagrams. Mechanix blends the traditional and authentic practice of hand-drawing diagrams, with a touch of artificial intelligence, to create a rich and visually enriching learning experience for students. The authors then document a myriad of interaction methods and geometric recognition techniques that have been used in Mechanix for features such as truss recognition, answer correction, scribbling, etc. It also discusses the overall distributed architecture of the system, and how students' solutions are evaluated securely at a server location, against the predefined sketches and equations designed by the instructor.

Discussion:

As I have used Mechanix before and am familiar with the user experience, the main takeaways from this paper are a couple of things - how Mechanix is better than its predecessors, and what algorithms does Mechanix use?

Prior to Mechanix:

1. Sketch-Worksheets: Generates raw facts about sketches, doesn't actually understand diagrams. Heavily dependent on the instructor to filter these facts.

2. Wintruss/Bridge Architect: No hand-drawn input, and offer only partially completed solutions

3. Andes Physics/Free Body Diagram Assistant: Pick and use shapes, no actual drawing, which is bad in the long run.

4. Newton's Pen: Forcing drawing in a very specific order.

Algorithms that power the various features of Mechanix:

1. Scribbling

Scribbles are understood as combinations of strokes within a small time interval. The algorithm also uses the region of scribbling whether to erase a single stroke, or a partial or full shape.

2. Steps in Geometric Recognition: This phase uses a kind of agglomerative clustering algorithm, whereby clusters of shapes are formed based on some pre-understood domain knowledge on the various possible complex shapes, and this goes on until no more clusters/groups of shapes can be merged.

(i) Stroke Segmentation: Cusp Detector (Wolin 2010)
(ii) Primitive Shape : PaleoSketch (Paulson and Hammond 2008)
(iii) Adding primitive shapes to collection of shapes.

(iv) Analyze various shape grouping using high level recognizers

(v) Replace low level shapes with high level shapes, and return to step (iii)

3. Truss Recognition

Trusses are recognized primarily by detecting shared edges. To find out if two polygons share a particular edge, we remove that edge from the graph form by the vertexes of polygon and the edges, and evaluate using BFS if there is still a way to reach B from A, where AB is the removed edge.

4. Answer Checking

(i) Checks for truss configuration, by representing the completed sketch as a graph, and compares with instructor's solution using Graph Isomorphism

(ii) Checks for axis and presence of forces and their directions

(iii) Checks force values 

5. Generic Closed-Shape Comparison Technique

Makes use of three similarity measures :  Hausdroff distance, Modified Hausdroff Distance, and Tanimoto Coefficient. They basically try to gague geometric similarity, as well as ratio of points overlapping between studen'ts solution and instructor's solution.

6. Creative Response

Uses the constraints provided by instructor to frame its own set of solutions against which the student's work is compared.

7. Distributed Architechture

The student's workspace is transferred in the form of XML document to a session at a server running Mechanix. Load balancing is done using HAProxy.

Mechanix: A Run Through

Our first assignment in Sketch Recognition was to go through a software called Mechanix, that has been design for high school and engineering students to work on solid mechanics problem effectively. It has a lucid user interface that provides a rich visual experience, and is powered by an AI core that is able to understand sketches and shapes drawn by the student, and also compare it with instructor provided templates.

Although the software on a whole was really impressive, for someone like me with butterfingers, it was fairly difficult to get through all the problems, as I found it hard to even make lines intersect correctly! When I told Dr. Tracy Hammond about this, she suggested that I can possibly work on algorithms that makes Mechanix more friendly towards poor artists! That is probably an excellent direction in the future, to help artistically deprived people like me!

Some thoughts on Mechanix:

1. Professors can view the student's submissions. What if a student has been trying to submit an imperfect diagram for a long time, which Mechanix didn't accept, but is acceptable to a human professor? Can the professor then tell Mechanix that the solution is indeed right (i.e Correct Solution, Detected as Wrong - True Negative?)? Will Mechanix be able to perform fresh analysis and learn to widen its range of accepted solutions?

I will be adding more thoughts here on Mechanix as and when I figure out new things!