add Reviews

reviews.md

@@ -0,0 +1,163 @@
+----------------------- REVIEW 1 ---------------------
+SUBMISSION: 4
+TITLE: Global Type Inference for Java using Answer Set Programming
+AUTHORS: Andreas Stadelmeier
+
+----------- Overall evaluation -----------
+SCORE: 1 (weak accept)
+----- TEXT:
+The paper provides a good overview and introduction to the research topic but needs more specifics and comes across as a careless abridgment of a more comprehensive document.
+
+Initially, the author briefly outlines the primary problem and presents Answer Set Programming (ASP) as a more promising alternative to overcome the obstacles with existing algorithms. The problem is sufficiently motivated in its brevity, and the author pitches the idea behind ASP and its objective, unfortunately, without introducing ASP in detail or presenting any concrete results. The paper, therefore, reads more as motivation and a superficial introduction to the topic.
+
+Some further points:
+
+* The author should have better explained the evaluation rules on page 4, as well as the syntax and semantic domains used. Without a detailed explanation or a look at the (only incidentally) referenced paper ([3] and [6]), the evaluation rules are not self-explanatory.
+* Just adding evaluation rules without further using them makes no sense. The author could have used this space better.
+* Are the evaluation rules on page 4 inline or floating figures? There is no explanation or caption that one could use to reference the image later. Anyways, without use or explanation, all the evaluation rules appear context-free.
+* Knowledge of ASP is also assumed. More information and a better introduction would be desirable.
+* Overall, I would like to see some more concrete results.
+* "Chapter 3.1": The author can omit this sectioning as just one sub-section exists.
+* "Chapters" should probably be called "Sections".
+
+The paper fails to provide concrete results and reads more like a declaration of intent. The evaluation rules are not further used or discussed and seem unrelated to the paper; therefore, they are not helpful.
+
+To summarise. The paper can be seen as motivation and introduction to a research topic. Omitting the evaluation rules in favor of a better introduction to ASP and more detail on the results already obtained would significantly increase the comprehensibility of the paper.
+----------- Reviewer's confidence -----------
+SCORE: 2 ((low))
+
+
+
+----------------------- REVIEW 2 ---------------------
+SUBMISSION: 4
+TITLE: Global Type Inference for Java using Answer Set Programming
+AUTHORS: Andreas Stadelmeier
+
+----------- Overall evaluation -----------
+SCORE: -1 (weak reject)
+----- TEXT:
+Global Type Inference for Java using Answer Set Programming
+
+
+SUMMARY
+
+This work is about an improved implementation of a previously introduced
+global type inference algorithm for a subset of Java. Instead of using a
+standard type inference algorithm that relies on calls to a unification procedure,
+the whole type inference process is mapped to an ASP program.
+Thanks to some magic of ASP, this leads to a significant speed up to infer
+the types of programs.
+The paper presents an example to motivate the issue of type inference for Java.
+Some parts of the unification algorithm are presented plus its translation to ASP.
+The (performance) benefits of ASP are explained in text.
+No empirical measurements are provided.
+
+EVALUATION
+
+This work reminds me of some earlier work that shows how to map algorithm W to
+a logic program. It seems that ASP has an edge over Prolog when it comes
+to search problems that involve "or constraints". Unfortunately, the paper
+is very sketchy (possible to due to space constraints) and leaves many
+questions unanswered (see below for details).
+
+There are various places where the paper needs to be improved.
+I am also missing insights why ASP is "better" (compared to Prolog).
+
+
+DETAILS
+
+p3
+
+ "...construct type solutions in real time"    reasonable time?
+
+p3
+
+"Another use case ..."
+
+To me the most interesting use would case would be type error diagnosis.
+Just as a pointer for future work.
+
+p3
+
+"Our ASP implementation replaces the unification step of the type inference algorithm."
+
+At this point I'm getting lost.
+
+The rules below Figure 4 cover the "unification steps"?
+
+When/how are these rules applied?
+Motivation behind these rules?
+What about rule (Match)? There are no references to T1 and T1 in the premise.
+
+p3
+
+"The algorithm terminates if every type placeholder in the input constraint set has an assigned type."
+
+I couldn't find the entry point to this algorithm.
+Needs a short example.
+
+
+p5, Figure 5.
+
+This seems to be saying turn the whole type inference algorithm into a logic program.
+It seems that ASP has the best computational model to achieve an efficient implementation.
+Would be interesting to have a comparison with Prolog.
+
+p6
+
+"ASP handles Or-Constraints surprisingly well."
+
+I guess Prolog will perform a "naive" search.
+What's the magic? Stable model semantics of ASP?
+----------- Reviewer's confidence -----------
+SCORE: 4 ((high))
+
+
+
+----------------------- REVIEW 3 ---------------------
+SUBMISSION: 4
+TITLE: Global Type Inference for Java using Answer Set Programming
+AUTHORS: Andreas Stadelmeier
+
+----------- Overall evaluation -----------
+SCORE: -2 (reject)
+----- TEXT:
+Summary
+=======
+
+The paper describes an algorithm for global type
+inference in Java based on ASP (answer set programming).
+Essentially, subtyping rules are translated to ASP
+statements, which are then solved. To paper concludes
+that the implementation based on ASP offers much better
+performance than a "traditional" formulation of the
+inference algorithms.
+
+Evaluation
+==========
+
+The results presented in the paper are promising. But the
+article in its current form feels like a preliminary writeup
+and does a bad job in presenting the results. Hence, I suggest
+to reject the paper.
+
+Here are some of the points that definitely need to be improved:
+
+- The introduction does not mention the contributions of the article
+  (I guess that the first section is meant to be the introduction,
+   it is not called liked that.)
+- The algorithm from [6] should be explained at a high-level.
+- The purpose of section 3 is not clear to me. I was expecting
+  that the section would explain figures 3 and 4, but these figures
+  do not come with any explanation.
+- I am missing some kind of introduction to ASP.
+- Why is ASP so much faster? After all it also has to try it all
+  variants in some ways.
+- A discussion of related work is completely missing.
+
+There is still plenty of space that can be used to address the points
+mentioned. (The paper is only 6 pages long.)
+
+The paper also contains several typos, which I do not list explicitly.
+----------- Reviewer's confidence -----------
+SCORE: 4 ((high))