Fall 2018

Instructors: Martha Palmer
Time and Location: Tue/Thur, 11:00 - 12:15, ECCR 150
Assessment: Four homeworks, one Paper presentation, and a term project.
Office Hours: Martha Palmer, Monday/Tuesday 2-3, Hellems 295

Textbooks

Semantic Role Labeling (eBook), Martha Palmer, Daniel Gildea, Nianwen Xue, Synthesis Lectures on Human Language Technologies, ed., Graeme Hirst, Morgan & Claypool, 2010. ISBN: 9781598298321 available on line on campus through Chinook

Representation and Inference for Natural Language. A First Course in Computational Semantics. Patrick Blackburn and Johan Bos, 2005, CSLI Publications. ISBN: 1-57586-496-7. selected chapters, available from the CU bookstore and D2L

Theme

Lexical semantics is becoming an increasingly important part of Natural Language Processing (NLP), as the field is beginning to address semantics at a large scale. This graduate seminar covers key issues in computational lexical semantics. We start with an introduction to theoretical models of lexical semantics and events, considering both their adequacy as linguistic models and their place in NLP. We focus particularly on computational lexical resources such as PropBank, VerbNet, FrameNet and the Generative Lexicon, and examine their strengths and limitations with respect to NLP applications. We will introduce apporoaches to developing automatic classifiers that are intended to make use of these resources and to offer richer representations of sentences in context. These techniques can be fully supervised (requiring hand-labeled training data), semisupervised, or unsupervised (learning lexical information from unlabeled text). We will also discuss the impact of Word Embeddings as an approximmation of semantic similarity and the resulting implications for future research directions.

Suggested Schedule and Readings

Introduction and Module 1: the Lexical Semantics of Verbs - Chap 1

  • Jan 16 Course Overview and Natural Language Processing, the Pundit case study Palmer, Martha, Carl Weir, Rebecca Passonneau, and Tim Finin. "The Kernel Text Understanding System." Artificial Intelligence 63: 17-68: Special Issue on Text Understanding. October, 1993.
  • Jan 18 Thematic Roles in Linguistics, Assignment 1: Exercises 1, 2 and 3, p. 19, SRL book, Due Jan 30
  • Background reading for Assignment:
    • Fillmore, C. J. 1968 "The Case for Case" in E. Bach and R.T. Harms, eds. Universals in Linguistic Theory, 1-88. New York: Holt, Rinehart and Winston. Section 3.
    • Jackendoff, R.S. 1976 Towards an Explanatory Semantic Representation, Linguistic Inquiry, 7:1, pp. 89-150. 
    • Dowty D.R 1991 Thematic Proto-Roles and Argument Selection. Language 67: 547-619 sections 1-7 
    • Levin, B. English Verb Classes: A Preliminary Classification Introduction, MIT Press, pp. 1-23, 1990.

Module 2: Available Computational Lexicons - Chap 2

  • Jan 23 Word Senses, WordNet and the OntoNotes Groupings
    • Palmer, M., Dang, H. and Fellbaum, C, 2007, Making Fine-grained and Coarse-grained sense distinctions,both manually and automatically, Journal of Natural Language Engineering,13:2, 137-163. George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine Miller, 1993, Introduction to WordNet: An On-line Lexical Database, 5 Papers on WordNet availalbe from the WordNet web site.
    • Background Reading for Sense Distinctions:
      • Edmonds, P. and Hirst, G., Near-Synonymy and Lexical Choice, Computational Linguistics June, 2002, Vol. 28, No. 2, Pages 105-144
      • Atkins, S., Fillmore, C. J., Johnson, C. R., Lexicographic Relevance: Selecting Information from Corpus Evidence, International Journal of Lexicography, Vol. 16 No. 3, Oxford University Press, 2003
      • Hanks, P. and Pustejovsky, J., A Pattern Dictionary for Natual Language Processing, Revue francaise de linguistique appliquie 2005/2 (Vol. X), CAIRN, INFO, 2005. Paper
  • Jan 25 WSD as a Machine Learning Task/Preposition SuperSenses/The Generative Lexicon Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Meredith Green, Abhijit Suresh, Kathryn Conger, Tim O $A!/ (BGorman, and Martha Palmer (2016). A corpus of preposition supersenses. LAW, at ACL 2016, Berlin, Germany.
    • Pustejovsky, James, 1991, The Generative Lexicon, ComputationaI Linguistics, Volume 17, Number 4, December. 
  • Jan 30 Wrap up The Generative Lexicon/Review Ass 1
  • Feb 1 PropBank
    • Assignment 2: Exercises 2,3,4 p. 29, SRL book, Due Feb 20
      Martha Palmer, Dan Gildea, Paul Kingsbury, 2005, The Proposition Bank: An Annotated Corpus of Semantic Roles, Computational Linguistics, 31:1 , pp. 71-105.
  • Feb 6 VerbNet
    • Kipper, Karin, Anna Korhonen, Neville Ryant, Martha Palmer. "A Large-scale Classification of English Verbs." Language Resources and Evaluation Journal,42(1). Springer Netherland: 2008. pp. 21-40.
  • Feb 8 FrameNet
    • Fillmore et al 2001 "Building a large lexical databank which provides deep semantics", Proceedings of the 15th Pacific Asia Conference on Language, Information and Computation. Eds. Benjamin Tsou, and Olivia Kwong. Hong Kong 2001.
    • Fillmore, Charles J., Christopher R. Johnson, and Miriam R.L. Petruck. 2002. International Journal of Lexicography, 1'6(3):2435
  • Feb 13 Semantic Proto-Roles
    Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. *Semantic Proto-roles In Transactions of the Association of Computational Linguistics.
    • Access to Computational Lexicons:
      • WordNet
      • FrameNet
      • PropBank
      • VerbNet
      • SemLink
      • VerbCorner
  • Feb 15 Abstract Meaning Representations (AMRs) and After AMRs, QAMR
    • Laura Banarescu; Claire Bonial; Shu Cai; Madalina Georgescu; Kira Griffitt; Ulf Hermjakob; Kevin Knight; Philipp Koehn; Martha Palmer; Nathan Schneider (2013) Abstract Meaning Representation for Sembanking Linguistics Annotation Workshop and Interoperability with Discourse, held with ACL 2013, Sofia, Bulgaria.
    • Nianwen Xue, Ondrej Bojar, Jan Hajic, Martha Palmer, Zdenka Uresova, Xiuhong Zhang Not an Interlingua, But Close: Comparison of English AMRs to Chinese and Czech LREC 2014.
    • Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, Luke Zettlemoyer. *Crowdsourcing Question-Answer Meaning Representations arXiv, Nov, 2017.

Module 3: Beyond shallow semantics

  • Feb 20, 22 Predicate Logic, Susan Brown
    • B&B Chap 1 and 2
    • Artificial Intelligence: A Modern Approach, Stuart J. Russell and Peter Norvig, Pearson Education, 2003, ISrBN:0137903952, Chap 14 and 15
    • Symbolic Logic: A First Course, Gary Hardegree, UMASS,
  • March 6 Wrap up Predicate Logic, Event Variables Assignment 3: Predicate Logic, due March 22
    • Davidson D. 1967. "The Logical Form of Action Sentences," Reprinted in Davidson, D: Essays on Actions and Events, Oxford University Press(1980)
    • Events, Stanford Encyclopedia of Philosophy
    • Parsons T. 1990 Events in Semantics of English . MIT Press, Boston
    • Casati, R., and Varzi, A., editors. Events, Dartmouth, Aldershot, 1996. the introduction

Module 4: More Empirical Approaches

  • March 1 Deep Learning approaches to WSD and SRL James Gung
  • March 8 Word Embeddings, Term Project Proposals are due
    • Scott Denning Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, Noah A. Smith. Carnegie Mellon University *Retrofitting Word Vectors to Semantic Lexicons, NAACL 2015, Best Student Paper Award.
  • March 13 Event Force Dynamics Bill Croft
  • March 15 Student Presentations
    • Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. Deep contextualized word representations NAACL 2018, New Orleans, LA, June, 2018.
    • Sascha Rothe; Hinrich Schuetze, (2017), AutoExtend: Combining Word Embeddings with Semantic Resources Computational Linguistics, Volume 43, Issue 3 - September 2017
  • March 20 VerbNet and Generative Lexicon Event Structure
    • Susan Brown James Pustejovsky, The Syntax of Event Structure, Cognition, Volume 41, Issues 1-3, December 1991, Pages 47-81
  • March 22 Ontologies and Event Ontologies Susan Brown
    • Assignment 4: due Apr 10 Compare and contrast these two papers. Critique one of them.
      • L.K. Schubert, Semantic representation, 29th AAAI Conference (AAAI15), Jan. 25-30, 2015, Austin, TX.
      • Omri Abend; Ari Rappoport, The State of the Art in Semantic Representation, In the Proceedings of ACL 2017, Vancouver, BC, August, 2017

Module 5: Future Directions

  • March 27, 29 Spring Break
  • April 3 PredPatt compared to Automatic Semantic Role Labeling, Chapter 3 Sheng Zhang; Rachel Rudinger; Benjamin Van Durme *An Evaluation of PredPatt and Open IE via Stage 1 Semantic Role Labeling IWCS 2017.
  • April 5 Student Presentations
    • Ellie Pavlick, Johan Bos, Malvina Nissim, Charley Beller, Ben Van Durme, Chris Callison-Burch *Adding Semantics to Data-driven Paraphrasing In the Proceedings of ACL 2015, pp. 1512-1522
    • Marco Baroni; Roberto Zamparelli Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space In the Proceedings of EMNLP 2010, pp. 1183-1193. An interesting related paper. Pay special attention to Section 4 on the background to existing compositional methods.
    • Laura Rimell; Jean Maillard; Tamara Polajnar; Stephen Clark RELPRON: A Relative Clause Evaluation Data Set for Compositional Distributional Semantics, Computational Linguistics, Volume 42, Issue 4 - December 2016
  • April 10 Student Presentation and Automatic Event Extraction, THYME/RED,
    • Lifu Huang, Heng Ji, Kyunghyun Cho, Clare R. Voss, *Zero-Shot Transfer Learning for Event Extraction arXiv, July, 2017.
    • Danilo Croce; Alessandro Moschitti; Roberto Basili; Martha Palmer Verb Classification using Distributional Similarity in Syntactic and Semantic Structures , In the Proceedings of ACL 2012, pp. 263-272
  • April 12 Recovery of Implicit Arguments, Tim O'Gorman
  • April 17 Grounded Verb Semantics
    • Lanbo She; Joyce Chai (2017) *Interactive Learning of Grounded Verb Semantics towards Human-Robot Communication ACL 2017 Vancouver, BC, August, 2017.\
  • April 19 Logic plus distributional models
    • Islam Beltagy, Stephen Roller, Pengxiang Cheng, Katrin Erk,and Raymond J. Mooney. *Representing meaning with a combination of logical and distributional models. Computational Linguistics 42(4), special issue on formal distributional semantics.
    • Machine Learning links:
      • Deep Learning Summer School, Montreal 2016
      • 32nd International Conference on Machine Learning, Lille, 2015
    • Christopher Manning's Videos
      • Language Vectors
      • Deep Learning
    • Yoav Goldberg, primer and tutorial
      • Primer
      • T1: Practical Neural Networks for NLP: From Theory to Code

Module 6: Term Project Paper Presentations

  • April 24, 26 Student Presentations
  • May 1, 3 Student Presentations

Possbile Additional Papers on Events:

  • The NAACL and ACL Events Workshops
  • James Pustejovsky; Marc Verhagen, 2009, SemEval-2010 Task 13: Evaluating Events, Time Expressions, and Temporal Relations (TempEval-2) In the Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009) held with NAACL-2009, Boulder, CO.
  • Rei Ikuta and Martha Palmer, (2014) Challenges of Adding Causation to Richer Event Descriptions, In the Proceedings of the 2nd Events Workshop, held in conjunction with ACL 2014, Baltimore, MD.
  • McClosky, D., Surdeanu, M., & Manning, C. D. (2011). Event extraction as dependency parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1 (pp. 1626-1635).

Advanced topics: possible term projects and/or post-class readings for interest:

  • Steven White, Drew Reisinger, Keisuke Sakaguchi, Tim Vieira, Sheng Zhang, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2016. Universal Decompositional Semantics on Universal Dependencies. In Empirical Methods in Natural Language Processing (EMNLP 2016), Austin, TX.
  • Travis Wolfe, Mark Dredze, and Benjamin Van Durme. 2017. Pocket Knowledge Base Population. In The Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2017), Vancouver, BC.
  • Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017.
  • Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework. In The Proceedings of the 8th International Conference on Natural Language Processing (IJCNLP).
  • Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal Commonsense Inference. Transactions of the Association for Computational Linguistics, 5:379 $(G!9 (B395.
  • Ellie Pavlick and Chris Callison-Burch. Most babies are little and most problems are huge: Compositional Entailment in Adjective Nouns ACL 2016, Berlin, Germany, August, 2016.
  • Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2013. Concreteness ratings for 40 thousand generally known English word lemmas. Behavior research methods, pages 1-8.
  • Felix Hill and Anna Korhonen. 2014. Concreteness and subjectivity as dimensions of lexical meaning. In the Proceedings of ACL 2014
  • David R. Dowty, 1986, The effects of aspectual class on the temporal structure of discourse: semantics or pragmatics? Linguistics and Philosophy, February 1986, Volume 9, Issue 1, pp 37-61

Machine Learning Background

  • Tom M. Mitchell, 2006, Machine Learning Department technical report CMU-ML-06-108, Carnegie Mellon University, The Discipline of Machine Learning
  • Machine Learning Resources/Links
    • Dan Klein's Machine Learning for Natural Language Processing: New Developments and Challenges (slides and video)
    • Michael Collins tutorial on NLP
    • Introduction to Machine Learning, S V N Vishwanathan
    • Weka, a collection of machine learning algorithms for data mining tasks.
    • Orange, Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting.
    • Videos of Andrew Ng's Stanford ML course
    • Noah Smith's course titled Language and Statistics, at CMU

Background in Ontologies

  • SUMO
  • CYC
  • Description Logic, including CLASSIC and OWL