CSE/ISE 364: Advanced Multimedia
Lectures: T/Th 3:50-5:10pm, Rm 2205 Computer Science
Course Webpage: http://tamaraberg.com/teaching/Spring_09/

Instructor: Tamara Berg  (tlberg -at- cs.sunysb.edu)
Office: 1411 Computer Science
Office Hours: Tuesdays 5:10-6:10pm, Thursdays 2:00-3:00pm, 5:10-6:10pm



Topics will include:
  • Text
  • Sound
  • Images
  • Video
  • Tagging & Annotation
  • Social Media
  • Location Information
  • Speech Processing
  • Recommendation systems
The focus of this course will be on multimedia as accessed via the web. For each type of digital media studied we will discuss fundamentals including storage, compression, and use. We will also study various algorithms for organizing, retrieving, and manipulating digital media.

Assignments
Assignment 1 Due Feb 19 - Due date extended to March 3
Assignment 2 Due March 12
Assignment 3A Due April 2 - Assignment 3B will build on Assignment 3A. As such I will release a solution to 3A on Sat, April 4. So, no late homeworks will be accepted after that.
Assignment 3B Due April 16.

Projects
Project discussions Visit office hours March 10 or 12 to discuss project ideas. This is worth 20 pts based on well thought out project ideas.
Project proposal presentations In class March 17. Please prepare a 10 minute powerpoint presentation detailing: your problem description, challenges you will have to solve, a timeline for when and how you will solve each of your challenges (100 pts). Include image mock-ups showing what you hope to accomplish.
Project update presentations In class April 21. Please prepare a 10 minute powerpoint presentation outlining: your problem description, progress so far and what further steps you will take (100 pts).
Final project presentations In class May 5 & 7, see below for schedule. Please prepare a 20 minute presentation (longer for larger groups). This presentation should outline: your problem description, implementation details, results and a demo of your system.
Final project write-up Due Monday, May 11. Please prepare a web page write-up of your project. This page should outline: your problem description, implementation details, results and a demo of your system (use a video screen capture program to capture a video of you using your system).


Tentative Schedule:
DateTopicReadings
Jan 27Introduction - Slides
Jan 29Text basics & Web document retrieval- SlidesThe Anatomy of a Search Engine
Feb 3Text modeling and classification - Slides, Demo Code
Feb 5Introduction to Matlab for multimedia - Exercise 1, Exercise 2, Soln to Exercise 2Matlab tutorial, Assignment 1 out
Feb 10Document clustering - Slides
Feb 12Intro to sound & digitization - Slides, Demo1, Demo2Chapter 6 of Textbook
Feb 17Sound analysis - See slides for Feb 12
Feb 19Guest Lecture - Margaret Schedel - Digital MusicAssignment 1 due - Due date extended to March 3
Feb 24Sound analysis (cont) & retrieval - Slides, Filtering Demo, Composition Demo"Content-Based Music Information Retrieval", Assignment 2 out
Feb 26Intro to computer vision - Slides
March 3Images, cameras, & color - SlidesChapter 4 of textbook
March 5Image content analysis - Slides, Image Demo
March 10Image retrieval - SlidesChapter 18 of textbook. Please visit office hours this week to discuss project ideas.
March 12Image retrieval (cont) - See slides for March 10Assignment 2 due.
March 17Project Proposal PresentationsPlease prepare a 10 minute presentation describing your project
March 19Image Blending & compositing - SlidesA Multiresolution Spline With Application to Image Mosaics, Peter J. Burt and Edward H. Adelson
March 24Image warping & morphing - SlidesAssignment 3A out
March 26Intro to video & tracking - SlidesChapter 5 of textbook
March 31Meta data & Tagging SlidesHT06, Tagging Paper, Taxonomy, Flickr, Academic Article, ToRead
April 2Guest Lecture - Christa Erickson - Digital ArtAssignment 3A due, Assignment 3B out
April 7Spring break - no class
April 9Spring break - no class
April 14Location Information - SlidesIM2GPS: estimating geographic information from a single image
April 16Speech Processing - Guest Lecture - Erin Palmer - SlidesAssignment 3B due
April 21Project Update PresentationsPlease prepare a 10 minute presentation describing your project progress
April 23Project Help DayCome with any project questions or problems you are encountering and we will hold project group meetings in class
April 28Social Media
April 30Collaborative Filtering
May 5Final Project Presentations Winter, Chenjie, Hongliang & Zhiqin, Daniel
May 7Final Project Presentations Konstantin, Michael, Jory & Logan, Xiu


Grading:
 This course will focus on developing a hands on understanding of various types of multi-media. There will be 4 assignments related to the main course topics. Students will also be responsible for defining and developing a project related to multi-media over the course of the semester, including a project proposal, status update, and final project presentation with web documentation.

Late homeworks will be accepted with a 10% reduction in value per day late.

Grading will consist of 60% assignments, 30% project, 10% participation.

Prerequisites:
 You do not need to have taken CSE/ISE 334. Students are expected to be proficient in programming and the basics of digital media, but I will provide most necessary background for the course as we go. Come talk to me if you have any questions!


Useful links:
Matlab
 Matlab tutorial by Hany Farid and Eero Simoncelli - Link
A more comprehensive Matlab tutorial by David Griffiths - Link
Matlab Answers from MIT - Link
Online Mathworks Matlab documentationLink
Lots of Matlab tutorialsLink

Book
 Fundamentals of Multimedia, Ze-Nian Li, and Mark S. Drew.

Other useful reference books
 Artificial Intelligence: A Modern Approach, Russel and Norvig.
Computer Vision: A Modern Approach, Forsyth and Ponce.
Foundations of Statistical Natural Language Processing, Christopher D. Manning, and Hinrich Schutze.