Real-Time Media

This course focuses on designing, developing and delivering real-time, performative work using audio and video elements. The class will have an emphasis on using MaxMSPJitter and other tools to create performative experiences that dynamically combine interactive elements such as video, sound, and code, allow for the unfolding of engaging narratives, and generate compelling visuals in real time.

We will look at various examples of both multimedia performances and installations, explore how we can apply the technologies we have learned to design real-time systems, and discuss methods we can use to make our work more engaging.

The class is three-fold and divided into tech tutorials, discussions of existing examples, and in-class performances.

DIVERSE PERSPECTIVES AND LEARNING MODES

Your success in this class is important to me. We all learn differently and require different kinds of accommodations. If there are aspects of this course that prevent you from learning or exclude you in any way, I invite you to communicate this with me. Together we’ll develop strategies to meet both your needs and the requirements of the course.

OFFICE HOURS

You all enter this classroom with different sets of skills. In virtue of this fact, I open my office to you as an extension of the classroom. You can book office hours with me via email. There’s no incorrect way to approach office hours, and they can be as formal or informal as you would like. In this class, we will not shy away from difficult conversations and work closely together to cultivate a space of openness and mutual support.

COURSE OBJECTIVES

At the completion of this course, the student will:

Be able to strategize and build real-time systems for performative work using a combination of tools.
Gain greater perspective of how media (both audio and video) can be used as materials in performative work.
Have a good understanding of how real-time media is used in the art and performance landscape.
Learn node-based coding using MaxMSPJitter.
Explore a variety of technical tools and methods for multi-media performance.

Evaluation

Grades will be determined according to this criteria:

Participation and Attendance 45%
Final Project 30%
Midterm Audio Performance 20%
Timer Patcher Homework 5%

The performance projects will be evaluated based on:

Concept – clear intention to tell a story, express emotions, or convey messages
Process – active efforts to develop the piece, meeting in-progress milestones,
Execution – level of completion + how well the performance reflects the concept

ASSIGNMENTS

Homework assignments must be documented (written description, photos, screenshots, screen recording, code, and video all qualify based on the assignment). Each assignment has a due date as stated.

Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete, an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are simply not attempted it may be marked partially complete. If an attempt isn’t made to meet the criteria specified it will be marked incomplete.

COMMUNICATION

You must check your email for important updates about class.

PARTICIPATION

ITP/IMA is committed to facilitating the fullest possible participation of all students. There are many forms of participation. Please communicate what kinds of engagement are best for you so it can be taken into account.

Examples of modes of participation can look like: asking questions, going to office hours, sending and reading emails, class group discussion, arriving on time, going to class, taking notes, listening to peers, submitting responses to a form (anonymous or not), following instructions, active listening, etc.

Attendance

Attendance is mandatory. Please email your instructor if you are going to miss a class. After the first two weeks of the add/drop period, effective in week three onward, students are permitted the following number of absences: 3 absences.

Any more than 3 absences after add/drop will affect your grade. Two late arrivals (more than 10 minutes after start time) will count as 1 absence.

For example:

4 absences will result in your highest possible grade being a B instead of an A

5 absences will result in your highest possible grade being a C instead of an A

6 absences will result in your highest possible grade being a D instead of an A

7 absences will result in an automatic F for the course

STATEMENT OF ACADEMIC INTEGRITY

Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else.

Collaboration is highly valued and often necessary to produce great work. Students build their own work on that of other people and giving credit to the creator of the work you are incorporating into your own work is an act of integrity. Plagiarism, on the other hand, is a form of fraud. Proper acknowledgment and correct citation constitute the difference.

Link to the Tisch Student Handbook

STATEMENT ON ACCESSIBILITY

It’s crucial for our community to create and uphold learning environments that empower students of all abilities. We are committed to create an environment that enables open dialogue about the various temporary and long term needs of students and participants for their academic success. We encourage all students and participants to discuss with faculty and staff possible accommodations that would best support their learning. Students may also contact the Moses Center for Student Accessibility (212-998-4980) for resources and support. Link to the Moses Center for Student Accessibility

STATEMENT ON COUNSELING AND WELLNESS

Your health and safety are a priority at NYU. Emphasizing the importance of the wellness of each individual within our community, students are encouraged to utilize the resources and support services available to them 24 hours a day, 7 days a week via the NYU Wellness Exchange Hotline at 212-443-9999. Additional support is available over email at [email protected] and within the NYU Wellness Exchange app. Link to the NYU Counseling and Wellness Center

STATEMENT ON USE OF ELECTRONIC DEVICES

Laptops and other electronic devices are essential tools for learning and interaction in classrooms. However, they can create distractions that hinder students’ ability to actively participate and engage. Please be mindful of the ways in which these devices can affect the learning environment, please refrain from doing non-class oriented activities during class.

STATEMENT ON TITLE IX

Tisch School of the Arts is dedicated to providing its students with a learning environment that is rigorous, respectful, supportive and nurturing so that they can engage in the free exchange of ideas and commit themselves fully to the study of their discipline. To that end, Tisch is committed to enforcing University policies prohibiting all forms of sexual misconduct as well as discrimination on the basis of sex and gender. Detailed information regarding these policies and the resources that are available to students through the Title IX office can be found by using the following link: Link to the NYU Title IX Office

STATEMENT OF PRINCIPLE

Teachers and students work together to create a supportive learning environment. The educational experience in the classroom is one that is enhanced by integrating varying perspectives and learning modes brought by students.

CODE OF CONDUCT

Our signature is collaboration – Not competition.
– Red Burns

We live in a fractured world where communities are burdened by differences – so what we ask of you is nothing short of miraculous.
— Red Burns

ITP/IMA is a community whose mission is to explore the imaginative uses of emerging technologies — to make people’s lives safer, more just, more beautiful, more meaningful, and more fun. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.

The scope of this Code of Conduct applies to all ITP/IMA community members (faculty, staff, students, residents, fellows, and guests), ITP/IMA physical and digital spaces (e.g. Email listservs, Yorb, and ITP/IMA Discord server), and all courses, events, and projects created at or in service to ITP/IMA.

SOLIDARITY

We are a community of and in solidarity with, people from every gender identity and expression, sexual orientation, race, ethnicity, language, neuro-type, all visible or invisible disability, body size, ability, class, religion, culture, subculture, age, skill level, occupation and background. As a result, we continually work to make participation in our community a harassment-free experience for identities above as well as all levels of experience, education and socio-economic status.

INTERNATIONAL

Our student body typically has a large International student presence. We understand that, by the nature of being established in New York, our curriculum and social environment naturally centers events and histories specific to the United States of America. We know that a large percentage of our student body are international and we will continue to incorporate, celebrate and acknowledge events and histories from those countries with respect and an open mind.

We align ourselves with any values in pursuit of equity and freedom for all identities and, by that nature, are in alignment with similar views in International countries as well.

LAND ACKNOWLEDGEMENT

We acknowledge that we are gathered on the unceded land of the Lenape and Canarsie peoples. We acknowledge the Lenape and Canarsie communities, their elders both past and present, as well as future generations.

We also acknowledge that NYU, Tisch School of the Arts, and ITP were founded upon exclusions and erasures of many Indigenous peoples, including those on whose land this institution is located. This acknowledgement demonstrates a commitment to beginning the process of working to dismantle the ongoing legacies of settler colonialism.

PRIVILEGE

We recognize that simply being part of the ITP/IMA community places us in a privileged position as we have the opportunity to receive an education from a private, expensive and prestigious university. We know that there are many equally talented students around the world who would like to partake of this community and, due to academic inaccessibility, it is not possible. We know and will continue to acknowledge that our privilege does not mean that we have more value than other people. We acknowledge this inequality and hold personal responsibility to share what we do in the spirit of inclusivity and create access where we can, as often as we can. We will use this privilege as a tool to help those most in need.

TECHNOLOGY

We pledge to center creative and ethical uses of technology in our research, teaching, and making. We accept the claim that technology is a reflection of society, its histories, and its politics. We reject the claim that technology is neutral and acknowledge that every technology has the potential to do as much harm as good. We acknowledge that when technologies cause harm, the harm disproportionately affects Black, Brown, Indigenous, People of Color (BIPOC), queer, trans, disabled, femme, low-income, survivors, and all other marginalized bodies and communities worldwide.

Schedule

Week 1: Intro

Wednesday 09/04: SLIDES

In Class:

Introductions
What is this Class?
What are we doing in this class?
Technical Tools: Overview

Homework:

Watch Tutorial and Create a Patcher – Due Monday 9/23

Week 2: Practice Performance – Desktop

Monday 09/09: SLIDES

In Class:

Examples: Desktop Performance
Performance Critique
Desktop Performance: Prompt
OBS Studio
Ideation

Homework:

Create a Desktop Performance (SLIDE #24) – Wednesday 9/11
Watch Tutorial and Create a Patcher – Due Monday 9/23

Wednesday 09/11

In Class:

Show Desktop Performances
Feedback and Reflections

Homework:

Watch Tutorial and Create a Patcher – Due Monday 9/23

Week 3: Practice Performance – Connection

Monday 09/16: SLIDES

In Class:

Examples: Connection
Networked Performance Tools
Network Performance: Prompt & Groups
Group Work

Homework: In Class Performance (Due 09/15)

Create a Networked Performance (SLIDE 28) – Due Wednesday 9/18
Watch Tutorial and Create a Patcher – Due Monday 9/23

Wednesday 09/18

In Class:

Show Connection Performances
Reflections

Homework:

Watch Tutorial and Create a Patcher – Due Monday 9/23

Week 4: Audio I

Monday 09/23: SLIDES

In Class:

Timer Patchers
Basics of MIDI
Simple Piano
Player Piano
Computer Keyboard Piano

Homework:

Set up a virtual MIDI bus on your computer – Due Wednesday 9/25
Do some research on the basics of musical composition. – Due Wednesday 9/25

Wednesday 09/25: SLIDES

In Class:

Another Way to Synthesize Sound: Analog Sound Waves
Connecting to Arduino
Playing with the Max Interface
Experiment: Algorithmic Music with Interface Visuals

Homework:

Go through all the patcher examples. Review what we’ve covered this week.

Week 5: Audio II

Monday 09/30: SLIDES

In Class:

Sample and Manipulate Sound with Buffer and Groove
Sound Files & Buffer
Make Something
Sampling Sound and Using It as Musical Instrument

Wednesday 10/02: SLIDES

In Class:

Different Ways to Work with Sound in p5
Speech as Material for Audio Performance
Alternative Ways to Play Music/Sound

Homework:

Review what we’ve learnt about synthesizing, sampling, and manipulating sound. Play with the examples. Experiment with building your own audio performance system.

Week 6: Audio III

Tuesday 10/07: SLIDES

In Class:

A Few Audio-Centered Project Examples
A Closer Look
Audio Performance Assignment
Workshop: Three Ideas for Midterm

Homework:

Pick a preliminary idea for your Audio Performance. Experiment with different tools and methods to start building a performance system what works for you.
On Wednesday, we will continue with this and share the systems you’ve built at the end of the class.

Wednesday 10/09: SLIDES

In Class:

Pick a Preliminary Idea & Create Your Audio Performance System
Come up with a Mini Performance to Demonstrate How the System Works

Homework:

Audio Performance Project Progress Report + Tech Demo: Submit Slides Here – Due Tuesday 10/15

Week 7: Audio Performance Development

Tuesday 10/15: SLIDES

In Class:

Project Progress Report and Tech Check
Getting Ready for Performance

Homework:

Audio Performance: Work on your audio performance. Get ready for the performance next week:
Make a detailed list of every step you’ll need to take to set up the performance.
Practice to limit your setup time to under 3 minutes.

Wednesday 10/16: SLIDES

In Class:

Project Progress Report and Tech Check
Getting Ready for Performance

Homework:

Audio Performance – Due Monday 10/21

Week 8: Audio Performance

Monday 10/21:

In Class:

Audio Performances Part 01
Feedback and Reflections

Wednesday 10/23:

In Class:

Audio Performances Part 02
Feedback and Reflections

Homework:

Fill out Midterm Survey (LINK TBD) – Due Monday 10/28

Week 9: Video I

Monday 10/28: SLIDES

In Class:

Project Examples
Hydra

Wednesday 10/30: SLIDES

In Class:

Jitter Matrix Basics & Animation
Video Basics – Webcam Feed, Resolution, & Resequencing Frames
Video Basics – Playing & Controlling Videos

Homework:

Find or record a collection of videos that you would potentially use in a performance (be aware of copyright issues).

Week 10: Video II

Monday 11/04: SLIDES

In Class:

Video Effects
Audio Visualization
Video Playback Systems

Wednesday 11/06: SLIDES

In Class:

Intro to OpenGL (3D)
Working with 3D Shapes & Models
More Examples with OpenGL

Homework:

Review what we’ve learnt about video synthesizing with Hydra, pattern animation, video playback, video effects, and working with 3D models.
Play with the examples. Experiment with building your own real-time visual effects system.

Week 11: Video III

Monday 11/11

In Class:

Lighting (with guest speaker) – Zoom Class

Wednesday 11/13: SLIDES

In Class:

Final Project Assignment
More Project Examples
Form Groups & Come up with Ideas

Homework:

Final Project – Proposal: Submit Slides Here – Due Monday 11/18

Week 12: Final Project Proposal

Monday 11/18:

In Class:

Project Proposals + WIP Demo (Perform a Small Portion of Your Work)

Homework:

Final Project

Wednesday 11/20

In Class:

Project Proposals + WIP Demo (Perform a Small Portion of Your Work)

Homework:

Final Project

Week 13: Final Project Development & Rehearsal

Monday 11/25

In Class:

Studio & 1:1 Meetings: Equipment, Audio & Lighting Checklists

Wednesday 11/27

In Class:

Tech Check & Rehearsal in the Classroom & Other Spaces

Homework:

Final Project

Week 14: Final Project Rehearsals

Monday 12/02

In Class:

Tech Check & Rehearsal in the Garage, Timeslots TBD

Wednesday 12/04

In Class:

Final Performances in the Classroom & Other Spaces

Homework:

Final Project

Week 15: Final Project

Monday 12/09

Final Performances in the Garage