AI Interaction Design

Please read here carefully before the workshop.

This is a one-day workshop which introduces a workshop on the applying Deep Learning Technique for interaction design and this document is created for the class of National Cheng Kung University.

As you know, deep learning is used in various fields and is starting to be used for interaction design, which is the design of interactions between users and computers.

In this class, we will first learn the basics of neural networks and a simple perceptron model, and then we will try to implement a simple deep learning application using several libraries.

Basic knowledge of javascript is required as p5.js and teachable machines will be used as learning materials.To complete the course, students are required to have a laptop PC, a webcam, an internet connection and a Chrome browser.

The workshop will be conducted using ZOOM. Since the webcam will be used during the workshop, the camera display on the ZOOM side does not need to be turned on all the time. Please understand that you may need to turn on the ZOOM camera for commemorative photos.

If you have any questions, please feel free to ask them via ZOOM chat or commentable. Since we only have 2 hours, if you have any difficulties in solving them on the spot, please review the archived videos that will be distributed later.

• Our comment app: https://commentable.lolipop.io/

Let's start with the basics of p5.js. p5.js is a coding environment based on javascript so that we can code on a browser without any specific IDE, like a Visual Studio, Xcode and so on., which is developed by the processing foundation to support creative coding. It runs in the browser, so there is no need to build an environment and you can start developing right away. Go to the below link and open up with Chrome browser.

• https://p5js.org

Now you got p5.js website and will see a menu on the left side. Just click Editor on the menu.

Here is a first template when you got to start the p5.js coding.

On an initial window, we get three files.

• index.html
• sketch.js
• style.css

For more information on the role of each file, please see following page.

• cssとhtmlとjs

On our workshop, we just check and play below examples.

• Drawing: Continuous Lines
• Dom: Capture
• Simulate: Particle

If you want to get to know about P5.js more, please visit below site and check beginners videos out!!

• https://thecodingtrain.com

The goal of this workshop is to apply deep learning techniques to interaction design. Interaction design is the design of interaction. The research area is how to design the relationship between computers and humans, or humans and humans. In this workshop, we will use Super Mario Bros. as an example, and work together to create a case study that uses deep learning for its controls.

Start with this template and save it on your account.

Now, we can control Mario by pressing a keyboard.

• d: move left
• a: move right
• w: jump
• s: crouch

Let's change keyboard inputs to mouse inputs. This is a simple transformation of interaction.

• What kind of mouse control design would you use?

It is a simple question, but fundamental of interaction design. The key is to notice the experiential value that comes from not changing the game content, but only the interaction design.

Here, we try to control Mario with keyboard and mouse inputs referring a traditional game pad mental model. cited from ファミリーコンピュータ.

If you are accustomed to this type of general controller operation, you may find it easier to assign the jump button to your right hand and the directional controls to your left.

I'm going to try to use something different instead of the usual keyboard and mouse for the controls. Yes, today I use camera. Here is my design.

• up right hand: move right
• up left hand: move left
• up both hands: jump

In order to achieve this kind of operation design, it used to be necessary to use computer vision to build advanced algorithms. However, recent AI technology, specifically deep learning, has made this kind of implementation easier. There is a lot of knowledge and methods available in libraries for implementing deep learning, and it is impossible to introduce them in today's workshop. Today we will try to use deep learning with ml5.js, which can work in conjunction with p5.js.

• https://ml5js.org/

Now, let's put a camera captured image on the game screen, just like below.