Research

 

This page showcases our projects (published after August 2014).

本ラボの研究プロジェクト(2014年8月以降)を紹介します.

ElasticPlay: Interactive Video Summarization with Dynamic Time Budgets
Video consumption is being shifted from sit-and-watch to selective skimming. Existing video player interfaces, however, only provide indirect manipulation to support this emerging behavior. Video summarization alleviates this issue to some extent, shortening a video based on the desired length of a summary as an input variable. But an optimal length of a summarized video is often not available in advance. Moreover, the user cannot edit the summary once it is produced, limiting its practical applications. We argue that video summarization should be an interactive, mixed-initiative process in which users have control over the summarization procedure while algorithms help users achieve their goal via video understanding. In this paper, we introduce ElasticPlay, a mixed-initiative approach that combines an advanced video summarization technique with direct interface manipulation to help users control the video summarization process. Users can specify a time budget for the remaining content while watching a video; our system then immediately updates the playback plan using our proposed cut-and-forward algorithm, determining which parts to skip or to fast-forward. This interactive process allows users to fine-tune the summarization result with immediate feedback. We show that our system outperforms existing video summarization techniques on the TVSum50 dataset. We also report two lab studies (22 participants) and a Mechanical Turk deployment study (60 participants), and show that the participants responded favorably to ElasticPlay.



Haojian Jin, Yale Song, and Koji Yatani. ElasticPlay: Interactive Video Summarization with Dynamic Time Budgets. To appear in MM 2017 (oral presentation).

Strummer: An Interactive Guitar Chord Practice System
Musical instrument playing is a skill many people desire to acquire, and learners now have a wide variety of learning materials. However, their volume is enormous, and novice learners may easily get lost in which songs they should practice first. We develop Strummer: an interactive multimedial system for guitar practice. Strummer provide data-driven and personalized practice for learners in order to identify important and easy-to-learn chords and songs. This practice design is intended to encourage smooth skill transfers to songs that learners even have not seen. Our user study confirms the benefits and possible improvements of the Strummer system. In particular, participants expressed their positive impressions on lessons provided by the system.

楽器の演奏は多くの人が望むスキルであり,様々な練習方法や教材が考案されているが,その種類の多さゆえ初心者は何を練習すればよいのか迷いがちである.さらに独学では自分の演奏の可否を判断して修正することが難しく,これらはモチベーション低下につながる問題となっている.本稿では,データドリブンでインタラクティブなギターコード練習システム 「Strummer」 を提案する.Strummer システムの開発にあたり,コードラベルが付与された 727 曲分の楽曲データを解析し,コードの出現率と難易度を考慮した曲の重要性の尺度を新たに定義した.これをシステム中で用いることで,ユーザは簡単かつ重要な楽曲から練習が可能になる.さらに,本システムは音響信号解析によってユーザが正しいコードを弾いたのかを認識してフィードバックする機能を有しており,インタラクティブかつ段階的な練習を提供する.実験では 5 人のギター初心者に計 5 時間ずつ本システムを用いてギター練習をしてもらい,インタビューやアンケートを実施し,本システムの評価を行った.



Shunya Ariga, Masataka Goto, and Koji Yatani. Strummer: An Interactive Guitar Chord Practice System. In Proc. ICME 2017, pp. 1057-1062.
有賀竣哉,後藤真孝,矢谷浩司.「Strummer:インタラクティブなギターコード練習システム」第114回報処理学会音楽情報科学研究会,2017-MUS-114(24),1-6,2017年2月.学生奨励賞受賞

指尖容積脈波を同時に取得する指紋認証システムの試作と評価
スマートデバイスの普及により生体認証は日常的に使用できる技術となっている.生体認証においては認証に必要な情報だけではなく,ユーザの健康に関する情報も同時に取得できる可能性がある.これを実現することにより,例えばユーザは携帯電話をアンロックする際に自身の健康情報を同時に記録することが可能となり,ライフログなどのアプリケーションに活用できる.我々は,このようなシステムをDual-purpose biometrics と呼んでいる.本論文では,Dual-purpose biometrics の一例として,指紋認証と同時に指尖容積脈波を取得するシステムを提案し,その試作と評価を行った.予備実験の結果,指の中心部だけでなく周辺部において,脈拍と呼吸数を指紋認証時に取得できることがわかった.この実験結果から,指尖容積脈波を同時に取得する指紋認証システム実現可能性と有用性の高いアプリケーションを議論する.

橋爪崇弘,矢谷浩司.「指尖容積脈波を同時に取得する指紋認証システムの試作と評価」第53回報処理学会UBI研究会,2017年3月.優秀論文賞受賞.

重心移動が可能なダンベル型デバイスの製作とその知覚に関する実験
筋力トレーニングは健康の増進に寄与することが知られている.しかしながら,従来の筋力トレーニングは単純な反復運動で構成されているため,ユーザが動機を維持しにくい課題がある.本研究では,この課題を解決するための手法として,重心の移動が可能なダンベル型デバイスを提案する.このデバイスでは左右のウェイトプレート間で水を移動させることにより重心の移動が可能である.重心を移動させることにより,デバイスの総重量に変化がなくとも,負荷が変化したような知覚を与えられる.この技術によって,ゲームをプレイしながら筋力トレーニングを行う新しいexergame などのアプリケーションが実現できる.本稿では,アクリルを用いて製作したダンベル型デバイスと,このデバイスがユーザに与える負荷の知覚に関する実験について報告する.

有薗拓也,橋爪崇弘,矢谷浩司.「重心移動が可能なダンベル型デバイスの製作とその知覚に関する実験」第53回報処理学会UBI研究会,2017年3月.学生奨励賞受賞

車椅子アクセシビリティ情報収集における飲食店の外観写真の有用性
車椅子利用者が飲食店を訪れる際,目的地にて車椅子が支障なく利用できるかという情報を知ることが必要となる.このようなアクセシビリティ情報をいかにして入手しているのかを調査するため,本研究ではまず7名の車椅子利用者にインタビューを行った.結果,飲食店口コミサイトにおける外観写真をよく利用していることがわかった.本研究ではさらに,車椅子利用者がアクセシビリティ情報の判断に有用だと感じたレストランの写真を収集・分析し,それらの特徴について整理を行った.本稿ではインタビュー調査の結果とともに,収集した写真の分析結果を報告し,これらの知見が車椅子アクセシビリティの向上を目指すユビキタス・モバイルシステムにどのように応用できるかを議論する.

宮崎章太,矢谷浩司.「車椅子アクセシビリティ情報収集における飲食店の外観写真の有用性」第53回報処理学会UBI研究会,2017年3月.

Phones on Wheels: Exploring Interaction for Smartphones with Kinetic Capabilities
This work introduces novel interaction and applications using smartphones with kinetic capabilities. We develop an accessory module with robot wheels for a smartphone. With this module, the smartphone can move in a linear direction or rotate with sufficient power. The module also includes rotary encoders, allowing us to use the wheels as an input modality. We demonstrate a series of novel mobile interaction for mobile devices with kinetic capabilities through three applications.



Takefumi Hiraki, Koya Narumi, Koji Yatani, and Yoshihiro Kawahara. 2016. Phones on Wheels: Exploring Interaction for Smartphones with Kinetic Capabilities. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct), 121-122.

インフォグラフィックスの作成をインタラクティブに支援するシステム(An interactive system for authoring infographics)

データ可視化の手段としてinfographics を用いる機会が近年増えている。データ可視化においては、読み手により良い理解を促すためにストーリーテリングが重要である。ストーリーテリングには重要なデータ探索、ストーリー作り、ストーリーの伝え方の3つのプロセスがある。本稿ではinfographics にてよく用いられるスタイルの画像を自動的に生成するシステムによって特にストーリーの伝え方を支援するシステムを提案する。画像は、インターネットからピクトグラムを取得しそれを加工することによって生成を行う。

岩下知美,矢谷浩司.「インフォグラフィックスの作成をインタラクティブに支援するシステム」第170回情報処理学会HCI研究会,2016年10月.

lumio
LumiO: A Plaque-aware Toothbrush
Toothbrushing plays an important role in daily dental plaque removal for preventive dentistry. Prior work has investigated improvements on toothbrushing with sensing technologies. But existing toothbrushing support focuses mostly on estimating brushing coverage. Users thus only have indirect information about how well their toothbrushing removes dental plaque. We present LumiO, a toothbrush that offers users continuous feedback on the amount of plaque on teeth. Lumio uses a well-known method for plaque detection, called Quantitative Light-induced Fluorescence (QLF). QLF exploits a red fluorescence property that bacterium in the plaque demonstrates when a blue-violet ray is cast. Blue-violet light excites this fluorescence property, and a camera with an optical filter can capture plaque in pink. We incorporate this technology into an electric toothbrush to achieve improvements in performance on plaque removal in daily dental care. This paper first discusses related work in sensing for oral activities and interaction as well as dental care with technologies. We then describe the principles of QLF, the hardware design of LumiO, and our vision-based plaque detection method. Our evaluations show that the vision-based plaque detection method with three machine learning techniques can achieve F-measures of 0.68 – 0.92 under user-dependent training. Qualitative evidence also suggests that study participants were able to have improved awareness of plaque and build confidence on their toothbrushing.



T. Yoshitani, M. Ogata, and K. Yatani. LumiO: A Plaque-aware Toothbrush. in Proceedings of UbiComp 2016, 2016.
[Paper] [Video]

Lock n' LoL: Group-based Limiting Assistance App to Mitigate Smartphone Distractions in Group Activities
Prior studies have addressed many negative aspects of mobile distractions in group activities. In this paper, we present Lock n' LoL. This is an application designed to help users focus on their group activities by allowing group members to limit their smartphone usage together. In particular, it provides synchronous social awareness of each other's limiting behavior. This synchronous social awareness can arouse feelings of connectedness among group members and can mitigate social vulnerability due to smartphone distraction (e.g., social exclusion) that often results in poor social experiences. After following an iterative prototyping process, we conducted a large-scale user study (n = 976) via real field deployment. The study results revealed how the participants used Lock n' LoL in their diverse contexts and how Lock n' LoL helped them to mitigate smartphone distractions.

Minsam Ko, Seungwoo Choi, Koji Yatani, and Uichin Lee. Lock n' LoL: Group-based Limiting Assistance App to Mitigate Smartphone Distractions in Group Activities. In Proc. CHI 2016, 998-1010.
[Paper] Best Paper Award winner

animation
Autocomplete Hand-drawn Animations
Hand-drawn animation is a major art form and communication edium, but can be challenging to produce. We present a system to help people create frame-by-frame animations through hand-drawn sketches. We design our interface to be minimalistic; it contains only a canvas for sketches and a few controls. When users are drawing on the canvas, our system silently analyzes all past sketches and predicts what might be drawn in the future across both spatial locations and temporal frames. The interface also offers users suggestions to beautify existing drawings. These predictions and suggestions greatly reduce the workload on creating multiple frames for animation and also help to create desirable results. Users can accept, ignore, or modify such predictions visualized on the canvas by simple gestures. Our method considers both high level structures and low level repetitions, and can significantly reduce manual workload while help produce better results. We evaluate our system through a preliminary user study and confirm that it can enhance both users’ objective performance and subjective satisfaction.



J. Xing, L.-Y. Wei, T. Shiratori, and K. Yatani. “Autocomplete Hand-drawn Animations,” in Proceedings of SIGGRAPH Asia 2015, 2015.
[Paper]

stylesnap
Mixed-Initiative Approaches to Global Editing in Slideware
Good alignment and repetition of objects across presentation slides can facilitate visual processing and contribute to audience understanding. However, creating and maintaining such consistency during slide design is difficult. To solve this problem, we present two complementary tools: (1) StyleSnap, which increases the alignment and repetition of objects by adaptively clustering object edge positions and allowing parallel editing of all objects snapped to the same spatial extent; and (2) FlashFormat, which infers the least-general generalization of editing examples and applies it throughout the selected range. In user studies of repetitive styling task performance, StyleSnap and FlashFormat were 4-5 times and 2-3 times faster respectively than conventional editing. Both use a mixed-initiative approach to improve the consistency of slide decks and generalize to any situations involving direct editing across disjoint visual spaces.



Darren Edge, Sumit Gulwani, Natasa Milic-Frayling, Mohammad Raza, Reza Adhitya Saputra, Chao Wang, and Koji Yatani. Mixed-Initiative Approaches to Global Editing in Slideware,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2015), 2015.
[Paper]

nugu
NUGU: A Group-based Intervention App for Improving Self-Regulation of Limiting Smartphone Use
Our preliminary study reveals that individuals use various management strategies for limiting smartphone use, ranging from keeping smartphones out of reach to removing apps. However, we also found that users often had difficulties in maintaining their chosen management strategies due to the lack of self-regulation. In this paper, we present NUGU, a group-based intervention app for improving self-regulation on smartphone use through leveraging social support: groups of people limit their use together by sharing their limiting information. NUGU is designed based on social cognitive theory and it has been developed iteratively through two pilot tests. Our three-week user study (n = 62) demonstrated that compared with its non-social counterpart, the NUGU users’ usage amount significantly decreased and their perceived level of managing disturbances improved. Furthermore, our exit interview confirmed that NUGU’s design elements are effective for achieving limiting goals.

M. Ko, S. Yang, J. Lee, C. Heizmann, J. Jeong, U. Lee, D. H. Shin, K. Yatani, J. Song, and K. Chung, “NUGU: A Group-based Intervention App for Improving Self-Regulation of Limiting Smartphone Use,” in Proceedings of the ACM conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2015), 2015.
[Paper]

reviewcollage
ReviewCollage: A Mobile Interface for Direct Comparison Using Online Reviews
Review comments posted in online websites can help the user decide a product to purchase or place to visit. They can also be useful to closely compare a couple of candidate entities. However, the user may have to read different webpages back and forth for comparison, and this is not desirable particularly when she is using a mobile device. We present ReviewCollage, a mobile interface that aggregates information about two reviewed entities in a one-page view. ReviewCollage uses attribute-value pairs, known to be effective for review text summarization, and highlights the similarities and differences between the entities. Our user study confirms that ReviewCollage can support the user to compare two entities and make a decision within a couple of minutes, at least as quickly as existing summarization interfaces. It also reveals that ReviewCollage could be most useful when two entities are very similar.



H. Jin, T. Sakai, and K. Yatani, “ReviewCollage: A Mobile Interface for Direct Comparison Using Online Reviews,” in Proceedings of the ACM SIGCHI International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI 2014), 2014.
[Paper] [Video] Honorable Mention Award

talkzones
TalkZones: Section-based Time Support for Presentations
Managing time while presenting is challenging, but mobile devices offer both convenience and flexibility in their ability to support the end-to-end process of setting, refining, and following presentation time targets. From an initial HCI-Q study of 20 presenters, we identified the need to set such targets per “zone” of consecutive slides (rather than per slide or for the whole talk), as well as the need for feedback that accommodates two distinct attitudes towards presentation timing. These findings led to the design of TalkZones, a mobile application for timing support. When giving a 20-slide, 6m40s rehearsed but interrupted talk, 12 participants who used TalkZones registered a mean overrun of only 8s, compared with 1m49s for 12 participants who used a regular timer. We observed a similar 2% overrun in our final study of 8 speakers giving rehearsed 30-minute talks in 20 minutes. Overall, we show that TalkZones can encourage presenters to advance slides before it is too late to recover, even under the adverse timing conditions of short and shortened talks.



B. Saket, S. Yang, H. Z. Tan, K. Yatani, and D. Edge, “TalkZones: Section-based Time Support for Presentations,” in Proceedings of the ACM SIGCHI International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI 2014), 2014.
[Paper] [Video] Honorable Mention Award