Published at the Wireless Communication and Information (WCI) – Berlin, 2012
Download Paper as PDF
Author & Co-Author:
Sebastian Meier, Johannes Landstorfer, Julia Werner, Prof. Reto Wettach, Andre Knörig, Jonathan Cohen — Interaction Design Studios (IxDS), Berlin, Andreas Sommerwerk — Creation Center, Deutsche Telekom AG, Berlin
The increasing number of location-aware devices along with faster wireless communication technologies have led to more and more location-based services. But so far most of them do not take in-depth user preferences and context into account. In this paper we present a new framework, based on the „Wizard of Oz“ paradigm, for prototyping a mobile location-based and context based service in the real world. Simulating a complex application before the actual development phase enables a much earlier harvesting of feedback and insights which leads to a more focused product and a less expensive development cycle.
In late 2011 the „Creation Center“ of Deutsche Telekom approached us to further develop and evaluate an idea for a new service which would be built around a complex algorithm that took into account a user‘s context and location. 1 No actual implementation of the algorithm existed at this point. Thus, our starting point was the question how to evaluate a non-existing contextual and location-based service.
Building an actual working service prototype was not within our time-frame or budget, as it required a huge dataset of locations and location-based information along with in-depth multi-dimensional user preferences, plus a set of undeveloped context-aware algorithms [ML09]. Prototyping often gives good results in a lab environment, but the only way to generate valid results for a contextual and location-based service is to essentially test it in the wild, in real contexts and in real locations [KKC*05, KSA*09, RCT*07, NOP*06, HRKS05].
Looking through a variety of methods and tools, we couldn‘t find a good fit for our requirements. So we turned to an approach based on the Wizard of Oz (WOZ) prototyping technique. „The WOZ paradigm utilizes a wizard (experimenter), behaving as an intelligent computer application, that interacts with a participant […]“ [LCW*04]. Originating in the field of natural language processing applications, the WOZ technique is an easy way to simulate an otherwise complex and often not yet existing technology [GCH*83]. Our obstacle of an intelligent contextual and location based service algorithm was unable to build within our time-frame, so we replaced the actual algorithm with our experimenters. Our process around the WOZ technique was split into three major steps.
2.1 In-depth interviews
To create a truly personalized service we needed profound knowledge about our test participants. Thus we started our process with in-depth interviews 1 This paper is focusing only on the framework used for testing the service. The service itself and insights gathered for Deutsche Telekom are confidential. with our recruited test participants to gain knowledge of their interests, likes, dislikes, lifestyle, behaviors, etc. With this background we were able to create very detailed profiles of each user, which would help us to aggregate and distribute the best matching recommendations and information in the working prototype.
2.2 Tracking and Feedback
The next phase consisted of two weeks of tracking and feedback. The participants were tracked via GPS-enabled apps on their smartphones, allowing us to log the geographical parameters of their daily routines. This was combined with a feedback-application, which allowed the participants to give us audio, video, image and text feedback about their actions, as well as their needs and wishes at any time. The method of receiving feedback derived from other probe-methods [GR08, GDP99], which provide rich feedback by creating easy-to-use feedback channels. The collected information about the participants‘ everyday lives was then combined with the results from the previous phase, creating not only insights into their favorite locations and everyday places, but helping us to chart their daily routines and behaviors, and to prepare location-based information and personalized recommendations.
2.3 Wizard of Oz
In the third and final phase we built a mobile application to simulate a real contextual and location-based service. The application was developed using a cross-device-framework so that it ran on both iOS and Android devices. Three important features were the ability to load the application onto a participant‘s device without going through an app store, which saved time and allowed us to use experimental features. A framework for delivering push notifications to our participants from an app that cannot use the apple standardized push notifications and a framework for allowing our participants to send us video, audio, photo and text feedback.
To replace the non-existent algorithm for providing context- and locationbased information, we used a team of information-gathering-specialists. The team members used the insights from the in-depth interviews and the data from the feedback- and tracking-phase to gather useful information and push it at the right time to the specific user.
The participants were able to rate this pushed information and give direct feedback about it, and the team members used this feedback to further finetune the service for each participant.
2.4 Simplified example use case
In the in-depth interviews the experimenter learned that the participant is vegetarian and has a medium income. In the tracking phase the experimenter got to know where the participant usually spent his lunch-times and what his favourite restaurants were. Based on those insights the experimenter could collect similar restaurants in the area that might fit the participants requirements. The recommendation would then be pushed to the participant‘s smartphone the next time the participant is in the area and it is time for lunch. After receiving the information the user would be able to give feedback on the recommendation and thus help the experimenter to refine future recommendations even more.
3. Experimental Setup
The prototype for our qualitative study was tested with a group of 12 participants, equally split between men and women, covering ages from 25 to 49 years. Half of the participants experienced the service as described above, the other half was used as a control group, receiving no personalized but rather random push notifications and random recommendations, the received random information was only location- or time-based but not specifically customized for each participant. All participants participated in all three phases. The prototype was evaluated through the feedback delivered by the participants during the two last phases, the statistics generated by the mobile application itself, and an intensive post-workshop with six of the participants, conducted after completing the experiment.
4. Related Works
Many papers show the advantage of testing mobile applications in real contexts [KKC*05, KSA*09, RCT*07, NOP*06, HRKS05]. Marco de Sá and Luís Carriço have even been working with a similar approach in their In-Situ Prototyping Tool [SC09]. Our method differentiates itself through the technique of splitting the app up into a native frame and a web app and by that the ability to modify the app on the fly in the background. Although the aim of our experiment, testing a location-based and contextual service and finally through our overall 3-phase experimental setup.
We were able to produce unusually precise insights for our client regarding their service and business models, enabling them to confidently proceed with the further development of the service. The quality of the insights delivered to the client was partly due to the detailed feedback and tracking data along with the statistics gathered throughout the process, but was also due to the realistic set-up of the service.
It even turned out that the „Wizard of Oz“ technique was sometimes too precise. The participants who received the „Wizard of Oz“ information perceived the service as backed by a powerful algorithm, not realizing that the data was actually produced by our team. When the information was too perfect, participants got suspicious. This response seemed to be similar to the „uncanny valley effect“ [Mor70] in which a machine is perceived as behaving eerily like a human. On the other hand this might also imply that personalization can indeed provide decent benefits but, besides all privacy questions, needs to be adjusted finely, because if the matching algorithm meets personal desires too precisely people feel insecure and spied upon. Clearly this finding needs more research.
In regard to our iterative rapid prototyping approach, the choice to use a mobile web app framework for creating the application and the ability to change the application without the participants able to notice, enabled us to react in real-time to user feedback to change the application or fix bugs. The overall approach also proved to be cost-effective, meeting or exceeding needs throughout the development and testing process.
But of course the method had its downsides and difficulties. Such a test can only be conducted for a small group of participants, as you need a lot of active “Wizards” to generate the content. Even though we found a good solution for loading the application onto our participants‘ devices, it wasn‘t completely perfect, especially in cases of inexperienced iOS participants we had to help install the application on their devices.
We learned from the participants‘ feedback that speed was crucial to the overall perception of the app and the information within. This turned out to be problematic in cases of slow wireless networks or old smartphones. Similar findings regarding the acceptance of mobile applications can be found in corresponding literature [MK09, VRP03].
We had to work through a number of privacy issues, and there was a deep question about the balance between clarity and transparency on the one hand, for example making sure users understood and accepted that they would be geographically tracked, yet on the other hand not revealing our full experimental set-up using Wizard of Oz instead of a real algorithm.
Overall we would highly recommend our user-centered approach in similar cases. Wherever a not yet existing complex mobile technology is setting the stage for an open question. Asking for a wide range of feedback. The „Wizard of Oz“ technique applied like in our setup can be very beneficial in the initial project phase for all follow-up processes of a project. Allowing more focused development in the subsequent production-steps.
[GCH83] John D. Gould, John Conti, and Todd Hovanyecz. 1983. Composing letters with a simulated listening typewriter. Commun. ACM 26, 4 (April 1983), 295-308.
[GDP99] Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: Cultural probes. interactions 6, 1 (January 1999), 21-29.
[GR08] Connor Graham and Mark Rouncefield. 2008. Probes and participation. In Proceedings of the Tenth Anniversary Conference on Participatory Design 2008 (PDC ‚08). Indiana University, Indianapolis, IN, USA, 194-197.
[HRKS05] Penny Hagen, Toni Robertson, Melanie Kan, and Kirsten Sadler. 2005. Emerging research methods for understanding mobile technology use. In Proceedings of the 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future (OZCHI ‚05). Computer-Human Interaction Special Interest Group (CHISIG) of Australia, Narrabundah, Australia, Australia, 1-10.
[KKC*05] Anne Kaikkonen, Aki Kekäläinen, Mihael Cankar, Titti Kallio, and Anu Kankainen. 2005. Usability testing of mobile applications: a comparison between laboratory and field testing. Journal of Usability Studies, volume 1, number 1, pages 4-16.
[KSA*09] Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, Rune Thaarup Høegh: Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Mobile HCI 2004: 61-73
[LCW*04] Lathrop, B., Cheng, H., Weng, F., Mishra, R., Chen, J., Bratt, H., Cavedon, L., Bergmann, C., Hand-Bender, T., Pon-Barry, H., Bei, B., Raya, M., Shriberg, L., 2004. A Wizard of Oz framework for collecting spoken human-computer dialogs: an experiment procedure for the design and testing of natural language in-vehicle technology systems. In: Proc. 12th Internat. Congress on Intelligent Transportation Systems, San Francisco CA, USA.
[MK09] Arvind Malhotra and Claudia Kubowicz Malhotra. 2009. A relevancy- based services view for driving adoption of wireless web services in the U.S.. Commun. ACM 52, 7 (July 2009), 130-134.
[ML09] Mohamed F. Mokbel and Justin J. Levandoski. 2009. Toward context and preference-aware location-based services. In Proceedings of the Eighth ACM International Workshop on Data Engineering for Wireless and Mobile Access (MobiDE ‚09). ACM, New York, NY, USA, 25-32.
[Mor70] The Uncanny Valley, Masahiro Mori, 1970, Energy, 7(4), pp. 33-35. Translated by Karl F. MacDorman and Takashi Minato.
[NOP*06] Nielsen, C. M., Overgaard, M., Pedersen, M. B. , Stage, J., & Stenild, S. (2006). It’s Worth the Hassle!: The Added Value of Evaluating the Usability of Mobile Systems in the Field. In: Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. (pp. 272-280). Association for Computing Machinery. (ACM International Conference Proceeding Series; 189).
[RCT*07] Yvonne Rogers, Kay Connelly, Lenore Tedesco, William Hazlewood, Andrew Kurtz, Robert E. Hall, Josh Hursey, and Tammy Toscos. 2007. Why it‘s worth the hassle: the value of in-situ studies when designing Ubicomp. In Proceedings of the 9th international conference on Ubiquitous computing (UbiComp ‚07), John Krumm, Gregory D. Abowd, Aruna Seneviratne, and Thomas Strang (Eds.). Springer-Verlag, Berlin, Heidelberg, 336-353.
[SC09] Marco de Sá and Luís Carriço. 2009. A mobile tool for in-situ prototyping. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI ‚09). ACM, New York, NY, USA, , Article 20 , 4 pages.
[VRP03] Viswanath Venkatesh, V. Ramesh, and Anne P. Massey. 2003. Understanding usability in mobile commerce. Commun. ACM 46, 12 (December 2003), 53-56.