PRECOG: a Descriptive Cognitive Model for predicting usability issues in a Low Code Development Platform
Low-code development platforms (LCDPs) address the need for increased productivity in software development. There is still, however, a considerable number of relevant gaps in current knowledge about how people reason during programming and development tasks, as well as a need for new tools to fill these gaps.
Low-code development platforms (LCDPs) address the need for increased productivity in software development. By raising the abstraction level at which software is developed, LCDPs automate low-level and routine development tasks, effectively contributing to solve the problem of global shortage of professional software developers. According to MarketsandMarkets (2020), the Global LCDPs market is projected to grow from €11,2 billion in 2020 to €38,8 billion by 2025. At the same time, they lower the entry barrier to software development. As these low-level tasks become automated, developers are not required to carry them out (or even to know how to carry them out). Low-level technical details are effectively hidden by the platform.
There is still, however, a considerable number of relevant gaps in current knowledge about how people reason during programming and development tasks (Sajaniemi, 2008). According to Myers, Pane, and Ko (2004), conventional programming languages require programmers to make “tremendous transformations” from what they intend to accomplish, to what they should code. Visual modelling languages, typically adopted by low-code development platforms, aim to mitigate this problem, but their actual effectiveness is still subject to debate.
The challenges of Low-code Development Platforms
In any case, the users’ prior knowledge plays a relevant role in the learning and use of a platform (Dijkstra, 1982). In the case of LCDPs, there is the double challenge of supporting users with little or no knowledge of programming, while also supporting expert programmers. Indeed, understanding individual differences and expectations, and identifying the sources of variation among different users will help this type of platform to be more broadly adopted (Blackwell, 2017). Since low-code development platforms aim at reducing the learning burden while providing powerful tools to address a wide range of application domains, a trade-off must be established between the scope of application and the learning costs of the platforms and their languages. This necessarily implies building an understanding of how different types of users approach the platforms.
The challenge then is how to reduce the learning effort of users without reducing the scope of the possible application domains.
PRECOG: a model to predict usability issues in LCDPs
As a contribution to this long-term goal, we have been looking at the difficulties faced by potential programmers with different expectations and academic backgrounds when using LCDPs. To achieve this, we developed PRECOG (see Figure 1), a descriptive cognitive model designed to help predict usability issues in LCDPs.
PRECOG starts with an Hierarchical Task Analysis (HTA) of the particular development use case under analysis. The purpose of the HTA is to decompose the development task into all its sub-tasks (see Dix et al., 2004). A tailored version of an External-Internal Task Mapping Analysis (ETIT — Moran, 1983) is then applied to analyse relevant sub-tasks output by the HTA. To this end, sub-tasks are described both from the perspective of naïve users, using their own terms and concepts (the External task space, henceforth referred to as the Knowledge-Based Description — KBD), and from the perspective of what the LCDP entails to accomplish a given sub-task (the internal task space, henceforth called the System-Based Description — SBD). The KBD relies on user’s data prior to any interaction with the system. With only a brief description of each task, participants describe how they would reach each task’s final goal using their current knowledge about programming and familiar development tools. This information provides the analysts with the participant’s mental model of the tasks in hand, before interaction with the system under evaluation. The SBD, on the other hand, is a step-by-step description of the actions performed by an expert user of the system under analysis.
Mapping rules between the naive user’s mental model (KBD) and the system-based description (SBD) support the identification of possible conflicts. Three types of conflicts can be uncovered by looking at the mapping between the two spaces:
● Under decomposition conflict — occurs when a procedure that is considered a single step in the KBD requires multiple steps in the SBD. This type of conflict might lead to an underestimation of the sub-task’s complexity.
● Over decomposition conflict — occurs when a procedure that is considered as having multiple steps in the KBD requires only one step in the SBD. This type of conflict might lead to an overestimation of the sub-task’s complexity and failure to identify and take shortcuts during the development task.
● No Correspondence conflict — occurs when there is no link between a step in the KBD and one (or several) steps in the SBD. This might occur because the user is not aware of the appropriate steps required by the LCDP or because the system does not include a feature representative of a mental step the user thinks is needed.
Once one of these types of conflicts is found, a risk analysis of the conflict is performed, which can help define priorities in the continuous improvement of the system.
The risk analysis of a conflict includes:
● Frequency — An ordinal scale from 0 (never) to 5 (frequent)
● Criticality — All issues are evaluated regarding the level of criticality as described in Figure 2, ranging from 2 (not a problem) to 10 (serious).
● Pure Risk — The Pure Risk of an error is a single value representing the weight that should be attributed to that error (see Frequency / Criticality matrix in Figure 3). The higher the value, the more critical should the error be considered.
Application of the PRECOG
We have successfully applied PRECOG to a relevant use case, which allowed us to validate the viability of the proposed approach, as a way of uncovering mismatches between user’s mental models of a development task and the LCDPs’ support to solve the task. Twenty programmers (10 experts, 10 novices) participated in the analysis of a market leading LCDP. The application of the PRECOG methodology was followed by user tests for direct observation of programmers interaction with the platform.
The methodology proved to correctly identify relevant issues from both user profiles, prior to the user studies that confirmed them.
● The patterns of issues observed were qualitatively different between groups.
● Experts mainly faced interaction related problems
● Novices faced problems attributable to a lack of knowledge on key programming concepts.
The use case helped us establish that PRECOG can also be used retrospectively to understand the root-cause of an issue identified through user testing.
Previous knowledge as predictor of interaction difficulties
Low-code development platforms have the potential to dramatically change how software is developed, making it possible, at least for particular domains, for someone without a formal education in computer science to develop quality software, and for experienced developers to significantly speed up the development process.
Understanding how programmers and non-programmers approach this type of platform is key to support their design and evolution. By developing and applying PRECOG, we were able to gain insights into potential hurdles different user profiles might feel when attempting, or transitioning to, low code development. The method is particularly beneficial when it is relevant to prospectively identify possible interaction problems, resulting from the users’ background knowledge being insufficient to guarantee a successful completion of the task at hand. Another strength of the PRECOG method is that it is not platform-dependent, being easily adaptable to different types of Information and Communication Platforms.
For more information please see (Silva et al., 2020) or contact us.
Short bio of authors
João Abril de Abreu is a Program Manager at OutSystems, in charge of R&D outreach and product innovation. João holds a PhD. degree in Computer Science from the University of Leicester (UK, 2010).
José Creissac Campos is a Senior researcher at HASLab/INESC TEC (@inesctec), and an Auxiliary Professor at the Department of Informatics of the University of Minho. He currently chairs the IFIP working group on User Interface Engineering (WG 2.7/13.4) and the Steering Committee of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS). José holds a DPhil. degree in Computer Science from the University of York (UK, 2001).
Rui Couto is a Postdoc researcher at HASLab/INESC TEC (@inesctec). He holds a PhD. degree in Informatics from the University of Minho (Portugal, 2017).
Carina Fonseca is a Product Designer at OutSystems. Carina holds a MSc. degree in Computer Science from Instituto Superior Tecnico (Portugal, 2017)
Antonio Nestor Ribeiro is an Auxiliary Professor at the Department of Informatics of the University of Minho and a Senior researcher at HASLab/INESC TEC. He holds a PhD. degree in Informatics from the University of Minho (Portugal, 2008).
Carlos Silva is the Head of the Perception, Interaction, and Usability group at CCG — Centro de Computação Gráfica. He holds an MSc in Experimental Psychology from the University of Minho (Portugal, 2011) and a PhD degree in Informatics from the University of Minho (Portugal, 2019).
Joana Vieira is a Senior UX Researcher at CCG — Centro de Computação Gráfica. She holds an MSc in Experimental Psychology from the University of Minho (Portugal, 2008) and is concluding a PhD in Ergonomics at the University of Lisbon.
References
Blackwell, A. F. (2017). End-user developers–what are they like? In New perspectives in end-user development (pp. 121–135). Springer International Publishing. https://doi.org/10.1007/978-3-319-60291-2_6
Dijkstra, E. W. (1982). How do we tell truths that might hurt? In Selected writings on computing: A personal perspective (pp. 129–131). Springer New York. https://doi.org/10.1007/978-1-4612-5695-3_22
Dix, A., Finlay, J., Abowd, G. D., & Beale, R. (2004). Human-Computer Interaction. Pearson Education.
Moran, T. P. (1983). Getting into a system: External-Internal task mapping analysis. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (December), 45–49. ACM. https://doi.org/10.1145/800045.801578
MarketsandMarkets (2020) Low-Code Development Platform Market by Component (Platform and Services), Application Type, Deployment Type (Cloud and On-Premises), Organization Size (SMEs and Large Enterprises), Industry, and Region — Global Forecast to 2025. MarketsandMarkets, Report number TC 5873, April 2020.
Myers, B. A., Pane, J. F., & Ko, A. (2004). Natural programming languages and environments. Communications of the ACM, 47, 47. https://doi.org/10.1145/1015864.1015888
Sajaniemi, J. (2008). Guest editor’s introduction: Psychology of programming: Looking into programmers heads. Human Technology: An Interdisciplinary Journal on Humans in ICT Environments, 4, 4–8. https://dx.doi.org/10.17011/ht/urn.200804151349
Silva, C., Vieira, J., Campos, J.C., Couto, R. and Ribeiro, A.N. (2020) Development and validation of a Descriptive Cognitive Model for a Low Code Development Platform. Human Factors. (online first) http://dx.doi.org/10.1177/0018720820920429
