Software crowdsourcing (SW CS) is an evolving software development paradigm, in which crowds of people are asked to solve various problems through an open call (with the encouragement of prizes for the top solutions). Because of its dynamic nature, SW CS has been progressively accepted and adopted in the software industry. However, issues pertinent to the understanding of requirements among crowds of people and requirements engineers are yet to be clarified and explained. If the requirements are not clear to the development team, it has a significant effect on the quality of the software product. This study aims to identify the potential challenges faced by requirements engineers when conducting the SW–CS based requirements engineering (RE) process. Moreover, solutions to overcome these challenges are also identified. Qualitative data analysis is performed on the interview data collected from software industry professionals. Consequently, 20 SW–CS based RE challenges and their subsequent proposed solutions are devised, which are further grouped under seven categories. This study is beneficial for academicians, researchers and practitioners by providing detailed SW–CS based RE challenges and subsequent solutions that could eventually guide them to understand and effectively implement RE in SW CS.
The crowdsourcing (CS) concept has arisen from new-fangled collaboration technologies such as social media and Web 2.0. The term CS was invented by Howe and Robison [
SW CS is the assignation of a worldwide pool of online developers who can be appointed on-demand to contribute to several types of software development tasks [
Traditional approaches to conducting the RE process typically involve an inadequate number of representatives in requirement-gathering sessions through interviews or focus groups [
CS-based RE is an umbrella term used for computerised or semi-computerised approaches to elicit and analyse information from a crowd to stem authenticated user requirements [
Although CS has noteworthy possibilities for RE, there remains scarce knowledge regarding how to set-up a CS-based project to fit the RE task [
It is essential to closely observe the conventional requirements gathering process to report the attributes of a software RE process conducted based on crowdsourcings. The RE process execution involves gathering, analysing, specifying and validating the requirements from stakeholders. In RE, multiple team members repeatedly converse with stakeholders in regard to requirements and continue to do so until a mutual agreement is reached on gathered requirements [
In CS-based RE, it is unlikely that requirements for developing a new system or version of an existing system are based on the crowd’s perspective in an open call. A study conducted by Howe [
Several studies attempted to use the abilities of the crowd and end-users to resolve RE problems. It was found that these studies primarily focused on research areas such as requirements-driven social adaptation [
Similarly, research on feedback-based RE has been conducted. Martens et al. [
Further, previous literature has specified appropriate requirements identification as another issue. In various software paradigms, users are usually diverse and unpredictable. It becomes costly to rely on an elite set of users for the functionality and quality attributes of the software. CS could assist in tackling the crowd’s capability to comprehend their requirements in a more desirable way. CrowdREquire [
Qualitative research was conducted in this study because it is a type of scientific research that answers questions, gathers evidence, yields findings that were not found in advance and generates findings that are relevant outside the study’s immediate limitations. Qualitative research is especially effective in gaining culturally detailed data about the values, opinions, behaviours and social contexts of specific populations [
In-depth interviewing is a qualitative research technique that encompasses thorough individual interviews with a fewer number of respondents to discover perspectives of a particular idea or situation. In-depth interviews are valuable when comprehensive information is required about a person’s opinion or behaviour or for discovering and discussing novel issues in depth [
Company names | Requirementanalysts | Softwaredevelopers | Businessanalyst | Requirementsengineers |
---|---|---|---|---|
Constant Variables | 1 | 2 | 3 | 1 |
Average year of experience | 7 | 6 | 8 | 4 |
Smart IS | 1 | 3 | 3 | 2 |
Average year of experience | 6 | 9 | 6 | 4 |
Adlabs | 1 | 3 | 3 | 1 |
Average year of experience | 4 | 7 | 5 | 5 |
One Bite | 2 | 2 | 3 | 2 |
Average year of experience | 4 | 8 | 4 | 6 |
Ninesol Technologies | 1 | 3 | 3 | 2 |
Average year of experience | 7 | 5 | 5 | 5 |
Fortlogics | 2 | 2 | 3 | 1 |
Average year of experience | 4 | 5 | 6 | 4 |
The in-depth interview plan comprised the following primary tasks. First, stakeholders were identified. The survey invitation was delivered to various software development companies to identify the most relevant stakeholders. The Pakistan software engineering board database was used to select the companies. A total of 12 companies were contacted and six responded to the invitation. These companies exist in all major Pakistani cities; however, the head offices are situated in Islamabad, Lahore and Rawalpindi. The selected software development companies that participated in this study are displayed in
In total, 73 respondents from the selected companies were willing to participate in the survey. Following this, an appointment was scheduled with the willing respondents to conduct the interview. However, only 50 respondents were interviewed—23 respondents could not give their responses because of their work commitments. In this research, the participating stakeholders (respondents) were requirements engineers (also called business analysts) who worked in crowdsourcing platforms. Once the stakeholders were listed, the interview questions were established. For this research, stakeholders were asked about the challenges they face when conducting CS-based RE and the strategies used to overcome those challenges.
This stage involved developing the interview instrument. Initially, an interview protocol was developed. This protocol contained the guidelines that were followed for each interview to certify uniformity between interviews and thus increase the trustworthiness of the findings. Moreover, an interview guide was generated, which contained the interview questions. The instrument was validated for face validity and content validity through two methods: Average congruency percentage (ACP) and content validity index (CVI). The instrument was validated through four experts who had educational, research and industry background in CS-based RE. All experts were specialised in crowdsourcing and had more than 10 years of industry and academic experience.
In ACP, experts computed the percentage of questions deemed to be relevant. Conversely, CVI calculated the content validity index for the individual item (I-CVI). Experts 1 and 3 found one out of 12 questions irrelevant, which resulted in a 91.66% relevancy rate at their level. However, Experts 2 and 4 rated all questions relevant, which resulted in a 100% relevancy rate at their level. The average value of the experts’ congruency percentage was 95.5%, which was considered valid. For CVI, the four experts were asked to evaluate each question’s content relevancy on a 4-point Likert scale, in which 1 = not relevant, 2 = somewhat relevant, 3 = relevant and 4 = very relevant. To decide the relevancy criteria for each question, the number of experts who gave three or four scores was counted as relevant and one or two scores were considered not relevant. It was found that the majority of questions (Q2, Q5, Q7–Q12) scored an I-CVI value of 1.00 (Q1, Q3–Q4 and Q6 scored an I-CVI value of 0.75). The resultant mean I-CVI value was 0.9, which was considered valid. Moreover, the proportion relevancy of questions was also calculated. It was found that all the experts rated the questions with high proportion relevancy, which resulted in a mean expert proportion of 0.87 (considered high). Based on the results, the face and content validity of the questions asked in the interview session were found to be significantly high, which ensured the quality of the instrument.
At this stage of the in-depth interviewing technique, interviewers were identified and trained. Two final-year research students of a bachelor’s in software engineering programs were trained and involved in the data collection process. Irrespective of the data collectors’ prior experience, comprehensive training was provided over two weeks. The training process comprised an introduction to the evaluation objectives, an analysis of data collection techniques, a detailed analysis of the data collection items and instruments, preparation in the instrument usage, skill-building drills on interpersonal communication and conversation of ethical issues.
Interviews were conducted with the 50 stakeholders (respondents). The interviews were conducted in groups with respondents from each participating company. First, interviewees’ consent was taken and re-explained according to the interview’s purpose, interviewees were notified as to why they were selected, and the interview session’s expected duration was indicated. Once the interview was conducted, the interview minutes were summarised in the form of an interview report—a process that was completed for each interview within 2–3 h after the interview session.
The gathered data were transcribed and analysed through grounded theory [
The data were collected through interviews before being transcribed. Transcription is significant for qualitative research because it assists the researcher in analysing data easily and creating a narrative with the data. The transcription allowed the researchers to find data patterns and save the accuracy of the data. Initial reads were given to the data to achieve understanding regarding data. Once the researchers understood the data, they identified critical segments and assigned codes. Every response was allocated with a letter code, which was next entered into the software. Afterwards, the similar codes were grouped and redundancies were removed. Consequently, themes were generated and categorised according to their similarities. The researchers observed the created codes and identified various patterns, which resulted in themes. These themes are wider than codes. Researchers combined various codes into a unique theme. In this study, the codes represented the challenges and their solutions and the categories represented the various RE phases.
In this section, the detailed result of the identified challenges and suggestions for SW CS are discussed. Moreover, the categories based on similarities and mapping of these challenges to RE phases are explained.
In classical software development, RE is critical; rather, it becomes more decisive in SW–CS based RE. To respond to the complexity of RE in SW CS, this research identified 20 challenges that requirements engineers face when conducting various phases of the RE process.
As described in
RE phase | Challenges | Challenge category | Suggested solution |
---|---|---|---|
Requirement elicitation | Time-consuming to gather the requirements [c1] | Time | Time should be managed by proper planning of the requirements elicitation phase. |
Hard to obtain quality work [c2] | Quality work | Requirements engineers must define protocols to obtain quality work. | |
Require more experts and resources [c3] | Experts required | More experts should be hired who are capable of handling requirements elicitation. | |
Lack of confidentiality and communication [c4] | Privacy communication issues | Disclosure information should be enhanced to support privacy. Communication tools should be used for frequent communication. | |
Requirement analysis and design | Difficulty in the effective interpretation of user requirements [c5] | Requirements understanding because of a lack of crowdsourcing knowledge | Requirements engineers should consider the users’ perspective to understand the requirements more effectively. |
Lack of conformity between the idea and solution [c6] | Conformity | The idea should have a context or purpose to conduct requirements analysis. | |
Lack of face-to-face communication, which results in changing requirements [c7] | Lack of face-to-face communication | Use online tools to conduct online interviews and gather complete requirements at the start of the project, which would avoid changes in the future. | |
Difficulty in user requirements fulfilment [c8] | Quality work | More than one RE technique should be used to have a detailed understanding of requirements for their complete fulfilment. | |
Difficulty in designing requirements because people give requirements according to their culture and needs [c9] | Cultural issues | Filter the requirements according to users’ needs to gain knowledge of users’ cultures and moral values. | |
Designed requirements conflict with user perspectives [c10] | Culture | Design requirements from the perspective of a user. | |
Difficult to prepare, initialise, decompose and aggregate requirements [c11] | Requirements understanding because of lacking crowdsourcing knowledge | Platforms such as Crowd REquire should be used to conduct the RE process. | |
Requirements engineers do not have the complete knowledge of the requirements in a crowdsourced environment, which leads to difficulty in requirements design [c12] | Requirement understanding because of lacking crowdsourcing knowledge | Platforms such as Crowd REquire should be used to conduct the RE process. | |
Requirement specifications | Difficult to manage and decide in finalising the requirements of numerous people [c13] | Requirement understanding because of a lack of CS knowledge | Enhance communication with clients and discuss their ideas with deeper attention. The communication mediums play a vital role in effectively discussing the problems and their solutions. The selection of the appropriate communication medium is vital to finalise the requirements among the stakeholders and discuss the shortcomings in the gathered requirements. |
Designing Software Requirement Specification (SRS) is complex because stakeholders are not on common ground [c14] | Complexity | Requirements engineers and customers must be on common ground. | |
Requires more experts [c15] | Experts required | Experts who are fit for projects and can deliver in time should be hired. | |
Lack of face-to-face communication between requirements engineers and clients [c16] | Confidence and confidentiality | Enhance communication with clients and discuss their ideas with deeper attention. | |
Difficult to maintain a proper document presented according to the customer’s needs [c20] | Quality work | Requirements engineers should check the quality of the drafts alongside each other by specifying them. | |
Requirement validation | Lack of quality assurance [c17] | Quality work | Targets should be achieved carefully and under serious observation. |
Difficult to obtain requirements for feedback from numerous people, which invalidates the requirements [c18] | Conformity | Requirements engineers should take precautions on earlier stages. A selected sample of people should perform validation. | |
Time-consuming to manage or validate the gathered requirements from the stakeholder [c19] | Time | Requirements engineers should take precautions on the earlier stages so that there is no need to do so afterwards. |
It was reported that eight challenges were most faced by requirements engineers when they performed requirement analysis and design: difficulty in the effective interpretation of user requirements [c5] lack of conformity between the idea and solution [c6] lack of face-to-face communication, which results in changing requirements [c7] difficulty in user requirements fulfilment [c8] difficulty in designing requirements because people give requirements based on their culture and needs [c9] designed requirements conflict with user perspectives [c10] difficult to prepare, initialise, decompose and aggregate requirements [c11] requirements engineers do not have the complete knowledge of the requirements in a CS environment, which leads to difficulty in requirements design [c12].
Subsequently, the respondents provided solutions to overcome these challenges. The respondents noted that, when they performed requirements specification activities, they predominantly encountered five challenges: difficult to manage and decide in finalising the requirements of numerous people [c13] designing SRS is complex because stakeholders are not on a common ground [c14] requires more experts [c15] lack of face-to-face communication between requirements engineers and clients [c16] difficult to maintain a proper document according to the customer’s needs [c20].
Solutions to overcome these challenges are also described in detail. Last, requirement validation included three common challenges: lack of quality assurance [c17] difficult to obtain requirements for feedback from numerous people, which invalidates the requirements [c18] time-consuming to manage or validate the gathered requirements from the stakeholder [c19].
The research results (challenges) were analysed thoroughly and grouped into various categories based on their similarity. In particular, there were 20 challenges, in which each challenge pertinent to a specific aspect was grouped under a category.
As demonstrated in ‘Time-consuming to gather the requirements [c1]’, and ‘time-consuming to manage or validate the gathered requirements from the stakeholder [c19]’ are challenges related to ‘time’. ‘Hard to obtain quality work [c2]’, ‘difficulty in user requirements fulfilment [c8]’, ‘lack of quality assurance [c17]’ and ‘difficult to maintain a proper document presented according to the customer’s needs [c20]’, are challenges related to ‘quality work’. ‘Difficulty in designing requirements because people give requirements according to their culture and needs [c9]’ and ‘designed requirements conflict with user perspectives [c10]’ are challenges related to ‘culture’. ‘Require more experts and resources [c3]’ and ‘requires more experts [c15]’, are categorised under ‘require more experts’ because both relate to the aspect of lacking experts’ comprehensiveness. ‘Lack of face-to-face communication, which results in changing requirements [c7]’, ‘designing SRS is complex as stakeholders are not on a common ground [c14]’, ‘lack of face-to-face communication between requirements engineers and clients [c16]’ and ‘lack of confidentiality and communication [c4]’ are challenges related to ‘communication and confidentiality’. ‘Lack of conformity between the idea and solution [c6]’ and ‘difficult to obtain requirements for feedback from numerous people, which invalidates the requirements [c18]’ are categorised under ‘conformity’ because both relate to the aspect of lacking conformity between requirements. ‘Difficulty in the effective interpretation of user requirements [c5]’, ‘difficult to prepare, initialise, decompose and aggregate requirements [c11]’, ‘requirements engineers do not have the complete knowledge of the requirements in a CS environment, which leads to difficulty in requirements design [c12]’ and ‘difficult to manage and decide in finalising the requirements of numerous people [c13]’ are categorised under ‘requirement understanding based on a lack of CS knowledge’ because the challenges occur from a lack of CS knowledge.
The research findings also generated mapping patterns between the RE process and challenge categories.
As demonstrated in
RE activities | Time | Quality work | Culture | More experts required | Confidence and confidentiality | Conformity | Requirement understanding based on a lack of CS knowledge |
---|---|---|---|---|---|---|---|
Requirements elicitation | X | X | X | X | |||
Requirements analysis and design | X | X | X | X | X | ||
Requirements specification | X | X | X | X | |||
Requirements validation | X | X | X |
According to the respondents, the challenges regarding ‘conformity’ occurred while the RE team analyses, designs and validates the gathered requirements. Further, the challenges regarding ‘requirement understanding based on a lack of CS knowledge’ were faced by most requirements engineers when they performed requirement analysis, design and specification. To increase an in-depth understanding and visualisation of the findings,
The focus group was conducted to validate the research findings (SW–CS based RE challenges and solutions). Jay’s [
The detailed focus group process consisted of six steps. First, it was necessary to define the focus group’s objective and validate the outcome of this research by comprising CS-based RE challenges and their solutions. The second step involved establishing a timeline. Planning the focus group session began six weeks before the session occurred. Eight senior requirements engineers were contacted, each with at least 10 years of experience, and six consented. Next, formal invitations were delivered. These six experts were serving in the industry as ‘principal software engineer’, ‘software project manager’, ‘senior software developer’ and ‘senior team leads’. It was ensured that all experts had been working in a CS environment. The interview questions were prepared thoroughly (see
Sr. No. | Questions |
---|---|
1 | What do you think about CS-based RE? |
2 | Do you think that the proposed study is comprehensive in terms of CS-based RE challenges and their solutions? (How?) |
3 | What are the strengths and weaknesses of the research outcome (CS-based RE challenges and their solutions)? |
4 | Do you see any problem with the recommended solution comprehension? |
5 | In your opinion, what are potential barriers of adopting these suggested solutions? |
6 | What aspects of the suggested solutions can be improved and how? |
As demonstrated in
The session began with a general open-ended question—‘what do you think about SW–CS based RE?’. It was ensured that all opinions regarding various questions were conveyed and considered. The participants’ comments or feedback regarding the identified CS-based RE challenges and their solutions were recorded. The participants suggested some modifications in regard to the challenges and solutions; consequently, a modified list of CS-based RE challenges and solutions was generated.
All the participants agreed on the study’s comprehensiveness. It was noted that the study could act as a guide for academicians and practitioners working in the domain of CS-based RE. According to Expert 6, categorising challenges and mapping with RE phases would further the reader’s understanding. Further, Expert 4 noted that the detailed solutions against the challenges enhance the study’s comprehensiveness.
According to the participants, the study findings were easy to understand and grasp. The experts reported the authenticity of the identified challenges by mentioning the comments of other session participants. There was a consensus regarding the practicality of the identified challenges and their solutions in a real environment. The experts suggested some modifications to the findings to enhance their understandability (see
Interestingly, the participants described various organisational and personal barriers in regard to adopting the study findings. The experts mentioned the problem of mindset in any organisation—organisations could be reluctant to consider these findings when working in CS-based RE. Further, in regard to personal barriers, there could be a lack of understanding or passion in understanding the challenges and their solutions for effective CS-based RE.
In addition, the organisation’s employees may possess a reluctant attitude towards adopting novel things. All these barriers have significance in their context; however, it was concluded that these barriers could be easily managed if careful attention is applied.
There are often multiple validity threats in survey-based studies. One of the difficulties with the survey is that it occasionally had a low response rate and the likelihood of individual biases. Previous literature has reported that thoughts and views obtained through a survey could be prejudiced and dissimilar from the real-world population distribution. Another study limitation regards the instrument (questionnaire) that covered most aspects related to the potential challenges and solutions. The instrument was statistically analysed in terms of face and content validity tests and the results demonstrated the instrument’s authenticity and comprehensiveness. However, it is possible that various significant questions that could have been further explored were overlooked.
The authors gathered qualitative data by conducting interviews with 50 respondents from the software industry who worked in a CS platform. In particular, the interviewees were all requirements engineers. Although their responses were quite comprehensive, the sample size is limited in terms of generalising the results. This limitation was attempted to be overcome through conducting a focus group, which would ideally validate and generalise the list of challenges and solutions. Last, the findings of this research is based on a human–understanding based data extraction strategy, which may have affected the outcome of this research. To address this limitation, experts were involved from both academia and the industry to ensure the contextual insights of the results.
This research study has identified 20 SW–CS based RE challenges and their subsequent proposed solutions through an industrial survey. These challenges were further grouped into seven categories. The authors validated the findings by including eight experts in a focus group session. A total of 12 companies were contacted and six were willing to respond to the invitation. In total, 73 relevant respondents from the selected companies were willing to participate in the survey; however, only 50 respondents were interviewed because of attrition. SW CS exhibited a basic pattern swing in regard to how the software would be established in the future. Consequently, this has lifted numerous issues and RE has become one of the most pressing issues. In this regard, if RE is not properly handled in SW CS, the quality of the software work product will be greatly affected. In response, this study has involved a qualitative data analysis through conducting interviews with software industry professionals. Consequently, 20 SW–CS based RE challenges and their proposed solutions were devised. The factors were categorised under seven categories—time, quality work, culture, more experts required, confidence and confidentiality, conformity and requirement understanding based on lacking CS knowledge. Next, the research findings were validated through a focus group, in which experts evaluated the comprehensiveness of the findings, the strengths and weaknesses of the results and potential barriers against adopting the findings. It is believed that SW CS platforms can be assisted by this research in terms of obtaining a better understanding and gathering of software requirements. Researchers, academicians and practitioners can benefit from this research by utilising the research outcome. Moreover, future research could engage in the following activities: using empirical evidence to evaluate the proposed mapping between CS-based RE challenges and strategies to solve them using a wider audience to validate whether lists of challenges and solutions are exhaustive.
We deeply acknowledge Taif University for Supporting this study through Taif University Researchers Supporting Project number (TURSP-2020/115), Taif University, Taif, Saudi Arabia.