|
Technical Reports
USC-CSSE-2009-516
Qi Li , "Using Additive Multiple-Objective Value Functions for Value-Based Software Testing Prioritization" (pdf)
Testing is one way of validating that the customers have specified the right product and verifying that the developers have built the product right. Traditional testing methodologies such as: path, branch, instruction, mutation, scenario, or requirement testing usually treat all aspects of software as equally important [1], and is often reduced to a purely technical issue leaving the close relationship between testing and business decisions unlinked and the potential value contribution of testing unexploited [2]. Value-based software testing serves to bridge the gap between the internal software testing process and business value that comes from customers and market [3]. The essential of Value-based software testing is feature prioritization that brings business importance that comes from CRACK key customers, software quality risk from developing team, software testing cost, process estimation and controlling from the testing team, and the external commercial factors, e.g., market pressure from the market into consideration. This decision making process could be seen as a multiple-objective decision making process that tries to maximize the business importance, identify the most risky features, minimize the testing cost and minimize the market erosion. The prioritization strategy could be reflected and influenced by different multiple-objective decision value functions. In [3], a multiplicative multiple-objective value function is used to generate the testing priority ranking. In this paper, we will use additive multiple-objective value function to generate testing priority ranking and compare the result in the end.
Added October 1st, 2009
USC-CSSE-2009-515
Vu Nguyen, Barry Boehm, Phongphan Danphitsanuphan, "Assessing and Estimating Corrective, Enhancive, and Reductive Maintenance Tasks: A Controlled Experiment" (pdf)
This paper describes a controlled experiment of student programmers performing maintenance tasks on a C++ program. The goal of the study is to assess the maintenance size, effort, and effort distribution of three different maintenance types and to describe estimation models to predict the programmer’s effort on maintenance tasks. The results of our study suggest that the corrective maintenance is much less productive than the enhancive and reductive maintenance. Our study also confirms the previous results which conclude that corrective and reductive maintenance requires large proportions of effort on the program comprehension activity. Moreover, the best effort model we obtained from fitting the experiment data can estimate the time of 79% of the maintainers with the error of 30% or less.
Added August 18th, 2009
USC-CSSE-2009-514
George Edwards, Nenad Medvidovic, "Model Interpreter Frameworks" (pdf)
One of the practical challenges in utilizing domain-specific modeling technologies is the construction of model interpreters that leverage domain-specific models for code generation and quality analysis. Model interpreter frameworks (MIFs) simplify and reduce model interpreter development effort by providing off-the-shelf analysis and synthesis capabilities, while still permitting domain-specific customization. The XTEAM model-driven design environment implements highly extensible MIFs that generate system simulations and source code from domain-specific models. Drawing on our experience building MIFs within XTEAM, as well as the related (though independent) experience of other research groups, we identify the shared, essential characteristics of MIFs.
Added July 15th, 2009
USC-CSSE-2009-513
Qi Li, "The Effort Distribution Pattern Analysis of Three Types of Software Quality Assurance Activities and Process Implication: an Empirical Study" (pdf)
Review, process audit, and testing are three main Quality Assurances activities during software development life cycle. They complement each other to examine work products for defects and improvement opportunities to the largest extent. Understanding the effort distribution and inter-correlation among them will facilitate software organization project planning, improve the software quality within the budget and schedule and make continuous process improvement. This paper reports some empirical findings of effort distribution pattern of the three types of QA from a serial of incremental projects in China. The result of the study gives us some implications on how to identify which type of QA activity is insufficient while others might be overdone, and how to balance the effort allocation and planning for future projects, improve the weak part of QA activities and finally improve the whole process effectiveness under the specific organization context.
Added July 5th, 2009
USC-CSSE-2009-512
Ali Afzal Malik, Barry Boehm, "Predicting Understandability of a Software Project: A Comparison of Two Surveys" (pdf)
This paper summarizes the results of an empirical study conducted to explore the relative importance of eight pertinent COCOMO II model drivers in predicting the understandability of a software project. Here understandability of the project is measured solely from the perspective of the development team. This empirical study employed a survey that was targeted towards experienced practitioners. Results from this survey are compared with the corresponding results from a prior similar survey conducted on graduate students. While this comparison reveals some interesting differences between the importance given to each of the eight model drivers by these two categories of individuals it also highlights some subtle similarities.
Added June 10th, 2009
USC-CSSE-2009-511
George Edwards, Nenad Medvidovic, "A Highly Extensible Simulation Framework for Domain-Specific Architectures" (pdf)
Discrete event simulation is a powerful and flexible mechanism for analyzing both the functional characteristics and quality properties of software design models. However, software engineers do not utilize simulations in many situations where it has significant potential benefits because (1) the programming model used by existing discrete event simulators is not well-suited for capturing domain-specific software designs, and (2) customizing and optimizing core simulation algorithms based on domain-specific requirements and constraints is difficult and, in some cases, not possible. In this paper, we describe a new approach to constructing discrete event simulations of software systems that removes both of these obstacles. First, our simulation approach supports a software architecture-based programming model and allows software engineers to define domain-specific modeling constructs. Second, our simulation approach allows domain-specific optimization of simulation engine functions through modification or replacement of core framework components. To demonstrate the implementation of our approach, we present XDEVS, is a highly extensible discrete event simulation framework that provides both of the above capabilities. We evaluated the utility and efficiency of XDEVS using a large-scale simulation of a volunteer computing system, and this evaluation confirmed that XDEVS represents an improvement over the current state-of-the-art.
Added May 13th, 2009
USC-CSSE-2009-510
Yuriy Brun, George Edwards, Jae young Bang, Nenad Medvidovic, "Online Reliability Improvement via Smart Redundancy in Systems with Faulty and Untrusted Participants" (pdf)
Many software systems today, such as computational grids, include faulty and untrusted components. As faults are inevitable, these systems utilize redundancy to achieve fault tolerance. In this paper, we present two new, "smart" redundancy techniques: iterative redundancy and progressive redundancy. The two techniques are efficient, adaptive, and automated. They are efficient in that they leverage runtime information to improve system reliability using fewer resources than existing methods. They are automated in that they inject redundancy in situations where it is most beneficial and eliminate it where it is unnecessary. Finally, they are adaptive in that they increase redundancy on-the-fly when component reliability drops and decrease redundancy when component reliability improves. We enumerate examples of systems that can benefit from our techniques but focus in this paper on computational grid systems. We present formal analytical and empirical analyses, demonstrating our techniques on a real-world computational grid and comparing them to existing methods.
Added May 13th, 2009
USC-CSSE-2009-509
Ye Yang, Qi Li, Mingshu Li, Qing Wang, "An Empirical Analysis on Distribution Patterns of Software Maintenance Effort," 24th IEEE International Conference on Software Maintenance (ICSM 2008), Beijing, China, September 28 - October 4, 2008 (pdf)
Distribution of effort in software engineering process has been the basis for facilitating more reasonable software project planning. This paper reports empirical results on activity effort distribution patterns of a series of industrial software maintenance projects. The results show that with respect to different influencing factors, the projects demonstrate large variations in their activity effort distribution, which necessitates appropriate adjustments to strategic planning.
Added April 20th, 2009
USC-CSSE-2009-508
Qi Li, Qing Wang, Ye Yang, Mingshu Li, "Reducing Biases in Individual Software Effort Estimations: A Combining Approach," Empirical Software Engineering and Measurement (ESEM 2008), Kaiserslautern, Germany, October 9-10, 2008 (pdf)
Software effort estimation techniques abound, each with its own set of advantages and disadvantages, and no one proves to be the single best answer. Combining estimating is an appealing approach. Avoiding the difficult problem of choosing the single “best” technique, it solves the problem by asking which techniques would help to improve accuracy, assuming that each has something to contribute. In this paper, we firstly introduce the systematic “external” combining idea into the field of software effort estimation, and estimate software effort using Optimal Linear Combining (OLC) method with an experimental study based on a real-life data set. The result indicates that combining different techniques can significantly improve the accuracy and consistency of software effort estimation by making full use of information provided by all components, even the much “worse” one.
Added April 20th, 2009
USC-CSSE-2009-507
Qi Li, Mingshu Li, Ye Yang, Qing Wang, Thomas Tan, Barry Boehm, Chenyong Hu, "Bridge the Gap between Software Test Process and Business Value: A Case Study," ICSP2009, Vancouver, Canada, May 16-17, 2009 (pdf)
For a software project to succeed, acceptable quality must be achieved within an acceptable cost, providing business value to the customers, and keeping delivery time short. Software testing is a strenuous and expensive process and is often not organized to maximize business value. In this article, we propose a practical value based software testing method which aligns the internal test process with the value objectives coming from the customers and the market. Our case study in a real-life business project shows that this method helps manage testing process effectively and efficiently.
Added April 20th, 2009
USC-CSSE-2009-506
Ali Afzal Malik, Barry Boehm, "An Empirical Analysis of Effort Distribution of Small Real-Client Projects" (pdf)
This paper presents an analysis of the weekly effort distribution of 23 small real-client projects. The procedure used in collecting the effort data is described and the major findings are summarized. The results indicate the crests and troughs of project effort in relation to the major project milestones. Possible reasons for the pecularities in the effort distribution are also discussed. Moreover, an attempt is made to analyze the impact of project type on the effort distribution by comparing the effort profiles of web-based projects with non-web-based projects.
Added April 15th, 2009
USC-CSSE-2009-505
Ali Afzal Malik, Barry Boehm, "Software Size Analysis of Small Real-Client Projects" (pdf)
This paper presents an analysis of the results of an empirical study conducted to examine the relationship between the estimated and actual sizes of small real-client projects. Estimates of software size are collected at two different points of the software development life cycle – one immediately before rebaselining the project mid-stream and the other immediately after rebaselining. It is found that estimates made after rebaselining are usually more accurate. The accuracy of estimates based on the category of projects is also analysed. The results indicate that the correlation between estimated and actual size is much stronger in case of web-based projects vis-à-vis non-web-based projects.
Added April 7th, 2009
USC-CSSE-2009-504
Supannika Koolmanojwong, Barry Boehm, "Using Software Project Courses to Integrate Education and Research: An Experience Report," 22nd IEEE-CS Conference on Software Engineering Education and Training (CSEET) 2009 (pdf)
At University of Southern California (USC), CSCI577ab is a graduate software engineering course that teaches best software engineering practices and allows students to apply the learned knowledge in developing real-client projects. The class is used as an experimental test-bed to deploy various research tools and approaches for validation of new methods and tools. Various research data have been collected as partial basis for twelve PhD dissertations. This paper reports how research and education are integrated via project experiments and how the results strengthen future educational experiences.
Added March 23rd, 2009
USC-CSSE-2009-503
Jo Ann Lane, "Cost Model Extensions to Support Systems Engineering Cost Estimation for Complex Systems and Systems of Systems," 7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) (pdf)
Considerable work has been done in the areas of software development, systems engineering, and Commercial Off-the-Shelf (COTS) integration cost modeling, as well as some preliminary work in the area of System of System Engineering (SoSE) cost modeling. These cost models focus on the engineering product, the processes used to develop the product, and the skills and experience levels of the technical staff responsible for the development of the product. The nature of complex systems and systems of systems (SoS) is that they are often composed of many subsystems, some relatively simple and others rather complex. The Constructive Systems Engineering Cost Model (COSYSMO) is a calibrated cost model that most closely estimates the systems engineering effort associated with these complex systems and SoS engineering activities. However, the current version of COSYSMO only allows the user to characterize the system using a single set of parameters, with no ability to generate multiple characterizations for the various subsystems comprising a complex system or SoS. This paper presents an extension to the COSYSMO cost model that allows a user to characterize multiple subsystems within a complex system or multiple systems within an SoS and illustrates how this can be applied to an example SoS.
Added March 16th, 2009
USC-CSSE-2009-502
Barry Boehm, Jo Ann Lane, Supannika Koolmanojwong, "A Risk-Driven Process Decision Table to Guide System Development Rigor," INCOSE 2009 (pdf)
The Incremental Commitment Model (ICM) organizes systems engineering and acquisition processes in ways that better accommodate the different strengths and difficulties of hardware, software, and human factors engineering approaches. As with other models trying to address a wide variety of situations, its general form is rather complex. However, its risk-driven nature has enabled us to determine a set of twelve common risk patterns and organize them into a decision table that can help new projects converge on a process that fits well with their particular process drivers. For each of the twelve special cases, the decision table provides top-level guidelines for tailoring the key activities of the ICM, along with suggested lengths between each internal system build and each external system increment delivery. This paper elaborates on each of the twelve cases and provides examples of their use.
Added March 16th, 2009
USC-CSSE-2009-501
Barry Boehm, Jo Ann Lane, "Better Management of Development Risks: Early Feasibility Evidence," 7th Annual Conference on Systems Engineering Research 2009 (CSER 2009) (pdf)
The Incremental Commitment Model (ICM) organizes more rapid and thorough concurrent systems engineering and acquisition processes in ways that provide points at which they can synchronize and stabilize, and at which their risks of going forward can be better assessed and fitted into a risk-driven stakeholder resource commitment process. In particular, its concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their system constraints and environments and on opportunities to deal with them. This paper describes in detail the content, preparation, and role of feasibility evidence at key decision points in the system development cycle and how this can be used to effectively identify and manage risks throughout the development cycle. The feasibility evidence is not intended to assess just a single sequentially developed system definition element, but rather to assess the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of anchor point milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase using developer-produced, expert-validated feasibility evidence.
Added March 16th, 2009
USC-CSSE-2009-500
Barry Boehm, Jo Ann Lane, "Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects" (pdf)
Future mission-critical DoD combat platforms, systems of systems, and network-centric services will have many usage uncertainties and emergent characteristics. Their hardware, software, and human factors will need to be concurrently engineered, risk-managed, and evolutionarily developed to converge on cost-effective system operations and mission success.
The recent revision of DoDI 5000.02 provides a much improved policy for dealing with these future challenges, including its starting with Needs and Opportunities, its emphasis on early Analysis of Alternatives as an entry condition for Milestone A, its inclusion of a passed Preliminary Design Review as an entry condition for Milestone B, and its emphasis on evolutionary versus single-increment or prespecified increments of development.
These emphases on DoDI 5000.02 are also key emphases in the Incremental Commitment Model (ICM), along with more detailed guidance on how to apply them. The ICM has been developed based on best commercial practices for addressing such future challenges; on discussions with contributors to DoDI 5000.02; on successful use of the ICM principles on previous DoD projects; and on its exploratory use on highly complex, future-precursor, net-centric DoD systems of systems.
The ICM builds on the experience-based critical success factor principles of stakeholder satisficing, incremental definition, iterative evolutionary system growth, concurrent engineering, and evidence-based, risk-driven management milestones. It is not a one-size-fits-all process model, but has risk-driven options at each milestone decision point to enable projects to adapt to their particular situations. It provides a decision table for early determination of common special-case life cycle processes to avoid overkill in project ceremony and documentation.
It also incorporates the strengths of existing V, concurrent engineering, spiral, agile, and lean process models. As with these other models, it is not fully tested for its ability to support DoDI 5000.02 in all project situations, but it has been formulated to make it straightforward to do so.
Added March 16th, 2009
Copyright 2008 The University
of Southern California
The written material, text,
graphics, and software available on this page and all related
pages may be copied, used, and distributed freely as long as the
University of Southern California as the source of the material,
text, graphics or software is always clearly indicated and such
acknowledgement always accompanies any reuse or redistribution
of the material, text, graphics or software; also permission to
use the material, text, graphics or software on these pages does
not include the right to repackage the material, text, graphics
or software in any form or manner and then claim exclusive proprietary
ownership of it as part of a commercial offering of services or
as part of a commercially offered product.
|
|
