Talks
- Managing Uncertainty in Value-Based SE
- Talk at Google: Learning Defect Predictors from Static Code Attributes; Oct 29 '08
- Creation is a predictable, controllable process that can be engineered
- Why I think about Turing Machines
- Learning Near Optimum Inspection Policies
- The Business Case for Automated Software Engineering
- Nighthawk: Two-Level Genetic-Random Unit Test Data Generator
- If you fix everything, you have nothing left to lose
- Multi-implications of multi-dimensional authoring: or, everything you wanted to know about geek herding, but were afraid to ask
- "STAR"- seeking new frontiers in cost modeling
- "2cee" a 21st century effort estimation method
- Improving IV&V Techniques Through the Analysis of Anomalies
- Data Mining, Truth, Justice, the American Way, and the Flying Spaghetti Monster
- Open source- can you ignore it?
Managing Uncertainty in Value-Based SE
A presentation to the COCOMO forum 2008. October 28. 2008.
Fri Nov 7 14:27:08 PST 2008
Talk at Google: Learning Defect Predictors from Static Code Attributes; Oct 29 '08
Download slides (PDF)
For six years, I have worked on learning quality predictors from NASA data. Based on that experience, this talk offers the following lessons from the trenches:
- Real world data collection is more like ambulance chasing that bus driving. The old DoD model of rigorous process control just breaks down in the modern era of distributed software development. Rather than lament lack of formal process, we should adapt our learning methods to handle the idiosyncrasies of our data.
- Static code attributes are a wide and shallow well- easy to get to the bottom, very hard to get much further. Our learners may have learned all they can learn from these attributes.
- The only way up is sideways. My data miners have struck a performance ceiling and the only way up is to change the performance target.
- The performance ceiling is very close- we can exploit that. Rather than large-scale automatic methods, it may be more productive to explore human-in-the-loop interactive learning strategies.
Tue Nov 4 18:13:58 PST 2008
Creation is a predictable, controllable process that can be engineered
Based on the success of the software engineering, I want argue that creativity can be engineered and, routinely, is done so in a commercial setting. To do so I must demythologize "creation". Rather than focus obsessively on that "a ha!" moment when the world spins on a dime and the new paradigm falls like an apple in Newton's garden, we should look at the conditions that preceded creation and increases the probability of finding shiny new apples.
To any individual, creation may feel like a special process but, as Hari Seldon will one day observe, given a large enough community, then the outputs of that communities creativity and individuality can be predicted.
Sat Apr 5 12:31:29 PDT 2008
Why I think about Turing Machines
CS and UI is the art of building glass walls to hide users from the computational cliff. Hopefully, most users are blind to how narrow is their treacherous path. But without the language of the Turing machine, you are blind to certain restrictions to writing, unable to explain why your system is broken, unable to avoid that problem in the future.
Sat Apr 5 12:28:02 PDT 2008
Learning Near Optimum Inspection Policies
Wed Fed 6,11:30am, 2008
WHERE: room 225b, IVV building
WHO: (Dr. Tim Menzies, Zach Milton) wvu, csee
ABSTRACT: High assurance software requires extensive and expensive assessment. There are many forms of software assessment, ranging from manual inspections to automatic formal methods. These assessment methods differ in their effectiveness and the effort required to apply them. Typically, the more effective methods are more expensive. Hence, project managers often "bias" the assessment resources and apply more effort where they think that extra effort might be most useful.
If most of the assessment effort explores project artifacts A,B,C,D, then that leaves a "blind spot" in E,F,G,H,I,.... Blind spots can compromise high assurance software. It is therefore important to discuss the bias introduced by the inspection policy. To but the matter in a nutshell, we need to ask "how blinding is our bias?"
This talk contrasts three different kinds of "bias" in selecting what code modules to inspect:
- Manual methods such as "read the biggest thing first/last"
- Traditional data mining methods such as those advocated by author and those deployed in NASA-funded inspection tools.
- A new data miner called "WHICH"
We find that #1 usually outperforms #2. This result calls into question many years of research by the speaker (translation: "oh dear.....").
But we also find that #3 almost always out-performs #1 or #2 (translation: "phew!!").
In fact #3 works so well that we speculate that it could be used as a proxy for determining the actual number of defects remaining to be found, after inspecting Z% of the code.
Sat Feb 9 20:08:13 PST 2008
The Business Case for Automated Software Engineering
IEEE Automated Software Engineering, Nov, 2007, Atlanta, USA. ASE 2007. Download paper on-line.
Adoption of advanced automated SE (ASE) tools would be more favored if a business case could be made that these tools are more valuable than alternate methods. In theory, software prediction models can be used to make that case. In practice, this is complicated by the "local tuning" problem. Normally, predictors for software effort and defects and threat use local data to tune their predictions. Such local tuning data is often unavailable.
This paper shows that assessing the relative merits of different SE methods need not require precise local tunings. STAR1 is a simulated annealer plus a Bayesian post-processor that explores the space of possible local tunings within software prediction models. STAR1 ranks project decisions by their effects on effort and defects and threats. In experiments with NASA systems, STAR1 found one project where ASE were essential for minimizing effort/ defect/ threats; and another project were ASE tools were merely optional.
Sat Nov 10 12:42:59 PST 2007
Nighthawk: Two-Level Genetic-Random Unit Test Data Generator
ASE 2007. Paper available on-line.
Randomized testing has been shown to be an effective method for testing software units. However, the thoroughness of randomized unit testing varies widely according to the settings of certain parameters, such as the relative frequencies with which methods are called. In this paper, we describe a system which uses a genetic algorithm to find parameters for randomized unit testing that optimize test coverage. We compare our coverage results to previous work, and report on case studies and experiments on system options.
Sat Nov 10 12:35:30 PST 2007
If you fix everything, you have nothing left to lose
On the Value of Stochastic Abduction (if you fix everything, you lose fixes for everything else) International workshop on living with uncertainty. 2007.
Creative solutions can sometimes be found in larger space of possible qualitative behaviors than in the tighter space of precise quantitative behaviors. To put that another way: If you fix everything, you loss fixes for everything else. We illustrate this effect with an example from software process theory.
Mon Nov 5 19:52:35 PST 2007
Multi-implications of multi-dimensional authoring: or, everything you wanted to know about geek herding, but were afraid to ask
If a stone tablet doubles in size, it weighs eight times as much. What is true for tablets is true for other media. As the dimensionality of our media grows, the authoring effort increases super-linearly. With open source tools, and utilizing crowd souring, massively multi-dimensional multi-media products can be produced on minuscule budgets. In open source, all products are available at all times for copying, or branching the project in an alternate direction. Unless the geeks work together, they will splinter into less productive sub-groups. Gangs of geeks generating new media become reliant on each other. Individuals learn to serve the needs of the group, often constraining their own ideas to those endorsed by the group. In one sense, this is old news. Open source multi-media generation is just relearning a centuries-old lesson that crowds can generate more than individuals. Shakespeare's portfolios are excellent examples of crowd sourced content generation. The plays themselves were remixes of older stories. Actors improvised portions of the plays before they were recorded in the portfolios.
But whats old is now back in the news. Our legal institutions, fixated on ownerships or corporate property rights, actively block crowd sourcing. Academic institutions (read tenure committees) give little credence to team players. Yet modern media authors must enlist in an army to complete multi-dimensional masterpieces. Do you like the ten people sitting next to you? You'd better, they've just become as important as your heart beat for completing your next project. But before you get together, you'd better generate an acceptance of modern models of accreditation that move beyond concepts of I, and that acknowledge us.
Sat Nov 3 12:51:59 PDT 2007
"STAR"- seeking new frontiers in cost modeling
2007 COCOMO forum.
STAR has 3 key advantages over traditional methods and even 2cee:
- Provides an integrated set of COCOMO models
- Can be used to systematically analyze strategic and tactical policy decisions
- Can be tuned/calibrated with constraint sets instead of traditional historical records.
STAR is an abductive inference engine that applies simulated annealing to a treatment learner.
Thu Oct 25 17:52:32 PDT 2007
"2cee" a 21st century effort estimation method
2007 COCOMO forum.
It became quickly apparent in the early states of our research that there was a major disconnect between the techniques used by estimation practitioners and the numerous ideas being addressed in the research community. Many fundamental estimation questions were not being addressed. We had to fix all that.
Thu Oct 25 17:45:14 PDT 2007
Improving IV&V Techniques Through the Analysis of Anomalies
We seek an active monitoring framework where data collected during an active IV\&V project will trigger an alert if a project becomes "unusual" (and defining "unusual" is one of the goals of this project).
Sat Sep 29 17:25:21 PDT 2007
Data Mining, Truth, Justice, the American Way, and the Flying Spaghetti Monster
President Bush advises that students should be exposed to many different schools of though. In this matter, he is less than totally correct.
An important part of science is to expose people to the critical and continual (re)evaluation of ideas. Otherwise, any nonsense can be shoved down our throats such as pirates are causing global warming.
So this talk concerns notions of certainty and standards for debate. One surprise will be that nothing is "true" (in a scientific sense) but many more things are false. Implications for humility, tolerance, and justice will be discussed.
Sat Sep 22 06:56:42 PDT 2007
Open source- can you ignore it?
- What is open source?
- Where did it come from?
-
Why can't you ignore it?
- Anthropologically: has been with us for centuries
- Economics: too powerful to ignore
- Legally: no future without it
- Technical: unavoidable
- So what social institutions will be adopted to handle it?
Sat Nov 10 18:30:15 PST 2007
