July 29 - Aug 1, 2003 I went to the
NASA's OSMA SAS03
- the annual SARP showcase.
(Btw, OSMA= office of safety and mission assurance;
SAS= software assurance symposium;
and SARP= software assurance research program).
Here's my diary from that trip. This is not some official record
by any means- just my personnel notes. If you want the official
web site for this conference, go see http://sas.ivv.nasa.gov/.
Oh,
if you want slides for these talks,
they should be available soon from
the SAS'03 page.
|
Topics |
All the presentations here were about stuff funded
out of the NASA SARP program. Topics included:
- security
- V&V and VR
- V&V and OO
- text mining
- IV&V processes
- web-applications
- code inspections
- real-time systems
- adaptive systems
- open-source systems
- software fault trees
- requirements engineering
- software fault injection
- static vs dynamic metrics
- static code defect detectors,
- software probabilistic risk assessment
- reliability and safety, reliability and operational profiles
- and other stuff as well
|
|
Attendance |
A lot of folks came to see the presentations.
170 attendees this year which was up from 110 in 2002 and
80 in 2001.
My own (private) view is that the
increased attendance was due to how
we monitor the projects.
See, the more serious we treat the PIs
during the year, the
more seriously they treat the program.
Of the attendants, several deserve special attention:
Bryan O'Conner, head, code Q;
Nelson Keeler, director, IV&V facility;
John Kelly: chief engineer;s office;
Martha Wetherholt, Code Q;
Ken McGill, research lead, Fairmont IV&V.
Ken is in charge of the day-to-day running of SARP.
Bryan, Ned and Martha can turn up the heat on Ken if
SAS or SARP goes off course.
Byran O'Conner gave the keynote address at the dinner.
Spoke much on the need for driving our test procedures
from exploring just the "known knowns" into the zone of
"unknown knowns" (i.e. the things other folks know and somehow,
we never hear about) and the "unknown unknowns" (i.e.
the things we don't even know we don't know).
Ned
opened the SAS.
Then
Martha Wetherholt briefed the SAS on software assurance
at NASA.
John, Martha, and Ken are on the source selection committee
for new proposals. They meet every morning for breakfast to
discuss what was to come and what they had seen the day before.
The source selection committee meeting was on the last afternoon
of SAS.
Here's Ned thanking Martha for
her work on software assurance across
NASA in general and her work on the program in particular
(the cup is the
"Quality Cup"
which was our general award for folks participation at the SAS).
Here's Martha thanking Ned for
handling the day-to-day management of the program.
SAS also attracted three special guests
from
Japan.
Naoki Ishihama works directly for the Japanese
NASDA (National Space Development Agency) and
Hideki Nomoto, and Haruka Nakao from
the Japan Manned Space Systems Corporation (JAMSS)
are contractors to that agency.
Here they are with
Bojan Cukic, who was kind enough to help host them and show them
around Morgantown.
The day before SAS, our Japanese guests
attended the IV&V facility and
gave an excellent seminar on Japanese IV&V work for space
applications.
Here's Ken presenting
them gifts.
Our Japanese guests
were kind enough to stay on and give us the benefit
of their experience for the SAS.
|
|
At SAS'03 |
A very lively conference.
Some folks coming up to us saying
"wow, these presentations are good- we'll all have to lift
our game to keep up".
Like most meetings, the real action was in the corridor when folks meet
to talk about coffee.
Martin Feather and Alice Lee pulled together
an informal working group of folks from around the country who
want to work on probabilistic risk assessment. It was a nice
example of what can happen at venues like SAS.
Also, SAS lets researchers
work in depth with their government points-of-contact.
Here's David Raffo in intense dialogues with
Judy Connelly and
John Marinaro
More David Raffo stuff, here discussing how to get project
data on software processes with Forrest Shull.
Mike Beims, asking questions.
This was the first year that the team
from West Virginia college were involved
with us. SAS must have been very useful to them- it let them get
a good overview of our work.
Here's Jeff Voas and Bryan O'Conner talking about finding the
"unknown unknowns".
|
|
Awards |
The award ceremony was an important event. They
were designed to tell folks what we value around here:
-
"Best paper"
is for those who get a paper into a widely read
forum.
-
"Best research" was given to those who had transitioned their
work into NASA applications (and this is a higher award
than "Best paper").
-
Other awards congratulate the researchers
with most daring or the research centers that generate
the best research.
Kalynnda Berens
won a "Best Research of the Year" award for her
fault injection work at Glenn.
Martin Feather's cat (pictured here) likes
his "Best Research" award for producing
research that got picked up by NASA customers.
The
"Best Research Organization" went to
the JPL gang, some of which are shown here:
(left to right)
Robyn Lutz, John Powell, Martin Feather, David Gilliam,
and Michael Gayle
(Manager, Software Assurance Group).
JPL got the award this year cause
of Robyn's work on orthogonal
defect classification
and David's work on security.
To honor daring resarch,
we had a new award this year: "the Buzz". Buzz Aldrin's
Ph.D. was all about orbital rendezvous. The thing of it was that, six years
later, he needed that orbital rendezvous stuff to get home. See, no
rendezvous for Buzz coming up from the moon, no ride home.
So "the Buzz" is given to researchers who take a risk
based on their work.
Jane Hayes (left)
and Alexander Dekhtyar (2nd from left)
won a "Buzz" award for most daring research: they dared
to compare their experimental technique with a state-of-the-art
commercial tool with the work of an experienced analysts.
Carol Smidts (see here with Bojan Cukic) also won a
"Buzz"
for daring to hold a peer review panel on her work back in 2001.
At that panel, she
invited all the who's who of reliability research plus all
the government folk who could slash her funding. She came through
with flying colors.
(Actually, Carol was so unaware she was going to win the "Buzz" that she did not attend the dinner when the award was given- it had to be passed to one
of her students.)
Katerina
Goseva-Popstajanova won "Best Paper" for an IEEE Transactions paper that we co-authored
with a whole gang
of folks, most of whom come from WVU. Not that I want to brag, of course,
but if you dropped WVU from the program
you'd lose most of the SARP-generated journal articles.
The awards
were held at the formal dinner on the first night.
I'm leaving the research chair position this year so,
at the dinner, they gave me
an award. Wes Deadrick's speech with the award told folks that I
was always late for meetings and the first thing I asked when I arrive
was "what is the time".
He then compared me with the white
rabbit from Alice in Wonderland.
So, naturally, he gave me this great huge fob watch- just like the
white rabbit.
Lisa Montgomery
was so good at organizing this SAS
that we gave her an award.
Sorry it was not two.
Some more awards, that were slightly sillier, were given next night
Here's Al Gallo getting a Wit award from Goddard giving
us the shortest set of presentation slides (brevity is the soul of wit).
Here's Ken McGill
getting the most verbose award cause the IV&V presentations were
the longest.
|
|
Talks |
Here's some of the speakers.
Note that I don't have pictures from everyone
since the flash on my
little camera couldn't
handle the large hall too well.
Hany Ammar: presented more talks than anyone else
at SAS. Here he is presenting
Relations Between Static & Dynamic SW Metrics.
This presentation scared Bill Jackson who said "cripes,
what if there is no relationship? that would put IV&V
out of business.". Happily, Hany had some good news.
Kalynnda Berens
doing Injecting Faults for Software Error Evaluation.
She was also on my panel on the future of V&V and did a great job.
Check out this pic:
her outfit matches the
color scheme of her slides.
Bojan Cukic
Verification and Validation of Adaptive Systems.
After this talk, Bill Jackson was heard to say that he wanted
to quit his admin job and go work for Bojan.
Alexander Dekhtyar (background) and
Jane Hayes (foreground) doing
Robust Requirements Tracing Via Internet Tech: Improving an IV&V Technique.
Martin Feather Requirements Decomposition Analysis.
Al Gallo presenting
Fault Tree Analysis Application for Safety and Reliability
with Massood Towhidnejad (behind).
David Gilliam
Reducing Software Security Risk Through an Integrated Approach.
Katerina Goseva-Popstajanova
Sensitivity of Software Reliability to Operational Profile Errors.
Mats Heimdahl A Spectrum of IV&V Modeling Techniques
Robyn Lutz
Adapting ODC for Empirical Evaluation of Pre-Launch Anomalies.
Me (Tim Menzies):
See More! Learn More! Tell More!.
John Powell (middle) and David Gilliam (right).
John gave Allen Nikora's talk on
Infusing Software Fault Measurement and Modeling Techniques
(since Allen had to be on sick leave that week).
Tom Robinson:
IV&V Cost Estimation-A Joint NASA & Navy Collab
Chris Rouff Formal Approaches to Swarm Technologies.
Chris's shirts can be really, really loud.
Here's Mike Hinchey
wearing dark glasses to stop
Chris's shirt from blinding him.
Yann-hang Lee
Timing and Race Condition Verification of Real-Time Systems
Forrest Shull State-of-the-Art Software Inspections & Reading at NASA
Jon Whittle Transitioning from Software Requirements Models to Design Models.
|
|
Panel |
Very proud of the panel that I ran
with Kalynnda Berens
and Mats Heimdahl
on the future of V&V. We spoke on
how open-source, off-shore, and model-based
software development could change how we do IV&V in the future.
We lead the whole room in a lively debate that folks spoke of
for hours afterwards. I got invites to two other conferences
to run the same panel: all the way from
Pittsburgh to Pakistan.
|
|
The workers |
Meetings like SAS don't work unless
a lot of people spend a lot of time
moving around mountains.
The Queen of SAS: Lisa
Montgomery. She was there from crack of dawn till crack of next
dawn every day, fighting fires, organizing the rooms, organizing
the receipts,
organizing the this, and organizing the that. She
had this whole mind-reading thing going on. There was all
these times when I'd rush up to Lisa with the latest
problem and she was already on it.
Woman with very
strong claim
to pay rise (IMHO).
Lisa, with the CSEE IT folks, plus Andreas Orrego
and Dustin Geletko (two of the CSEE GRAs who pulled an all-nighter
to get the presentations together). Not shown here
was Greg Mundy, another CSEE GRA,
who also went the extra mile for SAS.
The Fairmont IT gang who
provided computer support.
|
|
Games |
If you build it, they will come.
Putt-putt competitions, toy plane competitions,
they played it all.
Like some puppet master, I watched with
more glee than I should as the assembled masses grabbed the
offered golf sticks.
I was not the only one to find putt-putt
to be a great spectator sport.
Some
spectators got very excited. Here's Bill Jackson getting
perhaps too excited.
Whenever John Marinaro plays any game,
his whole body radiates this intense
"do not disturb" sign.
Here's Wes Deadrick's
(nearly) winning golf round. He was real gracious
about this. I believe his winning speech was
"You lot play golf every week? You should
be ashamed! I only get to the green twice a year."
Next night, it rained
so the action moved inside. The great airplane competition.
Lines and lines of folks trying to land a plane closest to the wall.
Many experiments. The winner, Andres Orrego (left)
spent the day working out glide paths.
David Gillian (right) tried this double-wing design thing
that totally failed to do anything except impress me
with his ingenuity. David Raffo's (middle) design was amazing- he took
off all the wings and threw the thing like a missile. Sadly, it bounced
back so far from the wall that Andreas's plane won.
And after the planes, came the frisbees.
|
|
The Future |
At the SAS dinner, Ken McGill spoke of
changes to SARP.
Four years ago, SARP proposals had a 67% chance
of success.
This year, the acceptance rate will be much more
competitive: acceptance rates of around 10%.
Ken was
quick to add the obvious- these low acceptance rates
are both good and bad.
The good news is that this program is becoming a very serious scientific
endeavor and lots of folks want to join in.
The bad news is that it can no longer fund merely good research.
Rather, the program will only be able to fund a
limited number of truly great research projects.
What he didn't mention is that, next year,
the good and bad news will be even louder:
- If we project forward what
projects are scheduled to finish in 2004, then there will 50% less
bucks for new projects next year.
-
So that 10% acceptance rate will
drop to, say, 5%.
That is,
unless we can find some way to top up the available bucks.