Open Source Software Evolution

Transcription

1 Open Source Software Evolution (Examining Lehman's Eight Laws of Software Evolution in the context of OSS) Shaun Brandt 2/6/2013

2 Contents About change About Manny Lehman and his laws Applying Lehman's laws to open source software 'Towards a Better Understanding of Software Evolution' Concluding thoughts

3 Change is inevitable For any non-trivial program, the first version released is never the only version released If users like your program, they're going to want more features and bug fixes If users hate your program (but are forced to use it), they're still going to want more features and bug fixes

4 Change is inevitable As expensive (cost and time-wise) as programs are to initially develop, the costs of maintaining programs is even higher In 1980, the ratio of development to maintenance cost was estimated to be 30% development, 70% maintenance Today, estimates range between 50% and 90% maintenance cost, depending on the age and complexity of the system

5 So, why is understanding change important? See the previous slide (Hint: it's cost!) The cost isn't necessarily (directly) monetary especially in the case of open source software: Opportunity costs (engineers that are constantly fighting complex, poorly written code are less efficient) Programs that grow too large and complex for their own good may need to be redesigned, potentially at great expense (especially but not exclusively - so in the commercial world)

6 (An aside: why do *I* care, and why should you?) My day to day job involves developing and maintaining a software system with a long history Pieces date back to the late 1980s/early 1990s, but most of the code is newer Decisions made by developers years ago affect how I do work today, and excessive cost can make old mistakes hard to fix later If you haven't worked on a mature software code base (open source or closed source) yet, you will.

7 About Manny Lehman Meir Manny Lehman ( ), was a professor at Middlesex University, and later at Imperial College London During the late 1960s/early 1970s, he worked at IBM's research division, and this is where he started studying software evolution Lehman's laws of software evolution derive from his study of IBM's programming process

8 Lehman's work in software evolution Lehman produced many papers on the topic of software evolution, and his eight laws were developed over time based on his work and the contributions of others: The first 3 laws were formalized in 1974 Two additional laws were formalized in 1978 The sixth law was formalized in 1991 The seventh and eighth laws were formalized in 1996 (Law 8 Feedback System, was described by Lehman in 1974 but was formalized later after some system dynamics work by W.M. Turski in 1996)

9 E-type systems? (a historical aside) Lehman's laws (as you'll see soon) refer to what are called 'E-type' systems This is based on a (1970's) method of dividing software programs into three types: S-type, P-type, and E-type S-type programs are those whose function is formally defined by and derivable from a specification

10 E-type systems? P-type programs are those which solve problems that can be precisely defined, but the acceptability of the solution is dependent on the environment in which the program is used The key difference between S-type and P-type programs is that S-type programs only care about adherence to the specification, but P-type programs are more concerned with the validity of the solution in a real world context

11 E-type systems? E-type programs that 'mechanize a human or societal activity', as described by Lehman The program itself is not the only component anymore; it is part of a larger feedback system including humans, interactive inputs, and other features. Every part directly affects the behavior of the system, and contributes to the evolution of the system as a whole Even the installation of the program in its operational domain changes the domain Air traffic control systems, stock trading systems, even operating systems

12 E-type systems? Lehman's laws are only concerned with E-type applications, since they're the most interesting (and most complex) In other words, you can forget about the other two types now. :)

13 Lehman's Laws of Software Evolution So, what are Lehman's laws of software evolution?

14 Lehman's Laws of Software Evolution Eight laws total I Continuing Change (1974) II Increasing Complexity (1974) III Self Regulation (1974) IV Conservation of Organizational Stability (1978) V Conservation of Familiarity (1978) VI Continuing Growth (1991) VII Declining Quality (1996) VIII Feedback System (1974, 1996)

15 Lehman's Laws of Software Evolution I -- Continuing Change (1974): From Lehman: E-type systems must be continually adapted or they become progressively less satisfactory From Xie et al: A program must continually adapt to its environment, otherwise it will become progressively less useful

16 Lehman's Laws of Software Evolution II Increasing Complexity (1974): From Lehman: As an E-type system evolves its complexity increases unless work is done to maintain or reduce it From Xie et al: As a program evolves, its complexity increases, unless proactive measures are taken to reduce or stabilize the compexity

17 Lehman's Laws of Software Evolution III Self Regulation (1974): From Lehman: E-type system evolution process is self-regulating with distribution of product and process measures close to normal From Xie et al: The evolution of large software systems is selfregulating, i.e. the system will adjust its size throughout its lifetime

18 Lehman's Laws of Software Evolution IV Conservation of Organizational Stability (1974): From Lehman: The average effective global activity rate in an evolving E-type system is invariant over product lifetime From Xie et al: The rate of productive output tends to stay constant throughout a program's lifetime

19 Lehman's Laws of Software Evolution V Conservation of Familiarity (1974): From Lehman: As an E-type system evolves, all associated with it (developers and users) must maintain mastery of its content and behavior to achieve satisfactory evolution From Xie et al: The authors state that this law suggests that incremental growth tends to remain constant or decline because developers need to understand the program's source code and behavior

20 Lehman's Laws of Software Evolution VI Continuing Growth (1991): From Lehman: The functional content of E-type systems must be continually increased to maintain user satisfaction over their lifetime From Xie et al: Programs grow over time to accommodate pressure for change and to satisfy an increasing set of requirements

21 Lehman's Laws of Software Evolution VII Declining Quality (1996): From Lehman: The quality of E-type systems will appear to be declining unless they are rigorously maintained and adapted to operational environment changes From Xie et al: Over time, software quality appears to be declining, unless proactive measures are taken to adapt the software to its operational environment

22 Lehman's Laws of Software Evolution VIII Feedback System (1974, 1996): From Lehman: E-type evolution processes constitute multi-level, multi-loop, multi-agent feedback systems and must be treated as such to achieve significant improvement over any reasonable base From Xie et al: Software projects are self-regulating systems with feedback.

23 Lehman's Laws of Software Evolution Lehman's laws were developed in the context of closed source software (more specifically, software running on IBM's OS/360) In the mid-1970s, most large software systems were commercial, closed source products Large, mature open source projects came later The GNU Project officially formed in 1983 The Linux kernel was first released in 1991

24 Applying Lehman's Laws to OSS While many studies have been successfully done to verify the validity of Lehman's laws on closed source applications, this is generally hard to do openly and transparently To monitor change, you must be able to see the things that have changed! Source code (especially the source code) Release notes SCM commit logs

25 Applying Lehman's Laws to OSS Getting information about exactly what's changing in a commercial, closed source product can be very difficult: Get permission / sign a NDA Find someone familiar with the system to get the information you need, which may be stored/accessed using non-standard/archaic tools Publish your work, but probably without referring to the product(s) by company or by name (you even see this in Lehman's own work!)

26 Applying Lehman's Laws to OSS Getting information about exactly what's changing in an open source project is really easy! Get a copy of the project's source repository Get a copy of the project's release notes Write and/or run some off the shelf tools to pull out the information you want Publish a paper that describes everything, because there are no NDAs or other legal constructs preventing you from doing so

27 Applying Lehman's Laws to OSS However, 'because it's easier to study' isn't the only (or even the main) motivation there are important questions raised by OSS, mainly: How does a large, distributed group of people work in what almost seems like an ad hoc manner at times, yet manage to produce (and maintain) software of equal or better quality than closed-source equivalents? Does the open source development model help or hinder software evolution? Do Lehman's laws even apply?

28 Applying Lehman's Laws to OSS As a result of this, a number of studies have been done to verify Lehman's laws with open source software The optional paper I assigned is full of references to them 7 are discussed in detail, all pre-2005 In my presentation, I will only discuss one paper Towards a Better Understanding of Software Evolution: An Empirical Study on Open Source Software by Xie et al.

29 Towards a Better Understanding of Software Evolution A 2009 study of the evolution of open source software, written by Guowu Xie, Jianbo Chen, and Iulian Neamtiu In this paper, the authors attempt to apply Lehman's laws to a body of large, mature open source software projects Other previous works concentrated on a subset of Lehman's laws. The authors of this paper attempt to address all of them

30 Towards a Better Understanding... Seven projects (all various servers) were selected in total Samba (a UNIX/Windows interoperability server) BIND (a or, more accurately, the - DNS server) OpenSSH (a Secure Shell server suite) SQLite (a SQL server library) Vsftpd (the default FTP server in many Linux distros) Quagga (a server suite for software network routing) Sendmail (a mail transfer agent, and the piece of software on this list with the longest history)

31 Towards a Better Understanding... The selected programs all have a long history, a large number of releases, medium-to-large code base sizes, and are widely used. A couple of the programs (BIND, Sendmail) don't include complete histories in the results due to technical issues

32 Towards a Better Understanding... - Methodology For every application, the team Downloaded every available public release Removed all test programs, client applications and other non-server components, and ran the rest through the C preprocessor Merged all the source code into a single.c file using a tool called CIL One such.c file (and resulting program) was created for each release

33 Towards a Better Understanding... - Methodology After noting the total LOC and confirming that the resulting file could be built, linked, and run, they then ran two source code analysis tools on the results: ASTdiff RSM ASTdiff a tool written by the authors; uses abstract syntax trees to 'diff' C programs and collects various change metrics, including changes to types, globals, function signatures, function calls per function, and so on

34 Towards a Better Understanding... - Methodology RSM (Resource Standard Metrics) a commercial tool that computes (among other things) cyclomatic complexity Cyclomatic complexity is a measurement of the number of linearly independent paths through a program's source code (yes, I had to go look this up!) Some researchers believe that there's positive correlation between cyclomatic complexity and code defect rates, but the belief in this isn't universal

35 Towards a Better Understanding... - Results Using the previously described tools and gathered metrics, the team then attempted to confirm whether each of Lehman's eight laws was true. The following slides summarize any additional methodology for each test, and the final results

36 Towards a Better Understanding... - Results I Continuing Change All seven programs demonstrated change over time Mainly additions and modifications, some deletions Most change occurred to functions (as opposed to structs or globals)

37 Towards a Better Understanding... - Results Changes to interface happened much less than changes to implementation Graphs were only provided for Samba, but the other programs exhibited similar behavior Authors' conclusion: Lehman's first law is confirmed for the test applications

38 Towards a Better Understanding... - Results II Increasing Complexity The authors used metrics based on function calls per function, cyclomatic complexity, and common coupling Common coupling is the case where multiple modules/functions share a piece of global data These three methods were described in works by other authors, and were selected by this paper's authors as reasonable ways to measure complexity that can 'hinder evolution' by making programs hard to change

39 Towards a Better Understanding... - Results

40 Towards a Better Understanding... - Results Just what do all those numbers mean? A series of linear regressions where the independent variable is the number of days since the initial release, and the dependent variable is the complexity metric The resulting slopes measure relative complexity over time for a particular metric Both total and normalized (for program size) results are given tl;dr as programs get older, they continue to get more complex, and any decrease in complexity is usually due to increased program size dragging down the average

41 Towards a Better Understanding... - Results Authors' conclusion: While determining whether steps are being made to reduce complexity is difficult without looking over lots of commit logs and determining intent, the increasing complexity over time as measured by each metric means that Lehman's second law is also confirmed.

42 Towards a Better Understanding... - Results III Self Regulation The authors describe this law by stating that software systems are self-regulating; that the system will adjust its size throughout its lifetime. In the closed source world, regulation occurs due to lack of resources, and the need to develop and support other products. A single software system is rarely isolated and can't be treated as such The authors of the paper state that evidence of self-regulation translates to observing 'ripples' small adjustments in the growth trend of the system

43 Towards a Better Understanding... - Results The authors measured module increment per release for each test program, and observed the expected ripple behavior. Authors' conclusion: Lehman's third law is confirmed for the test applications

44 Towards a Better Understanding... - Results IV Conservation of Organizational Stability Measuring 'work rate' (to prove an invariant work rate), is difficult 'Effort' doesn't always mean 'progress' Lehman suggested using 'changes per release' The authors attempted to do this, and found that the work rate, when calculated this way, is not invariant

45 Towards a Better Understanding... - Results There's an important caveat Remember, Lehman's laws were developed with commercial software in mind Commercial products usually have limited development resources Successful open source projects gain additional developers over time Authors' conclusion: Lehman's fourth law cannot be confirmed in the context of OSS without more information

46 Towards a Better Understanding... - Results V Conservation of Familiarity Lehman's law implies that growth will stay constant or decline over time, due to the need for developers to fully understand the system's behavior However, the authors find that the growth of modules in the seven test programs is neither constant nor decreasing This appears to be a particular behavior of open source software, which the authors call 'super-linear growth'

47 Towards a Better Understanding... - Results Authors' conclusion: Lehman's fifth law is not confirmed (at least for the test programs)

48 Towards a Better Understanding... - Results VI Continuing Growth The authors tested this by using two different metrics: LOC and module count. Unsurprisingly, all of the test programs grew over time!

49 Towards a Better Understanding... - Results BIND was the only program that showed any major decline in code size during its history, due to some restructuring work (more of a rewrite, actually) BIND 9 is an interesting case what does BIND's code rewrite have to say about how the software had evolved to that point? Authors' conclusion: Lehman's sixth law is confirmed for the test applications

50 Towards a Better Understanding... - Results VII Declining Quality To test this, the authors considered two different types of quality metrics 'internal' and 'external' External: quality as perceived by users of the software Internal: a little harder to define, but the authors believe that lower internal quality may be related to increased code complexity over time

51 Towards a Better Understanding... - Results External quality The authors examined defect tracking systems where available (OpenSSH, Samba, Quagga, and BIND) Main metrics Known defects for each release Defect density (defects per line of code) Results Over long periods of time, the number of known issues associated with a program tends to decline Defect density results were inconclusive

52 Towards a Better Understanding... - Results Internal quality In demonstrating the second law, the authors established that code complexity tends to increase in the test programs Authors' conclusion: the combination of mixed results in external quality, and the complexity issues resulting in decreased internal quality, means that Lehman's seventh law isn't confirmed for the test programs

53 Towards a Better Understanding... - Results VIII Feedback System Based on the concept of feedback in system dynamics In essence, the size of a system can be described in terms of the size of the previous release and the effort for that release While the details are more technical, the resulting idea is that system growth slows down over time, assuming that growth is inversely proportional to complexity

54 Towards a Better Understanding... - Results If this law were true, then the rate of growth of the test programs should decline over time. The author's results demonstrate that program growth rate is actually somewhat steady over time The spikes in the graph look like traditional feedback behavior, but the overall growth behavior contradicts expectations

55 Towards a Better Understanding... - Results Authors' conclusion: Lehman's eighth law isn't confirmed for the test applications

56 Towards a Better Understanding... - Other Observations The authors of the paper also discussed a few items not directly related to Lehman's laws Parallel evolution Many open source applications have points where they 'fork' into development and stable branches. The Linux kernel does this too Researchers have observed that growth of software on the stable branches tends to be linear, and that growth on development branches is super-linear Approximately quadratic in the case of Linux

57 Towards a Better Understanding... - Other Observations The two test programs that used parallel development (BIND and Samba) both demonstrate super-linear growth on their development branches, and generally linear growth on their stable branches

58 Towards a Better Understanding... - Other Observations Super-linear growth implies that development will require an ever-increasing amount of resources Some solutions suggested by other authors: splitting the project, moving parts into smaller satellite projects

59 Towards a Better Understanding... - Other Observations Distribution of changes 'hot spots' in the code that change frequently Frequently changed code may be due to poor design, and redesign may help facilitate future code evolution The authors found that SQLite, OpenSSH, Samba and Sendmail have a fairly large percentage of their changes occurring in a smaller number of functions. This implies the presence of more 'hot spots'.

60 Towards a Better Understanding... - Other Observations Interfaces versus Implementation The authors examined the ratio of changes to the code's interface versus the code's implementation A rapidly changing interface indicates that a system is young and actively evolving For all versions of a single program, the vast majority of changes are in implementation However, changes in interface are more common earlier in the development of the test programs

61 Towards a Better Understanding... - Final Words Threats to Validity I like to see authors actually attempt to explain where their reasoning may be flawed. Any other ideas they may have missed? Related work the authors did make an attempt to improve upon the work from previous authors, both by expanding the range of software tested, and by implementing new tools/metrics Unlike most of the previous work they looked at, their work attempts to confirm all eight laws, with various degrees of success

62 Final Words Understanding how code changes over time is important! You must understand what makes code evolve poorly before you can work to fix it. However, change itself is not a bad thing. Modern, sophisticated applications absolutely do, and absolutely must change to stay useful and relevant. This applies to all types of software, both open source and closed source

63 Final Words Open source software provides a useful playground to study software evolution and complexity However, commercially developed closed source software and OSS do have differences! Many of Lehman's laws of software evolution apply to open source software, even though they were designed with closed-source, commercial software in mind

64 The End Questions? Comments? Pitchforks? Torches?