Code Merge Browser

[tc]

Hello,

I'd like to explore what can be done in a reasonable amount of time. My goal is to have something that is usable and functional and once everyone is confident that it is heading in the right direction, features can be added/refined. Here are some points:

~~~~~~~~
RELEASE
~~~~~~~~

It would be nice to have it in a release but if not, it can be handled through http://www.vastgoodies.com/. However, whatever is uploaded to http://www.vastgoodies.com/, I'd like it to be correct.


~~~~
GUI
~~~~

All of this is mainly about the GUI. Talking to different folks has resulted in the whole gamut, from a MED type browser, to one like Pharo, to 'it should be done in Seaside'. So, here are some basics (from a Windows perspective. Unix isn't forgotten since a signicant amount of work is being done but it isn't ready yet):

1. The browser will be done as a GUI, not in Seaside.

2. The widgets available are 'WIN95' widgets. I use that designation because the MS SDK header files use it in a '#define'. This means there won't be fancy widgets because the ST product uses what is available from the OS, not third party libraries.


~~~~~~~~~~~
SCENARIOS
~~~~~~~~~~~

A user opens a browser, on the left side is what is available in the image. On the right side are apps/classes/methods from either the image or the repository. The goal is to merge code from several different methods on the right side into a selected method on the left.

Issue: How to present the classes and methods. Other environments have an easier time because it requires two list boxes, a 'Change Set' list and what classes are in the change set. With Envy, for the repo, six list boxes are required, 'App/Edition' -> 'Class/Edition' -> 'Method/Edition'.

Possible Solution: Use a list box or table that expands, the expansion would be:

app hierarchy.JPG (3.74 KiB) Viewed 235 times

. . . but any one of those should be expandable to all its editions. The app would have its editions under it, the same for the class, etc.

app hierarchy II.JPG (20.91 KiB) Viewed 235 times

The user would select the 'App', expand it, select 'Version 1', expand it, select 'version 2' of the class, expand the methods, select 'edition 3' of the method. The asterisks indicate what is released. The above list would be in a hierarchical table or list.

The left side of the screen would be a little easier since it represents what is currently in the image. The hierarchy would still be there but not all the editions and versions as shown above. The reason is code can only be moved into classes and methods loaded in the image.

The above could be split into two list boxes. The topmost box has the apps and classes and the hierarchical box underneath has the method hierarchy. Underneath that is the text pane.


~~~~~~~~~~~~~~~~~~~~~~~~~~
HIGHLIGHTING DIFFERENCES
~~~~~~~~~~~~~~~~~~~~~~~~~~

This is a tough one, because for the initial release, I won't be trying for graphics, i.e. draw a box around text on the left side (or an underline) and expand it out to a large shaded paren going to the text on the right like Apple's XCode browser. I'm looking for suggestions on that.


~~~~~~~~~~~~~~~~~
OTHER SCENARIOS
~~~~~~~~~~~~~~~~~

- Both sides of the window show what is in the image.
- Both sides show what is in the repo. No code can be moved though.
- If 'image' components are being shown, switch to showing classes/methods only, no apps.


Suggestions welcome.

Thanks.

--tc

[jtuchel]

Hi TC,

great to hear you are working on an improved code merge tool. I hope I can help a little by commenting, and I am sure other users will come up with even more feedback.

First of all, I think not going for a Seaside based solution is a good idea. We'd lock out many users from a new tool. For now, VAST is a local, GUI-based development tool. I guess the Win95 part will not be desirable for several users on Linux and AIX or Solaris. So if there is a chance to go with Extended Widgets alone and keep it portable, it's sure better to make adjustments to existing text widgets rather than limiting the changes tool to Windows.

First of all the highlighting of differences:
--------------------------------------------------

I think that highlighting only the changed characters and using colors (either by using text selection or the text color itself) is sufficient. It should be easy to understand. When comparing the image on the left and the repo version on the right, I should be able to find additions, removals and changes in the image as compared to the repo. (or, maybe even switchable between which side the differences should be highlighted.) So inserted text should be highlighted (or printed) in green, removals in red and changes in blue (or whatever). Maybe a little marker at the left border of the edit pane (where we usually see breakpoints) which marks all lines with relevant changes would also help when skimming thorugh longer methods.

The coloured areas that show longer passages between left and right would be even better, but maybe the added value as compared to color highlighting of the text is much smaller than one might think.

Another Idea: if there is an insertion of multiple lines, the area where the insertion is could be left blank in the older version, so that the unchanged portion is clearly visible. The same goes for removals

Let me try to clarify it a little: The left text is my changed method in the image, and the right one is the older edition we compare with. I have trouble formatting this in any useful way with this editor here, but I guess you get the message.

Code: Select all

This is unchanged text which has not been      This is unchanged text which has not been
touched in the method                                       touched in the method

This is an insertion of two lines which has been
changed in my image (printed in green)

                                                                        This text was removed in the newer edition
                                                                        in the image and is printed in red here

And from here on there are no more changes.      And from here on there are no more changes.

I must say I hardly ever merge code from several methods or editions into one, so I am not in a position to say much about that. The way we currently compare class/app/map editions from the navigation point of view seems reasonable to me and doesn't require too much change in the first place. Sometimes I wish I could switch between editions in the library to see the history of changes, so a drop-down-box for method editions may or may not be helpful here...

One thing I really would like to see improved is the merging actions. After several years of using VAST, I still find "Load alternatives" and "Release loadad" confusing and un-intuitive. And I think it would be nice to really merge only parts of editions, meaning accepting some chnages/editions/removals and forgetting others. So when you go thorugh the differences, I'd like to have a button to either accept a change or leave it out of a new edition. The Browser should then update and show this resolved difference as equal between both editions.

Not being a native english speaker, there is a good chance I couldn't formulate my thoughts in an understandable way, so please feel free to ask if you think there might be something to what I wrote but cannot really understand.

Joachim

[marten]

First of all, I think not going for a Seaside based solution is a good idea. We'd lock out many users from a new tool. For now, VAST is a local, GUI-based development tool. I guess the Win95 part will not be desirable for several users on Linux and AIX or Solaris. So if there is a chance to go with Extended Widgets alone and keep it portable, it's sure better to make adjustments to existing text widgets rather than limiting the changes tool to Windows.

I would like to see a fancy GUI - but that would take time and using only the base OS stuff ( ;-() Ok, that's a product strategy - but I'm happy, that even Microsoft does not do this with its VS2010.

One thing I really would like to see improved is the merging actions. After several years of using VAST, I still find "Load alternatives" and "Release loadad" confusing and un-intuitive. And I think it would be nice to really merge only parts of editions, meaning accepting some chnages/editions/removals and forgetting others. So when you go thorugh the differences, I'd like to have a button to either accept a change or leave it out of a new edition. The Browser should then update and show this resolved difference as equal between both editions.
Joachim

I also have problems with that. What is the alternative and offering me the timestamp of the edition is quite nice, but I rather would like so see the application version or class version number. I am always confused with it. Also showing which text windows contains the alternative version is pretty needed.

[tc]

Hello,

This is an example of me experimenting, but it doesn't do the job since Envy has 'apps & editions', 'classes & editions', 'methods & editions'. Putting six list boxes on a screen, 12 total (six for each side), clutters it up too much. Pharo has a much easier time since it is 'change set & classes'. This organizing of screen real estate into something useful but not cluttered is a 'sticking' point.

merge browser sm.jpg (95.41 KiB) Viewed 214 times

--tc

[daswartz]

This is an example of me experimenting, but it doesn't do the job since Envy has 'apps & editions', 'classes & editions', 'methods & editions'. Putting six list boxes on a screen, 12 total (six for each side), clutters it up too much. Pharo has a much easier time since it is 'change set & classes'. This organizing of screen real estate into something useful but not cluttered is a 'sticking' point.

How about taking a different approach for the select what to merge (top) portion of the browser?

Using your example of an "Application" compare/merge: Three columns. First column shows list of application editions/versions. I select two of them. The second column fills with the list of classes that are different between the app versions. based on the version of the class that is released into the selected app editions. Choose a class from column two, and the list of differing methods between those editions of the class shows in column three. Choose a method to display differences in the text panes.

If this is a "Class history" compare/merge browser, two columns: Class edition/versions, "released" method.

if this is a "Method history" compare/merge browser, one column of method editions to choose from.

If this is a compare/merge methods with different names - can't picture how to make it work, yet.

Merge features are only enabled when one of the selected "things" is what is currently in the image.

Since the source of the two things to be compared is multiple selections from the leftmost column, there needs to be an indicator to tie the selected items to the text panels below. Perhaps a color annotation which sets the color of the header above the text panel to match the color of a graphical annotation in the selection list.

I think this approach would fit my own uses, anyway, and doesn't require huge real estate.

It might even extend to a three-way merge/compare: select three things in the left column, get three text panels below.

Doug Swartz

[jtuchel]

Doug, Taylor,

to me this is way too theoretical. How often do you use a changes Browser or diff tool and switch between editions of classes, apps and whatnot, just like it were sunday afternoon and you have some time to take a stroll along envy's shopping windows...

Most of the time, I am doing one of two things:

* See what changes have been made to one method or a set of methods over a certain period of time
* Do my best to merge changes in different branches of a methods history into a new one

Most of the times, the context of what to compare comes from outside the actual changes/merge tool. And I think that is a good thing. When I'm down to the method editions to compare, I want to merely concentrate on that. It's hard enough and needs almost all my brain. All that selection stuff is another level that I've dug through top get here, now I want to fully concentrate on the job.

So the tool at hand should provide two things, and only these two:

* An easy and obvious way to detect changes between two method editions
* An easy and obvious way to merge different changes of a method into one current edition - including undo and a preview of the result

I'd vote for as little choice of editions/classes/apps/maps as possible in a diff/merge tool.
I'm okay with calling the merge tool from an editions list and getting a new merge browser. It's more important for me to understand what's on the left and what's on the right, and maybe I even want to change what's left and what's right within that Merge tool.

I like the way the existing changes Browser builds a stack of "Merges in my Todo-List", e.g. when I do a "Browse Changes between two map editions including required Maps". In such a case, I am not really interested if there are possibly a ton of even more editions of everything that I need to compare. Going through 113 changes between two map editions is enough already.

I am not looking for the "universal compare anything tool", but for a Browser that presents me two or three editions of a method or class or application or map that I have defined in an outer context. And when I'm down to that, I need tool support that makes the job easy.

So I'd vote for a tool that does one thing in the best possible way: merge two method editions into another one with as little effort as possible. The way to find and determine these two method editions does not necessarily belong into the tool.

Or, to put it in a more positive way: once we have a tool to merge code easily, we can have a look at the next higher level and see how we can extend it to support certain workflows around it. In an OO design way, the code merge tool can be reused in many contexts. If it has a clean API that says: compare these two editions of this method and let the user merge changes of both into the currently open edition of the method, then we can embed it in any kind of context and build great tools around it that add loads of value. Bottom-Up or Composition is the key here.

An over-engineered tool that cares too much about what could possibly ever be compared and how many permutations envy could ever surprise us with will probably cost a lot of money and effort and be of no help at all. Whatever context we come from: we are talking about editions of a method or class definition (and in a wider context: piece of text like XML, JSON or whatever file).

Sorry if this sounds negative or arrogant, it's not meant that way. I just want to make a very clear point of where I think priorities should be: *identifying* and *merging* changes of code artefacts. It's the basic building block for all possible ways to use the functionality.

Joachim

[tc]

Hello,

If I had my druthers, I'd prefer to improve something already there.

So I'd vote for a tool that does one thing in the best possible way: merge two method editions into another one with as little effort as possible.

Perhaps add a third pane to the changes browser? Or add merge capabilities to the current one like XCode, or two text panes (like now) with merging, or ?

--tc

[jtuchel]

Hi again,

Ideas evolve, and sometimes you first need to find the base from where to build them up on.

So let's start at the very bottom. What are the jobs we want to achieve with a changes tool?
a) understand the changes between two editions of a method (I'll completely ignore anything else for now)
b) merge text from one or up to two editions of a method into a current third one

Maybe we should start by looking at these two jobs separately and later see where we see commonalities that help.

So what's needed for Job a) understand changes?
First of all, we need a way to select two editons to compare, if these are not provided by the context from which the diff browser was called.
This could be a very simple chronological list of editions, latest editions on top, oldest at the bottom of the list, like we know it already.
Then we need two text panes, side by side.
And we need a way to pick two editions from the list (if it is there) and to determine whether the edition should be shown in the left or right pane. I guess the two best methods for this would be drag&drop and a context menu. There should also be a button to switch the two (As an example, look at how google maps lets you swap an address between start and destination when you want it to calculate a root from point a to b).
In the editions list, there should be an icon that signals what side the edition is shown on (a simple left/right arrow will be good enough).
I've already posted a few ideas of what could be done to mark additions/changes/removals. Text color (or text background color) and blank space for complete lines that were removed or added will surely do at first.
I think I haven't mentioned anything yet that would be hard to implement with existing widgets and techniques.
This can be accompanied by a drop down-list above/below both panes that also show the current edition and in which you can change the edition to show.

A vertical scrollbar should always scroll both methods in sync, while I am not sure what horizontal scroll bars should do... So in fact each pane could have its individual scrollbars, but they'd stay in synch (maybe that could be a preference that can be changed - I don't care)

In this scenario, there is no editing of code or loading/moving of code, we can just load one of the shown editions

And what about merging - Job b)?
Merging is a real monster.

Maybe we could start with merging one edition with the loaded edition, and ignore merging two editions from two branches into a new one for the moment. This would work the same way as diffing (Job a) ) and it is possibkly even enough if we use one branch as the loaded edition into which to merge stuff from another (most CVS and SVN based tools offer no more than that anyways).

In that scenario we could simply accept / ignore each individual change by using a context menu item and/or a Button which indicates we're moving a change from (,say,) the left pane to the right one. Plus, we need an undo option that can go back step by step. The coloring would update with every accept/ignore setting (and we should possibly color ignored changes and mark them as ignored - maybe using some shade of grey)

This way, the right pane would be the preview for the merging.

For me it would be completely okay to not be able to edit the loaded edition by hand.


So these are my ideas for a starting point. What do others think?

Joachim

[daswartz]

Joachim,

My earlier comments were on "How to select the two, or three, methods to compare and merge". While I think we need a better method to manage that side of the common cases of "merge two versions of an app, or two versions of a class", I certainly have no issue with focusing on the "bottom" of the screen first.

I like most of your ideas of how to make the code display and merge better. It's reasonable to only make changes to the loaded edition of a method. Undo is important.

The only idea I'm not comfortable with is the one to put a selection drop-down above each comparison pane to change which edition of the method I'm comparing. I would almost always use this tool in a larger context, ie, the merge of two different application or class versions. So, easily changing which editions of an individual method I'm comparing would almost never be needed and might be undesirable

One other note: I haven't looked at what algorithm the current changes browser uses. But, we might want to enhance it. While smalltalk methods are usually short enough that a simplistic compare works OK, I've certainly seen cases where the current algorithm doesn't seem to do a very good job.

Doug swartz

[tc]

I haven't looked at what algorithm the current changes browser uses. But, we might want to enhance it.

I can help with that. In an internal meeting, it was pointed out that the changes browser does a string compare. That appears to be true. It would be better to use EsParser. That has some issues, like, it takes blocks as a whole and some kind of recursion would be needed to go deeper. However, my initial goal is to have something that at least heads in the right direction, like the interface and use EsParser and the other issues can be dealt with later (like recursion for blocks).

--tc

[jtuchel]

I can help with that. In an internal meeting, it was pointed out that the changes browser does a string compare. That appears to be true.
--tc

As long as we're mostly talking of changed text, added characters and maybe added/removed lines, a String compare will surely get us quite far, even if a Parse Tree is even better...

Joachim,

My earlier comments were on "How to select the two, or three, methods to compare and merge". While I think we need a better method to manage that side of the common cases of "merge two versions of an app, or two versions of a class", I certainly have no issue with focusing on the "bottom" of the screen first.

I like most of your ideas of how to make the code display and merge better. It's reasonable to only make changes to the loaded edition of a method. Undo is important.

Thanks, Doug, good to hear others share my ideas. So I'm not too far off

The only idea I'm not comfortable with is the one to put a selection drop-down above each comparison pane to change which edition of the method I'm comparing. I would almost always use this tool in a larger context, ie, the merge of two different application or class versions. So, easily changing which editions of an individual method I'm comparing would almost never be needed and might be undesirable

I could live without the drop down box, but depending on what exactly I try to do, some kind of timeline (a vertical descendedt list of editions is good imo) to understand the evolution of code can be helpful. Of course necessarily not in the case where I need to compare or merge two defined editions of a method. But sometimes I need to know when a bug was introduced, and would like to go back in time step by step on one side of the comparison. So this is probably a third use case. But I have no strong feelings about the drop-downs. I'd rather see the merge tool start simple and evolve with ideas that come up when it's in use. That's why I like the bottom-up idea so much.

Joachim