Author Topic: Upstream Synchronization Strategy  (Read 6436 times)

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Upstream Synchronization Strategy
« on: January 20, 2014, 09:36:34 AM »
I'd like to take the lead getting our stuff synched back upstream over the next few weeks.  I thought I'd get the discussion going here on a few strategic matters.  There has also been some discussion in the upstream channels about libretro-mupen synchronization.

1. In case anyone's not aware, the upstream source has been migrated to github, and now contains some additional video and rsp plugins, e.g. the ones wahrhaft was maintaining.  This makes it easier than ever now to sync with upstream.

2.  Paul and I formed the mupen64plus-ae organization on github.  The idea is that the organization may be easier to find, and we can collect all mupen-ae related business under one place.  This is where we would house our forks of the various upstream repositories.  Right now I'm using it as a sandbox to test and verify some git mechanics, so don't put any stock into it for now (i.e. don't submit pull requests).

3. I did some experimentation and discovered that it is feasible to move the plugins outside the JNI folder and still get proper Android builds.  It only required a few changes to jni/Android.mk.  So organizing it something like this will work, if that's what we want:

    - (new repository root)
        - mupen64plus-ae
            - (contents of current repository, minus the jni subdirectories listed below)
        - mupen64plus-audio-sdl
            - (contents of current jni/audio-sdl)
        - mupen64plus-core
            - (contents of current jni/core)
        - mupen64plus-input-android
            - (contents of current jni/input-android)
        - mupen64plus-rsp-hle
            - (contents of current jni/rsp-hle)
        - mupen64plus-video-glide64mk2
            - (contents of current jni/gles2glide64)
        - mupen64plus-video-gln64
            - (contents of current jni/gles2n64)
        - mupen64plus-video-rice
            - (contents of current jni/gles2rice)

4. For best upstream synching, we should maintain our plugins as separate, independent repositories, each a fork of its upstream parent.  So we'd have a git repo for mupen64plus-audio-sdl, another for mupen64plus-core, etc.  We should try to keep the master branch as clean as possible to make it easy to diff/merge/cherry-pick between up- and downstream.  We could generally push new changes to branches, then clean them up before putting them over to master.

5. This then leads to a big question: What do we keep in the mupen64plus-ae repository?  The options that I can imagine are
    A. Keep a copy of plugin source in the ae repository.  This is what we're doing now.  The only thing we'd change is the upstream location in the git-subtree pull script (see tools folder of current repo).
    B. Remove all plugin source from the ae repository.  The user would clone all the mupen64plus-ae organization's repos into a common directory, then ndk-build from the mupen64plus-ae subdirectory.
    C. Same as B, but add the compiled binaries to the ae repository.  That would make it much easier to traverse the git history for bisecting and so forth.  The right plugin version would be associated with each commit in the ae repo.
    D. Use git-submodule.  I'm only saying it for completeness.  Everything I read about it is that it's a nightmare to use and requires all devs to be git gurus.

I personally feel that git history traversal with the correct plugin versions is very important, so I favor options A and C.  I think I lean towards option A since it keeps things simple (no change to current practice, and we wouldn't even need to reorganize directories as in item 3) and makes it easy for new devs to jump in and add fixes.  I personally don't mind if the commits are messy, because I can always clean them up before synching back our own plugin fork.  That would keep our fork repos clean for easy upstream synchronization and reuse by other projects (e.g. mupen64plus-pandora).  To maintain cleanliness, we might consider giving only a few devs write access to the plugin forks, but support pull requests.
« Last Edit: January 20, 2014, 09:39:21 AM by littleguy »
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #1 on: January 20, 2014, 09:54:15 AM »
I like option C in the sense that it greatly reduces build times when doing a git bisect.  Problem I see is that although things like the RSP plugin would change infrequently, the video plugins are the focus of constant change, which means the size of those repositories would grow enormous over time.

So I think I am leaning more toward option A.  Changes and branches off the mupen64plus-ae repository would have all the plugins' source included.  Then when changes to a video plugin are ready to submit upstream for discussion, we can merge those changes first into our repositories, and making pull requests for upstream pretty simple.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #2 on: January 21, 2014, 07:42:59 PM »
Not sure if anyone else will weigh in, but I suppose we could go ahead with A since it doesn't really change anything for most devs.

Would you mind transferring ownership of the paulscode/acra* repositories over to the mupen64plus-ae organization?  At some point we'll want to do it for the main repo too.  Not sure how big a deal it would be to most people... just changing a url in a config file to point to the new one.  Maybe change it over before F-Droid tries their next pull or something.

edit: There's a bit of a lull for me in the version 3.0 implementation so I'll probably spend the next few days getting most of the plugins back in synch, and start pushing clean versions to our official forks.
« Last Edit: January 21, 2014, 07:45:15 PM by littleguy »
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #3 on: January 21, 2014, 09:01:38 PM »
Also, would anyone mind if we changed the ./jni/ subdirectories to match their upstream names?  e.g. front-end -> mupen64plus-ui-console
« Last Edit: January 23, 2014, 09:20:51 PM by littleguy »
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #4 on: January 23, 2014, 09:23:50 PM »
Ok, four repos synced, two to go (video plugins).  The diff between upstream and downstream for audio, core, rsp, ui-console is pretty minimal, as you can see here:

https://github.com/mupen64plus/mupen64plus-audio-sdl/network
https://github.com/mupen64plus/mupen64plus-core/network
https://github.com/mupen64plus/mupen64plus-rsp-hle/network
https://github.com/mupen64plus/mupen64plus-ui-console/network

On to rice, then glide.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #5 on: January 24, 2014, 07:47:37 AM »
Would you mind transferring ownership of the paulscode/acra* repositories over to the mupen64plus-ae organization?
Sure, sorry for not responding yesterday.  Busy with work again.  There is a big reorganization in the works over here, and I'm hoping to solidify my indispensability/ competitiveness in case a FTE position opens up.

There's a bit of a lull for me in the version 3.0 implementation so I'll probably spend the next few days getting most of the plugins back in synch, and start pushing clean versions to our official forks.
Excellent, thanks for taking on the challenge  :)

Also, would anyone mind if we changed the ./jni/ subdirectories to match their upstream names?  e.g. front-end -> mupen64plus-ui-console
No objections on my end.  Looks like we are heading in a really good direction here IMO.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #6 on: January 24, 2014, 12:26:10 PM »
I'm going to split out the gles2n64 plugin and put it in a new repository in the mupen64plus-ae organization, so that it's easy for other embedded ports to pull it.  (Also did this for input-android fwiw.)  In principle, upstream could one day take ownership, if they ever wanted it.

Question: Which name should we use?
  mupen64plus-video-gles2n64
  mupen64plus-video-gln64

I'm inclined to say the latter so that its heritage is clear, but I thought I'd ask first.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #7 on: January 25, 2014, 10:33:56 AM »
Yes, I'd go with gln64.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #8 on: January 25, 2014, 10:51:59 AM »
Awesome, I'll take care of that as part of the sync.

Any thoughts on the best time to transfer ownership of the acra* and mupen64plus-ae repositories?  It might be kind of nice to do it all as part of this one-time migration/re-organization.  Then there's sort of one mental epoch.

In case you've never done it, you just go into the "danger zone" of the repository's settings.  I tested it on the input-android plugin (moved from owner littleguy77) and it was trivial.  The difference between transferring and forking is that the Issues, Wiki, comments, etc. are migrated.  Devs will need to change their origin URL, but that should be obvious next time they try to fetch or push (they'll get an error saying the paulscode repo doesn't exist).  If we're worried about that, I've seen some groups fork the new location back to the old location.  Then in the old location they restrict push permissions and change the front page readme to tell devs where the new location is.

Edit: actually forking back might not be a bad idea, we could use the paulscode location for the 2.x branch of development, and the new location for the 3.x branch of development.

« Last Edit: January 25, 2014, 10:55:00 AM by littleguy »
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #9 on: January 25, 2014, 12:59:03 PM »
Ok, i'll do that this evening when I get home from work.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #10 on: January 26, 2014, 08:52:14 AM »
Transfer of the repository ownership to the mupen64plus-ae organization is complete.  Once 3.0 is ready to publish, I'll delete my fork to avoid confusion.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #11 on: January 26, 2014, 09:27:11 AM »
Sounds good.  Don't forget acra-storage as well.  We had a few customizations in there.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Upstream Synchronization Strategy
« Reply #12 on: January 26, 2014, 01:04:42 PM »
Oops, missed acra-storage.  Its transferred now as well.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Upstream Synchronization Strategy
« Reply #13 on: January 26, 2014, 01:12:42 PM »
Thanks.

I made a branch to rename a bunch of stuff, and will do some testing on it before I push to master.  Just mentioning it so there's no duplication of effort with any other devs.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version