Author Topic: Video sizing implementation  (Read 6442 times)

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Video sizing implementation
« on: May 17, 2013, 01:04:50 PM »
Been curious about this for a long time, so I did some quick experimentation.  We can do all the video sizing operations in Java, during the initialization of the GameSurface.  All we need to do is set the layout parameters of the GameSurface view (size and position) and pass the size values accordingly through the standard config files.  All video plugins would be hard-coded to stretch, which would simply fill the GameSurface at whatever size it's defined in Java.  The benefits would be
  • Easy way to implement screen position (top/middle/bottom) for glide
  • Easy way to fix the junk graphics on the side in Rice on some devices
  • Reduces the amount of custom code we add to the video plugins and simplifies ae-bridge
  • Would be easy to extend the video preferences to allow more customization

For example, the video options might look like this:
  • Screen orientation
    • Landscape (default)
    • Reverse landscape
    • Portrait
    • Reverse portrait
  • Screen position
    • Top-left
    • Top-center (default)
    • Top-right
    • Middle-left
    • Middle-center
    • Middle-right
    • Bottom-left
    • Bottom-center
    • Bottom-right
  • Screen size
    • Maintain aspect ratio (default)
    • Fill screen
    • 800x600
    • 640x480
    • 480x360
    • 320x240

Has the Java-side video sizing been explored already?  Does this have any known drawbacks?
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #1 on: May 17, 2013, 01:20:56 PM »
It should be more efficient on the native side if we do the resizing in Java, because the frame being rendered on will always be same dimensions as the frame being drawn (I recall seeing at least for gles2n64 an optimization built in for this case), but of course any savings are then cancelled out on the Java side when it draws to the screen.  In theory it should work out to be the same as far as speed.  I say in theory, because Android is notoriously slow at certain drawing processes - we'll have to run tests on lots of devices to know what the net effect on performance is.

As far as the options (I know they were only a quick example), the only concern I have there is what you listed for screen position and screen size.  The way it is there seems to imply the frame would be the specified size (a very small 320x240 frame for example) floated to the specified screen position.  It seems to me that "Maintain aspect ratio" and "Fill screen" should be in their own category.  The "Screen size" category should also include something like "full resolution" and "half resolution" in addition to the N64 native resolutions.

Another good benefit of this method is it would solve the mathematical headache I was having by trying to implement the resolution selector for when screen stretch was not enabled on the native side.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #2 on: May 17, 2013, 01:24:16 PM »
Also, the screen position options could be just "Top/Left", "Center", and "Bottom/Right", since the frame would always be stretched to either full width or full height (depending on if the device was in landscape or portrait mode)
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #3 on: May 17, 2013, 01:33:43 PM »
Thanks for the input.  You're right, that was a quick example.  But I should clarify what I mean if a user chose Landscape-Top-Right-640x480 -- I attached an actual screenshot of this using the Java-side approach.  (On my screen you can actually see the game properly inside the box, but something about doing it this way must screw up the built-in screenshot feature of android.)

2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #4 on: May 17, 2013, 01:48:30 PM »
I guess I just don't see a floating small game-frame as being a very useful feature.  I think the majority of users who want the option to change the resolution (i.e. OUYA users) are doing it for performance reasons, but still want the game to fill the screen (with the option of maintaining aspect ratio).  That said, it is great to have the flexibility to be able to do something like what you have there (maybe for custom controls and what-not).  Perhaps there is a way to give folks these options without making the settings become too complex to figure out (really need a good term that distinguishes between an literal 640x480 frame vs a stretched 640x480 frame).
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #5 on: May 17, 2013, 01:49:53 PM »
Oh, I just got what you meant -- Top-Right means the frame is stretched to the Top-Right quadrant.  That's even a third option, LOL.  The customization potential is quite large :)
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #6 on: May 17, 2013, 02:03:53 PM »
I guess I don't know what a "stretched" 640x480 frame is.  Is that where the native code renders to a 640x480 context, and then android (or the connected HDTV) stretches the image to fill the physical screen?

All I've been talking about is a small rendering window that floats over the screen, with the thought that such a configuration might yield FPS improvements.  Where it floats to would be defined by the position preference.  Not sure how many people would use the left or right alignment ("gravity" in android parlance), but it would be trivial to add.

I would actually use an undersized render window if it made a slow game playable (I'm thinking a game that requires glide) or allowed me to get past a "problem area" of a glitchy game.  Would be pretty tough on a smartphone, but on a 7" tablet 640x480 is actually quite playable.  But anyhow, you're right, my main point here is that it would be super easy to add such flexibility; how much of that flexibility we use would just be up to what the users want.

2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #7 on: May 17, 2013, 03:16:17 PM »
I guess I don't know what a "stretched" 640x480 frame is.  Is that where the native code renders to a 640x480 context, and then android (or the connected HDTV) stretches the image to fill the physical screen?

Correct, like I have in the paulscode/lowres branch on github
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #8 on: May 17, 2013, 03:29:20 PM »
I guess I don't know what a "stretched" 640x480 frame is.  Is that where the native code renders to a 640x480 context, and then android (or the connected HDTV) stretches the image to fill the physical screen?

Correct, like I have in the paulscode/lowres branch on github

If there are performance gains and a demand for that, then I guess I'd call it "subsampling" and probably wouldn't state the pixel count for those configurations.  Probably just call it "Subsampling mode xxx" or "Subsampling 1/2x" sort of like in anti-aliasing when you see "Supersampling 2x"....  But yeah it could get confusing.

I'm going to try this alternate implementation on another branch and see if I can get it nailed down.  The graphics stuff is pretty easy, mostly just config file syncing and run-time layout adjustment.  But there are some touchscreen control issues that show up if you do it this way since touches are currently only registered if you touch inside the GameSurface.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #9 on: May 17, 2013, 04:57:14 PM »
If there are performance gains and a demand for that, then I guess I'd call it "subsampling" and probably wouldn't state the pixel count for those configurations.  Probably just call it "Subsampling mode xxx" or "Subsampling 1/2x" sort of like in anti-aliasing when you see "Supersampling 2x"....  But yeah it could get confusing.

I was thinking of using more descriptive terms -- not sure folks will associate the word subsampling with the behavior (although there are plenty of other such terms in there, LOL)  I feel like somehow the word "stretch" should be used.  So you select the "resolution" for one setting, then have a "stretch frame" setting that determines if the frame will be stretched to fit the screen or a small floating frame, and finally a third setting ("fill screen", or "maintain aspect ratio" or something?) for what we are currently calling stretch screen.
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #10 on: May 17, 2013, 05:00:34 PM »
LOL I'll go with whatever.  For now I'll just put some verbose placeholder strings in and we can figure out the text/features later.

Edit: Yeah, I think multiple prefs arranged like you say would make more sense in that case...
« Last Edit: May 17, 2013, 05:02:14 PM by littleguy »
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #11 on: May 17, 2013, 05:16:21 PM »
How about "zoom"?  Stretch might connote distortion, especially since that's how it's currently used.  Zoom is common on HDTV settings, websites, powerpoint, etc.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3496
  • Developer
    • View Profile
    • PaulsCode.Com
Re: Video sizing implementation
« Reply #12 on: May 17, 2013, 05:21:10 PM »
Excellent, you're better at this than I am, haha
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
RAM: 1 GB
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
RAM: 2GB
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #13 on: May 20, 2013, 09:11:45 AM »
For anyone watching this thread, I posted the changes to a new branch on github.

The main bug right now is the touchscreen controls when non-native or non-stretch is used.  It's probably not hard to fix, I just haven't had the time yet.  One thing I noticed, using the xperia play, is that the bug is only apparent with the vanilla Activity-based version of GameActivity.  Using the NativeActivity-based version seems to resolve the bug.  So that might point directly to the bugfix.  They are handling touch input through slightly different pathways.
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version

Offline littleguy

  • Moderator
  • double
  • *****
  • Posts: 1945
    • View Profile
Re: Video sizing implementation
« Reply #14 on: May 20, 2013, 06:14:06 PM »
The touch control stuff is fixed now and the branch is pretty much done except for two things.  Need to add some logic for PAL aspect ratio (11:9) and need to add the zoom feature for lower resolutions.

One question though: Does anyone really need the "stretch screen" option anymore?  I'm not talking about "zoom" where you scale say a 640x480 up to the maximum size and keep the aspect ratio (adding black bars on top&bottom or left&right).  I'm talking about "stretch" where the aspect ratio gets distorted and there are no black bars anywhere.  The reason I ask is because I could keep stretch option and add zoom, but that leads to a lot of combinations and is probably unnecessarily complicated for the vast majority of people.  For example, someone could zoom and stretch a 640x480 resolution to fill a 1080p display (which would be both pixellated and "squashed").  Would be much easier if the only option were zoom (and it was disabled if the person selected native resolution).
2012 Nexus 7, rooted stock Lollipop
Samsung Galaxy Victory, rooted stock Jelly Bean
Xperia PLAY, stock Gingerbread
OUYA, retail version