Flash Augmented Reality on iPhone and Android ? Not so fast..

May 20th, 2011 by Slav

This is one of those articles where title pretty much says it all.

Now with AIR 2.6 iOS camera support and the unrestricted camera feed access to developers in iOS4, we can finaly build some cool AR apps on the mobiles right ?. Well… in theory at least!

The Setup :
To see how it would perform, I made a quick AR demo using Flar toolkit 2.5.5 for motion tracking and pattern recognision and Papervision 2.1 for the 3D scene. Scene consist of a simple flat shaded 3Ds model,  plus 4 interactive buttons. The whole scene has less then 170 polygons. A very lightweight scene with very low poly count. To make the conditions fair for both devices I kept the AR scene dimensions the same (hence it looks smaller on HTC Desire HD). Both apps use gpu rendering.

I exported the app into an iPhone format using latest available AIR SDK – 2.6. For comparison I also converted the flash project into Android format. I tested the iPhone version on iPhone 4 and Android version on HTC Desire HD.

The demo has 2 buttons that enable the user to turn off the motion tracking/pattern recognition – (FLAR ON/OFF), and scene 3D rendering. By turning off the 3D rendering, and seeing the changes in frame rate,  we can see how much resources are consumed only by the rendering of the 3D scene. By turning off the 3D rendering we can see how much resources are spent on the motion tracking.

Here’s the quick video showing the iPhone, and Android version

Framerate results comparison:


(Swf had frame rate set to 25fps)

And here are the lessons learned :

Lesson 1: Augmented reality can be implemented on mobile devices via flash conversion, just not in any usable manner

Although app will still run and function, the poor frame renders the application unusable. The problem is not just visual – stuttering screen playback, it’s also functional. If you have interactive elements in the scene such as buttons, low frame rate will make a click / touch detection difficult and user would have to try multiple times for a touch event to register.

Lesson 2: iOS conversion performance is superior to Android one.
Although both devices run on a similar hardware (both have 1GHz processor) and the apps were converted from the identical swf file – Android performance was on average worse. Also, while framerate on the iPhone seemed to be pretty stable, on the Android it was generally erratic and fluctuating.

Why is this happening?

I personally think it’s due to the way the apps are being converted and run on the device. While iPhone app is generated by converting the swf file into a native app, android app is being converted into an android air app compatible file and run by the Air runtime wrapper that has to be separately installed on the device.
That , in my opinion , creates a performance decrease , as it requires an extra hardware abstraction layer between the code and the device hardware. Think of it as flash file being run inside a flash plugin in the browser, versus OS native app. By design, swf app will never achieve the same performance as the native one.

The question here is , why does Adobe approaches Android conversion in this way. As for the iOS, it is quite clear why the iPhone apps are converted to native format. Apple wouldn’t let them do the Air wrapper approach. For the Android , they don’t need to however, yet they choose to do it. From the marketing point of view , it makes perfect sense of course. In the same way the spread of flash plugin helps to spread the entire platform and the tools (mainly the creative suite).
From the developers perspective it’s not that great though.

It means that every Android user , in order to run your app, will have to download AIR for android runtime. Although this helps to keep the file size relatively small (android version 451Kb vs 5.3MB iPhone version) the air runtime itself has 17 MB!
I would really like to hear justification from Adobe, as they were really conscious about increasing the file size of flash plugin (still less then 3MB) when almost everybody has broadband at home, yet they’re OK with forcing people to download 17MB on 2G/3G mobiles? I think they might be shooting themselves (and developers ) in the foot here.

The extra AIR Runtime file size issue is bad enough, but if you add an inferior and erratic performance to the mix there is really no reason to prefer conversion to air vs native app.

I am aware that performance in AIR 2.7 is better (there’s no Android Air runtime available at this moment so can’t test this personally) and with flash 11 and molehill we might see even better increase when it comes to GPU rendering.

However; there will always be applications that rely heavily on the cpu and for those, native app conversion will be always superior option.

Posted in AIR, Android, AS3, Augmented Reality, flash

34 Responses

  1. Dave

    Flash + AR sucks even on desktops, so it’s not surprising that it’s unusable on mobile devices. You might see a small performance improvement using Away3d lite.

    I wonder if molehill will help much, especially if you can offload the marker tracking to the new gpu accelerated pixel bender since that seems to be the biggest framerate killer.

  2. Neil Crookes

    Nice one Slav, a very thorough evaluation and write up. Good work. I hope Adobe see this and answer your question.

  3. Ferro Boyd

    I am curious to see the difference in performace if you had used Away3D or Away3D Lite. In almost all instances I believe it performs better.

  4. Slav

    Neil: Cheers. I’m hoping somebody from Adobe will react :)

    Ferro / Dave : whether I use away3D lite or not will not make a lot of a difference here, it is realtime pattern recognision and tracking that’s mostly dragging the framerate down.
    Notice that turning on the 3D caused only 1 frame decrease in framerate on iPhone.

  5. Austin

    I think it’s because natively 3d calculations are being done as straight up pixel programming. The Papervision library is not being converted to native GL calls thus the huge bottle neck. The native code is actually rendering all of the display logic instead of using hardware GL. Correct? Molehill will solve this Im sure for mobile whenever this occurs.

  6. Slav

    Austin: again if you look carefuly at the framerates table, the biggest bottle neck is Flar. Turnin Flar on reduces framerate to only 6 percent. Turning papervision on reduces framerate by only half.

  7. Ferro

    Are you familiar with Blitting? I know that the technique works very well with improving framerate for game animation. I wonder if the same principals can be applied to this to increase performance and decrease the memory footprint somehow. You can read a little about it here: http://www.adobe.com/devnet/flash/articles/blitting_mc.html

  8. Shawn

    You should test the iOS ones again in AIR 2.7, I’m seeing a solid 2-3x increase in performance.

  9. Slavomir

    Shawn: is AIR 2.7 SDK out yet ?

  10. Shawn

    Just via pre-release forums. I think it’s coming in June though…

  11. Anton

    Hi Slav I was wondering if you had time to test with the new AIR 2.7 SDK as the guys above suggested and if you have seen any difference? If you any exemple that I could download to test would be great! Thank you.

  12. Slavomir

    Anton, want to try today actually, wanted to use new FDT 4.5 mobile templates but there were a few issues..

  13. =*^_^*=

    maybe try alchemy?
    Flash really sucks on mobile if u wanna 3d go android native, opengl or something like that but not flash. flash is only 2d on mobile-forget it

  14. fideldonson

    Hi Slav
    Have you tried compiling with the 2.7 sdk yet?

    You should look into makc3d’s qtrack AR-tracker in my tests AIR AR on IOS is doable…

    Best regards

  15. tomph

    Hey there.

    This is a great early example of air on ios. Thanks for sharing it.

    I was wanting to know. Have you used the native CameraUI (air) component for this? If so, how did you overlay the camera image with graphics?

    I’m having trouble placing a layer over the CameraUI.

    Any help would be great. Thanks.

  16. Julian Patino

    Hi Slavomir ,

    Currently I have a problem because I need to do something similar with adobe air for IOS and AR.

    My problem is the superposition of elements and images on the camera in a similar way as you do in this example.

    It’s possible that you may share me the portion of source code that handles this task.

    I would appreciate endlessly.

    Best regards.

  17. Slavomir

    Since couple of you requested this, I have uploaded the source code for the core of the app to

    http://durej.com/download/src.zip

    (without papervision, flare and utils packages)

  18. FlashPad » Blog Archive » Mobile development with FDT5 – a disastrous start

    […] Air 3.2 gpu accelerated molehill capbilities on the iOS and android devices , perhaps revisit my Augmented Reality Demo so I can compare the performance […]

  19. Bloooom

    Hey Slavomir, I am trying to run your code, but I am not sure I know how to handle it :)

    Can you please give me the steps to do that? I already have the FLAR tookit, and I am running on Flash CS5.5 with AIR 3.2.

    Is your project FLEX or Flash? Please tell me how to proceed…

    Cheers,

    Jo.

  20. Slavomir Durej

    Jo, just the fact you have to ask me if is this is Flex or Flash, means you know too little about this and I won’t be able to help you.
    Try sites like gotoandlearn.com , but you might want to start with something easier then AR on mobiles..
    But fiy, I am using Flex SDK to compile this, my ide of choice is FDT, but flex or FlashDevelop will do fine as well.

  21. lala

    hi,
    wow,coincidence. i am exactly on same page with almost same conclusions.
    i had setup flartoolkit with papervision 3d scene, simple and it runs slow enough on my android (glaxy note). then i start searching and found this page.
    which version of flartoolkit you are using??
    Anyways, as i have been googling all day, here are my rough thoughts right now..
    1- there is flarmanager (modified version for ease of use and hopefully speedy)
    2- there is flartoolkit alchemy brach / version. i heard its much faster..
    3- papervision is not a choice to move on by all means

    i’ll try to test flartoolkit-alchemy branch with alternativa and try to post my results here..

  22. Slavomir Durej

    Lala.
    To save you some time here are some later findings of mine.
    AR on mobiles is very problematic as all modern packages use stage 3D. The problem with stage 3D it has to be rendered natively UNDER the video , not over it (as required) , so it makes the whole AR concept quite problematic to implement especialy on mobile phones. You basicaly have to stream a video feed from your camera to a texture that’s mapped on a 3D plane and place it under the scene, but there are issues with the texture sizes, and other things..
    I did some testing with Away 3D 4, and newest AIR SDK and there were quite a few issues with it, but haven’t tried Alternativa or Flare. Even tried in2ar (http://www.in2ar.com/) which works well on desktop, but again Away 3D was giving me grief about stencil and depth buffer on mobiles, and I basically just kept hitting a different walls (bug/error messages) and ran out of time.
    As a side note, in2ar, although very promising project, has an incredibly unrealistic price (almost 3000 EURos) I think Eugene although certainly a smart guy, but he’s being completely unreasonable to expect somebody to pay much more for a actionscript library then for Adobe Master Collection CS6.
    Such library would be typically used by a studio for a client that wants some fun facebook application involving AR, and budget’s from 5 – 15K , which makes the whole thing completely unaffordable.

  23. lala

    thanks for details Slavomir Durej.

  24. Alecia

    Fantastic goods from you, man. I have understand your stuff prior
    to and you’re simply too magnificent. I really like what you’ve got here, really like what
    you are saying and the way wherein you say it. You’re making it entertaining and you still care for to stay it sensible. I can’t wait to read much more from you.
    That is really a terrific site.

  25. Ferro

    Check out the performance on this AIR AR app running on android. It uses in2ar.

  26. Joe

    Here is a new Library for Flex to develop Mobile Augmented Reality: http://www.in2ar.com

    :)

  27. Slavomir Durej

    in2air is great , but they have a horribly expensive licence model. over 2000K for a commercial licence! No client we have ever worked with can afford to pay this much!
    So in practice (in real life) this library is unusable :(

  28. kredyt konsolidacyjny bez bik

    My family all the time say that I am wasting my time here at net, however I know I am getting familiarity everyday by reading such pleasant posts.

  29. whey protein

    It’s going to be ending of mine day, however before end I am reading this impressive piece of writing to increase my experience.

  30. Jay Martin

    Hi Slavomir

    I managed to get it to work on desktop web camera, now i am trying to publish the flar tool kit for mobile (IOS), im using flash, I have been looking around the web about replacing the desktop camera for a mobile camera (cameraUI). do you have any suggestions which I can try.

  31. fast payday loans

    Greetings! Quick question that’s entirely off topic. Do you know how to make your site mobile friendly? My site looks weird when browsing from my apple iphone. I’m trying to find a theme or
    plugin that might be able to correct this issue.

    If you have any suggestions, please share. Thanks!

    Look into my blog post; fast payday loans

  32. prestamos personales

    Do you have a spam problem on this blog; I also am a blogger, and I was wanting
    to know your situation; many of us have created some nice procedures and we are
    looking to swap strategies with others, why not shoot me an e-mail if interested.

  33. oprocentowanie kredytu samochodowego w mbanku

    Wspaniałagłębiа:-D

  34. download hack mobile

    I’m not sure if tyere was any neutralizer or any other type oof solution
    added to my hair att that time, I was too relaxed to notice
    much of anything since getting my hair done always makes me a zombie.

    “I Feel You” – Songs of Faith and Devotion This song iss a bit harsher than anything else Depeche
    Mode has ever done, which makes it distinct and
    interesting. Often, comfrey is used as a poultice or compress to heal
    wounds, but add this plant to your compost tea and you’ll have a
    high potassium fertilizer that caan replace the traditional seaweed fertilizers used on tomatoes.

    Feel free to visit my weblog: download hack mobile