[MacRuby-devel] [ANN] AXElements: UI Automation with MacRuby

[Brice Ruth]

I followed the steps in the radar below to activate VO and had the same
results as described. Interesting to see which of my non-Apple menu items
are using the private API ;-)

SwitchResX
MenuMeters

And those that are using the public API

Quicksilver
pyTivoX
Jumpcut
gfxCardStatus
vmware
dropbox
(mine)

The workaround listed in the radar talks about associating a global shortcut
with a NSStatusItem. Does anyone know how to do that? And for AX - can that
be accessed then via the global shortcut?

Thanks!

Brice Ruth, FCD
Software Engineer, Madison WI


On Thu, Sep 29, 2011 at 7:40 PM, Brice Ruth <bdruth at gmail.com> wrote:

> It appears that there's a known bug in the public NSStatusBar /
> NSStatusItem API that prevents these from being selected by the
> Accessibility API. At least that's what I'm assuming - see here:
>
> http://openradar.appspot.com/6832098
>
> Granted, this is from 2009, but I'm actually unable to access my
> application as you indicated below .. I can access the volume control, but
> when I try to find mine, I can't ... is there any way to list all the
> menu_extra descriptions that AX can see? That way I can confirm that I'm not
> just going after the wrong description.
>
> Thanks!
>
> Brice Ruth, FCD
> Software Engineer, Madison WI
>
>
> On Thu, Sep 29, 2011 at 11:47 AM, Mark Rada <mrada at marketcircle.com>wrote:
>
>> Whether or not an app is supported depends on if it implements the
>> accessibility protocol. All Cocoa applications, including NSStatusBar apps,
>> do implement this by default.
>>
>> However, sometimes the default implementation is not sufficient or is just
>> buggy. So far, the only places I have had issues is with things in the menu
>> bar; I have logged bugs with Apple, but only in the last few months and so
>> they haven't been fixed yet. However, there is nothing stopping you from
>> overriding the appropriate methods in the NSAccessibility protocol for the
>> class/object that isn't behaving properly. Customizing accessibility is
>> supported by Apple; though you usually will only have to override/implement
>> the protocol if you have a lot of custom UI elements that either directly
>> subclass from NSView or change the default behaviour a lot. I would check
>> things out using the Accessibility Inspector first to see how viable
>> AXElements would be in this case.
>>
>> The difficult part about menu bar objects is getting the initial reference
>> to the object; in a worst case scenario you would have to scan across the
>> menu bar to find the object (use Accessibility.element_at_position for
>> various coordinates). Normally, though, you can just get the application
>> reference, just  as you would for an app that uses a window. Here is a short
>> example for using the standard menu bar items to futz with the volume
>> slider:
>>
>> ui = Accessibility.application_with_bundle_identifier
>> 'com.apple.systemuiserver'
>> volume = ui.menu_extra(description: 'system sound volume')
>>
>> click volume # this will click the icon so the slider is visible,
>> otherwise it won't work
>>
>> 15.times { decrement volume.slider }
>>  15.times { increment volume.slider }
>>
>>
>> HTH,
>> Mark
>>
>>
>> On 2011-09-29, at 11:24 AM, Brice Ruth wrote:
>>
>> This looks very useful. I skimmed through the documentation and didn't see
>> anything for accessing NSStatusBar applications. Is this possible? Or would
>> I need to add some hooks in my application to allow the test to open the
>> windows when launched versus via clicking a menuitem from the NSStatusBar
>> menu?
>>
>> Brice Ruth, FCD
>> Software Engineer, Madison WI
>>
>>
>> On Thu, Sep 29, 2011 at 9:55 AM, Mark Rada <mrada at marketcircle.com>wrote:
>>
>>> Hey all,
>>>
>>> On behalf of Marketcircle Inc., I am open sourcing a library that I have
>>> been working on for a while now. AXElements is an alternative to using
>>> Auotmator or the ScriptingBridge frameworks. AXElements provides more
>>> generic actions than it's alternatives and should work with almost any Cocoa
>>> app without needing to be explicitly supported; you can think of it as
>>> Capybara for desktop apps.
>>>
>>> It is meant to be used for automated testing of GUI apps, but can be used
>>> for general UI automation. You can combine it with testing libraries like
>>> minitest, Rspec, or macbacon; but there is nothing stopping you from
>>> combining AXElements with James to voice command some workflows.
>>>
>>> The source is available on github and I have put together some small
>>> tutorails in the documentation. I've also made a beta release to rubygems,
>>> and will make a proper release soon. To get it now:
>>>
>>>     macgem install AXElements --pre
>>>
>>> And then require 'ax_elements' to get started in macirb or a script.
>>>
>>> Source: http://github.com/Marketcircle/AXElements
>>> Documentation (including some tutorials):
>>> http://rdoc.info/github/Marketcircle/AXElements/master/frames/file/README.markdown
>>>
>>> Cheers,
>>>        Mark Rada
>>>
>>> _______________________________________________
>>> MacRuby-devel mailing list
>>> MacRuby-devel at lists.macosforge.org
>>> http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
>>>
>>
>> _______________________________________________
>> MacRuby-devel mailing list
>> MacRuby-devel at lists.macosforge.org
>> http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
>>
>>
>>
>> _______________________________________________
>> MacRuby-devel mailing list
>> MacRuby-devel at lists.macosforge.org
>> http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.macosforge.org/pipermail/macruby-devel/attachments/20110929/8bbdd2e6/attachment-0001.html>