AFour Technologies
Get Started

Sikuli Automation Tool with Coded UI

Sikuli:

Sikuli automates anything you see on the screen. It uses image recognition to identify and control GUI components. It is useful when there is no easy access to the GUI elements.

Advantages of using Sikuli :

  1. Using Sikuli we can automate a desktop as well as a Web application
  2. Easy and simple to use
  3. Can be easily integrated with Java, C# and other popular programming languages
  4. Can be used to automate Flash Objects / Flash Websites
  5. Uses powerful “”Visual Match” mechanism to automate applications
  6. Open source

There are certain situations in the automation where automation tool does not find appropriate elements.

Eg: Consider a desktop application which is completely developed in C# but HTML layer is also embedded in it. Now, if we are using CodedUI, AutoIt, TestStack,White or any other automation tool for desktop automation, it is very hard or sometimes impossible to locate elements efficiently. Sikuli can be very helpful in such scenarios.

My aim in this blog is to show how we can integrate Sikuli with Visual studio so that we can use Sikuli with C#.

Sikuli Integration with Visual Studio for Coded UI test cases:

Pre-requisites :

  1. Visual Studio should be installed on the system; preferably Visual Studio 2012
  2. Any screen-shot capturing tool like Greenshot etc.
  3. Java 1.6 tested with jre-6u35-windows-i586.exe
  4. Microsoft .Net framework 4.5
  5. NuGet

Steps :

  1. Create new project in Visual Studio 2012
  2. In the “Solution Explorer” – right-click on the solution
  3. Choose Manage NuGet Package for Solution

Search for “SikuliIntegrator” and install

  1. After the installation “JSikuliModule.jar” should be part of your solution, together with some additional files
  2. Right-Click on “JSikuliModule.jar” and choose “Properties”
  3. Set “Copy to Output Directory” property to “Copy always”

After performing step 7, there can be issues in copying the jar into the results folder of your test case. We can write code to copy jar from source to destination.

Once all the 7 steps are performed successfully, it’s very easy to use Sikuli in your C# code. Let’s see some useful methods of Sikuli.

Add SikuliModule in the namespace in the following way so that we can use Sikuli related classes and methods

using SikuliModule

General Syntax :
SikuliAction.[ACTION](ARGUMENTS);

Example of some of the methods supported by SikuliIntegrator :

Click
SikuliAction.Click(pattern);
Double Click
SikuliAction.DoubleClick(pattern);
Right Click
SikuliAction.RightClick(pattern);
Hover
SikuliAction.Hover(pattern);
Wait
SikuliAction.Wait(pattern, timeout);
Drag and Drop
SikuliAction.DragAndDrop(fromPattern, toPattern);

Where pattern is path to image

I hope this blog will help in solving your element identification problems to a great extent.

References :

https://qaagent.wordpress.com/2013/07/17/how-to-use-sikuli-in-my-c-coded-tests/

http://www.sikuli.org/

Picture of Mahesh Kulkarni
Mahesh Kulkarni

4 Comments

  • Aditya Katiyar
    March 11, 2016 at 3:50 pm Reply

    Worth for Automation framework

  • karthick
    June 7, 2017 at 1:59 pm Reply

    Visual Studio: 2017 Enterprise Edition
    I am getting the below error, can anybody help me on this, please
    Test method CodedUITestProject31.CodedUITest1.CodedUITestMethod1 threw exception:
    System.Exception: Exception in thread “main” java.lang.UnsatisfiedLinkError: C:\Users\karth\AppData\Local\Temp\tmplib\Win32Util.dll: Can’t find dependent libraries
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(Unknown Source)
    at java.lang.ClassLoader.loadLibrary(Unknown Source)
    at java.lang.Runtime.load0(Unknown Source)
    at java.lang.System.load(Unknown Source)
    at com.wapmx.nativeutils.jniloader.NativeLoader.loadLibrary(NativeLoader.java:44)
    at org.sikuli.script.Win32Util.(Win32Util.java:19)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.sikuli.script.Env.getOSUtil(Env.java:91)
    at org.sikuli.script.ScreenHighlighter.init(ScreenHighlighter.java:180)
    at org.sikuli.script.ScreenHighlighter.(ScreenHighlighter.java:293)
    at org.sikuli.script.Screen.initBounds(Screen.java:105)
    at org.sikuli.script.Screen.(Screen.java:117)
    at com.sikuliintegrator.Operation.RightClick(Operation.java:116)
    at com.sikuliintegrator.Pointer.main(Pointer.java:106)

  • uttam
    December 7, 2017 at 5:03 pm Reply

    How we can scroll down . I am using desktop application

  • Dipa
    May 21, 2019 at 11:44 am Reply

    Getting below error after performing all the steps

    driver = new ChromeDriver(@”C:\\”);
    Thread.Sleep(5000);
    driver.Navigate().GoToUrl(“http://www.google.com”);
    Screen screen = new Screen();

    Pattern googletextbox = new Pattern(@”C:\Users\DWS\Desktop\genter.png”);
    screen.Wait(googletextbox, 5);

    screen.Type(googletextbox,”hello”);

    Error is : Message: Test method DesignSystemTestPro.CodedUITest1.CodedUITestMethod1 threw exception:
    System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:8080

Leave a Reply