The most popular tools for test automation are Selenium, Appium, Cucumber, Jmeter, Postman..
Let’s get to know the problems that you will encounter in the future with Open Source Testing Tools.
1. Hardware Requirements
A powerful Macbook needed for automation engineers. Because You will need lots of things for development and debugging test automation. Mac-mini, USB extender, iOS and Android Devices. Some guys still have to code with VBscript and use Windows laptops. It’s not your fault 🙁
Conclusion: There must be a different and common way of doing this, we can’t dedicate all devices to all engineers. First of all it should be in a shareable environment.
2. Software Installations & Configurations
Installation of Appium and all dependencies on your computer for mobile test automation development is killer. It should take your 1 day to just install and configure the dependencies.
Conclusion: I am not a mobile developer, do I really have to install so many things on computer? I am installing more than an Android or iOS developer. How will I keep them updated? How do I upgrade to the new SDK version?
3. Get Required Access Permissions
- Test Database Access Permissions
- Backend System Access Permissions
- Git Repository Access
- Test Management Tool Access
- Issue management Access
- Notification app Access
Conclusion: If I have just started this job, I may not even know which systems I should have access to. It will also take a while for anyone new to the team to open, close, or update account / access to these systems. there must be a definitive solution to doing this in a different way.
4.Inspect Existing Test Automation Project
- *Automation Project’s Programming Language and Framework.
- Supported Platforms (iOS, Android, Web, API, Hybrid, Desktop..)
- Test Automation Scenarios
- Merge and branching strategies
5.Application package to be tested
*You have to test the right packages at all times. You have to access APK(Android) and IPA(iOS Real Device) files on mobile projects.
*Package from the Latest Branch / Specified branch
Conclusion: Will I compile and create the application package to be tested, or will it come to me automatically or manually from the continuous integration tool?
6. Emulator vs Simulator vs Real Device
- Emulators and Simulators are free and they have no duration limitations like cloud services. But can’t replicate real user conditions like hardware and software configurations, work slower than real devices, consume a lot of memory and CPU on your computer.
- Emulators / Simulators are good at Unit Testing, Static Code analysis, development. They are weak at UI Regression Testing and Compatibility testing
Conclusion: Real devices provide test runs under real-world conditions. If your test is passing on an emulator / simulator that doesn’t mean it will pass on Real Device.
7. On-Premise (In-House) vs Cloud Device Farm
You need a device farm for compatibility testing and high device coverage.
Conclusion: Cloud farm has got limitation, pricing and slowness problems. Can I believe that the mobile phone on which I ran the test is real and only dedicated to me? I prefer other users not to access my application and automation code.
8.Interactive Live Mobile App Testing on Real Device
- We are going to automate 100 percent of our regressions.
- Automation and manual test engineers are independent teams.
Conclusion: Manually exploratory testing is very important now and will remain so in the future. There is a problem of device sharing between manual and automation test engineers. Devices in on-premise device farms also have the same problem. You should not separate Manual and automation engineers. The goals of both teams are the same. Measuring the quality of the software. There is actually 1 team. You should not automate everything, Welcome to the real world.
9.Start to Automate
- Inspect ID, XPATH,.. of the mobile elements on APP and write your automation code on IDE. ()
- Element locator problems (There is no ID or Name, Dynamic ID’s,..)
Conclusion: Using Appium with your favorite development IDE (IntelliJ, Pycharm, WebStorm, VisualStudio, Eclipse, Ride.. ) Can’t we do this with a single application instead of 2 different client applications with hours of installation and configurations? Should we centrally manage Appium server and client applications? Can I get support from the development team for element locator problems? HTML concepts and basic programming knowledge is required. Reusable elements can help you.
10. Integration Requirements
- JIRA issue management / Slack notification / Email / Jenkins..
- Auto-fill and submit test run Defects to Jira
- Basic reporting with Email
- In Real Time Get Slack notification messages for test execution.
- CI tool plugin to trigger Test Run.
- Git integration
- Cloud Device Farm integration
Conclusion: Developing dozens of integrations will require serious effort. If I cannot solve this work with open source resources in a short time, I may have to replace the complete test automation framework. We may even need to get test consultancy.
11. Code Quality
- Use PageObject Pattern
- Fix Static Code Analysis issues.
- Apply SOLID..
- Write BDD TestCases
- Apply GitFlow branching model
Conclusion: What is expected of me? I am not a developer..
12. TestCases in BDD (Cucumber)
- Given-When-Then formatted Acceptance criterias needed
Conclusion: JIRA issues are not in BDD format. How can I adapt the current format to this?
13. Test Data problem
- Reading Data from Database/File/..
- Writing Data to Database/File..
- Create Test Data sets for your test automation suite
- Use Data Fakers
- Can’t generate Test Data, Mock the middleware
Conclusion: The test data problem will always be your problem. The important thing is how much support is built into which tool. Data-driven testing can help you.
14. Debugging Features
- Check your test’s screenshot, video on each step.
- Check the console logs, application logs.
- Debug your test run locally.
Conclusion: Debugging the test run is very important. Without debugging, it’s like a blind man driving a car.
15. Run Your Test
- Check the availability of the devices / browsers
- Multi device / browser & version selection
- Check the availability of CI / CD tools
- On demand or Scheduled test running
- Separated Test Scenarios
- Running Test Code on Docker Container
Conclusion: Where is my project built? Where is the test running on? How can I run it locally or remotely? How can I watch the stage of the test?
16.Running on Different devices (PARALLEL) / OS / Versions
- Check that all test automation results on all popular devices / versions available are as you expect.
- Running Tests with Selenium Grid
- Using Local or Remote Devices.
- Emulator / Simulator / Real Device again.
Conclusion: If all tests take two hours to run on 1 device, it should take an average of two hours to run on 100 devices. Other options are unacceptable and unexpected for me.
17. Reporting test results
- Test Case / Test Run report
- JIRA Defect Reporting
Conclusion: Reporting is not a nice to have feature, it is a must.
18. Community Support
- If you started with java, there is a lot of community support for issues / libraries. You may not have much chance for other programming languages.
- The incompatible tool versions should conflict with each other and You should get fail when you want to run your test
- You should get no support / response on some issues with open source. You are alone.
- Risk of violation and license restriction risk.