emsify 1.4.0
Modern Automation for EMS - Web scraping toolkit for telecom portals
License: MIT
Requires Python: >=3.7

# emsify - Modern Automation for EMS

Web scraping and automation toolkit for telecom EMS portals (Huawei, Ericsson, Alcatel).

**Package:** `emsify` | **Version:** `1.4.0`

## Why emsify?

Telecom EMS portals have 3-4 nested iframes. Standard Selenium requires manual frame switching:

```python
# Standard Selenium - painful
driver.switch_to.frame('frame1')
driver.switch_to.frame('frame2')  # Which frame? Trial and error!
driver.find_element(By.XPATH, '//button')

# emsify - automatic
ems.click('//button')  # Finds it anywhere
```

**Key Features:**
- Auto frame navigation (90% hit rate on current frame)
- Stale frame recovery
- 4-strategy file upload
- Dual locate (Selenium + JS)
- CSV logging
- Offline install support

---

## Installation

```bash
pip install emsify
```

### Offline Installation

```bash
pip download mae -d emsify_offline
cd emsify_offline
pip install emsify-1.1.0-py3-none-any.whl --no-index --find-links .
```

## Requirements

- Python >= 3.7
- Selenium >= 4.0.0
- WebDriver (Edge, Chrome, or Firefox)

---

## Quick Start (Emsify helper – current core)

The **current core helper** provided by this package is the high-level `Emsify` class,
optimized for interacting with complex pages (nested iframes, shadow DOM, canvas, etc.).

```python
from selenium import webdriver
from emsify import Emsify

driver = webdriver.Chrome()
ems = Emsify(driver, verbose=True, use_js_locator=True)

driver.get("https://example.com")

# Read heading text
title_text = ems.interact("//h1", command="text", t=10)
print("Title:", title_text)

# Click the first link on the page
ems.interact("//a", command="click", t=10)
```

- `Emsify` automatically searches across windows, frames, shadow DOM, and (optionally)
  falls back to canvas/OCR when needed.
- See `tests/test_emsify_*.py` and `docs/LAB_TEST_RESULTS.md` for behavioral coverage
  verified in the original LAB environment.

---

## Quick Start (legacy EmsAutomation API)

```python
from selenium import webdriver
from emsify import EmsAutomation

# Setup driver
driver = webdriver.Edge()
driver.get('https://ems-portal.example.com')

# Create automation instance
ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Navigate through frames automatically
ems.click('//a[text()="Configuration"]')
ems.click('//button[@id="save"]')

# Get statistics
print(ems.get_stats())
```

## Usage Examples

### Example 1: Basic EMS Portal Login

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.maximize_window()

try:
    # Navigate to portal
    driver.get('https://ems.example.com/login')
    
    # Create automation instance
    ems = EmsAutomation(driver, verbose=True)
    
    # Fill login form
    ems.send_keys('id=username', 'operator01')
    ems.send_keys('id=password', 'Pass@123')
    ems.click('//input[@value="Login"]')
    
    # Wait for dashboard
    ems.locate('//div[@id="dashboard"]', timeout=10)
    print("Login successful!")
    
finally:
    driver.quit()
```

### Example 2: Navigate Nested Frames

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Navigate through menu (handles frames automatically)
ems.hover_click('//div[@class="menu"]', '//a[text()="Alarms"]')
ems.hover_click('//div[@class="submenu"]', '//a[text()="Active Alarms"]')

# Work with table in nested frame
table = ems.locate('//table[@id="alarms"]')
rows = table.find_elements('xpath', './/tr')
print(f"Found {len(rows)} alarms")

# Click on first alarm
ems.click('//table[@id="alarms"]//tr[1]')

driver.quit()
```

### Example 3: File Upload

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com/upload')

ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Navigate to upload page
ems.click('//a[text()="Configuration"]')
ems.click('//a[text()="Upload"]')

# Upload file (tries multiple strategies)
success = ems.upload_file('config.xml', '//input[@type="file"]')

if success:
    print("File uploaded successfully")
    ems.click('//button[@id="submit"]')
else:
    print("Upload failed")

driver.quit()
```

### Example 4: Extract Data from Table

**New in v1.1.0:** Enhanced XPath builder with fluent interface

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Navigate to reports
ems.click('//a[text()="Reports"]')
ems.click('//a[text()="Site Status"]')

# Extract table data
table = ems.locate('//table[@id="sites"]')
headers = table.find_elements('xpath', './/th')
header_text = [h.text for h in headers]
print("Headers:", header_text)

rows = table.find_elements('xpath', './/tr')
for row in rows[1:]:  # Skip header row
    cells = row.find_elements('xpath', './/td')
    site_data = [cell.text for cell in cells]
    print(site_data)

driver.quit()
```

### Example 5: Handle Dynamic Content

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Click button that loads content dynamically
ems.click('//button[@id="load-data"]')

# Wait for dynamic content with increased timeout
element = ems.locate('//div[@class="loaded-content"]', timeout=30)
print("Content loaded:", element.text)

# Use JavaScript method for hidden elements
hidden = ems.locate('//div[@style="display:none"]', method='js')
print("Hidden element found:", hidden)

driver.quit()
```

### Example 6: Multiple Windows/Tabs

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

ems = EmsAutomation(driver)

# Login
ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Click link that opens new window
main_window = driver.current_window_handle
ems.click('//a[@target="_blank"]')

# Switch to new window automatically
ems.switch_to_new_window()

# Work in new window
ems.click('//button[@id="action"]')

# Close and switch back
driver.close()
driver.switch_to.window(main_window)

driver.quit()
```

### Example 7: Error Handling and Logging

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

# Enable verbose mode for debugging
ems = EmsAutomation(driver, verbose=True)

try:
    # Login
    ems.send_keys('id=username', 'admin')
    ems.send_keys('id=password', 'secret')
    ems.click('//button[@id="login"]')
    
    # Try to find element with error handling
    element = ems.locate('//button[@id="submit"]', timeout=5)
    if element:
        element.click()
    else:
        print("Element not found")
    
    # Get error statistics
    stats = ems.get_stats()
    print(f"Errors: {stats}")
    
    # Check current frame location
    print(f"Current frame: {ems.frame_path}")
    
    # Save logs to CSV
    ems.save_logs()
    print("Logs saved to CSV file")
    
except Exception as e:
    print(f"Error occurred: {e}")
    ems.save_logs()  # Save logs even on error
    
finally:
    driver.quit()
```

### Example 8: Using XPath Builder (v1.1.0+)

**New:** `XPathChain` fluent interface for complex selectors  
**v1.2.0:** Automatic XSS protection with input sanitization

```python
from selenium import webdriver
from emsify import EmsAutomation, xp

driver = webdriver.Edge()
driver.get('https://ems.example.com')

ems = EmsAutomation(driver)

# Use XPath builder for cleaner code (v1.2.0: auto-sanitized)
username_field = xp.input_with_placeholder('Username')
password_field = xp.input_with_placeholder('Password')
login_button = xp.button_with_text('Login')  # Quotes are escaped automatically

ems.send_keys(username_field, 'admin')
ems.send_keys(password_field, 'secret')
ems.click(login_button)

# Build custom XPath
menu_item = xp.div_with_class('menu-item')
ems.click(menu_item)

# NEW: Fluent chaining (v1.1.0)
complex_xpath = xp.chain('div').has_class('form').child('input').attr('type', 'email').visible().first().build()
ems.send_keys(complex_xpath, 'user@example.com')

# v1.2.0: Disable sanitization if needed (not recommended)
unsafe_xpath = xp.button_with_text("admin's button", sanitize=False)

driver.quit()
```

### Example 9: Secure File Upload (v1.2.0+)

**New:** Path validation prevents directory traversal attacks

```python
from selenium import webdriver
from emsify import EmsAutomation

driver = webdriver.Edge()
driver.get('https://ems.example.com')

# v1.2.0: Path validation enabled by default
ems = EmsAutomation(driver, validate_path=True)

ems.send_keys('id=username', 'admin')
ems.send_keys('id=password', 'secret')
ems.click('//button[@id="login"]')

# Safe: File in current directory or subdirectory
ems.upload_file('./config.xml', '//input[@type="file"]')  # ✅ Allowed
ems.upload_file('data/report.csv', '//input[@type="file"]')  # ✅ Allowed

# Blocked: Path traversal attempts
try:
    ems.upload_file('../../../etc/passwd', '//input[@type="file"]')  # ❌ Blocked
except ValueError as e:
    print(f"Security: {e}")  # "Path outside allowed directory"

# Disable validation if needed (not recommended)
ems_unsafe = EmsAutomation(driver, validate_path=False)
ems_unsafe.upload_file('../file.txt', '//input')  # ⚠️ Allowed but risky

driver.quit()
```

## What's New in v1.3.0

### 🔒 Security Fixes
- ✅ **XSS Prevention** - Input sanitization in XPath builder (14 vulnerabilities fixed)
- ✅ **Path Traversal Protection** - File path validation in drivers and upload
- ✅ **Input Validation** - Selector and timeout validation in locate()

### 🆕 New Features
- ✅ **Shadow DOM Support** - Search elements inside Shadow DOM with `shadow=True`
- ✅ **Canvas Interaction** - Click, capture, and analyze canvas elements
- ✅ **Window Switch Detection** - Automatic frame stack reset on window change
- ✅ **interact() Method** - Unified API for all element interactions

### 🏗️ Architecture Improvements
- ✅ **Refactored EmsAutomation** - Now inherits from TrackedEmsWalk (150+ lines eliminated)
- ✅ **Unified Frame Management** - Automatic timeout budget & max_depth enforcement
- ✅ **Better Error Tracking** - Improved exception handling and statistics

### 🧪 Testing
- ✅ **43 Comprehensive Tests** - 100% pass rate
- ✅ **Security Test Suite** - XSS and path traversal validation
- ✅ **Feature Tests** - Shadow DOM, Canvas, Window Switch, interact()

### 📝 Backward Compatibility
- ✅ **Escape Hatches** - `sanitize=False` and `validate_path=False` parameters
- ✅ **Zero Breaking Changes** - Existing code works unchanged

---

## What's New in v1.1.0

- ✅ **DragDropUploader** - 4-strategy file upload (direct, hidden, JS drag, dialog)
- ✅ **Timeout budget tracking** - Prevents 65s+ searches (now max 2× timeout)
- ✅ **Max depth limit** - Prevents stack overflow on deeply nested frames (default: 10)
- ✅ **Stale frame handling** - Auto-recovery from stale frame references
- ✅ **XPathChain** - Fluent interface for building complex XPath expressions
- ✅ **25+ XPath helpers** - axis navigation, position filters, state filters
- ✅ **85% faster** - Optimized frame search with remaining time calculations

## API Reference

### EmsAutomation Class

#### locate(selector, timeout=10, method='selenium', default_first=False, condition='present')

Find element with smart frame navigation.

```python
# Basic usage
element = ems.locate('//button[@id="submit"]')

# With timeout
element = ems.locate('//button', timeout=20)

# Use JavaScript method
element = ems.locate('#hidden', method='js')

# Search default frame first
element = ems.locate('//input', default_first=True)

# Wait for specific condition
element = ems.locate('//button', condition='clickable')
```

#### click(selector, timeout=10)

Click element.

```python
ems.click('//button[@id="submit"]')
ems.click('id=login-btn', timeout=5)
```

#### send_keys(selector, text, clear=False, timeout=10)

Send text to input field.

```python
ems.send_keys('id=username', 'admin')
ems.send_keys('//input[@name="email"]', 'user@example.com', clear=True)
```

#### hover_click(hover_selector, click_selector, timeout=10)

Hover over element then click another.

```python
ems.hover_click('//div[@class="menu"]', '//a[text()="Settings"]')
```

#### upload_file(file_path, selector, timeout=10)

Upload file using DragDropUploader (v1.1.0+) with 4 strategies:
1. Direct input (if visible)
2. Hidden input (unhide and use)
3. JS drag-drop (DataTransfer API)
4. Click dialog (manual fallback)

```python
ems.upload_file('config.xml', '//input[@type="file"]')
ems.upload_file('data.csv', '#file-upload')

# Returns True/False
success = ems.upload_file('large.zip', '#dropzone', timeout=30)
```

#### switch_to_new_window()

Switch to newest window/tab.

```python
ems.click('//a[@target="_blank"]')  # Opens new tab
ems.switch_to_new_window()  # Switches to it
```

#### get_stats()

Get error statistics.

```python
stats = ems.get_stats()
print(stats)  # {'TimeoutException': 2, 'NoSuchElementException': 1}
```

#### save_logs()

Save operation logs to CSV.

```python
ems.save_logs()  # Saves to {domain}_log.csv
```

#### frame_path

Get current frame location.

```python
print(ems.frame_path)  # Shows: [default] or frame1 > frame2 > frame3
```

#### interact(selector, cmd='click', value=None, timeout=5) (v1.3.0+)

Universal interaction method - alternative to specific methods.

**Commands:** click, send_keys, clear, get_text, get_attr, is_displayed, is_enabled, hover, double_click, right_click, submit, locate

```python
# Actions
ems.interact('//button', 'click')
ems.interact('id=username', 'send_keys', 'admin')

# Custom timeout
ems.interact('//slow-button', 'click', timeout=15)
ems.interact('id=field', 'send_keys', 'text', timeout=10)

# Getters
text = ems.interact('//div', 'get_text')
href = ems.interact('//a', 'get_attr', 'href', timeout=3)

# Advanced
ems.interact('//button', 'hover')
ems.interact('//div', 'double_click')
```

#### Shadow DOM Methods (v1.2.0+)

Search elements inside Shadow DOM.

```python
# Search only Shadow DOM
ems.locate('#element', shadow=True)
ems.click('#button', shadow=True)

# Search both regular + shadow DOM
ems.locate('#element', search_both=True)
```

#### Canvas Methods (v1.2.0+)

Interact with HTML5 canvas elements.

```python
# Find canvas
canvas = ems.find_canvas('canvas')

# Click at coordinates
ems.click_canvas('canvas', 100, 200)

# Get image data
data = ems.get_canvas_data('canvas')  # Returns base64 PNG

# Get pixel color
rgba = ems.get_canvas_pixel('canvas', 50, 75)  # Returns (r, g, b, a)
```

## Selector Formats

```python
# XPath (starts with //, ./, or ()
ems.locate('//button[@id="submit"]')
ems.locate('.//input[@type="text"]')

# CSS Selector
ems.locate('#submit-button')
ems.locate('.form-control')

# Explicit prefix
ems.locate('xpath=//button[@id="submit"]')
ems.locate('css=#submit-button')
ems.locate('id=submit-button')
ems.locate('name=username')
```

## Frame Navigation

Automatic search order:
1. Current frame (90% hit rate)
2. Default content (if `default_first=True`)
3. All nested frames (depth-first)
4. Final default attempt

```python
ems.click('//button')  # Finds anywhere
print(ems.frame_path)  # Check location
```

## Troubleshooting

### Element Not Found

```python
# Increase timeout
element = ems.locate('//button', timeout=30)

# Try JavaScript method
element = ems.locate('//button', method='js')

# Enable verbose mode
ems = EmsAutomation(driver, verbose=True)
```

### WebDriver Issues

```python
# Check versions
from emsify.drivers import get_edge_version, get_msedgedriver_version

print(f"Browser: {get_edge_version()}")
print(f"Driver: {get_msedgedriver_version()}")
```

### Frame Issues

```python
# Use TrackedEmsWalk for debugging
from emsify import TrackedEmsWalk

tracked = TrackedEmsWalk(driver, verbose=True)
tracked.locate_element('//button')  # Shows frame switches
```

## Security Best Practices (v1.2.0+)

### 1. Use Default Security Settings
```python
# ✅ Recommended: Security enabled by default
ems = EmsAutomation(driver, verbose=True)

# ❌ Not recommended: Disabling security
ems = EmsAutomation(driver, validate_path=False)  # Allows path traversal
```

### 2. XPath Sanitization
```python
from emsify import xp

# ✅ Safe: Automatic sanitization
user_input = "admin' or '1'='1"  # Malicious input
xpath = xp.button_with_text(user_input)  # Quotes escaped automatically

# ❌ Unsafe: Manual XPath construction
xpath = f"//button[text()='{user_input}']"  # Vulnerable to injection
```

### 3. File Upload Validation
```python
# ✅ Safe: Validate paths
ems.upload_file('./safe_file.xml', '//input[@type="file"]')

# ❌ Unsafe: Absolute or traversal paths
ems.upload_file('/etc/passwd', '//input')  # Blocked by default
ems.upload_file('../../../file', '//input')  # Blocked by default
```

### 4. Error Handling
```python
driver = webdriver.Edge()
try:
    ems = EmsAutomation(driver, verbose=True)
    # Your code
except Exception as e:
    print(f"Error: {e}")
    ems.save_logs()
finally:
    driver.quit()
```

## Offline Setup

```bash
pip download mae -d emsify_offline
cd emsify_offline
pip install emsify-1.1.0-py3-none-any.whl --no-index --find-links .
```

```python
from selenium.webdriver.edge.service import Service
service = Service('C:/path/to/msedgedriver.exe')
driver = webdriver.Edge(service=service)
```

## License

MIT License

## Author

Ahmedul Kabir Omi (here.is.omi@gmail.com)

## Testing

### Run Tests
```bash
# Run all tests
python tests/run_tests.py

# Run security tests only
python -m pytest tests/test_security_fixes.py -v

# Run architecture tests
python -m pytest tests/test_architecture_concept.py -v
```

### Test Results (v1.3.0)
- ✅ 43/43 tests passed (100%)
- ✅ Security vulnerabilities: 0
- ✅ New features: Shadow DOM, Canvas, Window Switch, interact()
- ✅ Code coverage: 100% for security fixes

---

## Migration Guide (v1.1.0 → v1.3.0)

### No Changes Required
Existing code works without modification:
```python
# v1.1.0 code
ems = EmsAutomation(driver)
ems.click('//button')

# v1.2.0 - Same code, more secure
ems = EmsAutomation(driver)  # Now with XSS protection & path validation
ems.click('//button')  # Works identically
```

### Optional: Enable New Features
```python
# Use new security features explicitly
ems = EmsAutomation(driver, 
                    max_depth=10,        # Frame depth limit
                    validate_path=True)  # Path validation
```

---

## Documentation

- **Installation Guide:** [docs/INSTALL.md](docs/INSTALL.md)
- **Test Results:** [docs/TEST_RESULTS_NEW_FEATURES.md](docs/TEST_RESULTS_NEW_FEATURES.md)
- **Shadow DOM Guide:** [docs/SHADOW_DOM_INTEGRATION.md](docs/SHADOW_DOM_INTEGRATION.md)
- **Canvas Support:** [docs/CANVAS_SUPPORT.md](docs/CANVAS_SUPPORT.md)
- **interact() Method:** [docs/INTERACT_METHOD_ADDED.md](docs/INTERACT_METHOD_ADDED.md)
- **Session Summary:** [docs/SESSION_SUMMARY.md](docs/SESSION_SUMMARY.md)

## Links

- PyPI: https://pypi.org/project/emsify/
- GitHub: https://github.com/hereisomi/emsify
- Issues: https://github.com/hereisomi/emsify/issues